Successfully debugging your shader means following a few simple rules.
Many problems are related to the various environment variables that ray3.exe expects in order to find licenses, registries, etc. You can avoid this by starting Visual C++ using the msdev.exe command from within a Softimage Command Prompt window, which sets the Softimage environment variables automatically. You can display this window by running %HOME%\Application\bin\SIShell.bat or by choosing Programs > Softimage Products > Command Prompt.
Once you’ve ensured that the environments variables are set, do the following.
Within Softimage, create the .mi file using the shader to be debugged. Use the imf_disp utility to view the frame buffer while rendering using ray2.exe.
To create an .mi2 file in Softimage
1. From the Render toolbar, choose Render > Export to MI > Options. The Render Options property editor opens.
2. Set your options such as the path and file name, start and end frames, and so on.
3. From the Render toolbar, choose Render > Export to MI and export either the current pass or all passes. You may have to wait for the output file to be generated. You can then manually edit and render this file with mental ray from the command line.
To use the imf_disp utility while rendering
The imf_disp standalone displays image files, including memory-map texture images created by imf_copy. The imf_disp standalone displays all formats supported by the mental ray rendering software (except the SOFTIMAGE Zpic format).
imf_disp also supports image piping. Normally, mental ray prints connection information into the output image file that lets programs like imf_disp connect and display pictures while being rendered. If the -imgpipe option is used when rendering from the command line, the relevant information is printed to the given file descriptor instead:
ray3 -imgpipe 1 scene.mi | imf_disp
• Build the shader’s .dll using the Debug Multithreaded DLL option under Configuration Properties > C/C++ > Code Generation > Runtime Library.
• Type ray3.exe in Configuration Properties > Debugging > Command. Use the Command Arguments to specify the other switches that are necessary to render from the command line.
Set breakpoints and begin debugging. Remember that you won't be able to trace into mi_ functions and that mental ray is stripped.
Testing Shaders after Installation
After you install the shader in Softimage, you can make sure that it is properly installed and registered by following these steps:
To test shaders after installation
1. Open a command-prompt window. If your Start menu is in its default configuration, you can choose Start > All Programs > Softimage Products > Softimage_7.5 > Command Prompt.
2. Enter the following command:
ray3 -v
mental ray prints out a great deal of start-up information.
3. Enter the following, including the shader’s GUID:
$lookup "{GUID}"
4. mental ray postpones loading the shader library until required, so no feedback is given at this point. However, mental ray does report bad registry entries upon exiting. In the shell running mental ray, type Ctrl-C to terminate execution. If the shader library entry was not found or expanded properly, mental ray will most likely print out a message such as:
LIB 0.0 error 291825: {GUID}: registry entry
{GUID} not found^C
Autodesk Softimage v7.5