Building VisIt+OSPRay/SWR

Alok Hota, ahota@vols.utk.edu

The following instructions walk you through the steps of building
an OSPRay-enabled instance of VisIt. Additionally, you can use OpenSWR
(via Mesa) with VisIt, though it requires a separate download as VisIt's
Mesa version does not include SWR yet.
I tried to include any common problems encountered along with
troubleshooting steps to resolve them.

NOTE: These instructions are for Linux machine running on a Xeon
CPU or a KNL Xeon Phi. I have not tested with any other OS and don't
know how well or even if anything would work.
I have confirmed that this works on CentOS, so any RHEL-based distro
should be fine. Ubuntu and similar distros should work, but I
haven't had the chance to test it.
We have tested with Sandy Bridge, Ivy Bridge, and Haswell Xeon CPUs.
Single- or dual-socket is fine.
Some steps in the process differ on KNL; I tried to make sure to
mention the differences when applicable.

I am currently working on a build/binary for newer versions of
VisIt, OSPRay, and SWR.
Newer versions of VisIt (possibly 2.13+) will be incorporating VTK 7,
which has OSPRay support built in. After that transition, the visit_ospray
source will not be necessary.

Intel SIMD Program Compiler binaries v. 1.9.1
EXAMPLE: wget https://downloads.sourceforge.net/project/ispcmirror/v1.9.1/ispc-v1.9.1-linux.tar.gz?r=&ts=1495658150&use_mirror=iwebNOTE: This is a precompiled binary, which needs to be placed alongside the OSPRay source directory from above

This will get the whole process started. Many packages will be downloaded.
This process should be fully automatic at this point. If you run into issues,
please refer to section 2.1. This process will probably last around 2 hours.
In the example above, the -j4 indicates 4 threads will be used during the build process.
Feel free to use more than 4 threads if you have the cores available.

In this example, the ISA being used is AVX2, e.g. on a Haswell/Broadwell Xeon.
If you are building on KNL, change the
-DOSPRAY_BUILD_ISA=AVX2 option to
AVX512.
For other variations, please refer to Section 3.

Assuming everything has gone perfectly until this point, the next step is to
build the OSPRay-enabled VisIt. The source for OSPRay-enabled VisIt is available
on a VisIt development branch, shown below.

In this example, /home/alok/visit_ospray/build/bin/visit is the OSPRay-enabled
application executable.

If you're ready to run a quick test, go straight to Section 6.
Of course many things could have happened during this process. The remainder of this document
contains more detailed information and troubleshooting steps for the process.

About the above components

Use VisIt's build_visit script to get all necessary libraries and to
build a working copy of VisIt. The script will install everything into
one place (called the
VISIT_HOME directory) which makes everything much
easier.
At the time of writing, 2.10 is the most recent version
confirmed to work. This process should also work with 2.8.1 and 2.9.x.
Versions 2.11+ are not yet guaranteed to work.

OSPRay source is available in the different releases on their GitHub
repository. The most recent version confirmed to work
is 1.1.0. Higher versions are not yet confirmed or guaranteed to work.
Lower versions will not work as the code uses features not
present until 1.1.0.

OSPRay requires ISPC, which is a SIMD compiler.
At the time of writing, the highest available version is 1.9.1, which
is known to work with OSPRay 1.1.0.

The VisIt+OSPRay source is a modified version of the VisIt code that
includes some piping to call OSPRay for rendering. It is currently
available as a branch in VisIt's svn repository. Run
svn co <the above URL> in
/home/alok/visit_ospray/ to get
the code.

Notes

In general I prefer out-of-source builds to keep things clean, hence
the separate build and src directories in the example structure above.
The build_visit script unfortunately does an in-source build by default.
This might be configurable by setting a
VISIT_BUILD_DIR environment
variable. I only recently found this option and have not tested it,
however.

Building VisIt (part 1)

Using the build_visit script

First the vanilla VisIt instance needs to be installed using the
build_visit script downloaded in the last step. The script will
download several tar files and create build directories for each
library it needs. You shouldn't need to worry about these. The two
directories it creates that are important are
visit and
visit2.10 (or whichever version you
use).
The former will contain all installations of the libraries and is
referred to as the
VISIT_HOME
directory. The latter will contain a
src
directory with the source code and compiled binaries.

It is NOT recommended to use system libraries, even for things like
Qt or Python. I have found that system installations are often missing
certain options that will break the VisIt build process. I suggest
saving yourself the headache and letting VisIt download and install the
necessary libraries.

If you want to specify a compiler, use the
--cc and
--cxx flags to
specify a C and C++ compiler, respectively. Feel free to specify more
or fewer threads for make (the -j4).

This will ask you to agree with Qt's license at the beginning.
Building and installing will take some time, but in general I have
found this command to be fairly reliable.

Notes

In the directory containing the build_visit script,
/home/alok/visit in this case,
there will be a cmake file named
<hostname>.cmake.
This contains the settings for the
VISIT_HOME directory mentioned above.
You will need this when building the OSPRay-enabled instance of Visit.

Building OSPRay

Untar the ISPC binaries. I like to leave them in the
ispc directory that the tar file
contains, and keep it in the
ospray
directory alongside
build and
ospray-1.1.0.

Go to the build directory.
You can run the CMake command given at the beginning of this document.
An alternative to this is using
ccmake, which provides a terminal
interface to the CMake process.

To use ccmake,
run
ccmake ../ospray-1.1.0.
In ccmake, press c to configure.
OSPRay may not find the ISPC executable.
The executable is located within the
ispc-v1.9.1-linux
directory from the tar file. I suggest giving an absolute path.
Set the following variables:

OSPRAY_BUILD_ISA to the
highest level your CPU supports (AVX2 for Haswell or later). If you
are compiling for KNL, either use ALL or AVX512

OSPRAY_BUILD_MIC_SUPPORT to
OFF if you are on a Xeon CPU

OSPRAY_BUILD_MPI_DEVICE to
OFF

OSPRAY_USE_EXTERNAL_EMBREE to
OFF

ISPC_EXECUTABLE to
/home/alok/ospray/ispc-v1.9.1-linux/ispc

NOTE: Warnings about OSPRay not finding Embree installed on your system are
okay. It will use a pre-packaged Embree bundled with OSPRay 1.1.0. You don't
need to download or build Embree.

NOTE:
If you are building on KNL and need AVX512, you will have to use the Intel compiler.

You can turn off the different example applications. If you would like
to verify OSPRay was built correctly, you can leave the volume viewer
application enabled. You will need to download a readable dataset to
test with, such as the ones at the bottom of
this page.

Finally, run
make -j 4 for however many threads
you like. This may take some time, especially when building and linking
the different AVX libraries.
Once compilation is done, make sure that you have the files
libembree.so and
libospray.so.

Troubleshooting

If you don't have a libembree.so
shared library after compiling, double check that you set
OSPRAY_USE_EXTERNAL_EMBREE to OFF.

Building VisIt (part 2)

Copy the <hostname>.cmake file from the vanilla VisIt build into
the visit_ospray/src/config-site
directory. You will need to modify it to add four CMake configuration
variables:

I usually place these before the database reader plugin libraries.
Of course, replace the directories above with the actual ones
you are using. Note that this CMake file has settings for the
VISIT_HOME directory.
This allows the VisIt+OSPRay build to use the
libraries that were installed in the vanilla build.

Change to the visit_ospray build directory and run
cmake ../src, then run
make with the desired number of threads. This may take some time.
If the compile is successful, you should be able to go to the
bin/ directory and run the
following:

./visit -gui -ospray [-o /your/data/file/here]

Open a volume of your choice and render it. Currently, if the VisIt
volume rendering method is set to Splatting, it will use OSPRay
instead. OSPRay will also work when rendering a mesh, e.g. with a
Pseudocolor plot or a Contour plot.

Troubleshooting

If your compiler gives an error stating it cannot find
vtkMultiProcessController.h,
you can safely comment out the line that includes it.
You can resume compiling after making this change without cleaning.

If you are getting errors about unresolved symbols, you can try
modifying a line in your <hostname>.cmake file. Find the two
lines setting C/C++ compiler flags. They are usually near the top and
commented. In the flags given, remove the
-fvisibility flag from both
the C and C++ flags. This keeps symbols from being marked as hidden.
Rebuild VisIt after making this change.

Notes

Some of the annotations in VisIt do not currently work well with OSPRay.
This includes the color map legend, dataset information, and user
information. Bounding box and axis/ticks work, as shown in the examples
below.

You may sometimes see some white/black lines in the bottom left corner.
These are artifacts of the annotations and triad attempting to be
rendered. I'm hoping to fix this soon.

On our dual-socket Haswell machine, we typically see 1-2 fps for large
volumes when rendering with OSPRay. Note that this is with 8 samples
per pixel in OSPRay. A lower sample rate results in faster rendering at
the expense of image quality. The frame rate is expected to increase
when VisIt moves to VTK 7.x; development for that is underway.

Using Mesa/OpenSWR

VisIt already uses Mesa, but includes a version that does not have
OpenSWR built in. The newer version is a drop-in replacement, making it
easy to get OpenSWR's performance. If you were able to get Mesa through
your package manager, you can skip building. If you downloaded the
source, first make sure you have the dependencies listed
here.
Building Mesa is a simple process using
configure. To specifically use
OpenSWR, run

This will launch the OSPRay-enabled VisIt built above and run the Python script.
The script simply loads the ironProt dataset, adds a Volume plot, and renders it.
The volume render will be done with OSPRay.

Examples

The following show examples of VisIt rendering with OSPRay and Mesa/SWR
using various different plots and on different platforms.