OpenVDB - Sparse volume data structure and tools

Aims of this PR:

• Validate CMake build system for Linux/MacOs
• Homogenize FindXxx.cmake modules and all CMakeLists
• Improve and document dependency requirements for all OpenVDB components
• Add missing build options for the core, python, binaries and houdini components
• Doxygen documentation for dependencies and cmake installation.

Incorporates the following PRs:

• https://github.com/AcademySoftwareFoundation/openvdb/pull/150 - OpenEXR fixes
• https://github.com/AcademySoftwareFoundation/openvdb/pull/268 - ILMBase fixes

Does not address:

• https://github.com/AcademySoftwareFoundation/openvdb/pull/160 - I've added NumPy support, but it requires CMake 3.14. This PR adds NumPy support for any version using another custom Find module. We should discuss minimum CMake version.
• https://github.com/AcademySoftwareFoundation/openvdb/pull/171 - GNU vars. Not incorporated, but should be done
• https://github.com/AcademySoftwareFoundation/openvdb/pull/240 - Some additional Windows and Maya updates which need investigating

Some important things that have not been done in this PR:

• glfw, blosc, log4cplus. numpy auto options (use if available)

Known issues:

• I am getting cmake warnings on some paltforms when building against houdini and providing my own boost version or my own ilmbase/openexr version (for Houdini 17.5). To be honest I think these warnings are valid as libHalf, for example, is called the same thing in dsolib/ as it is in any old install of ilmbase. But I'd like to try and resolve these as loads are output.
• NumPy support does not work with Boost 1.65 as it needs Boost::numpy. Better NumPy support will be added in a separate PR.

I realise there are a lot of changes here. I can refactor the branch history if it's desired.

Test environments include CentOS, travis and MacOS. Note this PR does not include the travis update. This can be found here:

https://github.com/Idclip/openvdb/tree/cmake_travis

• CentOS 7 - g++, runtime tested with and without a Houdini 17 installation. In both cases, uses the same library versions as Houdini.
• MacOS 10.14.3 - clang++, runtime tested with and without a Houdini 17 installation. Of note, tested with Blosc 1.16 where there are issues, Boost 1.68, TBB 2019, ZLIB 1.2.11.
• Travis - Latest build - https://travis-ci.com/Idclip/openvdb/builds/104392461

To Test:

Note that you can provide the ROOT variables as CMake defines instead of exporting the environment. These minimal option sets may not work first time, but will provide a good starting point.

#!/bin/bash

rm -rf .build
mkdir .build
cd .build

# No pkg-config for tbb
export TBB_ROOT=

cmake \
 -D CMAKE_INSTALL_PREFIX=<install location> \
 ../

#!/bin/bash

rm -rf .build
mkdir .build
cd .build

export HOUDINI_ROOT= # Or source houdini env

cmake \
 -D CMAKE_INSTALL_PREFIX=<install location> \
 -D OPENVDB_BUILD_HOUDINI_SOPS=ON \
 ../

Copy the half implementation from openexr2.4.

Inline the lookup tables rather than using separate files. Converts OpenVDB-style export tags. Namespace into math library.

Getting error about buffer exceeds BLOSC_MAX_BUFFERSIZE see:

https://github.com/dreamworksanimation/openvdb/blob/a7a1abde7c955d5a83acac2c82341497d73576c2/openvdb/points/StreamCompression.cc#L69

I am behind a firewall or I would upload the vdb in question.

It is a points vdb, which we are rendering with our own neat little Arnold Points VDB Procedural in HtoA in Arnold.

Houdini gives the same error:

Blosc decompress failed due to exceeding maximum buffer size.

Is this a known issue, what is the work around?

Renders are failing, people are wailing.

Hi,

Per https://github.com/SteveMacenski/spatio_temporal_voxel_layer/issues/167 and https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=951704, there's an issue with loading OpenVDB 6.2 in Ubuntu Focal due to the dependency on jemalloc. It has been reproduced by multiple users and traced back to OpenVDB.

There's a note that it needs to be compiled with --disable-initial-exec-tls to make things work properly

After installing all dependencies and building OpenVDB from https://github.com/AcademySoftwareFoundation/openvdb, I ran TestOpenVDB.py and got the following message:

libboost_python37.so.1.68.0: cannot open shared object file: No such file or directory

I ran "apt-file search libboost_python" in the Terminal to make sure the package was nowhere in my computer and it was not. I then ran an extensive "aptitude search boost 1.68" search in the Terminal and the closest named file to the requirement was lib64boost-python37-1.68.0. I installed it and then ran the TestOpenVDB.py file again but it still gave me the same message, seeing as it is not the same file. I did extensive searches of lib64 and libboost with the 1.68 1.67 and 1.71 extension and I could not find the elusive libboost_python37.so.1.68.0 which python is asking for.

I am running Ubuntu 20.04 and Python 3.8.3, which are currently the most updated versions. Please help with this difficult issue, I would sincerely appreciate it. It is likely that I am not the only person with this problem right now.

Sincerely, Andrew

I’ve noticed you use lookup tables and bit tricks in performance-critical parts of the code, for functionality that’s implemented in hardware.

Intel introduces BSR and BSF instructions in 80386, i.e. they support them since 1985.

POPCNT is slightly newer, but it’s reasonable to assume it’s present, introduced in SSE4a by AMD in 2007, supported by Intel since 2008. Most people don't use 12 years old computers.

This pull request implements them for vc++, gcc and clang compilers. For unknown ones it falls back to your original software implementation.

Please let me know if you’re willing to accept the PR, is yes I should probably implement a couple of tests, so far I've only built with vc++ but haven't tested yet. Also you wrote you want me to sigh something.

If you’re not interested, I’ll just keep these changes in my private branch.

This PR introduces the core OpenVDB AX Library, binaries, documentation and required CMake changes. The Houdini AX SOP can come after this has been reviewed. A few "high level" thoughts:

• Language docs. Still need work, but the core specification w.r.t types and operators is available https://idclip.github.io/openvdbax-doxygen/ax.html I think this may require its own meeting to fully review.
• vdb_ax thoughts on the command line binaries interface and commands would be good.
• Directory structure. With the new nested infrastructure, it's entirely possible to instead of openvdb_ax have openvdb/ax as an include path.
• Boost::random. As far as the dependency goes, I'm planning to remove it and switch to std::mt19937/mt19937_64. I can update this PR once this is available - however I'd like to find some time in a TSC meeting to discuss the actual implementation of methods such as rand().
• AX CI - Currently AX is tested for vfx platforms 19 and 20, and llvm versions 6.0 through to 10.0. It builds AX separately and depends on an existing installation of VDB. I like this as it splits out the CI somewhat, but means that building VDB and AX together isn't tested.
• Lib version. It currently has it's own version.h and .so version. Does anyone feel strongly on this being kept separate?

There is one final change that I'm hopeful will make it in to AX before 7.2 is released in regards to how exceptions are handled throughout the lib - however this doesn't stall reviewing/merging this work and actually may function as a good first test PR once this is all in!

/var/tmp/portage/media-gfx/openvdb-4.0.2/work/openvdb-4.0.2/openvdb/python/pyFloatGrid.cc: In function ‘void exportFloatGrid()’: /var/tmp/portage/media-gfx/openvdb-4.0.2/work/openvdb-4.0.2/openvdb/python/pyFloatGrid.cc:50:9: error: ‘py::numeric’ has not been declared py::numeric::array::set_module_and_type("numpy", "ndarray"); ^~~~~~~

python/numeric.hpp removed from boost https://github.com/boostorg/python/commit/2d1f66fd1948fa28fde05f4c533b13fb3c96d782

• Added options to enable/disable the python test and excluded the interpreter requirement if disabled
• Improved the report/error logic if python/numpy is not found
• Fixed Windows C4146/C4996 warnings

Main changes in this PR fix issues using python3, different versions of boost python (which can end up being called boost_python, boost_pythonX or boost_pythonXx) and adds support for boost numpy post boost 1.65.

Operating System: (e.g. Windows 2019, macOS-10.15, Linux CentOS 7.4) Version / Commit SHA: (e.g VDB 7.2, SHA: 1a2b3c4d5e...) CMake Version: (e.g. 3.18) Compiler: (e.g. gcc 10, msvc 2019)

Hi, is there possibility to use OpenVDB in Python on Win10? Because when I install openvdb package and import library

>>> import pyopenvdb or >>> import openvdb

I obtain following error: ImportError: LD_LIBRARY_PATH environment variable not set. The pyopenvdb package currently only work on Linux x86_64 systems.

Is there a way how can I use openvdb library on Win 10 (exclude c++) or how can I create openvdb volume model in Python by using your library?

Thank you very much for your advice. Have a nice day.

When I made the changes to switch from GEOvdbProcessTypedGrid to GEOvdbApply, I accidentally introduced this bug in the projection mode of the VDB Advect Points SOP:

https://github.com/AcademySoftwareFoundation/openvdb/pull/591/files#diff-3da92c28c566b6736ba31e46a2c8a84a00c8beada13f7c3cda90de1782014874R1022

This change fixes the bug so that this mode no longer crashes.

@jmlait

Environment

Windows 10 10.0.0 through vcpkg Visual Studio 2020

Describe the bug

Any function working on a grid will result in an unresolved external symbol imath_half_to_float_table error.

• Creating a grid will work
• getting the library version works.

good -> openvdb::FloatGrid::Ptr grid; error -> openvdb::FloatGrid::Ptr grid = openvdb::FloatGrid::create();

To Reproduce

Steps to reproduce the behavior:

1. Build with vcpkg
2. IN Visual Studio try to build the 'hello world' example from the cookbook
3. See error
4. Try using the various 'use imath half' or 'internal half' settings to either have nothing change or complain that openexr doesn't have a half.h file.

Expected behavior

I expected it to work like before upgrading to v10.

Additional context

It seems like any time that the 'half' code is touched it breaks something with the windows build. Searching for "unresolved external symbol imath_half_to_float_table" actually showed up on the openexr repository issues so it seems like it could be more than a bad openvdb build.

Environment

Operating System: Win10 Version / Commit SHA: VDB 10.0 CMake Version: 3.24.2 Compiler: msvc 2019

Describe the problem

When setting -DOPENVDB_SIMD=SSE42 OpenVDB's cmake passes the GCC style flags directly to MSVC over in CMakeLists.txt@386

To Reproduce

Steps to reproduce the behavior:

1. Build with -DOPENVDB_SIMD=SSE42 or -DOPENVDB_SIMD=AVX with MSVC
2. cl : Command line warning D9002: ignoring unknown option '-msse4.2' [c:\db\build\S\VS1564D\openvdb.vcxproj]

Additional context

MSVC doesn't have a flag for SSE42, but if you call the sse42 intrinsics like openvdb does the right opcodes will still be generated, AVX can be enabled with /arch:AVX source

Environment

Operating System: Gentoo 2.9 Version / Commit SHA: VDB 10.0.1 CMake Version: 3.25.1 Compiler: gcc 11

Describe the problem

Trying to build the vdb_tool binary fails in cmake configuration, because it uses OpenEXR-2 library OpenEXR:IlmImf. Replacing the link line and using OpenEXR:OpenEXR as the link library for OpenEXR-3 let the tool be built. The error is:

-- Configuring done
CMake Error at openvdb_cmd/vdb_tool/CMakeLists.txt:96 (target_link_libraries):
  The link interface of target "vdb_tool_common" contains:

    OpenEXR::IlmImf

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.



-- Generating done

To Reproduce

Steps to reproduce the behavior:

1. Download the 10.0.1 release tarball from GitHub
2. Run CMake

cmake -G Ninja -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_DOCDIR=share/doc/openvdb-10.0.1/ -DOPENVDB_ABI_VERSION_NUMBER=10 -DOPENVDB_BUILD_DOCS=no -DOPENVDB_BUILD_UNITTESTS=yes -DOPENVDB_BUILD_VDB_LOD=yes -DOPENVDB_BUILD_VDB_RENDER=yes -DOPENVDB_BUILD_VDB_TOOL=yes -DOPENVDB_BUILD_VDB_VIEW=yes -DOPENVDB_CORE_SHARED=ON -DOPENVDB_CORE_STATIC=no -DOPENVDB_ENABLE_RPATH=OFF -DUSE_AX=no -DUSE_BLOSC=no -DUSE_CCACHE=OFF -DUSE_COLORED_OUTPUT=ON -DUSE_EXR=yes -DUSE_IMATH_HALF=ON -DUSE_LOG4CPLUS=ON -DUSE_NANOVDB=no -DUSE_PKGCONFIG=ON -DUSE_PNG=yes -DUSE_STATIC_DEPENDENCIES=no -DUSE_TBB=ON -DUSE_ZLIB=no -DBUILD_TEST=yes -DOPENVDB_BUILD_VDB_AX=no -DOPENVDB_TOOL_NANO_USE_BLOSC=OFF -DOPENVDB_TOOL_USE_ABC=OFF -DOPENVDB_TOOL_USE_EXR=ON -DOPENVDB_TOOL_USE_JPG=ON -DOPENVDB_TOOL_USE_NANO=no -DOPENVDB_TOOL_USE_PNG=ON -DOPENVDB_SIMD=AVX -DCMAKE_BUILD_TYPE=RelWithDebInfo

3. See error above

Additional context

I removed the command line options for our toolchain file and initial cache from the above command for better readability. They don't have an influence on the outcome.

This is documented at:

include_directories(${LLVM_INCLUDE_DIRS})
separate_arguments(LLVM_DEFINITIONS_LIST NATIVE_COMMAND ${LLVM_DEFINITIONS})
add_definitions(${LLVM_DEFINITIONS_LIST})

• https://github.com/llvm/llvm-project/blob/main/llvm/docs/CMake.rst#embedding-llvm-in-your-project

See also discussion at:

• https://discourse.cmake.org/t/how-to-use-target-compile-definitions/6982