VDBFusion: Flexible and Efficient TSDF Integration

Overview

VDBFusion: Flexible and Efficient TSDF Integration

C++ Python Linux PyPI version shields.io PRs Welcome Paper MIT license Open In Colab

example

This is a small utility library that implements the VDBFusion algorithm, similar to TSDF-based reconstruction pipelines but using a different data-structure (VDB).

Installation

Take a seat and relax, you only need to:

pip install vdbfusion

If you plan to use our C++ API then you should build this project from source. More details in the Installation instructions.

Usage

The code shown below is not intended to be copy pasted but rather be a spiritual guide for developers. If you really want to give this library a try you should consider checking the standalone Python, Notebooks, and C++ examples.

Data loading

NOTE: This step is not mandatory. Our API only expects points and poses but this is the easiest way to deal with 3D data.

Python C++
class Dataset:
    def __init__(self, *args, **kwargs):
        # Initialize your dataset here ..

    def __len__(self) -> int:
        return len(self.n_scans)

    def __getitem__(self, idx: int):
        # Returns a PointCloud(np.array(N, 3))
        # and sensor origin(Eigen::Vector3d)
        # in the global coordinate frame.
        return points, origin
class Dataset {
  // Initialize your dataset here ..
  Dataset(...);

  // Return length of the dataset
  std::size_t size() const { return n_scans_; }

  // Returns a Cloud(std::vector
       
        )
       
  // and the sensor origin(Eigen::Vector3d) in the
  // global coordinate frame.
  std::tuple
       Point> 
       operator[](
       int idx) 
       const;
};
      

TSDF Fusion pipeline

Python C++
import vdbfusion

vdb_volume = vdbfusion.VDBVolume(voxel_size,
                                 sdf_trunc,
                                 space_carving
dataset = Dataset(...)

for scan, origin in dataset:
    vdb_volume.integrate(scan, origin)
#include "vdbfusion/VDBVolume.h"

vdb_fusion::VDBVolume vdb_volume(voxel_size,
                                 sdf_trunc,
                                 space_carving);
const auto dataset = Dataset(...);

for (const auto& [scan, origin] : iterable(dataset)) {
  vdb_volume.Integrate(scan, origin);
}

Visualization

For visualization you can use any 3D library you like. For this example we are going to be using Open3D. If you are using the Python API make sure to pip install open3d before trying this snippet.

Python C++
import open3d as o3d

# Extract triangle mesh (numpy arrays)
vert, tri = vdb_volume.extract_triangle_mesh()

# Visualize the results
mesh = o3d.geometry.TriangleMesh(
    o3d.utility.Vector3dVector(vert),
    o3d.utility.Vector3iVector(tri),
)

mesh.compute_vertex_normals()
o3d.visualization.draw_geometries([mesh])
#include <open3d/Open3D.h>

// Extract triangle mesh (Eigen).
auto [verts, tris] = vdb_volume.ExtractTriangleMesh();

// Visualize the results
auto mesh = o3d::geometry::TriangleMesh(
    verts,
    tris,
)

mesh.ComputeVertexNormals()
o3d::visualization::DrawGeometries({&mesh})

LICENSE

The LICENSE can be found at the root of this repository. It only applies to the code of VDBFusion but not to its 3rdparty dependencies. Please make sure to check the licenses in there before using any form of this code.

Credits

I would like to thank the Open3D and OpenVDB authors and contributors for making their implementations open source which inspired, helped and guided the implementation of the VDBFusion library.

Citation

If you use this library for any academic work, please cite the original paper.

@article{vizzo2022sensors,
  author         = {Vizzo, Ignacio and Guadagnino, Tiziano and Behley, Jens and Stachniss, Cyrill},
  title          = {VDBFusion: Flexible and Efficient TSDF Integration of Range Sensor Data},
  journal        = {Sensors},
  volume         = {22},
  year           = {2022},
  number         = {3},
  article-number = {1296},
  url            = {https://www.mdpi.com/1424-8220/22/3/1296},
  issn           = {1424-8220},
  doi            = {10.3390/s22031296}
}
Comments
  • cannot make install succeed

    cannot make install succeed

    hello, I want to use C++ API, then I followed your instructions Build from source in Linux. But, when I tried to use sudo make instal, I got some messages. make_install.log

    You know, most of the CMakeLists.txt projects can be installed as the below instructions:

    git clone https://github.com/PRBonn/vdbfusion.git
    cd vbdfusion && mkdir build && cmake ..
    make -j4
    

    Some things looked better than before( Because I can see the download packages process). However, I also got some messages; make_j4.log

    As the terminal output to screen, I tried to find what happened in make. bootstrap.log

    It just looks like having some problems with boost, I found some answers to this question(undefined reference to), but I don't know if it's correct. By the way, my system(Ubuntu 20.04) has installed boost(1.71), and your packages also need to reinstall boost as 3third libraries(the required boose version also is 1.71).

    I don't know the next what I should do. Please give some tips. So, I wish you can help me solve this problem when you have little time. Thank you!

    opened by yinloonga 5
  • Comparison to Voxblox in paper

    Comparison to Voxblox in paper

    Hallo, since I have been using voxblox for a while. So far I am pretty satisfied with the performance it can bring. Especially the speed of its fast integrator. So as for the runtime, the experiments in paper point out the VDB's runtime is even faster than voxblox. Since there are three integrator types in voxblox, i.e. simple, merged and fast and apparently it means a lot to runtime, I am wondering which integrator type specifically does the comparison use? Thanks in advance for your reply!

    opened by Willyzw 5
  • Worse results than the paper displayed

    Worse results than the paper displayed

    I'm running the kitti 00 sequence with python API, but I got the result below: image image

    which is much worse than the paper displayed:

    image

    I got this result without editing the config file.

    And I repeated it by changing the fill_holes to True, but the result did not change.

    Why there are two different results?

    opened by Rotatingpencil 4
  • ROS Integration example

    ROS Integration example

    Hi,

    I'm currently looking at integrating this algorithm to our ROS based robot software stack. Our robot is equipped with a LiDAR, a RealSense D435i and encoders. We're currently using the gmapping SLAM algorithm to provide the transform from the /map frame to the /realsense_depth_optical_frame.

    I was wondering if you managed to integrate this library with ROS in the past and if you have pointers on how to integrate it with our robot?

    Thanks! Alex

    opened by alex-ssom 4
  • Extrinsics in RGBD integration

    Extrinsics in RGBD integration

    Hallo, while trying out the example for RGB-D image based integration, I notice that the extrinsic/global poses are inverted. Since normally the global poses are given as camera to world, shall I interpret the inverted pose as world to camera transformation?

    Lets take an example, the global pose is inverted here for scannet dataset. https://github.com/PRBonn/vdbfusion/blob/40ad3021c271c26d638672531818f86d2c91d3b8/examples/python/datasets/scannet.py#L33

    Then the RGBD point cloud is transformed using the inverted pose here https://github.com/PRBonn/vdbfusion/blob/40ad3021c271c26d638672531818f86d2c91d3b8/examples/python/datasets/scannet.py#L61-L63

    So far is fine, as I vaguely remember open3d indeed assumes an extrinsic from world to camera. However, this inverted pose is further fed to the integration pipeline then apparently regarded as the frame origin. I am wondering whether this is really the purpose? https://github.com/PRBonn/vdbfusion/blob/40ad3021c271c26d638672531818f86d2c91d3b8/examples/python/datasets/scannet.py#L74

    opened by Willyzw 3
  • installing VDBFusion C++ API

    installing VDBFusion C++ API

    Hello!

    I am attempting to install VDBFusion, including the C++ API, using the Docker image.

    This is what I get as a message towards the end of the Docker Compose process:

    #14 3.984 -- ----------------------------------------------------
    #14 3.984 -- ----------- Configuring OpenVDBBinaries ------------
    #14 3.984 -- ----------------------------------------------------
    #14 3.989 -- Found TBB: /usr/include (found suitable version "2020.1", minimum required is "2019.0") found components: tbbmalloc 
    #14 3.992 -- Configuring done
    #14 4.032 -- Generating done
    #14 4.034 -- Build files have been written to: /openvdb/build
    #14 4.175 Scanning dependencies of target openvdb_static
    #14 4.179 Scanning dependencies of target openvdb_shared
    #14 4.200 [  2%] Building CXX object openvdb/openvdb/CMakeFiles/openvdb_static.dir/instantiations/VolumeAdvect.cc.o
    #14 4.200 [  2%] Building CXX object openvdb/openvdb/CMakeFiles/openvdb_static.dir/instantiations/VolumeToSpheres.cc.o
    #14 4.200 [  2%] Building CXX object openvdb/openvdb/CMakeFiles/openvdb_static.dir/instantiations/VolumeToMesh.cc.o
    #14 4.201 [  2%] Building CXX object openvdb/openvdb/CMakeFiles/openvdb_shared.dir/instantiations/VolumeToSpheres.cc.o
    #14 46.93 c++: fatal error: Killed signal terminated program cc1plus
    #14 46.93 compilation terminated.
    #14 46.93 make[2]: *** [openvdb/openvdb/CMakeFiles/openvdb_static.dir/build.make:63: openvdb/openvdb/CMakeFiles/openvdb_static.dir/instantiations/VolumeToSpheres.cc.o] Error 1
    #14 46.93 make[2]: *** Waiting for unfinished jobs....
    #14 47.07 [  3%] Building CXX object openvdb/openvdb/CMakeFiles/openvdb_shared.dir/instantiations/VolumeToMesh.cc.o
    #14 97.09 c++: fatal error: Killed signal terminated program cc1plus
    #14 97.09 compilation terminated.
    #14 97.10 make[2]: *** [openvdb/openvdb/CMakeFiles/openvdb_shared.dir/build.make:63: openvdb/openvdb/CMakeFiles/openvdb_shared.dir/instantiations/VolumeToSpheres.cc.o] Error 1
    #14 97.10 make[2]: *** Waiting for unfinished jobs....
    #14 118.5 make[1]: *** [CMakeFiles/Makefile2:166: openvdb/openvdb/CMakeFiles/openvdb_static.dir/all] Error 2
    #14 118.5 make[1]: *** Waiting for unfinished jobs....
    #14 133.5 make[1]: *** [CMakeFiles/Makefile2:193: openvdb/openvdb/CMakeFiles/openvdb_shared.dir/all] Error 2
    #14 133.5 make: *** [Makefile:141: all] Error 2
    ------
    failed to solve: executor failed running [/bin/sh -c git clone --depth 1 https://github.com/nachovizzo/openvdb.git -b nacho/vdbfusion     && cd openvdb     && mkdir build && cd build     && cmake     -DOPENVDB_BUILD_PYTHON_MODULE=ON     -DUSE_NUMPY=ON     -DCMAKE_POSITION_INDEPENDENT_CODE=ON     -DUSE_ZLIB=OFF     ..    && make -j$(nproc) all install     && rm -rf /openvdb]: exit code: 2
    

    Would you happen to have insights as to what went wrong? Many thanks.

    opened by doctorsylvainpronovost 2
  • How to get TSDF volume?

    How to get TSDF volume?

    Dear authors, thank you for such awesome tool, it is working like a charm! I wonder, what is the fastest way to get TSDF volume as numpy array, as well its volume bounds?

    opened by anastasiia-kornilova 2
  • Fix the inverted origin in dataloaders

    Fix the inverted origin in dataloaders

    As describe in #2, the global pose was inverted to conform to the open3d convention, which assumes a transformation from world to camera. This however leads to the wrong origin position. This pull request fixed this problem at three RGBD image based examples.

    opened by Willyzw 2
  • Dataset example for Argoverse v2?

    Dataset example for Argoverse v2?

    First, I'd like to thank you for creating a such nicely developed library and introducing me to OpenVDB!

    The Argoverse v2 dataset (https://www.argoverse.org/av2.html#sensor-link) was recently released. I wonder if you could provide a quick sample of Dataset and Pipeline, similar to the other ones (such as KITTI and nuScenes). Thank you again!

    opened by changyuan79 1
  • Python examples cache.py error

    Python examples cache.py error

    args_to_key() is missing the ignore parameter.

    https://github.com/PRBonn/vdbfusion/blob/8d01832419050aed1105e666040c4e5d3a14c96c/examples/python/utils/cache.py#L38

    opened by swarmt 1
  • Build VDBFusion on MSVC 2019/Win 10

    Build VDBFusion on MSVC 2019/Win 10

    I am trying to build this repo on MSVC 2019/Windows 10. I used Cmake to configure and Generate, it was smooth. When I build the generated project, I face the error below, does anyone have any idea to fix it? image

    opened by SimonNgj 1
  • Realtime Point Cloud extraction

    Realtime Point Cloud extraction

    Introduction

    For real-time applications, it is very useful to visualize the volume, for instance for debugging. This can be done currently by using marching cubles which generates a mesh. The problem is that generating meshes is very computationally expensive, unless recent commits to OpenVSB make it fast and high-quality enough. This implies, the volume currently cannot be displayed in real time.

    Besides, to do ICP to correct TFs/Frames, which is necessary to do real-time, it'd be necessary to have cloudpoints in realtime, which, as said, it is not currently possible. Note that to correct errors that occur during integration ( for instance "Lady and Cow" ), ICP would potentially help. Also note, this is used in Voxblox

    Note that Open3D, in its tensor SLAM pipeline, does implement this and does work realtime ( example )

    Proposed Work

    1 - Test whether recent changes in OpenVDB make generation of meshes quality and speed wise good enough. Also analyze different MeshToVolumeFlags 2 - In case point (1) does not hold true, then implement either in OpenVDB or in vdbfusion a way to quickly extract pointclouds ( maybe optionally with normals) in realtime.

    Questions

    I'd like to know. 1- Generally what's your opinion/POV regarind this line of work in terms of utility, feasabliitty etc. 2- Whether that's something you have planned to do or will do 3- Whether you need extra help with that.

    opened by paucarre 1
  • Any trial on ESDF and gradient computation

    Any trial on ESDF and gradient computation

    Hi, thanks for your great work! From the SLAM perspective, it would be better if the residual and gradient can be computed from the map to support frame-to-mesh registration (as you did in the PUMA paper, this is another good work). Also, the generation of the ESDF can be used for robotic planning. Have you tried these two directions? Or could you please share some hints on how to achieve them? I will try to make some contributions ^v^. Thanks.

    opened by gogojjh 8
Releases(v0.1.6)
  • v0.1.6(Mar 23, 2022)

    • Minor changes on Python API.
    • New method to merge incoming TSDF grids following the traditional TSDF equations, both for the update_tsdf and integrate methods.
    • Add prune method to clean up the grid according to a weight threshold.
    • Python bindings build now optional.
    Source code(tar.gz)
    Source code(zip)
  • v0.1.5(Feb 8, 2022)

Owner
Photogrammetry & Robotics Bonn
Photogrammetry & Robotics Lab at the University of Bonn
Photogrammetry & Robotics Bonn
A terse, flexible language and runtime for creating and executing visual novels.

Fabulist A terse, flexible language and runtime for creating and executing visual novels. Contributing We're open to contributions from anyone and eve

NovelRT 6 Sep 21, 2022
foxBMS is a free, open and flexible development environment to design battery management systems.

foxBMS is a free, open and flexible development environment to design battery management systems. It is the first modular open source BMS development platform.

The foxBMS Team 78 Oct 1, 2022
☕ GDBFrontend is an easy, flexible and extensionable gui debugger.

GDBFrontend is an easy, flexible and extensionable gui debugger. Installing Requirements GDB => 8.2 (with python3) python3 => 3.2 tmux PIP Package (Py

Oğuzhan Eroğlu 2.4k Sep 26, 2022
the implementations of 'A Flexible New Technique for Camera Calibration' and Bouguet's method

StereoCameraCalibration MonocularCameraCalibration/StereoCameraCalibration/StereoCameraRectification 1、Class "MonocularCameraCalibration" provides the

gtc1072 7 Sep 23, 2022
Raspberry Pi Pico (RP2040) and Micro-ROS (ROS 2) Integration

The Pico is an amazing microcontroller and I couldn't wait for ROS 2 support or Arduino Core, so here is my approach. Once the Arduino Core for RP2040 is out it will be easier to use micro_ros_arduino.

Darko Lukić 19 Jun 19, 2022
JWM is a cross-platform Java window management and OS integration library.

JWM aims to expose an interface that abstracts over window creation, input handling and OS integration

JetBrains 361 Sep 30, 2022
2D/3D Registration and system integration for image-based navigation of orthopedic robotic applications, inculding femoroplasty, osteonecrosis, etc.

Registration and System Integration Software for Orthopedic Surgical Robotic System This repository contains software programs for image-based registr

Cong Gao 15 Sep 3, 2022
JWM - Cross-platform window management and OS integration library for Java

Java Window Management library Vision JWM is a cross-platform Java window management and OS integration library. JWM aims to expose an interface that

Humble UI 361 Sep 30, 2022
BNFLite is a C++ template library for lightweight flexible grammar parsers

BNFLite is a C++ template library for lightweight flexible grammar parsers. BNFLite offers creative approach when the developer can specify a language for further parsing directly in the C++ code. Moreover, such "specifications" are executable now!

Alexander S 61 Aug 18, 2022
Flexible, portable, high-performance bit fields C++ library. unsigned a:13 becomes F<13> a;

C-plus-plus-library-bit-fields Flexible, portible, high-performance bit fields C++ library. The bit fields are specified with a dummy structure where

Walt Karas 26 Aug 31, 2022
FlexOS: Towards Flexible OS Isolation (ASPLOS'22) Artifact Evaluation Repository

FlexOS ASPLOS'22 Artifact Evaluation This repository contains the artifacts, including experiments and graphs, for the paper: FlexOS: Towards Flexible

null 12 Aug 24, 2022
A flexible image resampling library

Image Resampling Library This is a simple yet flexible image resampling library that supports the following image samplers: Nearest Average Bilinear G

null 39 Sep 15, 2022
Solving Kepler's equation via contour integration, implemented in C++

Kepler's Goat Herd Code for solving Kepler's equation using contour integration, following Philcox et al. (2021, arXiv). This uses a method originally

Oliver Philcox 45 Sep 11, 2022
ESPHome integration for the Xiaomi Mijia Bedside Lamp v2.

esphome-xiaomi_bslamp2 Warning: this code is still under development This code might not yet be production-ready. At this point, it is declared beta-q

Maurice Makaay 124 Sep 28, 2022
font rendering, Zep vim emulator, microui integration

LabFont This project is an exploration of getting text into a rendeing pipeline based on @floooh's sokol. Frameworks such as Dear ImGui solve text ren

Nick Porcino 14 Sep 11, 2022
A gdnative plugin for Godot's UWP export to add xbox live integration

GodotXbox Current instructions for setting up DLL, steps to setup project for contributing/modifying are coming soon. It should be noted that Godot's

Cregg Hancock 34 Sep 27, 2022
Yet Another Ghidra Integration for IDA

Yagi Yet Another Ghidra Integration for IDA Overview Yagi intends to include the wonderful Ghidra decompiler into both IDA pro and IDA Free. ?? You ca

Airbus CERT 378 Sep 23, 2022
X-CUBE-AZRTOS-F4 (Azure RTOS Software Expansion for STM32Cube) provides a full integration of Microsoft Azure RTOS in the STM32Cube environment for the STM32F4 series of microcontrollers.

X-CUBE-AZRTOS-F4 Azure RTOS Software Expansion for STM32Cube With Azure RTOS complementing the extensive STM32Cube ecosystem providing free developmen

STMicroelectronics 26 Sep 22, 2022
Integration of Rust code with a C Kernel Module

Rust-Kernel-Mod Description Integration of Rust code with a C Kernel Module This project sets up a minimalistic Linux kernel module in C which it uses

Dylan Turner 1 Nov 1, 2021