A RGB-D SLAM system for structural scenes, which makes use of point-line-plane features and the Manhattan World assumption.

Related tags

Machine Learning PlanarSLAM
Overview

PlanarSLAM

This repo proposes a RGB-D SLAM system specifically designed for structured environments and aimed at improved tracking and mapping accuracy by relying on geometric features that are extracted from the surrounding. More details can be found in our papers (RGB-D and Monocular).

Authors: Yanyan Li, Raza Yunus, Nikolas Brasch, Nassir Navab and Federico Tombari

License

PlanarSLAM is released under a GPLv3 license.

For commercial purposes, please contact the authors: yanyan.li (at) tum.de. If you use PlanarSLAM in an academic work, please cite:

inproceedings{Li2021PlanarSLAM,
  author = {Li, Yanyan and Yunus, Raza and Brasch, Nikolas and Navab, Nassir and Tombari, Federico},
  title = {RGB-D SLAM with Structural Regularities},
  year = {2021},
  booktitle = {2021 IEEE international conference on Robotics and automation (ICRA)},
 }

1. Prerequisites

We have tested the library in ubuntu 16.04 and ubuntu 18.04, but it should be easy to compile on other platforms. A powerful computer (e.g. i7) will ensure real-time performance and provide more stable and accurate results.

C++11 or C++0x Compiler

We use the new thread and chrono functionalities of C++11.

Pangolin

We use Pangolin for visualization and user interface. Dowload and install instructions can be found at: https://github.com/stevenlovegrove/Pangolin.

OpenCV and OpenCV_Contrib

We use OpenCV and corresponding OpenCV_Contrib to manipulate images and features. Dowload and install instructions can be found at: http://opencv.org. Tested with OpenCV 3.4.1

Eigen3

Required by g2o (see below). Download and install instructions can be found at: http://eigen.tuxfamily.org. Required at least 3.1.0.

DBoW2 and g2o (Included in Thirdparty folder)

We use modified versions of the DBoW2 library to perform place recognition and g2o library to perform non-linear optimizations. Both modified libraries (which are BSD) are included in the Thirdparty folder.

PCL

We use PCL to reconstruct and visualize mesh. Download and install instructions can be found at: https://github.com/ros-perception/perception_pcl. Tested with PCL 1.7.0 and 1.9.0.

  1. https://github.com/PointCloudLibrary/pcl/releases

  2. compile and install

    cd pcl 
mkdir release 
cd release

cmake  -DCMAKE_INSTALL_PREFIX=/usr/local \ -DBUILD_GPU=ON -DBUILD_apps=ON -DBUILD_examples=ON \ -DCMAKE_INSTALL_PREFIX=/usr/local -DPCL_DIR=/usr/local/share/pcl  .. 

make -j12
sudo make install

2. Test the system

Structural Public Scenes

ICL NUIM, Structural TUM RGB-D, All types of Corridors

Test the system locally

  1. Download 'freiburg3_structure_texture_far' and associate RGB-D pairs based on associate.py provided by the dataset.

    python associate.py PATH_TO_SEQUENCE/rgb.txt PATH_TO_SEQUENCE/depth.txt > associations.txt

  2. Compile the system

./build.sh

​ 3. Run the system

./Examples/RGB-D/Planar_SLAM Vocabulary/ORBvoc.txt Examples/RGB-D/TUM3.yaml PATH_TO_SEQUENCE_FOLDER .PATH_TO_SEQUENCE_FOLDER/ASSOCIATIONS_FILE

similar command for testing ICL-NUIM sequences

./Examples/RGB-D/Planar_SLAM Vocabulary/ORBvoc.txt Examples/RGB-D/ICL.yaml PATH_TO_SEQUENCE_FOLDER  PATH_TO_SEQUENCE_FOLDER/ASSOCIATIONS_FILE

Citation

inproceedings{Li2021PlanarSLAM,
  author = {Li, Yanyan and Yunus, Raza and Brasch, Nikolas and Navab, Nassir and Tombari, Federico},
  title = {RGB-D SLAM with Structural Regularities},
  year = {2021},
  booktitle = {2021 IEEE international conference on Robotics and automation (ICRA)},
 }

inproceedings{Li2020SSLAM,
  author = {Li, Yanyan and Brasch, Nikolas and Wang, Yida and Navab, Nassir and Tombari, Federico},
  title = {Structure-SLAM: Low-Drift Monocular SLAM in Indoor Environments},
  year = {2020},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
 }

Acknowledgement

ORB_SLAM2 and the corresponding community.

Comments
  • slow running

    slow running

    Hi. First, thank you for the great work. I run it on my own rgb-d dataset, which img size is 480x640. It takes approximately 2-3 seconds to track a frame. I cannot understand the reason why it works so slowly. Can you suggest anything to improve the speed? Thanks in regard.

    opened by artykov1511 1
  • one question

    one question

    Excuse me， Why are the displayed keypoints so small, did you use an algorithm to improve the feature points? If not, can you tell me how to set it up? I want him to be as big as the key point of ORB-SLAM2. Thank you very much!

    opened by Xuheyi 1
  • Segmentation fault (core dumped)

    Segmentation fault (core dumped)

    Hi there,

    I followed the similar issue and commented set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wall -O3 -march=native ") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -O3 -march=native") as described in #3

    But then the system dead after the 0th image, as shown below:

    __PlanarSLAM Copyright (C) 2021 Yanyan Li, Technical University of Munich (TUM). This program comes with ABSOLUTELY NO WARRANTY; This is free software based on ORB-SLAM2, and you are welcome to redistribute it under certain conditions. See LICENSE.txt.

    Loading ORB Vocabulary. This could take a while... Vocabulary loaded in 5.67s Vocabulary loaded!

    img_width = 640 img_height = 480 mUndistX size = 480 x 640mUndistY size = 480 x 640

    Camera Parameters:

    • fx: 481.2
    • fy: -480
    • cx: 319.5
    • cy: 239.5
    • k1: 0
    • k2: 0
    • p1: 0
    • p2: 0
    • fps: 30
    • color order: RGB (ignored if grayscale)

    ORB Extractor Parameters:

    • Number of Features: 1000
    • Scale Levels: 8
    • Scale Factor: 1.2
    • Initial Fast Threshold: 20
    • Minimum Fast Threshold: 7

    Depth Threshold (Close/Far Points): 3.32502 PlanarSLAM Printer: This is the 0th image Matching planes... plane 0: p1 [-0.00024552766, -0.99999911, -0.0013000764, 0.90096229]: 1, p2 : [-0.0001090349, -0.00013670312, -1, 3.9469013] 1, angle : 0.00143681 vertical! plane 1: p1 [-0.0001090349, -0.00013670312, -1, 3.9469013]: Matching done p1: [0.00024552766; 0.99999911; 0.0013000764] p2: [0.0001090349; 0.00013670312; 1] p3: [0.99999893; -0.00024538589; -0.00010900124] PlanarSLAM Printer: This is the 1th image Segmentation fault (core dumped)__

    any hints for this? thanks in advance

    opened by dwindy 1
  • Removing hardcoded shape and depth map factor parameters from PlaneExtractor.h file

    Removing hardcoded shape and depth map factor parameters from PlaneExtractor.h file

    Now parameters such as image size and depth map factor are hardcoded in the PlaneExtractor.h file. This causes an error when trying to run program with images that do not match these parameters.

    This PR contains the necessary fixes for reading shape and depth map factor from the config file.

    It is important to mention that in PlaneExtractor.cpp:45 the division sign was replaced with the multiplication sign, since the inverse element is taken from config file.

    opened by vnmsklnk 0
  • double free or corruption (out) Signal: SIGABRT (Aborted)

    double free or corruption (out) Signal: SIGABRT (Aborted)

    Thank you for your great job. When I perform test on 'rgbd_dataset_freiburg3_structure_texture_far', "double free or corruption (out) Signal: SIGABRT (Aborted)" appears. Details can be found in the attached pictures. My configure is: Ubuntu18.04 PCL1.9.0 Opencv3.4.0 Eigen3.3.4 Look forward to hearing you. Best regards.

    1 2 3

    opened by 546454596 2
  • vtkPNGWriter error

    vtkPNGWriter error

    Hello, when I run PlanarSLAM on the freiburg3_structure_texture_far dataset, I am getting the following error:

    PlanarSLAM Printer: This is the 1th image ERROR: In /build/vtk6-VHOYAG/vtk6-6.3.0+dfsg1/IO/Image/vtkPNGWriter.cxx, line 260 vtkPNGWriter (0x7f5d80348240): Unable to open file mesh/000000.png ERROR: In /build/vtk6-VHOYAG/vtk6-6.3.0+dfsg1/IO/Image/vtkImageWriter.cxx, line 505 vtkPNGWriter (0x7f5d80348240): Ran out of disk space; deleting file(s) already written Segmentation fault (core dumped)

    Compilation is passed. After checking, it seems that it is not the reason for insufficient disk space, because there are more than 10 G free spaces.

    The versions of some third-party libraries I use are as follows：

    1. Pangolin = 0.5
    2. OpenCV & OpenCV_Contrib = 3.4.5
    3. Eigen3 = 3.3.7
    4. PCL = 1.8 (coming with ROS Melodic )

    Is it caused by the incompatibility of PCL and VTK versions? Whether the version of PCL must be 1.7 or 1.9? Can you give me some advice? or which version are you using？Best regards.

    opened by Rothschild-hhu 2
  • Segmentation fault on the TranslationOptimization function

    Segmentation fault on the TranslationOptimization function

    Hello, I have only tried some datasets (OF_KT0, LR_KT2, and the rgbd_dataset_freiburg3_long_office_household) with segmentation fault result. It is produced in: https://github.com/yanyan-li/PlanarSLAM/blob/2720f5c112bbb401604623fc3df716936b857894/src/Optimizer.cc#L3462 More accurately, when it is called after the bool Tracking::TranslationEstimation() function. I use Ubuntu 18.04, PCL 1.9 and the version implemented of DBoW2 and g2o by ORB-SLAM runs correctly in my system.

    Thanks in advance.

    opened by joanpepcompany 3
Owner
Yanyan Li
Ph.D. candidate in Technical University of Munich
Yanyan Li
GitHub
A toolkit for making real world machine learning and data analysis applications in C++

dlib C++ library Dlib is a modern C++ toolkit containing machine learning algorithms and tools for creating complex software in C++ to solve real worl

Davis E. King 11.6k Dec 31, 2022
R2LIVE is a robust, real-time tightly-coupled multi-sensor fusion framework, which fuses the measurement from the LiDAR, inertial sensor, visual camera to achieve robust, accurate state estimation.

R2LIVE is a robust, real-time tightly-coupled multi-sensor fusion framework, which fuses the measurement from the LiDAR, inertial sensor, visual camera to achieve robust, accurate state estimation.

HKU-Mars-Lab 602 Jan 5, 2023
libsvm websitelibsvm - A simple, easy-to-use, efficient library for Support Vector Machines. [BSD-3-Clause] website

Libsvm is a simple, easy-to-use, and efficient software for SVM classification and regression. It solves C-SVM classification, nu-SVM classification,

Chih-Jen Lin 4.3k Jan 2, 2023
TensorFlow Serving is a flexible, high-performance serving system for machine learning models, designed for production environments.

A flexible, high-performance serving system for machine learning models

null 5.7k Jan 4, 2023
Colibri core is an NLP tool as well as a C++ and Python library for working with basic linguistic constructions such as n-grams and skipgrams (i.e patterns with one or more gaps, either of fixed or dynamic size) in a quick and memory-efficient way. At the core is the tool ``colibri-patternmodeller`` whi ch allows you to build, view, manipulate and query pattern models.

Colibri Core by Maarten van Gompel, [email protected], Radboud University Nijmegen Licensed under GPLv3 (See http://www.gnu.org/licenses/gpl-3.0.html

Maarten van Gompel 122 Nov 17, 2022
Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

eXtreme Gradient Boosting Community | Documentation | Resources | Contributors | Release Notes XGBoost is an optimized distributed gradient boosting l

Distributed (Deep) Machine Learning Community 23.6k Dec 30, 2022
Minerva: a fast and flexible tool for deep learning on multi-GPU. It provides ndarray programming interface, just like Numpy. Python bindings and C++ bindings are both available. The resulting code can be run on CPU or GPU. Multi-GPU support is very easy.

Minerva: a fast and flexible system for deep learning Latest News We've cleared quite a lot of Minerva's dependencies and made it easier to build. Bas

Distributed (Deep) Machine Learning Community 682 Dec 28, 2022
SSL_SLAM2: Lightweight 3-D Localization and Mapping for Solid-State LiDAR (mapping and localization separated) ICRA 2021

SSL_SLAM2 Lightweight 3-D Localization and Mapping for Solid-State LiDAR (Intel Realsense L515 as an example) This repo is an extension work of SSL_SL

Wang Han 王晗 337 Dec 27, 2022
A lightweight C++ machine learning library for embedded electronics and robotics.

Fido Fido is an lightweight, highly modular C++ machine learning library for embedded electronics and robotics. Fido is especially suited for robotic

The Fido Project 413 Dec 17, 2022
MITIE: library and tools for information extraction

MITIE: MIT Information Extraction This project provides free (even for commercial use) state-of-the-art information extraction tools. The current rele

null 2.8k Dec 29, 2022
LIO-SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping

A real-time lidar-inertial odometry package. We strongly recommend the users read this document thoroughly and test the package with the provided dataset first.

Tixiao Shan 2.2k Jan 4, 2023
Yggdrasil Decision Forests (YDF) is a collection of state-of-the-art algorithms for the training, serving and interpretation of Decision Forest models.

Yggdrasil Decision Forests (YDF) is a collection of state-of-the-art algorithms for the training, serving and interpretation of Decision Forest models. The library is developed in C++ and available in C++, CLI (command-line-interface, i.e. shell commands) and in TensorFlow under the name TensorFlow Decision Forests (TF-DF).

Google 268 Jan 9, 2023
MATLAB and C++ implementations of sideslip angle estimators

sideslip-angle-vehicle-estimation MATLAB and C++ implementations of sideslip angle estimators Factor graph sideslip angle estimator Papers: "A Factor

Libraries for Multibody Dynamics Simulation (MBDS) 10 Oct 15, 2022
LIDAR(Livox Horizon) point cloud preprocessing, including point cloud filtering and point cloud feature extraction (edge points and plane points)

LIDAR(Livox Horizon) point cloud preprocessing, including point cloud filtering and point cloud feature extraction (edge points and plane points)

hongyu wang 12 Dec 28, 2022
ORB-SLAM3 is the first real-time SLAM library able to perform Visual, Visual-Inertial and Multi-Map SLAM with monocular, stereo and RGB-D cameras, using pin-hole and fisheye lens models.

Just to test for my research, and I add coordinate transformation to evaluate the ORB_SLAM3. Only applied in research, and respect the authors' all work.

B.X.W 5 Jul 11, 2022
BAAF-Net - Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021)

Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021) This repository is for BAAF-Net introduce

null 90 Dec 29, 2022
✔️The smallest header-only GUI library(4 KLOC) for all platforms

Welcome to GUI-lite The smallest header-only GUI library (4 KLOC) for all platforms. 中文 Lightweight ✂️ Small: 4,000+ lines of C++ code, zero dependenc

null 6.6k Jan 8, 2023
Program will decode a PNG file into an array and apply the gaussian blur filter. Blurring an image reduces noise by taking the average RGB values around a specific pixel and setting it’s RGB to the mean values you’ve just calculated.

Gaussian Blur with multithreading in C Reading in an image file into a single or 2D array Applying Gaussian filter on image Using multithreading appro

Suman Raj Khanal 7 Nov 24, 2021
Real-Time SLAM for Monocular, Stereo and RGB-D Cameras, with Loop Detection and Relocalization Capabilities

Real-Time SLAM for Monocular, Stereo and RGB-D Cameras, with Loop Detection and Relocalization Capabilities

Raul Mur-Artal 7.8k Jan 5, 2023
Structural variant detection and association testing

wham The wham suite consists of two programs, wham and whamg. wham, the original tool, is a very sensitive method with a high false discovery rate. Th

Zev Kronenberg 93 Nov 14, 2022