A RGB-D SLAM system for structural scenes, which makes use of point-line-plane features and the Manhattan World assumption.

Overview

PlanarSLAM

This repo proposes a RGB-D SLAM system specifically designed for structured environments and aimed at improved tracking and mapping accuracy by relying on geometric features that are extracted from the surrounding. More details can be found in our papers (RGB-D and Monocular).

Authors: Yanyan Li, Raza Yunus, Nikolas Brasch, Nassir Navab and Federico Tombari

License

PlanarSLAM is released under a GPLv3 license.

For commercial purposes, please contact the authors: yanyan.li (at) tum.de. If you use PlanarSLAM in an academic work, please cite:

inproceedings{Li2021PlanarSLAM,
  author = {Li, Yanyan and Yunus, Raza and Brasch, Nikolas and Navab, Nassir and Tombari, Federico},
  title = {RGB-D SLAM with Structural Regularities},
  year = {2021},
  booktitle = {2021 IEEE international conference on Robotics and automation (ICRA)},
 }

1. Prerequisites

We have tested the library in ubuntu 16.04 and ubuntu 18.04, but it should be easy to compile on other platforms. A powerful computer (e.g. i7) will ensure real-time performance and provide more stable and accurate results.

C++11 or C++0x Compiler

We use the new thread and chrono functionalities of C++11.

Pangolin

We use Pangolin for visualization and user interface. Dowload and install instructions can be found at: https://github.com/stevenlovegrove/Pangolin.

OpenCV and OpenCV_Contrib

We use OpenCV and corresponding OpenCV_Contrib to manipulate images and features. Dowload and install instructions can be found at: http://opencv.org. Tested with OpenCV 3.4.1

Eigen3

Required by g2o (see below). Download and install instructions can be found at: http://eigen.tuxfamily.org. Required at least 3.1.0.

DBoW2 and g2o (Included in Thirdparty folder)

We use modified versions of the DBoW2 library to perform place recognition and g2o library to perform non-linear optimizations. Both modified libraries (which are BSD) are included in the Thirdparty folder.

PCL

We use PCL to reconstruct and visualize mesh. Download and install instructions can be found at: https://github.com/ros-perception/perception_pcl. Tested with PCL 1.7.0 and 1.9.0.

  1. https://github.com/PointCloudLibrary/pcl/releases

  2. compile and install

    cd pcl 
    mkdir release 
    cd release
    
    cmake  -DCMAKE_INSTALL_PREFIX=/usr/local \ -DBUILD_GPU=ON -DBUILD_apps=ON -DBUILD_examples=ON \ -DCMAKE_INSTALL_PREFIX=/usr/local -DPCL_DIR=/usr/local/share/pcl  .. 
    
    make -j12
    sudo make install
    

2. Test the system

Structural Public Scenes

ICL NUIM, Structural TUM RGB-D, All types of Corridors

Test the system locally

  1. Download 'freiburg3_structure_texture_far' and associate RGB-D pairs based on associate.py provided by the dataset.

    python associate.py PATH_TO_SEQUENCE/rgb.txt PATH_TO_SEQUENCE/depth.txt > associations.txt
    
  2. Compile the system

./build.sh

​ 3. Run the system

./Examples/RGB-D/Planar_SLAM Vocabulary/ORBvoc.txt Examples/RGB-D/TUM3.yaml PATH_TO_SEQUENCE_FOLDER .PATH_TO_SEQUENCE_FOLDER/ASSOCIATIONS_FILE

similar command for testing ICL-NUIM sequences

./Examples/RGB-D/Planar_SLAM Vocabulary/ORBvoc.txt Examples/RGB-D/ICL.yaml PATH_TO_SEQUENCE_FOLDER  PATH_TO_SEQUENCE_FOLDER/ASSOCIATIONS_FILE


Citation

inproceedings{Li2021PlanarSLAM,
  author = {Li, Yanyan and Yunus, Raza and Brasch, Nikolas and Navab, Nassir and Tombari, Federico},
  title = {RGB-D SLAM with Structural Regularities},
  year = {2021},
  booktitle = {2021 IEEE international conference on Robotics and automation (ICRA)},
 }
inproceedings{Li2020SSLAM,
  author = {Li, Yanyan and Brasch, Nikolas and Wang, Yida and Navab, Nassir and Tombari, Federico},
  title = {Structure-SLAM: Low-Drift Monocular SLAM in Indoor Environments},
  year = {2020},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
 }

Acknowledgement

ORB_SLAM2 and the corresponding community.

Comments
  • slow running

    slow running

    Hi. First, thank you for the great work. I run it on my own rgb-d dataset, which img size is 480x640. It takes approximately 2-3 seconds to track a frame. I cannot understand the reason why it works so slowly. Can you suggest anything to improve the speed? Thanks in regard.

    opened by artykov1511 1
  • one question

    one question

    Excuse me, Why are the displayed keypoints so small, did you use an algorithm to improve the feature points? If not, can you tell me how to set it up? I want him to be as big as the key point of ORB-SLAM2. Thank you very much!

    opened by Xuheyi 1
  • Segmentation fault (core dumped)

    Segmentation fault (core dumped)

    Hi there,

    I followed the similar issue and commented set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wall -O3 -march=native ") set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -O3 -march=native") as described in #3

    But then the system dead after the 0th image, as shown below:

    __PlanarSLAM Copyright (C) 2021 Yanyan Li, Technical University of Munich (TUM). This program comes with ABSOLUTELY NO WARRANTY; This is free software based on ORB-SLAM2, and you are welcome to redistribute it under certain conditions. See LICENSE.txt.

    Loading ORB Vocabulary. This could take a while... Vocabulary loaded in 5.67s Vocabulary loaded!

    img_width = 640 img_height = 480 mUndistX size = 480 x 640mUndistY size = 480 x 640

    Camera Parameters:

    • fx: 481.2
    • fy: -480
    • cx: 319.5
    • cy: 239.5
    • k1: 0
    • k2: 0
    • p1: 0
    • p2: 0
    • fps: 30
    • color order: RGB (ignored if grayscale)

    ORB Extractor Parameters:

    • Number of Features: 1000
    • Scale Levels: 8
    • Scale Factor: 1.2
    • Initial Fast Threshold: 20
    • Minimum Fast Threshold: 7

    Depth Threshold (Close/Far Points): 3.32502 PlanarSLAM Printer: This is the 0th image Matching planes... plane 0: p1 [-0.00024552766, -0.99999911, -0.0013000764, 0.90096229]: 1, p2 : [-0.0001090349, -0.00013670312, -1, 3.9469013] 1, angle : 0.00143681 vertical! plane 1: p1 [-0.0001090349, -0.00013670312, -1, 3.9469013]: Matching done p1: [0.00024552766; 0.99999911; 0.0013000764] p2: [0.0001090349; 0.00013670312; 1] p3: [0.99999893; -0.00024538589; -0.00010900124] PlanarSLAM Printer: This is the 1th image Segmentation fault (core dumped)__

    any hints for this? thanks in advance

    opened by dwindy 1
  • Removing hardcoded shape and depth map factor parameters from PlaneExtractor.h file

    Removing hardcoded shape and depth map factor parameters from PlaneExtractor.h file

    Now parameters such as image size and depth map factor are hardcoded in the PlaneExtractor.h file. This causes an error when trying to run program with images that do not match these parameters.

    This PR contains the necessary fixes for reading shape and depth map factor from the config file.

    It is important to mention that in PlaneExtractor.cpp:45 the division sign was replaced with the multiplication sign, since the inverse element is taken from config file.

    opened by vnmsklnk 0
  • double free or corruption (out) Signal: SIGABRT (Aborted)

    double free or corruption (out) Signal: SIGABRT (Aborted)

    Thank you for your great job. When I perform test on 'rgbd_dataset_freiburg3_structure_texture_far', "double free or corruption (out) Signal: SIGABRT (Aborted)" appears. Details can be found in the attached pictures. My configure is: Ubuntu18.04 PCL1.9.0 Opencv3.4.0 Eigen3.3.4 Look forward to hearing you. Best regards.

    1 2 3

    opened by 546454596 2
  • vtkPNGWriter  error

    vtkPNGWriter error

    Hello, when I run PlanarSLAM on the freiburg3_structure_texture_far dataset, I am getting the following error:

    PlanarSLAM Printer: This is the 1th image ERROR: In /build/vtk6-VHOYAG/vtk6-6.3.0+dfsg1/IO/Image/vtkPNGWriter.cxx, line 260 vtkPNGWriter (0x7f5d80348240): Unable to open file mesh/000000.png ERROR: In /build/vtk6-VHOYAG/vtk6-6.3.0+dfsg1/IO/Image/vtkImageWriter.cxx, line 505 vtkPNGWriter (0x7f5d80348240): Ran out of disk space; deleting file(s) already written Segmentation fault (core dumped)

    Compilation is passed. After checking, it seems that it is not the reason for insufficient disk space, because there are more than 10 G free spaces.

    The versions of some third-party libraries I use are as follows:

    1. Pangolin = 0.5
    2. OpenCV & OpenCV_Contrib = 3.4.5
    3. Eigen3 = 3.3.7
    4. PCL = 1.8 (coming with ROS Melodic )

    Is it caused by the incompatibility of PCL and VTK versions? Whether the version of PCL must be 1.7 or 1.9? Can you give me some advice? or which version are you using?Best regards.

    opened by Rothschild-hhu 2
  • Segmentation fault on the TranslationOptimization function

    Segmentation fault on the TranslationOptimization function

    Hello, I have only tried some datasets (OF_KT0, LR_KT2, and the rgbd_dataset_freiburg3_long_office_household) with segmentation fault result. It is produced in: https://github.com/yanyan-li/PlanarSLAM/blob/2720f5c112bbb401604623fc3df716936b857894/src/Optimizer.cc#L3462 More accurately, when it is called after the bool Tracking::TranslationEstimation() function. I use Ubuntu 18.04, PCL 1.9 and the version implemented of DBoW2 and g2o by ORB-SLAM runs correctly in my system.

    Thanks in advance.

    opened by joanpepcompany 3
Owner
Yanyan Li
Ph.D. candidate in Technical University of Munich
Yanyan Li
A toolkit for making real world machine learning and data analysis applications in C++

dlib C++ library Dlib is a modern C++ toolkit containing machine learning algorithms and tools for creating complex software in C++ to solve real worl

Davis E. King 11.6k Dec 31, 2022
R2LIVE is a robust, real-time tightly-coupled multi-sensor fusion framework, which fuses the measurement from the LiDAR, inertial sensor, visual camera to achieve robust, accurate state estimation.

R2LIVE is a robust, real-time tightly-coupled multi-sensor fusion framework, which fuses the measurement from the LiDAR, inertial sensor, visual camera to achieve robust, accurate state estimation.

HKU-Mars-Lab 602 Jan 5, 2023
libsvm websitelibsvm - A simple, easy-to-use, efficient library for Support Vector Machines. [BSD-3-Clause] website

Libsvm is a simple, easy-to-use, and efficient software for SVM classification and regression. It solves C-SVM classification, nu-SVM classification,

Chih-Jen Lin 4.3k Jan 2, 2023
null 5.7k Jan 4, 2023
Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

eXtreme Gradient Boosting Community | Documentation | Resources | Contributors | Release Notes XGBoost is an optimized distributed gradient boosting l

Distributed (Deep) Machine Learning Community 23.6k Dec 30, 2022
Distributed (Deep) Machine Learning Community 682 Dec 28, 2022
SSL_SLAM2: Lightweight 3-D Localization and Mapping for Solid-State LiDAR (mapping and localization separated) ICRA 2021

SSL_SLAM2 Lightweight 3-D Localization and Mapping for Solid-State LiDAR (Intel Realsense L515 as an example) This repo is an extension work of SSL_SL

Wang Han 王晗 337 Dec 27, 2022
A lightweight C++ machine learning library for embedded electronics and robotics.

Fido Fido is an lightweight, highly modular C++ machine learning library for embedded electronics and robotics. Fido is especially suited for robotic

The Fido Project 413 Dec 17, 2022
MITIE: library and tools for information extraction

MITIE: MIT Information Extraction This project provides free (even for commercial use) state-of-the-art information extraction tools. The current rele

null 2.8k Dec 29, 2022
LIO-SAM: Tightly-coupled Lidar Inertial Odometry via Smoothing and Mapping

A real-time lidar-inertial odometry package. We strongly recommend the users read this document thoroughly and test the package with the provided dataset first.

Tixiao Shan 2.2k Jan 4, 2023
Yggdrasil Decision Forests (YDF) is a collection of state-of-the-art algorithms for the training, serving and interpretation of Decision Forest models.

Yggdrasil Decision Forests (YDF) is a collection of state-of-the-art algorithms for the training, serving and interpretation of Decision Forest models. The library is developed in C++ and available in C++, CLI (command-line-interface, i.e. shell commands) and in TensorFlow under the name TensorFlow Decision Forests (TF-DF).

Google 268 Jan 9, 2023
MATLAB and C++ implementations of sideslip angle estimators

sideslip-angle-vehicle-estimation MATLAB and C++ implementations of sideslip angle estimators Factor graph sideslip angle estimator Papers: "A Factor

Libraries for Multibody Dynamics Simulation (MBDS) 10 Oct 15, 2022
LIDAR(Livox Horizon) point cloud preprocessing, including point cloud filtering and point cloud feature extraction (edge points and plane points)

LIDAR(Livox Horizon) point cloud preprocessing, including point cloud filtering and point cloud feature extraction (edge points and plane points)

hongyu wang 12 Dec 28, 2022
ORB-SLAM3 is the first real-time SLAM library able to perform Visual, Visual-Inertial and Multi-Map SLAM with monocular, stereo and RGB-D cameras, using pin-hole and fisheye lens models.

Just to test for my research, and I add coordinate transformation to evaluate the ORB_SLAM3. Only applied in research, and respect the authors' all work.

B.X.W 5 Jul 11, 2022
✔️The smallest header-only GUI library(4 KLOC) for all platforms

Welcome to GUI-lite The smallest header-only GUI library (4 KLOC) for all platforms. 中文 Lightweight ✂️ Small: 4,000+ lines of C++ code, zero dependenc

null 6.6k Jan 8, 2023
BAAF-Net - Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021)

Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021) This repository is for BAAF-Net introduce

null 90 Dec 29, 2022
Suman Raj Khanal 7 Nov 24, 2021
Real-Time SLAM for Monocular, Stereo and RGB-D Cameras, with Loop Detection and Relocalization Capabilities

Real-Time SLAM for Monocular, Stereo and RGB-D Cameras, with Loop Detection and Relocalization Capabilities

Raul Mur-Artal 7.8k Jan 5, 2023
Structural variant detection and association testing

wham The wham suite consists of two programs, wham and whamg. wham, the original tool, is a very sensitive method with a high false discovery rate. Th

Zev Kronenberg 93 Nov 14, 2022