Fisheye version of VINS-Fusion

Overview

VINS-Fisheye

This repository is a Fisheye version of VINS-Fusion with GPU and Visionworks acceleration. It can run on Nvidia TX2 in real-time, also provide depth estimation based on fisheye. This project stands as a part of Omni-swarm: A Decentralized Omnidirectional Visual-Inertial-UWB State Estimation System for Aerial Swarm. You may use it alone on any type of robot or as a part of Omni-swarm for swarm robots.

Only stereo visual-inertial-odometry is supported for fisheye cameras now. Loop closure module for fisheye camera will release later.

Image of PCL Drone path and RGB point cloud estimation

Image of fisheye Feature tracker for fisheye

Image of Disparity Disparity estimation for depth estimation

1. Prerequisites

The essential software environment is same as VINS-Fusion. Besides, it requires OpenCV cuda version.(Only test it on OpenCV 3.4.1). Visionworks: Optional

If you want to CUDA mode of this package, libSGM is required for depth estimation.

2. Usage

2.1 Change the opencv path in the CMakeLists

This package support CUDA mode and CPU mode with OpenMP enabled. If you are using on embedded device, I strongly recommend you to use this package with CUDA to achieve best performance.

By default, CUDA is automatically detected, however you can disable it by set

set(DETECT_CUDA false)

When using OpenCV, I recommend that you to compile and install opencv 3.4 with CUDA to /usr/local/

If your opencv with CUDA is installed in other localization, modify the

SET("OpenCV_DIR"  "/usr/local/share/OpenCV/")

If you don't have visionworks, please

set(ENABLE_VWORKS false)

NVIDIA VisionWorks gives slightly better performance, however, the VisionWorks support for this package is not stable yet.

Fisheye usage

Term 1

#If use CUDA
roslaunch vins fisheye_split.launch config_file:=/home/your_name/your_ws/src/VINS-Fusion-Fisheye/config/fisheye_ptgrey_n3/fisheye_cuda.yaml
#If use CPU
roslaunch vins fisheye_split.launch config_file:=/home/your_name/your_ws/src/VINS-Fusion-Fisheye/config/fisheye_ptgrey_n3/fisheye_cpu.yaml

Term 2

rosbag play fishey_vins_2020-01-30-10-38-14.bag --clock -s 12

Term 3(for visuallization only)

roslaunch vins vins_rviz.launch

GPU is default enabled, if you are not using CUDA, disable it in yaml config file.

For rosbag, you can download from https://www.dropbox.com/s/kmakksca3ns6cav/fisheye_vins_2020-01-30-10-38-14.bag?dl=0

Parameters for fisheye

depth_config: "depth_cpu.yaml" # config path for depth estimation, depth_cpu.yaml uses opencv SGBM, depth.yaml uses visionworks SGM, you must install visionworks before use visionworks sgm
image_width: 600 # For fisheye, this indicate the flattened image width; min 100; 300 - 500 is good for vins
fisheye_fov: 235 # Your fisheye fov
enable_up_top: 1 #Choose direction you use
enable_down_top: 1
enable_up_side: 1
enable_down_side: 1
enable_rear_side: 1
thres_outlier : 5.0 # outlier thres for backend
tri_max_err: 3.0 #outlier thres for triangulate

depth_estimate_baseline: 0.05 # mini baseline for pts initialization
top_cnt: 30 #number of track point for top view
side_cnt: 30 #number of track point for side view
max_solve_cnt: 30 # Max Point for solve; highly influence performace
show_track: 0 # if display track
use_vxworks: 0 #use vision works for front-end; not as stable as CUDA now

enable_depth: 1 # If estimate depth cloud; only available for dual fisheye now
rgb_depth_cloud: 0 # -1: point no texture,  0 depth cloud will be gray, 1 depth cloud will be colored;
#Note that textured and colored depth cloud will slow down whole system
use_gpu: 1 # If using GPU
use_vworks: 0 # If using visionworks

Parameter for depth estimation

#choose the depth you want estimate
enable_front: 1
enable_left: 1
enable_right: 1
enable_rear: 0
#downsample ration
downsample_ratio: 0.5
#choose use cpu or visionworks
use_vworks: 0

#Publish cloud jump step
pub_cloud_step: 1
#If show dispartity
show_disparity: 0
#If publish depth map image
pub_depth_map: 1
#Publish cloud in radius
depth_cloud_radius: 10
#If publish all depth cloud in a topic
pub_cloud_all: 1
#If publish all depth cloud in every direction
pub_cloud_per_direction: 0

Related Paper

Omni-swarm: A Decentralized Omnidirectional Visual-Inertial-UWB State Estimation System for Aerial Swarm The VINS-Fisheye is a part of Omni-swarm. If you want use VIN-Fisheye as a part of your research project, please cite this paper.

See also

Autonomous aerial robot using dual‐fisheye cameras, Wenliang Gao, Kaixuan Wang, Wenchao Ding, Fei Gao, Tong Qin, Shaojie Shen, 2020 Journal of Field Robotics (JFR) for Fisheye camera navigation. The basic idea of this project is from this paper.

H. Xu, L. Wang, Y. Zhang and S. Shen. (2020) Decentralized visual-inertial-UWB fusion for relative state estimation of aerial swarm. in 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE. for swarm localization which is this project developed for.

VINS-Fusion

An optimization-based multi-sensor state estimator

VINS-Fusion is an optimization-based multi-sensor state estimator, which achieves accurate self-localization for autonomous applications (drones, cars, and AR/VR). VINS-Fusion is an extension of VINS-Mono, which supports multiple visual-inertial sensor types (mono camera + IMU, stereo cameras + IMU, even stereo cameras only). We also show a toy example of fusing VINS with GPS. Features:

  • multiple sensors support (stereo cameras / mono camera+IMU / stereo cameras+IMU)
  • online spatial calibration (transformation between camera and IMU)
  • online temporal calibration (time offset between camera and IMU)
  • visual loop closure

We are the top open-sourced stereo algorithm on KITTI Odometry Benchmark (12.Jan.2019).

Authors: Tong Qin, Shaozu Cao, Jie Pan, Peiliang Li, and Shaojie Shen from the Aerial Robotics Group, HKUST

Videos:

VINS

Related Papers: (papers are not exactly same with code)

  • A General Optimization-based Framework for Local Odometry Estimation with Multiple Sensors, Tong Qin, Jie Pan, Shaozu Cao, Shaojie Shen, aiXiv pdf

  • A General Optimization-based Framework for Global Pose Estimation with Multiple Sensors, Tong Qin, Shaozu Cao, Jie Pan, Shaojie Shen, aiXiv pdf

  • Online Temporal Calibration for Monocular Visual-Inertial Systems, Tong Qin, Shaojie Shen, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS, 2018), best student paper award pdf

  • VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator, Tong Qin, Peiliang Li, Shaojie Shen, IEEE Transactions on Robotics pdf

If you use VINS-Fusion for your academic research, please cite our related papers. bib

1. Prerequisites

1.1 Ubuntu and ROS

Ubuntu 64-bit 16.04 or 18.04. ROS Kinetic or Melodic. ROS Installation

1.2. Ceres Solver

Follow Ceres Installation.

2. Build VINS-Fusion

Clone the repository and catkin_make:

    cd ~/catkin_ws/src
    git clone https://github.com/HKUST-Aerial-Robotics/VINS-Fusion.git
    cd ../
    catkin_make
    source ~/catkin_ws/devel/setup.bash

(if you fail in this step, try to find another computer with clean system or reinstall Ubuntu and ROS)

3. EuRoC Example

Download EuRoC MAV Dataset to YOUR_DATASET_FOLDER. Take MH_01 for example, you can run VINS-Fusion with three sensor types (monocular camera + IMU, stereo cameras + IMU and stereo cameras). Open four terminals, run vins odometry, visual loop closure(optional), rviz and play the bag file respectively. Green path is VIO odometry; red path is odometry under visual loop closure.

3.1 Monocualr camera + IMU

    roslaunch vins vins_rviz.launch
    rosrun vins vins_node ~/catkin_ws/src/VINS-Fusion/config/euroc/euroc_mono_imu_config.yaml 
    (optional) rosrun loop_fusion loop_fusion_node ~/catkin_ws/src/VINS-Fusion/config/euroc/euroc_mono_imu_config.yaml 
    rosbag play YOUR_DATASET_FOLDER/MH_01_easy.bag

3.2 Stereo cameras + IMU

    roslaunch vins vins_rviz.launch
    rosrun vins vins_node ~/catkin_ws/src/VINS-Fusion/config/euroc/euroc_stereo_imu_config.yaml 
    (optional) rosrun loop_fusion loop_fusion_node ~/catkin_ws/src/VINS-Fusion/config/euroc/euroc_stereo_imu_config.yaml 
    rosbag play YOUR_DATASET_FOLDER/MH_01_easy.bag

3.3 Stereo cameras

    roslaunch vins vins_rviz.launch
    rosrun vins vins_node ~/catkin_ws/src/VINS-Fusion/config/euroc/euroc_stereo_config.yaml 
    (optional) rosrun loop_fusion loop_fusion_node ~/catkin_ws/src/VINS-Fusion/config/euroc/euroc_stereo_config.yaml 
    rosbag play YOUR_DATASET_FOLDER/MH_01_easy.bag

4. KITTI Example

4.1 KITTI Odometry (Stereo)

Download KITTI Odometry dataset to YOUR_DATASET_FOLDER. Take sequences 00 for example, Open two terminals, run vins and rviz respectively. (We evaluated odometry on KITTI benchmark without loop closure funtion)

    roslaunch vins vins_rviz.launch
    (optional) rosrun loop_fusion loop_fusion_node ~/catkin_ws/src/VINS-Fusion/config/kitti_odom/kitti_config00-02.yaml
    rosrun vins kitti_odom_test ~/catkin_ws/src/VINS-Fusion/config/kitti_odom/kitti_config00-02.yaml YOUR_DATASET_FOLDER/sequences/00/ 

4.2 KITTI GPS Fusion (Stereo + GPS)

Download KITTI raw dataset to YOUR_DATASET_FOLDER. Take 2011_10_03_drive_0027_synced for example. Open three terminals, run vins, global fusion and rviz respectively. Green path is VIO odometry; blue path is odometry under GPS global fusion.

    roslaunch vins vins_rviz.launch
    rosrun vins kitti_gps_test ~/catkin_ws/src/VINS-Fusion/config/kitti_raw/kitti_10_03_config.yaml YOUR_DATASET_FOLDER/2011_10_03_drive_0027_sync/ 
    rosrun global_fusion global_fusion_node

5. VINS-Fusion on car demonstration

Download car bag to YOUR_DATASET_FOLDER. Open four terminals, run vins odometry, visual loop closure(optional), rviz and play the bag file respectively. Green path is VIO odometry; red path is odometry under visual loop closure.

    roslaunch vins vins_rviz.launch
    rosrun vins vins_node ~/catkin_ws/src/VINS-Fusion/config/vi_car/vi_car.yaml 
    (optional) rosrun loop_fusion loop_fusion_node ~/catkin_ws/src/VINS-Fusion/config/vi_car/vi_car.yaml 
    rosbag play YOUR_DATASET_FOLDER/car.bag

6. Run with your devices

VIO is not only a software algorithm, it heavily relies on hardware quality. For beginners, we recommend you to run VIO with professional equipment, which contains global shutter cameras and hardware synchronization.

6.1 Configuration file

Write a config file for your device. You can take config files of EuRoC and KITTI as the example.

6.2 Camera calibration

VINS-Fusion support several camera models (pinhole, mei, equidistant). You can use camera model to calibrate your cameras. We put some example data under /camera_models/calibrationdata to tell you how to calibrate.

cd ~/catkin_ws/src/VINS-Fusion/camera_models/camera_calib_example/
rosrun camera_models Calibrations -w 12 -h 8 -s 80 -i calibrationdata --camera-model pinhole

7. Acknowledgements

We use ceres solver for non-linear optimization and DBoW2 for loop detection, a generic camera model and GeographicLib.

8. License

The source code is released under GPLv3 license.

We are still working on improving the code reliability. For any technical issues, please contact Tong Qin <qintonguavATgmail.com>.

For commercial inquiries, please contact Shaojie Shen <eeshaojieATust.hk>.

You might also like...
Modified version of srlua for MSVC using version 5.4 of Lua

Modified version of srlua for MSVC using version 5.4 of Lua. Quote from the original README: This is a self-running Lua interpreter. It is meant to be

Semantic version library written in ANSI C

semver.c Semantic version v2.0 parser and render written in ANSI C with zero dependencies. Features Standard compliant (otherwise, open an issue) Vers

Updated version of Silicos-it's shape-based alignment tool

shape-it Description Code for shape-it with openbabel3 and rdkit INSTALL Following example is the basic way to install the tool: git clone https://git

AstoriaCore is a customized Version of AzerothCore and the private source of WaloriaCore by Fractional aka Fred.
AstoriaCore is a customized Version of AzerothCore and the private source of WaloriaCore by Fractional aka Fred.

Community driven Classless MMO Framework. Proudly founded by Lushen and based on AzerothCore and TrinityCore 😄

Firebase Arduino Client Library for ESP8266 and ESP32. The unified version of Firebase-ESP8266 and Firebase-ESP32 Realtime database libraries with Cloud Firestore, Firebase and Google Cloud Storage, Cloud messaging and Cloud Functions supports.
Enables Permanent UART output across boots. Supports PS4 version 6.72 only for now

Enables Permanent UART for PS4. Requires a 5.05/6.72/7.02 Jailbroken PS4. To build, you require ps4-sdk from Scene-Collective/ps4-payload-sdk In the t

A header-only version of Instagram Private API in C++

C++ Instagram API A header-only version of Instagram Private API in C++ Getting started. Writing your first bot General API Documentation Usage Error

Another version of EVA using anti-debugging techs && using Syscalls
Another version of EVA using anti-debugging techs && using Syscalls

EVA2 Another version of EVA using anti-debugging techs && using Syscalls First thing: Dont Upload to virus total. this note is for you and not for me.

A simpler version of the infamous zero kb virus written in C++.

A simpler version of the infamous zero kb virus written in C++. A few years back I was struck by the 'zero kb' virus, and so i decide to write my own zero kb virus as an act of vengeance (

Comments
  • Question about calibration

    Question about calibration

    Dear professor: Hello, I'm a freshman to the omni-based system. Recently, I have read your paper 《Autonomous aerial robot using dual‐fisheye cameras》. It's a wonderful idea to use the dual fisheye to improve the SLAM robustness and accuracy. However, there's one thing troubles me a lot: how to calibrate the omni-based cameras ? I use the kalibr [https://github.com/ethz-asl/kalibr] system to calibrate our device: like insta-360 or the Go-Pro. image image However, the reprojection error is large, always converged at 1 pixel above and when I use it in the SLAM system, it always failed. I'm confused is there some tricks about how to calibrate this kind of sensor or do I need some special movement to improve the calibration result? I'm looking forward for your advices. Thank you very much.

    Yours

    opened by Gatsby23 0
Owner
HKUST Aerial Robotics Group
HKUST Aerial Robotics Group
QPEP (Quadratic Pose Estimation Problems) Enhanced VINS-Mono SLAM System

VINS-Mono-QPEP The QPEP (Quadratic Pose Estimation Problems) Enhanced VINS-Mono

Jin Wu 36 Nov 2, 2022
VID-Fusion: Robust Visual-Inertial-Dynamics Odometry for Accurate External Force Estimation

VID-Fusion VID-Fusion: Robust Visual-Inertial-Dynamics Odometry for Accurate External Force Estimation Authors: Ziming Ding , Tiankai Yang, Kunyi Zhan

ZJU FAST Lab 86 Nov 18, 2022
A Multi-sensor Fusion Odometry via Smoothing and Mapping.

LVIO-SAM A multi-sensor fusion odometry, LVIO-SAM, which fuses LiDAR, stereo camera and inertial measurement unit (IMU) via smoothing and mapping. The

Xinliang Zhong 152 Dec 24, 2022
Visual-inertial-wheel fusion odometry, better performance in scenes with drastic changes in light

VIW-Fusion An visual-inertial-wheel fusion odometry VIW-Fusion is an optimization-based viusla-inertial-wheel fusion odometry, which is developed as a

庄庭达 261 Dec 30, 2022
Hands-On example code for Sensor Fusion and Autonomous Driving Stack based on Autoware

Autoware "Hands-On" Stanford Lecture AA274 / Graz University of Technology M. Schratter, J. Zubaca, K. Mautner-Lassnig, T. Renzler, M. Kirchengast, S.

Virtual Vehicle 27 Dec 12, 2022
Port of Adafruit / NXP Sensor Fusion filter

AHRS Fusion Port of Adafruit NXP sensor fusion algorithms based on Kalman filters for rust. Resources https://github.com/adafruit/Adafruit_AHRS https:

Gaute Hope 4 May 14, 2022
BAAF-Net - Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021)

Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021) This repository is for BAAF-Net introduce

null 90 Dec 29, 2022
Mars_lib - MaRS: A Modular and Robust Sensor-Fusion Framework

Introduction The Modular and Robust State-Estimation Framework, or short, MaRS, is a recursive filtering framework that allows for truly modular multi

Control of Networked Systems - University of Klagenfurt 146 Jan 5, 2023
EKF-based late fusion

深蓝学院多传感器融合感知课程 项目实现了Lidar与Camera的后融合感知算法,融合的算法基于扩展卡尔曼滤波(Extended Kalman Filter,EKF)。输入数据为Lidar检测结果以及Camera检测结果,检测算法与Apollo 6.0一致,Lidar检测算法为PointPillar

深蓝学院 9 Dec 7, 2022
Make CVE-2020-0668 exploit work for version < win10 v1903 and version >= win10 v1903

CVE-2020-0668 Made CVE-2020-0668 exploit work for version < win10 v1903 and version >= win10 v1903 Diaghub Exploit (< v1903) powershell exploit works

null 12 Nov 9, 2022