VID-Fusion: Robust Visual-Inertial-Dynamics Odometry for Accurate External Force Estimation

Overview

VID-Fusion

VID-Fusion: Robust Visual-Inertial-Dynamics Odometry for Accurate External Force Estimation

Authors: Ziming Ding , Tiankai Yang, Kunyi Zhang, Chao Xu, and Fei Gao from the ZJU FAST Lab.

0. Overview

VID-Fusion is a work to estimate odometry and external force simultaneously by a tightly coupled Visual-Inertial-Dynamics state estimator for multirotors. Just like VIMO, we formulate a new factor in the optimization-based visual-inertial odometry system VINS-Mono. But we compare the dynamics model with the imu measurements to observe the external force and formulate the external force preintegration like imu preintegration. So, the thrust and external force can be added into the classical VIO system such as VINS-Mono as a new factor.

We present:

  • An external force preintegration term for back-end optimization.
  • A complete, robust, tightly-coupled Visual-Inertial-Dynamics state estimator.
  • Demonstration of robust and accurate external force and pose estimation.

Simultaneously estimating the external force and odometry within a sliding window.

Related Paper: VID-Fusion: Robust Visual-Inertial-Dynamics Odometry for Accurate External Force Estimation, Ziming Ding, Tiankai Yang, Kunyi Zhang, Chao Xu, and Fei Gao, ICRA 2021.

Video Links: bilibili or Youtube.

1. Prerequisites

Our software is developed and only tested in Ubuntu 16.04, ROS Kinetic (ROS Installation), OpenCV 3.3.1.

Ceres Solver (Ceres Installation) is needed.

2. Build on ROS

cd your_catkin_ws/src
git clone [email protected]:ZJU-FAST-Lab/VID-Fusion.git
cd ..
catkin_make  --pkg quadrotor_msgs mav_msgs  # pre-build msg
catkin_make

3. Run in vid-dataset

cd your_catkin_ws
source ~/catkin_ws/devel/setup.bash
roslaunch vid_estimator vid_realworld.launch
roslaunch benchmark_publisher publish.launch #(option)
rosbag play YOUR_PATH_TO_DATASET

We provide the experiment data for testing, in which the vid-experiment-dataset is in ros bag type. The dataset provides two kinds of scenarios: tarj8_with_gt and line_with_force_gt.

  • tarj8_with_gt is a dataset with odometry groundtruth. The drone flys with a payload.

  • line_with_force_gt is a dataset with external force groundtruth. The drone connects a force sensor via a elastic rope.

A new visual-inertial-dynamics dataset with richer scenarios is provided in VID-Dataset.

The drone information should be provided in VID-Fusion/config/experiments/drone.yaml. It is noticed that you should use the proper parameter of the drone such as the mass and the thrust_coefficient, according to the related bag file.

As for the benchmark comparison, we naively edit the benchmark_publisher from VINS-Mono to compare the estimated path, and add a external force visualization about the estimated force and the ground truth force. The ground truth data is in VID-Fusion/benchmark_publisher/data. You should switch path or force comparison by cur_kind in publish.launch (0 for path comparison and 1 for force comparison).

As for model identification, we collect the hovering data for identification. For the two data bags, tarj8_with_gt and line_with_force_gt, we also provide the hovering data for thrust_coefficient identification. After system identification, you should copy the thrust_coefficient result to VID-Fusion/config/experiments/drone.yaml

roslaunch system_identification system_identify.launch 
rosbag play YOUR_PATH_TO_DATASET
#copy the thrust_coefficient result to VID-Fusion/config/experiments/drone.yaml

The external force is the resultant force except for rotor thrust and aircraft gravity. You can set force_wo_rotor_drag as 1 in config file to subtract the rotor drag force from the estimated force. And the related drag coefficient k_d_x and k_d_y should be given.

4. Acknowledgements

We replace the model preintegration and dynamics factor from VIMO, and formulate the proposed dynamics and external force factor atop the source code of VIMO and VINS-Mono. The ceres solver is used for back-end non-linear optimization, and DBoW2 for loop detection, and a generic camera model. The monocular initialization, online extrinsic calibration, failure detection and recovery, loop detection, and global pose graph optimization, map merge, pose graph reuse, online temporal calibration, rolling shutter support are also from VINS-Mono.

5. Licence

The source code is released under GPLv3 license.

6. Maintaince

For any technical issues, please contact Ziming Ding ([email protected]) or Fei GAO ([email protected]).

For commercial inquiries, please contact Fei GAO ([email protected]).

You might also like...
A dataset containing synchronized visual, inertial and GNSS raw measurements.
A dataset containing synchronized visual, inertial and GNSS raw measurements.

GVINS-Dataset Author/Maintainer: CAO Shaozu (shaozu.cao AT gmail.com), LU Xiuyuan (xluaj AT connect.ust.hk) This repository hosts dataset collected du

Visual Leak Detector for Visual C++ 2008-2015

Visual Leak Detector Introduction Visual C++ provides built-in memory leak detection, but its capabilities are minimal at best. This memory leak detec

Fisheye version of VINS-Fusion
Fisheye version of VINS-Fusion

VINS-Fisheye This repository is a Fisheye version of VINS-Fusion with GPU and Visionworks acceleration. It can run on Nvidia TX2 in real-time, also pr

Hands-On example code for Sensor Fusion and Autonomous Driving Stack based on Autoware
Hands-On example code for Sensor Fusion and Autonomous Driving Stack based on Autoware

Autoware "Hands-On" Stanford Lecture AA274 / Graz University of Technology M. Schratter, J. Zubaca, K. Mautner-Lassnig, T. Renzler, M. Kirchengast, S.

Port of Adafruit / NXP Sensor Fusion filter

AHRS Fusion Port of Adafruit NXP sensor fusion algorithms based on Kalman filters for rust. Resources https://github.com/adafruit/Adafruit_AHRS https:

BAAF-Net - Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021)
BAAF-Net - Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021)

Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021) This repository is for BAAF-Net introduce

EKF-based late fusion

深蓝学院多传感器融合感知课程 项目实现了Lidar与Camera的后融合感知算法,融合的算法基于扩展卡尔曼滤波(Extended Kalman Filter,EKF)。输入数据为Lidar检测结果以及Camera检测结果,检测算法与Apollo 6.0一致,Lidar检测算法为PointPillar

Brute force Mnemonic BIP39 Bip32 Bip44

BIP39 Experimental project BIP39/Bip32/Bip44. This is a modified version LostCoins The project needs the help of a programmer! We need to change the D

A Brute-Force Tool For Facebook Accounts
A Brute-Force Tool For Facebook Accounts

fblookup fblookup is a facebook password cracking tool written in C which allows you to hack every facebook account using a wordlist without any block

Owner
ZJU FAST Lab
ZJU FAST Lab
Visual-inertial-wheel fusion odometry, better performance in scenes with drastic changes in light

VIW-Fusion An visual-inertial-wheel fusion odometry VIW-Fusion is an optimization-based viusla-inertial-wheel fusion odometry, which is developed as a

庄庭达 241 Nov 19, 2022
A Robust LiDAR-Inertial Odometry for Livox LiDAR

LIO-Livox (A Robust LiDAR-Inertial Odometry for Livox LiDAR) This respository implements a robust LiDAR-inertial odometry system for Livox LiDAR. The

livox 347 Nov 22, 2022
LVI-SAM: Tightly-coupled Lidar-Visual-Inertial Odometry via Smoothing and Mapping

LVI-SAM This repository contains code for a lidar-visual-inertial odometry and mapping system, which combines the advantages of LIO-SAM and Vins-Mono

Tixiao Shan 1.1k Nov 15, 2022
Continuous-Time Spline Visual-Inertial Odometry

Continuous-Time Spline Visual-Inertial Odometry Related Publications Direct Sparse Odometry, J. Engel, V. Koltun, D. Cremers, In IEEE Transactions on

Minnesota Interactive Robotics and Vision Laboratory 67 Nov 7, 2022
Tightly coupled GNSS-Visual-Inertial system for locally smooth and globally consistent state estimation in complex environment.

GVINS GVINS: Tightly Coupled GNSS-Visual-Inertial Fusion for Smooth and Consistent State Estimation. paper link Authors: Shaozu CAO, Xiuyuan LU and Sh

HKUST Aerial Robotics Group 563 Nov 17, 2022
A Multi-sensor Fusion Odometry via Smoothing and Mapping.

LVIO-SAM A multi-sensor fusion odometry, LVIO-SAM, which fuses LiDAR, stereo camera and inertial measurement unit (IMU) via smoothing and mapping. The

Xinliang Zhong 150 Nov 7, 2022
Mars_lib - MaRS: A Modular and Robust Sensor-Fusion Framework

Introduction The Modular and Robust State-Estimation Framework, or short, MaRS, is a recursive filtering framework that allows for truly modular multi

Control of Networked Systems - University of Klagenfurt 132 Oct 24, 2022
Livox-Mapping - An all-in-one and ready-to-use LiDAR-inertial odometry system for Livox LiDAR

Livox-Mapping This repository implements an all-in-one and ready-to-use LiDAR-inertial odometry system for Livox LiDAR. The system is developed based

null 245 Nov 11, 2022
A ros package for robust odometry and mapping using LiDAR with aid of different sensors

W-LOAM A ros package for robust odometry and mapping using LiDAR with aid of different sensors Demo Video https://www.bilibili.com/video/BV1Fy4y1L7kZ?

Saki-Chen 51 Nov 2, 2022
This repo includes SVO Pro which is the newest version of Semi-direct Visual Odometry (SVO) developed over the past few years at the Robotics and Perception Group (RPG).

rpg_svo_pro This repo includes SVO Pro which is the newest version of Semi-direct Visual Odometry (SVO) developed over the past few years at the Robot

Robotics and Perception Group 1k Nov 16, 2022