Vision-based quadrotor fault-tolerant flight controller.

Overview

Summary

This repository contains the flight controller, and the vision-based state estimator for a quadrotor subjected to complete failure of a single rotor. The program runs in a ROS environment.

IMAGE ALT TEXT HERE

Citing

If you use this work in an academic context, please cite the following RA-L publication:

S. Sun, G. Cioffi, C. de Visser and D. Scaramuzza, "Autonomous Quadrotor Flight despite Rotor Failure with Onboard Vision Sensors: Frames vs. Events," IEEE Robot. Autom. Lett. (RA-L). 2021. PDF, VIDEO

@ARTICLE{SunAutonomous2021,
  author={S. {Sun} and G. {Cioffi} and C. {de Visser} and D. {Scaramuzza}},
  journal={IEEE Robotics and Automation Letters}, 
  title={Autonomous Quadrotor Flight despite Rotor Failure with Onboard Vision Sensors: Frames vs. Events}, 
  year={2021},
  volume={6},
  number={2},
  pages={580-587},
  doi={10.1109/LRA.2020.3048875}}  

Structure

\vio_estimator

Vision-based pose estimator using a standard camera.

\ftc_estimator

Rotation-corrected Complementary Filter and Extended Kalman Filter providing full pose and velocity estimates for the fault-tolerant flight controller.

\ftc_ctrl

Fault-tolerant flight controller that generates thrust command of individual rotors, using a Nonlinear Dynamic Inversion approach.

Installation

This installation guide was tested with Ubuntu 18.04

Requirements:

You also need to install the following packages for Ceres Solver:

sudo apt install liblapack-dev libblas-dev

Create a ROS workspace

mkdir -p ~/catkin_ws/src && cd ~/catkin_ws
catkin init

Configure the ROS environment

catkin config --extend /opt/ros/melodic --cmake-args -DCMAKE_BUILD_TYPE=Release

Clone the repository and dependencies

cd src/
git clone https://github.com/uzh-rpg/fault_tolerant_control.git
vcs-import < fault_tolerant_control/dependencies.yaml

Build the workspace

cd ~/catkin_ws && catkin build

Arm64

If you run the program in arm64 architecture, please add

set(ENV{ARM_ARCHITECTURE} aarch64)

in /vio_estimator/common/ze_common/CMakeLists.txt and /vio_estimator/common/ze_cmake/cmake/modules/ze_setup.cmake

in /fast_neon package, use branch /test/aarch64-compilation by

git checkout /test/aarch64-compilation

Test the vision-based state estimator

You can test the vision-based state estimator by running this rosbag recorded from real-flights using a standard camera (mvBLuefox).

roslaunch ze_vio_ceres live_Bluefox.launch
rosbag play /data_bluefox.bag

Test the Flight Controller in the Simulation

You can test the flight controller in RotorS, a MAV gazebo simulator. First of all, please install Gazebo.

Then run the following launch file

roslaunch ftc_ctrl testSim.launch

The drone is ready to fly in the simulator. We provide some simple commands in the /scripts folder

roscd ftc_ctrl && cd scripts

To take off, please type

./start_rotors.sh hummingbird

Then you can switch one motor off by running the script

./stop_rotor.sh hummingbird

You may decide which rotor to be turned off in simulation.yaml.

To control the quadrotor to a way point, e.g., xyz = [1.0 2.0 3.0], use the script

./reference.sh 1.0 2.0 3.0 hummingbird

Real flight test

You may reproduce our experiments in real flights, if you have

  • A mvBluefox camera and it's ROS driver.
  • A Terarranger-One range sensor and it's ROS_driver.
  • A quadrotor that you can control the thrust of each individual rotor.
  • A Joystick.
  • Safety nets ;)

To start the flight controller, please run

roslaunch ftc_ctrl test.launch

The controller will publish rotor thrust command in a rostopic

/control_command/rotor_thrust 

You can feed these command to your low-level motor controller.

Add the vision-based estimator into the loop

Once the drone starts hovering before one motor is switched off, you can start the state estimator by running the following launch files:

roslaunch ze_vio_ceres live_Bluefox.launch

The estimated state will be published in the ROS topic

/state_est

which is used by the flight controller.

Issues
  • Built fails

    Built fails

    Starting  >>> rotors_gazebo                                                    
    Finished  <<< opengv                                            [ 1 minute and 47.3 seconds ]
    _______________________________________________________________________________
    Errors     << ze_common:make /home/archit/catkin_ws/logs/ze_common/build.make.000.log
    In file included from /usr/include/c++/9/map:61,
                     from /home/archit/catkin_ws/src/fault_tolerant_control/vio_estimator/common/ze_common/include/ze/common/buffer.hpp:28,
                     from /home/archit/catkin_ws/src/fault_tolerant_control/vio_estimator/common/ze_common/include/ze/common/csv_trajectory.hpp:28,
                     from /home/archit/catkin_ws/src/fault_tolerant_control/vio_estimator/common/ze_common/src/csv_trajectory.cpp:26:
    /usr/include/c++/9/bits/stl_map.h: In instantiation of ‘class std::map<long int, Eigen::Matrix<double, 3, 1>, std::less<long int>, Eigen::aligned_allocator<Eigen::Matrix<double, 3, 1> > >’:
    /home/archit/catkin_ws/src/fault_tolerant_control/vio_estimator/common/ze_common/include/ze/common/buffer-inl.hpp:204:53:   required from ‘class ze::Buffer<double, 3>/home/archit/catkin_ws/src/fault_tolerant_control/vio_estimator/common/ze_common/include/ze/common/csv_trajectory.hpp:73:21:   required from here
    /usr/include/c++/9/bits/stl_map.h:122:71: error: static assertion failed: std::map must have the same value_type as its allocator
      122 |       static_assert(is_same<typename _Alloc::value_type, value_type>::value,
          |                                                                       ^~~~~
    /usr/include/c++/9/bits/stl_map.h: In instantiation of ‘class std::map<long int, Eigen::Matrix<double, 7, 1>, std::less<long int>, Eigen::aligned_allocator<Eigen::Matrix<double, 7, 1> > >’:
    /home/archit/catkin_ws/src/fault_tolerant_control/vio_estimator/common/ze_common/include/ze/common/buffer-inl.hpp:204:53:   required from ‘class ze::Buffer<double, 7>/home/archit/catkin_ws/src/fault_tolerant_control/vio_estimator/common/ze_common/include/ze/common/csv_trajectory.hpp:91:21:   required from here
    /usr/include/c++/9/bits/stl_map.h:122:71: error: static assertion failed: std::map must have the same value_type as its allocator
    make[2]: *** [CMakeFiles/ze_common.dir/build.make:63: CMakeFiles/ze_common.dir/src/csv_trajectory.cpp.o] Error 1
    make[2]: *** Waiting for unfinished jobs....
    make[1]: *** [CMakeFiles/Makefile2:2109: CMakeFiles/ze_common.dir/all] Error 2
    make: *** [Makefile:141: all] Error 2
    cd /home/archit/catkin_ws/build/ze_common; catkin build --get-env ze_common | catkin env -si  /usr/bin/make --jobserver-auth=3,4; cd -
    
    ...............................................................................
    Failed     << ze_common:make                                    [ Exited with code 2 ]
    Failed    <<< ze_common                                         [ 12.5 seconds ]
    Abandoned <<< rotors_simulator                                  [ Unrelated job failed ]
    Abandoned <<< ceres_catkin                                      [ Unrelated job failed ]
    Abandoned <<< imp_core                                          [ Unrelated job failed ]
    Abandoned <<< imp_bridge_opencv                                 [ Unrelated job failed ]
    Abandoned <<< imp_bridge_ros                                    [ Unrelated job failed ]
    Abandoned <<< imp_imgproc                                       [ Unrelated job failed ]
    Abandoned <<< ze_cameras                                        [ Unrelated job failed ]
    Abandoned <<< imp_features                                      [ Unrelated job failed ]
    Abandoned <<< ze_geometry                                       [ Unrelated job failed ]
    Abandoned <<< imp_correspondence                                [ Unrelated job failed ]
    Abandoned <<< ze_imu                                            [ Unrelated job failed ]
    Abandoned <<< ze_ros                                            [ Unrelated job failed ]
    Abandoned <<< ze_visualization                                  [ Unrelated job failed ]
    Abandoned <<< ze_vio_common                                     [ Unrelated job failed ]
    Abandoned <<< ze_ceres_backend                                  [ Unrelated job failed ]
    Abandoned <<< ze_data_provider                                  [ Unrelated job failed ]
    Abandoned <<< ze_vio_frontend                                   [ Unrelated job failed ]
    Abandoned <<< ze_vio_ceres                                      [ Unrelated job failed ]
    Finished  <<< rotors_gazebo                                     [ 6.7 seconds ]
    _______________________________________________________________________________
    Warnings   << suitesparse:make /home/archit/catkin_ws/logs/suitesparse/build.make.000.log
    --2022-02-08 09:58:53--  https://github.com/ethz-asl/thirdparty_library_binaries/raw/master/SuiteSparse-4.2.1.tar.gz
    Resolving github.com (github.com)... 140.82.112.4
    Connecting to github.com (github.com)|140.82.112.4|:443... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: https://raw.githubusercontent.com/ethz-asl/thirdparty_library_binaries/master/SuiteSparse-4.2.1.tar.gz [following]
    --2022-02-08 09:58:53--  https://raw.githubusercontent.com/ethz-asl/thirdparty_library_binaries/master/SuiteSparse-4.2.1.tar.gz
    Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8002::154, 2606:50c0:8001::154, 2606:50c0:8003::154, ...
    Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8002::154|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 22541409 (21M) [application/octet-stream]
    Saving to: ‘SuiteSparse-4.2.1.tar.gz’
    
    
    SuiteSparse-4.2.1.t   0%[                    ]       0  --.-KB/s               
    SuiteSparse-4.2.1.t  19%[==>                 ]   4.16M  20.7MB/s               
    SuiteSparse-4.2.1.t  43%[=======>            ]   9.39M  23.4MB/s               
    SuiteSparse-4.2.1.t  68%[============>       ]  14.62M  24.3MB/s               
    SuiteSparse-4.2.1.t  92%[=================>  ]  19.83M  24.7MB/s               
    SuiteSparse-4.2.1.t 100%[===================>]  21.50M  24.8MB/s    in 0.9s    
    
    2022-02-08 09:58:55 (24.8 MB/s) - ‘SuiteSparse-4.2.1.tar.gz’ saved [22541409/22541409]
    
    make[3]: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
    ar: creating libcerbla.a
    ar: creating libsuitesparseconfig.a
    ar: creating libamd.a
    ar: creating libbtf.a
    ar: creating libcamd.a
    ar: creating libccolamd.a
    ar: creating libcolamd.a
    ar: creating libcholmod.a
    ar: creating libklu.a
    ar: creating libldl.a
    ar: creating libumfpack.a
    ar: creating libcsparse.a
    ar: creating libcxsparse.a
    ar: creating libspqr.a
    ar: creating librbio.a
    make[3]: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
    cd /home/archit/catkin_ws/build/suitesparse; catkin build --get-env suitesparse | catkin env -si  /usr/bin/make --jobserver-auth=3,4; cd -
    
    ...............................................................................
    Finished  <<< suitesparse                                       [ 2 minutes and 11.8 seconds ]
    [build] Summary: 38 of 57 packages succeeded.                                  
    [build]   Ignored:   None.                                                     
    [build]   Warnings:  8 packages succeeded with warnings.                       
    [build]   Abandoned: 18 packages were abandoned.                               
    [build]   Failed:    1 packages failed.                                        
    [build] Runtime: 2 minutes and 14.7 seconds total.                             
    [build] Note: Workspace packages have changed, please re-source setup files to use them.
    
    opened by architjaiswal 0
  • vcs-import

    vcs-import

    Hi I want install your project when run this command "vcs-import < fault_tolerant_control/dependencies.yaml" got this error message : "Could not determine ref type of version: [email protected]: Permission denied (publickey). fatal: Could not read from remote repository.

    Please make sure you have the correct access rights and the repository exists." I search in google but I cant solve this problem.

    opened by enavid 1
  • Compile problem on ze_common

    Compile problem on ze_common

    Here is the error:

    /home/nswang1994/catkin_ws/src/fault_tolerant_control/vio_estimator/common/ze_common/include/ze/common/csv_trajectory.hpp:73:21: required from here /usr/include/c++/9/bits/stl_map.h:122:71: error: static assertion failed: std::map must have the same value_type as its allocator 122 | static_assert(is_same<typename _Alloc::value_type, value_type>::value, | ^~~~~ /usr/include/c++/9/bits/stl_map.h: In instantiation of ‘class std::map<long int, Eigen::Matrix<double, 7, 1>, std::less, Eigen::aligned_allocator<Eigen::Matrix<double, 7, 1> > >’: /home/nswang1994/catkin_ws/src/fault_tolerant_control/vio_estimator/common/ze_common/include/ze/common/buffer-inl.hpp:204:53: required from ‘class ze::Buffer<double, 7>’ /home/nswang1994/catkin_ws/src/fault_tolerant_control/vio_estimator/common/ze_common/include/ze/common/csv_trajectory.hpp:91:21: required from here /usr/include/c++/9/bits/stl_map.h:122:71: error: static assertion failed: std::map must have the same value_type as its allocator make[2]: *** [CMakeFiles/ze_common.dir/build.make:76: CMakeFiles/ze_common.dir/src/csv_trajectory.cpp.o] Error 1 make[1]: *** [CMakeFiles/Makefile2:2041: CMakeFiles/ze_common.dir/all] Error 2 make: *** [Makefile:146: all] Error 2

    opened by nswang1994 2
Owner
Robotics and Perception Group
Robotics and Perception Group
This repository is used for storing sourcecode related to final project of Computer Graphics and Computer Vision

Computer Graphics and Computer Vision Description: This repository is used for storing sourcecode related to final project of Computer Graphics and Co

null 12 Jun 16, 2022
physically based renderer written in DX12 with image-based lighting, classic deffered and tiled lighting approaches

Features Classical Deferred Renderer Physically Based shading Image Based Lighting BRDF Disney model (Burley + GGX) Tangent space normal mapping Reinh

Alena 35 Jun 2, 2022
Yocto/GL: Tiny C++ Libraries for Data-Driven Physically-based Graphics

Yocto/GL: Tiny C++ Libraries for Data-Oriented Physically-based Graphics Yocto/GL is a collection of small C++17 libraries for building physically-bas

Fabio Pellacini 2.3k Jun 26, 2022
An OCR Library based on PaddleOCR & OnnxRuntime

An OCR Library based on PaddleOCR & OnnxRuntime

Daniel 0 Mar 27, 2021
A path tracer based on hardware ray tracing

GoldenSun GoldenSun is a GPU path tracer. It uses hardware ray tracing APIs to do the tracing. As an experimental project, there is no release plan, n

Minmin Gong 20 Feb 27, 2022
[WIP] A media playback library for Dart & Flutter apps on Windows & Linux. Based on libVLC & libVLC++.

dart_vlc Bringing power of VLC to Flutter & Dart apps on Windows & Linux Installation dependencies: ... dart_vlc: ^0.0.1 Documentation Create a ne

Hitesh Kumar Saini 335 Jun 29, 2022
Vulkan physically based raytracer including denoising

VulkanPBRT Vulkan physically based raytracer including denoising. The GPU raytracer is based on Vulkan only, as well as for the denoising only the Vul

null 15 May 26, 2022
SoL (for Speed of Light, or sun in Spanish) is a Physically-based rendering library written in modern C++

SoL (for Speed of Light, or sun in Spanish) is a small rendering library written in C++20. Its goal is to strike a good balance between performance and usability, and allow easy experimentation for rendering researchers.

Arsène Pérard-Gayot 10 May 19, 2022
A terminal-based graphics library for both 2D and 3D graphics.

TermGL A terminal-based graphics library for both 2D and 3D graphics. Written in C, created for terminals supporting ANSI escape codes. Table of Conte

null 201 Jun 26, 2022
Sketch-Based Streaming Anomaly Detection in Dynamic Graphs

AnoGraph Implementation of Sketch-Based Streaming Anomaly Detection in Dynamic Graphs. Siddharth Bhatia, Mohit Wadhwa, Philip S. Yu, Bryan Hooi Existi

Stream-AD 51 May 10, 2022
Port of Doom 3 for PSVITA. (Based on https://github.com/emileb/d3es-multithread )

ABOUT Combination of the excellent dhewm3 and d3wasm projects, now with multithreaded frontend and backend renderering. Forked from: https://github.co

Rinnegatamante 22 Jun 12, 2022
🌞 A physically based monte carlo path tracer written in C++

Physically Based Path Tracer The below 3D model for the Head of Michelangelo's David bust was taken from this link. Other .obj files taken from The St

Aman Shenoy 34 Jun 8, 2022
work in progress 3d renderer based on sdl2

work in progress 3d software renderer based on SDL. (Only supports wireframe view for now) Building On Linux, install libsdl2 and then run the folowin

null 4 Sep 12, 2021
Monster Mash: New Sketch-Based Modeling and Animation Tool

Monster Mash is a new sketch-based modeling and animation tool that allows you to quickly sketch a character, inflate it into 3D, and promptly animate it. You can perform all interactions in the sketching plane. No 3D manipulation is required.

Google 1.1k Jul 2, 2022
Ksnip is a Qt-based cross-platform screenshot tool that provides many annotation features for your screenshots.

Ksnip is a Qt-based cross-platform screenshot tool that provides many annotation features for your screenshots.

ksnip 1.4k Jun 26, 2022
work in progress 3d renderer based on sdl2

work in progress 3d software renderer based on SDL. (Only supports wireframe view for now) Building On Linux, install libsdl2 and then run the folowin

Jayachandra Kasarla 4 Sep 12, 2021
A modern C++ physically based renderer

The Dakku Renderer Warning: This project is currently under developing and does not guarantee any consistency. About Dakku is a physically based rende

xehoth 6 Apr 15, 2022
Implementation of light baking system for ray tracing based on Activision's UberBake

Vulkan Light Bakery MSU Graphics Group Student's Diploma Project Treefonov Andrey [GitHub] [LinkedIn] EARLY STAGES OF DEVELOPMENT Project Goal The goa

Andrei Treefonov 5 May 24, 2022
2D lowpoly editor heavily based on this one

polyedit 2D lowpoly editor heavily based on this one Download Direct link Releases page Libraries This project uses: SFML Tiny File Dialogs jsoncpp im

null 19 Aug 7, 2021