EVO: Event based Visual Odometry
Credit, License, and Patent
Citation
This code implements the event-based visual odometry pipeline described in the paper EVO: A Geometric Approach to Event-Based 6-DOF Parallel Tracking and Mapping in Real-time by Henri Rebecq, Timo Horstschaefer, Guillermo Gallego and Davide Scaramuzza.
If you use any of this code, please cite the following publications:
@Article{RebecqEVO,
author = {Rebecq, Henri and Horstschaefer, Timo and Gallego, Guillermo and Scaramuzza, Davide},
journal = {IEEE Robotics and Automation Letters},
title = {EVO: A Geometric Approach to Event-Based 6-DOF Parallel Tracking and Mapping in Real Time},
year = {2017},
volume = {2},
number = {2},
pages = {593-600},
doi = {10.1109/LRA.2016.2645143}
}
@InProceedings{Gehrig_2020_CVPR,
author = {Daniel Gehrig and Mathias Gehrig and Javier Hidalgo-Carri\'o and Davide Scaramuzza},
title = {Video to Events: Recycling Video Datasets for Event Cameras},
booktitle = {{IEEE} Conf. Comput. Vis. Pattern Recog. (CVPR)},
month = {June},
year = {2020}
}
The code comes with a test dataset. If you don't have an event camera and want to run the code on more data, you can do one of the following:
- Using one of our event camera datasets:
- Generate one using event-camera simulator: http://rpg.ifi.uzh.ch/esim/index.html
- Convert a standard video to events using our "Video to Events: Recycling Video Datasets for Event Cameras." cited above, code here: https://github.com/uzh-rpg/rpg_vid2e
Patent & License
-
The proposed EVO method is patented.
H. Rebecq, G. Gallego, D. Scaramuzza Simultaneous Localization and Mapping with an Event Camera Pub. No.: WO/2018/037079. International Application No.: PCT/EP2017/071331 -
The license is available here.
Acknowledgements
The open sourcing was curated by Antonio Terpin and Daniel Gehrig.
Table of contents
Remark that this is research code, any fitness for a particular purpose is disclaimed.
Getting started
This software depends on ROS. Installation instructions can be found here. We have tested this software on Ubuntu 18.04 and ROS Melodic.
-
Create and initialize a new catkin workspace if needed
mkdir -p ~/catkin_ws/src && cd ~/catkin_ws/ catkin config \ --init --mkdirs --extend /opt/ros/melodic \ --merge-devel --cmake-args \ -DCMAKE_BUILD_TYPE=Release -
Clone this repository
cd src/ && git clone [email protected]:uzh-rpg/rpg_dvs_evo_open.git -
Clone (and fix) dependencies
./rpg_dvs_evo_open/install.sh [ros-version] # [ros-version]: melodic, ..Substitute
[ros-version]with your actual ROS distribution. For instance:./rpg_dvs_evo_open/install.sh melodicMake sure to install ROS and the commonly used packages (such as rviz, rqt, ..).
The above commands do the following:
-
First, we install the required packages.
-
Second, we clone the repositories evo relies on.
Remark that with the above commands we install also the dependencies required to run the pipeline live. If you do not need them, you can comment the unnecessary packages from the dependencies.yaml file (davis driver).
Please refer to this repo if there are issues with the driver installation (required for the live).
-
-
Build the packages
catkin build dvs_tracking
Building everything might take some time... this might be the right time for a coffee! :)
... and do not forget to source the workspace afterwards!
source ../devel/setup.bash
Examples
| Example | Launch file | Rosbag |
|---|---|---|
| Multi-keyframe sequence | flyingroom.launch | 538 MB |
| Desk sequence | desk.launch | 145 MB |
To download the rosbags, you can use the following command:
wget [rosbag link] -O /path/to/download/folder/
For instance, the following will download the multi-keyframe sequence rosbag:
wget http://rpg.ifi.uzh.ch/data/EVO/code_examples/evo_flyingroom.bag -O /tmp/evo_flyingroom.bag
To run the pipeline from a rosbag, first start the pipeline as
roslaunch dvs_tracking [launch-file] auto_trigger:=true
where the specific launch file (fine-tuned) for each example is listed in the following table. The most interesting and repeatible one (due to the bootstrapping sequence, see further improvements) is the multi-keyframe sequence.
Then, once the everything is loaded, run the rosbag as
rosbag play -r 0.7 [rosbag-file]
For instance,
rosbag play -r 0.7 /tmp/evo_flyingroom.bag
Remark that we set the auto_trigger parameter to true. You can also set it to false and follow the instruction on how to run it live.
If anything fails, just try it again (give it a couple of chances!), and make sure to follow exactly the above instructions.
In the further improvements section are outlined the things that might go wrong when running the code. To improve the reliability when playing the rosbags (running live is easier), consider using -r .7, to reduce the rate. This helps especially when the hardware is not powerful enough.
For instance,
rosbag play /tmp/evo_flyingroom -r .7
Eventually, setting bootstrap_image_topic:=/dvs/image_raw will bootstrap from traditional frames and later switch to only events. This is the most reliable way currently available to bootstrap.
roslaunch dvs_tracking [launch-file] bootstrap_image_topic:=/dvs/image_raw auto_trigger:=true
Running live
Run
The procedure is analogous to the one explained in the examples:
-
Run the ros core on the first terminal.
roscore -
On a second terminal, launch the event camera driver.
rosrun davis_ros_driver davis_ros_driver -
On another terminal, launch the pipeline, disabling the auto-triggering.
roslaunch dvs_tracking live.launch auto_trigger:=false camera_name:=[calibration filename] events_topic:=[events topic]For instance:
roslaunch dvs_tracking live.launch auto_trigger:=false camera_name:=DAVIS-ijrr events_topic:=/dvs/eventsIf your calibrations filenames are
my_camera.yaml, then usecamera_name:=my_camera. Make sure to use the same name for both calibration files. If your sensor outputs events under the topic/my_sensor/events, then useevents_topic:=/my_sensor/events.For the SVO based bootstrapper, proceed with step 4. For the fronto-planar bootstrapper, go to step 5. How to set up the fronto-planar bootstrapper is explained in the live.launch file.
If you want to bootstrap from traditional frames, you can use the command:
roslaunch dvs_tracking live.launch bootstrap_image_topic:=[topic raw frames] auto_trigger:=falseFor instance
bootstrap_image_topic:=/dvs/image_raw.In this case, it makes sense to use the SVO-based bootstrapping only. This option is recommended to debug/improve/extend the rest of the EVO pipeline, without worrying about the quality of the bootstrapping.
-
You should see two rqt GUI. One is the SVO GUI. Reset and start the pipeline until it tracks decently well. Make sure to set the namespace to
svo. -
Press the
Bootstrapbutton in the EVO GUI. This will automatically trigger the pipeline.Alternatively, it is also possible to trigger one module at the time:
- Press the
Start/Resetbutton inrqt_evo. Perform a circle (or more), and then pressUpdate. This will trigger a map creation. - If the map looks correct, press
Switch to trackingto start tracking with EVO. If not, reiterate the map creation. - As the camera moves out of the current map, the latter will be automatically updated if the
Map-Expansionis enabled. You may disableMap-Expansionto track high-speed motions using the current map (single keyframe tracking). - The scene should have enough texture and the motions should recall the ones that you can see in the provided examples.
- Press the
-
If anything fails, just press
Ctrl+Cand restart the live node ;)
Some remarks:
- The calibration files paths will be built as
$(find dvs_tracking)/parameters/calib/ncamera/$(arg camera_name).yamland$(find dvs_tracking)/parameters/calib/$(arg camera_name).yaml, wherecamera_nameis specified as argument to the launch file. You can also set it as default inlive.launch. - If your sensor provides frames under a topic
/my_sensor/image_raw, and you want to bootstrap from the traditional frames, you can usebootstrap_image_topic:=/my_sensor/image_raw.
To run the pipeline live, first adjust the template live.launch to your sensor and scene. You can follow the following steps. Further customization, such as which bootstrapper to use, are explained in the launch file itself.
Calibration
Make sure you have updated calibration files for your event camera, in the dvs_tracking/parameters/calib folder.
Make sure your .yaml files have the same format as the provided ones (single camera format, multiple cameras format).
Remark that we have two calibration files in two different format for the same sensor. The single camera format has to be placed in the dvs_tracking/parameters/calib folder and the multiple cameras format in the dvs_tracking/parameters/calib/ncamera folder.
See this section for further references on calibration.
Tuning
Adjust the parameters in the launch file dvs_tracking/launch/template.launch.
Tuning is crucial for a good performance of the pipeline, in particular the min_depth and max_depth parameters in the mapping node, and the bootstrap node parameters.
An explanation of all the parameters for each module of the pipeline can be found in the Wiki.
The main parameters can be found in the template launch file, and are explained contextually. We still invite you to have a look at the Wiki, to discover further interesting features ;)
| Module |
|---|
| Global parameters |
| Bootstrapping |
| Mapping |
| Tracking |
If you are not using the fronto-planar bootstrapper, then you might need to tune SVO.
Remark that this might not be needed. You can test the svo tuning bootstrapping from traditional frames:
roslaunch dvs_tracking live.launch bootstrap_image_topic:=[topic of raw frames] auto_trigger:=[true/false]
For instance, bootstrap_image_topic:=/dvs/image_raw.
Further improvements
In the following we outline the main problems currently known and possible remedies. You are very welcome to contribute to this pipeline to make it even better!
| What can go wrong | TODO |
|---|---|
| Randomness due to OS scheduler, unreliable rosbags | Implement rosbag data provider pattern, and ensure correctness of events consumption |
| Currently the pipeline uses multiple nodes. Switching to nodelets or using a single node could improve the repeatability of the rosbags. | |
| Robustness | bootstrapping: catch whenever SVO does not converge and trigger an automatic restart (what a human operator would eventually do manually). |
tracking: catch whenever the tracker diverges, and re-initialize. Currently we have two parameters to predict this situation, namely min_map_size and min_n_keypoints. |
|
| Improve bootstrapping robustness | Currently we have two working ways to bootstrap the pipeline from events: from SVO (feeding it with events frames) and with a fronto-planar assumption. |
Reducing the assumptions required and making them more reliable would allow a better bootstrapping, reducing the gap to the bootstrapping from traditional frames (bootstrap_image_topic:=/dvs/image_raw). |
and
How to address these problems? Are these two problems caused by the version mismatch of 
69 Sep 10, 2022
17 Mar 1, 2022
109 Dec 23, 2022
84 Dec 31, 2022
60 Nov 15, 2022
12 Feb 17, 2022
23 Oct 29, 2022
20 Dec 14, 2022
45 Dec 21, 2022
213 Jan 4, 2023
23 Dec 27, 2022
45 Dec 21, 2022
157 Dec 26, 2022
47 Nov 18, 2022
491 Jan 5, 2023
1 Mar 7, 2022
266 Dec 28, 2022
59 Jan 5, 2023
14 Aug 3, 2022