Docker files and scripts to setup and run VINS-FUSION-gpu on NVIDIA jetson boards inside a docker container.

Overview

jetson_vins_fusion_docker

This repository provides Docker files and scripts to easily setup and run VINS-FUSION-gpu on NVIDIA jetson boards inside a docker container.

This is tested on Xavier NX with Jetpack 4.4 [L4T 32.4.3]

This repository is inspired by jteson-containers and vins-fusion-gpu-tx2-nano

Docker image

The Docker image Docker.vins.gpu contains

Installtion steps

Hardware

  • It's recommended to use Xavier NX with SSD drive for best performance. Check this video to see how to use Xavier NX image from the SSD drive

Docker Default Runtime

To enable access to the CUDA compiler (nvcc) during docker build operations, add "default-runtime": "nvidia" to your /etc/docker/daemon.json configuration file before attempting to build the containers:

{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    },

    "default-runtime": "nvidia"
}

You will then want to restart the Docker service or reboot your system before proceeding.

Building the docker image

  • Open a terminal, and execute the folllwing commands

    # Create a directory to clone this package into
    mkdir -p $HOME/src && cd $HOME/src/
    # Clone this package
    git clone https://github.com/mzahana/jetson_vins_fusion_docker.git
  • Build the mzahana:vins_gpu Docker image

    cd $HOME/src/jetson_vins_fusion_docker
    ./scripts/setup_jetson.sh

    You may need to provide passowrd for sudo when asked

  • Once the image is built, you can verify that by listing Docker images docker images. You should see mzahana:vins_gpu availble in the listed images

  • An alias will be added in the ~/.bashrc for convenience. The alias is called vins_container. You can simply run the VINS container by executing vins_container in a terminal window

  • Once the container is running, an interactive terminal inside the container can be used.

NOTE The docker image includes installations of Realsense SDK and realsense-ros in case the D435i cameras is used with VINS. Make sure to donload the installRealSenseROS package on the Jetson board, and run the disableAutosuspend.sh to turn off the USB autosuspend setting on the Jetson so that the camera is always available. Then reboot for the changes to take effect.

Running VINS nodes

  • The Docker.vins.gpu image comes with a convenience ROS package, jetson_vins_fusion_scripts, which has some launch files and configuration files for the VINS-FUSION-gpu package

  • You can run the VINS node (loop_fusion, vins_node) using the following command

    roslaunch jetson_vins_fusion_scripts vins.launch 

    The vins.launch script runs two nodes, loop_fusion, and vins_node). To disable the loop_fusion,

    roslaunch jetson_vins_fusion_scripts vins.launch run_loop_fusion:=false
  • Then you can either run a bag file in another terminal (you can log into the container again using vins_container alias)

    rosbag play bag_name.bag

    You can downlod one of the EuROC bags, inside the container using wget command

  • The jetson_vins_fusion_scripts package also has example configuration files for D435i cameras to use with VINS-FUSION. You will need to calibrate the D435i IMU using this document. Then you need to calibrate the stereo setup+IMU using Kalibr

  • For easy calibration with Kalibr, a docker image is available here

  • NOTE There is a shared folder between the container (container name is vins_gpu) and the host system (Xavier) in order to easily share files between the two. The shared folder is located in the home folder of the host (Xavier) under the name vins_gpu_shared_volume, and available inside the container's home folder under the name shared_volume

Runing on Jetson Nano

Althouhg this package was not tested on Jetson Nano, in theory it should work. However, you need to make sure that you increase the swap size

Add a swap of 4GB

sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo swapon --show
sudo cp /etc/fstab /etc/fstab.bak
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab 

To lock Jetson Nano at its maximum frequency and power mode by running the following commands:

sudo jetson_clocks
sudo nvpmodel -m 0 

Using VINS with cameras

Intel Realsense D435i

  • Install the docker image as described above
  • Calibrate the D435i IMU as described here
  • Calibrate the D435i stereo infrared cameras using Kalibr
    • Prepare a calibration target. It is recommended to print an apriltag calibration target, you can download from here
    • Connect the D435i camera using the USB C cable
    • If you are using a PC, make sure to install Realsense SDK and realsense-ros in order to run the D435i. Otherwise, you can run it directly in the docker container provided by this repo
    • Run the rs_camera.launch script
      roslaunch jetson_vins_fusion_scripts rs_camera.launch
    • Record image and imu topics. For convenience, there is a launch file that you can use, and the recorded bag will be saved inside ~/shared_volume
      roslaunch jetson_vins_fusion_scripts rs_record_bag.launch
      NOTE Make sure that you move the camera in all axes. See this video
    • You will need to print a calibration target (apriltag target is recommended, see here )
    • Calibrate the D435i stereo camera using Kalibr and the recorded rosbag. For convenience, a docker image with Kalibr is available here
    • Use Kalibr camera-IMU calibration
      • You will need to provide IMU calibration Yaml file (you can use imu_utils package, and the multi-camera calibration results by Kalibr, and a ROS bag that have images and IMU topics
    • Use the camera-IMU calibration results from Kalibr to update VINS configuratoin files. The VINS configuration files would be available inside the vins_gpu container in ~/catkin_ws/jetson_vins_fusion_scripts/config/d435i, inside the container
  • Inside the container's terminal, run vins nodes
roslaunch jetson_vins_fusion_scripts vins.launch config_file:=/root/catkin_ws/src/
jetson_vins_fusion_scripts/config/d435i/rs_stereo_imu_confg.yaml
  • VINS configuration files can also be placed in the shared directory betwee the the vins container and Jetson file system. Simply place the config files in the ~/vins_shared_volume of Jetson, and it will be available inside the ~/shared_voluem inside the container. Then you can, for example, launch the nodes as follows
roslaunch jetson_vins_fusion_scripts vins.launch config_file:=/root/shared_volume/vins_config/d435i/rs_stereo_imu_confg.yaml

Here (Jetson), I created a folder ~/vins_shared_volume/vins_config/d435i and placed left.yaml & right.yaml & rs_stereo_imu_confg.yaml files in it

Issues
  • unauthorized: authentication required

    unauthorized: authentication required

    When I run ~/src/jetson_vins_fusion_docker$ ./scripts/setup_jetson.sh

    I get

    Docker configuration ...
     
    groupadd: group 'docker' already exists
    Adding user developer to group docker
    Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
    Executing: /lib/systemd/systemd-sysv-install enable docker
     
    Serial ports configuration (ttyTHS1) ...
     
     
    Adding udev rules for /dev/ttyTHS* ...
     
    KERNEL=="ttyTHS*", MODE="0666"
     
    Disabling nvgetty ...
     
    
    Install udev rules for Realsense D435...
    Realsense udev Rules installed
     
    Building Dockerfile.vins.gpu ...
     
    reading L4T version from /etc/nv_tegra_release
    L4T BSP Version:  L4T R32.5.2
    l4t-base image:  nvcr.io/nvidian/nvidia-l4t-base:r32.5.2
    Building containers for melodic...
    Building ros:melodic-ros-base-l4t-r32.5.2 container...
    Sending build context to Docker daemon  271.9kB
    Step 1/17 : ARG BASE_IMAGE=nvcr.io/nvidia/l4t-base:r32.5.0
    Step 2/17 : FROM ${BASE_IMAGE}
    unauthorized: authentication required
     
    Adding alias to .bashrc script ...
     
    alias vins_container='source $HOME/src/jetson_vins_fusion_docker/scripts/docker_run_vins.sh'
     
    #------------- You can run the VINS container from the terminal by executing vins_container -------------#
     
    #------------- Please reboot your Jetson before running executing the vins_container alias, for some changes to take effect -------------#
    
     
    

    The problem is:

    unauthorized: authentication required

    As a result, the build process is incomplete. This error usually happens if the docker file doesn't have the correct base image name. Let me know what you think.

    opened by canberkgurel 3
  • Using VINS_Fusion for OFFBOARD

    Using VINS_Fusion for OFFBOARD

    Hi @mzahana Thanks for this repo it help me alot...

    I am trying to use OFFBOARD mode in Pixhawk 2.4.8(PX4 Firmware) using VINS_FUSION with D435i..

    I'm getting two errors while doing mavsys mode -c OFFBOARD,in the terminal where i launched px4.launch I'm getting error saying UNSUPPORTED FCU and in the terminal where I'm writing mavsys mode -c OFFBOARD this command I'm getting Time out! error. Now the steps I'm doing is:- 1.) Launching the camera and the VINS_FUSION 2.) Launching the Fast planner 3.) Launching the px4.launch (roslaunch mavros px4.launch) 3.) Publishing the topic /mavros/setpoint_raw/local(frame id= map,x=0,y=0,z=2) 4.) Arming the drone

    These steps are working perfectly but 5.) While changing the mode to OFFBOARD**(mavsys mode -c OFFBOARD)** I'm getting an error here.. drone is unable to go to OFFBOARD mode.. I'm also having some doubts:-

    1.) Which parameters we've to change in px4 so that by drone will be able to take position inputs from odom? 2.) How I am gonna make PIXHAWK understand that the VINS is the Odometry data that it should work

    opened by EhrazImam 0
  • Significant delay when running in real time

    Significant delay when running in real time

    Hello,

    I installed everything as described, but when I ran the example, there was multiple seconds of delay between me moving the camera and the path updating in Rviz. Is this delay normal? D435i's IMU is calibrated using Intel's official calibration software and uploaded to the camera. I am using your yaml file as I was having problems with my calibration process.

    Thanks

    opened by HGuo2022 2
  • Camera orientation of D435i

    Camera orientation of D435i

    Hi @mzahana , I'm using vins with d435i camera and I'm using image of camera only not the imu of it because when we're enabling the imu the vins is getting extreme distorted,also we have done imu calibration perfectly and the calibration data is also perfect but i don't why it's getting distorted although vins is working great with image only so im using it for now.Now I'm facing an orientation problem in vins like in vins rviz the camera is in upward direction also when we move our drone upward the value of -y is increasing and when we are moving our drone forward the z is increasing... Please help regarding the issue

    opened by EhrazImam 3
  • failed to create shim OCI runtime create failed

    failed to create shim OCI runtime create failed

    When I ran ~/src/jetson_vins_fusion_docker$ ./scripts/setup_jetson.sh

    I got:

    Docker configuration ...
     
    [sudo] password for developer: 
    groupadd: group 'docker' already exists
    Adding user developer to group docker
    Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
    Executing: /lib/systemd/systemd-sysv-install enable docker
     
    Serial ports configuration (ttyTHS1) ...
     
     
    Adding udev rules for /dev/ttyTHS* ...
     
    KERNEL=="ttyTHS*", MODE="0666"
     
    Disabling nvgetty ...
     
    
    Install udev rules for Realsense D435...
    Realsense udev Rules installed
     
    Building Dockerfile.vins.gpu ...
     
    reading L4T version from /etc/nv_tegra_release
    L4T BSP Version:  L4T R32.5.2
    l4t-base image:  nvcr.io/nvidia/l4t-base:r32.5.0
    Building containers for melodic...
    Building ros:melodic-ros-base-l4t-r32.5.2 container...
    Sending build context to Docker daemon  282.1kB
    Step 1/17 : ARG BASE_IMAGE=nvcr.io/nvidia/l4t-base:r32.5.0
    Step 2/17 : FROM ${BASE_IMAGE}
     ---> d9141f00a2c3
    Step 3/17 : ARG ROS_PKG=ros_base
     ---> Using cache
     ---> ec3716bdfe01
    Step 4/17 : ENV ROS_DISTRO=melodic
     ---> Using cache
     ---> fdafbad3cf33
    Step 5/17 : ENV ROS_ROOT=/opt/ros/${ROS_DISTRO}
     ---> Using cache
     ---> 5e86a8e56055
    Step 6/17 : ENV DEBIAN_FRONTEND=noninteractive
     ---> Using cache
     ---> c84ee7847575
    Step 7/17 : WORKDIR /workspace
     ---> Using cache
     ---> 8a888747948a
    Step 8/17 : RUN apt-get update &&     apt-get install -y --no-install-recommends           git 		cmake 		build-essential 		curl 		wget 		gnupg2 		lsb-release 		ca-certificates     && rm -rf /var/lib/apt/lists/*
     ---> Running in 382e4b899c55
    failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: error adding seccomp filter rule for syscall clone3: permission denied: unknown
     
    Adding alias to .bashrc script ...
     
    alias vins_container='source $HOME/src/jetson_vins_fusion_docker/scripts/docker_run_vins.sh'
     
    #------------- You can run the VINS container from the terminal by executing vins_container -------------#
     
    #------------- Please reboot your Jetson before running executing the vins_container alias, for some changes to take effect -------------#
    

    For some reason, I'm getting the OCI runtime create failed error. My suspicion is that it's related to the docker version. See https://github.com/containerd/containerd/issues/6203

    @mzahana Did you run into this problem? Can you send me the result of dockerd --version?

    Mine looks like this:

    $uname -a
    Linux KimNx 4.9.201-tegra #1 SMP PREEMPT Fri Jul 9 08:56:59 PDT 2021 aarch64 aarch64 aarch64 GNU/Linux
    $dockerd -v
    Docker version 20.10.7, build 20.10.7-0ubuntu5~18.04.3
    
    opened by canberkgurel 2
  • Docker image on SD card (Jetson TX2)

    Docker image on SD card (Jetson TX2)

    Thanks for the docker image @mzahana.

    I followed the procedure mentioned in the README but due to large image size and storage constraints on Jetson TX2, I added the built docker image on external SD card. While testing Vins_fusion_gpu I faced latency issues. I suspect that, having the docker image on the SD card is leading to this latency. What are your views on the same?

    What is the size of built docker image for you? Have you used an external storage for your setup?

    opened by Pallav1299 4
Owner
Mohamed Abdelkader Zahana
interests: - Control - Robotics (multi-agent systems) - Drones - PX4
Mohamed Abdelkader Zahana
Hardware-accelerated DNN model inference ROS2 packages using NVIDIA Triton/TensorRT for both Jetson and x86_64 with CUDA-capable GPU.

Isaac ROS DNN Inference Overview This repository provides two NVIDIA GPU-accelerated ROS2 nodes that perform deep learning inference using custom mode

NVIDIA Isaac ROS 42 Jul 18, 2022
A C++ implementation of Yolov5 helmet detection in Jetson Xavier nx and Jetson nano

A C++ implementation of Yolov5 to detect head or helmet in the wild in Jetson Xavier nx and Jetson nano This repository uses yolov5 to detect humnan h

null 10 Jul 27, 2022
A C++ implementation of Yolov5 to detect mask running in Jetson Xavier nx and Jetson nano.

yolov5-mask-detect A C++ implementation of Yolov5 to detect mask running in Jetson Xavier nx and Jetson nano.In Jetson Xavier Nx, it can achieve 33 FP

null 3 Mar 19, 2022
3D reconstruction with L515, VINS-RGBD and voxblox

3D-Recon: VINS-RGBD voxblox Recently we are trying to create dataset for 3D perception, so we need to create a 3D scanned environment like meta Replic

Ran Cheng 25 Jul 29, 2022
GPU Cloth TOP in TouchDesigner using CUDA-enabled NVIDIA Flex

This project demonstrates how to use NVIDIA FleX for GPU cloth simulation in a TouchDesigner Custom Operator. It also shows how to render dynamic meshes from the texture data using custom PBR GLSL material shaders inside TouchDesigner.

Vinícius Ginja 37 Jul 27, 2022
GPU ray tracing framework using NVIDIA OptiX 7

GPU ray tracing framework using NVIDIA OptiX 7

Shunji Kiuchi 22 Jun 11, 2022
Golang bindings for Nvidia Datacenter GPU Manager (DCGM)

Bindings Golang bindings are provided for NVIDIA Data Center GPU Manager (DCGM). DCGM is a set of tools for managing and monitoring NVIDIA GPUs in clu

NVIDIA Corporation 29 Jul 6, 2022
waifu2x converter ncnn version, runs fast on intel / amd / nvidia GPU with vulkan

waifu2x ncnn Vulkan ncnn implementation of waifu2x converter. Runs fast on Intel / AMD / Nvidia with Vulkan API. waifu2x-ncnn-vulkan uses ncnn project

null 2.2k Aug 7, 2022
Openpilot on Jetson Xavier NX

Licensing xnxpilot is released under the MIT license. Some parts of the software are released under other licenses as specified. Any user of this soft

eFini 56 Jul 2, 2022
YoloX for a Jetson Nano 4 using ncnn.

YoloX Jetson Nano YoloX with the ncnn framework. Paper: https://arxiv.org/pdf/2107.08430.pdf Special made for a Jetson Nano, see Q-engineering deep le

Q-engineering 8 Aug 2, 2022
With deep learning to neuroscience world with shield for jetson nano - JNEEG (In progress)

With deep learning to neuroscience world with shield for jetson nano - JNEEG (In progress)

Ildaron 19 Jul 4, 2022
🐸 Coqui STT is an open source Speech-to-Text toolkit which can run in real time on devices ranging from a Raspberry Pi 4 to high power GPU servers

Coqui STT ( ?? STT) is an open-source deep-learning toolkit for training and deploying speech-to-text models. ?? STT is battle tested in both producti

Coqui.ai 1.4k Jul 29, 2022
DeepSpeech is an open source embedded (offline, on-device) speech-to-text engine which can run in real time on devices ranging from a Raspberry Pi 4 to high power GPU servers.

Project DeepSpeech DeepSpeech is an open-source Speech-To-Text engine, using a model trained by machine learning techniques based on Baidu's Deep Spee

Mozilla 20k Jul 29, 2022
4eisa40 GPU computing : exploiting the GPU to execute advanced simulations

GPU-computing 4eisa40 GPU computing : exploiting the GPU to execute advanced simulations Activities Parallel programming Algorithms Image processing O

Ecam 4MIN repositories 2 Jan 10, 2022
Time Series Quick Simulator - able to perform time series analysis and to setup validation experiments.

tsqsim Time Series Quick Simulator - able to perform time series analysis and to setup validation experiments. With its somewhat limited plotting capa

null 6 Mar 8, 2022
A modern object detector inside fragment shaders

YOLOv4 Tiny in UnityCG/HLSL Video Demo: https://twitter.com/SCRNinVR/status/1380238589238206465?s=20 Overview YOLOv4 Tiny is one of the fastest object

null 43 Jul 26, 2022
An x64 binary executing code that's not inside of it.

Remote Machine Code Fetch & Exec in other words, another self rewriting binary.. boy I just love doing these. Description The idea behind this one is

x0reaxeax 2 Jul 9, 2022
Visual odometry package based on hardware-accelerated NVIDIA Elbrus library with world class quality and performance.

Isaac ROS Visual Odometry This repository provides a ROS2 package that estimates stereo visual inertial odometry using the Isaac Elbrus GPU-accelerate

NVIDIA Isaac ROS 178 Aug 3, 2022