This repository is used for automatic calibration between high resolution LiDAR and camera in targetless scenes.

Overview

livox_camera_calib

livox_camera_calib is a robust, high accuracy extrinsic calibration tool between high resolution LiDAR (e.g. Livox) and camera in targetless environment. Our algorithm can run in both indoor and outdoor scenes, and only requires edge information in the scene. If the scene is suitable, we can achieve pixel-level accuracy similar to or even beyond the target based method.

An example of a outdoor calibration scenario. We color the point cloud with the calibrated extrinsic and compare with actual image. A and C are locally enlarged views of the point cloud. B and D are parts of the camera image corresponding to point cloud in A and C.

Related paper

Related paper available on arxiv:
Pixel-level Extrinsic Self Calibration of High Resolution LiDAR and Camera in Targetless Environments

Related video

Related video: https://youtu.be/sp6PZTL2StY

1. Prerequisites

1.1 Ubuntu and ROS

Ubuntu 64-bit 16.04 or 18.04. ROS Kinetic or Melodic. ROS Installation and its additional ROS pacakge:

    sudo apt-get install ros-XXX-cv-bridge ros-xxx-pcl-conversions

1.2 Eigen

Follow Eigen Installation

1.3 Ceres Solver

Follow Ceres Installation.

1.4 PCL

Follow PCL Installation. (Our code is tested with PCL1.7)

2. Build

Clone the repository and catkin_make:

cd ~/catkin_ws/src
git clone https://github.com/hku-mars/livox_camera_calib.git
cd ../
catkin_make
source ~/catkin_ws/devel/setup.bash

3. Run our example

Download Our recorded rosbag to your local path, and then change the path in calib.launch to your data path. Then directly run

roslaunch livox_camera_calib calib.launch

4. Run on your own sensor set

4.1 Record data

Record the point cloud and image msg to rosbag (15s or more for avia LiDAR). Then change the topic name in config_outdoor.yaml file

# Topic name in rosbag
PointCloudTopic: "/livox/lidar"
ImageTopic: "/camera/color/image_raw"

4.2 Modify some params

Modify the camera matrix and distortion coeffs in camera.yaml
Modify the initial extrinsic in config_outdoor.yaml if needed.

Comments
  • Issue with finding depth-continuous regions

    Issue with finding depth-continuous regions

    We were very impressed with the results seen running your sample datasets, but once we tried using our own data, the extrinsics calibration failed in 99% of tested scenarios. In every case, your tool miscalculates scene edges and shifts the image projection, in some cases, by as much as 50-60% off original estimates.

    For example, here is the projection of an ultra-HiRes image onto a cloud generated with the Livox Avia:

    image

    And here is the same cloud / image pair filtered through your calibration tool:

    image

    We provided the same exact intrinsics and initial extrinsics that were used to create the first projection, hoping that your tool would properly recognize scene edges and perfect / optimize the calibration. Instead, it identifies edges far away from their origin and attempts to shift the projection to match them. This results in a completely inaccurate calibration.

    Here is the planner cloud:

    image

    And the residuals seem to show that the lines are properly detected:

    image

    Nonetheless, the optimization shifts everything, producing the result I have posted above. If there are any settings you could recommend that might improve our result, please let me know. I presume that the reason your sample datasets work better is that they are based on scenes with very clean, broad lines, such as those found in artificial structures. We even tried indoor scenes with Aruco / checkerboard plates placed in various locations, and your tool does not recognize any of the lines in these plates. It completely ignores the plates and, as with the example above, "hunts" for edges far from the original extrinsic origin.

    Perhaps this tool is only applicable in industrial / city-centric settings where large buildings or artificial structures are present. We will attempt to re-test in a "cleaner" environment, but the fact that this tool cannot work with even standard Aruco / checkerboard patterns to properly orient itself makes it mostly unusable for the majority of contexts we would require.

    opened by VisionaryMind 13
  • os1 64 and hik vision camera caliberation.

    os1 64 and hik vision camera caliberation.

    Hi I have checked your git repository regarding lidar camera calibration. I am a research intern at my Home university NUST Pakistan. I have an os1 64 spinning lidar available at my lab. I need to calibrate the camera and lidar. Now kindly tell me the precise step I need to follow to get that calibration parameter. I don't know how to use Fast LIO for spinning lidar calibration. kindly outline the steps to follow to get extrinsic parameters with your package using os1 64 lidar. I would very thankful to you for this favor.I have bag file recorded with topic pointcloud2 and hikvision camera image. kindly help me in this regard

    opened by islamtalha01 7
  • Bad result of calibration

    Bad result of calibration

    Lidar: Livox Mid-100 Camera: Flir BlackFly Rosbag: https://cloud.tsinghua.edu.cn/f/09ceccd8e1c647f59650/?dl=1 Camera_config: https://cloud.tsinghua.edu.cn/f/2687d771b0ea421ea30d/?dl=1

    The calibration result seems promising after rough calibration, by collapsed after fine-tune optimization. rough_extrinsic optim_extrinsic

    opened by meng-zha 7
  • Rough Optimization looks good, then Optimization goes bad

    Rough Optimization looks good, then Optimization goes bad

    Hi there,

    Cool package, hoping to get this to work! I'm using an os1-64. When I launch the calib the Rough Optimization starts and is pretty good. However, when it moves onto the Optimization step, the point cloud is aligned very poorly. It seems when Optimization starts the cloud is rotated by some amount, maybe 90 degrees.

    Here are a couple photos. Please note that my camera images do not fit on my monitor so I can only show what I can see:

    Rough Optimization step: image

    Optimization Step: image

    Any thoughts or ideas would be appreciated. Thank you

    opened by pauldeee 6
  • Process dies on LiDAR edge extraction

    Process dies on LiDAR edge extraction

    I have setup an environment with Ubuntu 18.04 and ROS Melodic. All dependencies have been installed, however, using your sample ROS bag files, I am not able to get past the Lidar edge extraction phase. Here is the terminal output:

    Camera Matrix: 
    [1364.45, 0, 958.327;
     0, 1366.46, 535.074;
     0, 0, 1]
    Distortion Coeffs: 
    [0.0958277;
     -0.198233;
     -0.000147133;
     -0.000430056;
     0]
    Init extrinsic: 
    [0, -1, 0, 0;
     0, 0, -1, 0;
     1, 0, 0, 0;
     0, 0, 0, 1]
    [ INFO] [1617231323.709287053]: Loading the rosbag /home/visionarymind/Documents/scans/livoxcalib/outdoor1.bag
    [ INFO] [1617231328.493049027]: Sucessfully load Point Cloud and Image
    [ INFO] [1617231328.493564736]: Building Voxel
    [ INFO] [1617231357.841487217]: Extracting Lidar Edge
    [lidar_camera_calib-2] process has died [pid 2586, exit code -11, cmd /home/visionarymind/vision_ws/devel/lib/livox_camera_calib/lidar_camera_calib /home/visionarymind/vision_ws/src/livox_camera_calib/config/camera.yaml /home/visionarymind/vision_ws/src/livox_camera_calib/config/config_outdoor.yaml /home/visionarymind/Documents/scans/livoxcalib/outdoor1.bag /home/visionarymind/vision_ws/src/livox_camera_calib/result __name:=lidar_camera_calib __log:=/home/visionarymind/.ros/log/2c3b8fe2-9274-11eb-9545-000c29df74c2/lidar_camera_calib-2.log].
    log file: /home/visionarymind/.ros/log/2c3b8fe2-9274-11eb-9545-000c29df74c2/lidar_camera_calib-2*.log
    

    Would you have any advice on how to correct this problem?

    opened by VisionaryMind 6
  • pcl problem

    pcl problem

    Hi, thank you for your contribution to the society. When I run your code, I met a problem:

    [ INFO] [1626940489.756662588]: Loading the rosbag /home/catkin_ws/src/livox_camera_calib/data/merge.bag [ INFO] [1626940490.902931052]: Sucessfully load Point Cloud and Image [ INFO] [1626940490.903897494]: Building Voxel [ INFO] [1626940508.755300261]: Extracting Lidar Edge [ INFO] [1626940526.485055384]: Load all data! [ INFO] [1626940526.904265229]: Finish prepare! [pcl::KdTreeFLANN::setInputCloud] Cannot create a KDTree with an empty input cloud!

    Do you know in which function this error happens? So I can check my own data. Thank you for your time.

    opened by Zhu-Liyuan 4
  • Multi Calibration not Converging/gets stuck

    Multi Calibration not Converging/gets stuck

    Hi Team

    Thanks for the awesome repository.

    I was able to run Single Extrinsic Calibration but when I run multi calibration, it just gets stuck forever. Multi Calibration does not even work for 2 image and 2 pcd files. I am attaching my snapshot of yaml file I used for multi calibration. I am also attaching the output after running multi calibration. I have used same same multi calibration file without any change except the input yaml file.

    Please help me out if I am missing something.

    Regards

    multi_calib_yaml multi_calib_output .

    opened by ckgpeace 3
  • Question about vpnp cost function

    Question about vpnp cost function

    Dear author, thanks for your excellent work, but I am puzzled by the vpnp cost function, I did not understand why we need the line direction to compute the matrix V, and the physical meaning of this formula VRV.transpose

    1636686381(1)

    Look forward to your reply

    opened by RoseWillow 3
  • Issue while running on ros-kinetics based docker image on ubuntu machine18.04

    Issue while running on ros-kinetics based docker image on ubuntu machine18.04

    Hi, Because of lower version of PCL(1.17), i prepared a docker image(ros:kinetic-ros-core-xenial), with all dependencies. Also placed the respective files on path. While running i am getting the following error-

    [ INFO] [1629189514.066569998]: Loading the rosbag /opt/catkin_ws/src/livox_camera_calib/data/indoor1.bag process[rviz-3]: started with pid [381] process 381: D-Bus library appears to be incorrectly set up; failed to read machine uuid: UUID file '/etc/machine-id' should contain a hex string of length 32, not length 0, with no other text See the manual page for dbus-uuidgen to correct this issue. libGL error: No matching fbConfigs or visuals found libGL error: failed to load driver: swrast Could not initialize OpenGL for RasterGLSurface, reverting to RasterSurface. libGL error: No matching fbConfigs or visuals found libGL error: failed to load driver: swrast

    [rviz-3] process has died [pid 381, exit code -11, cmd /opt/ros/kinetic/lib/rviz/rviz -d /opt/catkin_ws/src/livox_camera_calib/rviz_cfg/calib.rviz __name:=rviz __log:=/root/.ros/log/81017cba-ff36-11eb-bf90-2c56dc38a7a2/rviz-3.log]. log file: /root/.ros/log/81017cba-ff36-11eb-bf90-2c56dc38a7a2/rviz-3*.log*

    [ INFO] [1629189515.516145283]: Sucessfully load Point Cloud and Image [ INFO] [1629189515.516546482]: Building Voxel [ INFO] [1629189539.422697927]: Extracting Lidar Edge [ERROR] [1629189544.520860218]: Can not load image from /opt/catkin_ws/src/livox_camera_calib/data/indoor1.bag [lidar_camera_calib-2] process has died [pid 374, exit code 255, cmd /opt/catkin_ws/devel/lib/livox_camera_calib/lidar_camera_calib /opt/catkin_ws/src/livox_camera_calib/config/camera.yaml /opt/catkin_ws/src/livox_camera_calib/config/config_outdoor.yaml /opt/catkin_ws/src/livox_camera_calib/data/indoor1.bag /opt/catkin_ws/src/livox_camera_calib/result __name:=lidar_camera_calib __log:=/root/.ros/log/81017cba-ff36-11eb-bf90-2c56dc38a7a2/lidar_camera_calib-2.log]. log file: /root/.ros/log/81017cba-ff36-11eb-bf90-2c56dc38a7a2/lidar_camera_calib-2*.log*

    I am not sure, why process has been died and telling indoor1.bag is not found but it has already started with the same.

    Any suggestions ? Thanks.

    opened by rahulsharma11 3
  • pnp_calib

    pnp_calib

    English version

    Thank you for the open source. This work is very perfect. However, I have the some question about the code. First, what's the vpnp. I don't know the Physical mean. Could you provide me reference? Second, why's the camera frame map into the pixel frame(image frame), you use the (u0-cx)/ fx. According to my knowledge, I would use (u0)/fx - cx. Thank you very much. image

    中文版 感謝您提供的開源程式碼,這個程式碼非常的棒 但是我有一些疑問是有關於code的部份

    1. 請問什麼是vpnp,我不太清楚其物理意義,請問能麻煩您提供我關於這個參考資料嘛?
    2. 在lidar_camera_calib.cpp裡面中,您透過相機內參投影到pixel frame,您使用的是(u0-cx)/ fx,您是先減掉像素中心在除以焦距長. 但照我知識中,應該是(u0)/fx - cx,先除以焦距長,在減掉像素的中心,請問我哪裡搞錯了嘛?
    opened by yanlong658 3
  • 关于vPnP的问题

    关于vPnP的问题

    非常感谢您的分享,有几个问题想请教下您:

    1. 下图这个公式可以解释下吗,代表什么含义?
    2. 代码里构建的残差公式是依据的下图这个公式,但是比这个公式多乘了一个V.transpose(),可以解释下吗?
    3. 方便的话,能否把这篇论文发我看看。地址: [email protected]

    再次感谢,期待您的回复

    image

    opened by codehory 2
  • 作者您好!使用outdoor.yaml时发生的一个严重错误,以及一个无法在粗标定以后显示rviz的错误

    作者您好!使用outdoor.yaml时发生的一个严重错误,以及一个无法在粗标定以后显示rviz的错误

    您好,我在室外场景尝试使用标定,但是在使用config_outdoor.yaml进行标定时(也就是尝试过使用indoor yaml文件不会出现的错误),在迭代完成计算结果的时候会出现一个错误,如图1,导致无法输出结果: 2022-12-01 20-45-39 的屏幕截图 另外不管使用哪种config的时候在粗标定结束后会无法正常使用rviz,报错如下: 2022-12-01 20-51-09 的屏幕截图 希望得到您的回复,谢谢!

    opened by lilrookie76 0
  • Depth discontinuous boundary problem

    Depth discontinuous boundary problem

    Thank you very much for your excellent work. We are experimenting with your method, but we have encountered some problems. I noticed that you compared the depth continuous boundary with the depth discontinuous boundary in the paper and provided a direct image (Figure 7 in the paper). Could you please help to provide the engineering code for depth discontinuous boundary extraction? I've looked for several articles that use this method, but they don't have the open source code. Thanks again and best wishes.

    opened by Xiaotianzhi 3
  • seg.segment 出现 std::bad_alloc 的错误

    seg.segment 出现 std::bad_alloc 的错误

    您好,我的设备是ubuntu20.04 pcl 1.10 ros中——catkin_make没有问题 在运行roslaunch _camera_calib calib.launch 时,出现了terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc错误 麻烦您帮忙看看,问题可能出在哪里? 2022-11-01 14-36-14屏幕截图

    opened by hhz334 1
Owner
HKU-Mars-Lab
Mechatronics and Robotic Systems (MaRS) Laboratory
HKU-Mars-Lab
ROS package to calibrate the extrinsic parameters between LiDAR and Camera.

lidar_camera_calibrator lidar_camera_calibrator is a semi-automatic, high-precision, feature-based camera and LIDAR extrinsic calibration tool. In gen

Networked Robotics and Sytems Lab 74 Nov 22, 2022
This repository uses a ROS node to subscribe to camera (hikvision) and lidar (livox) data. After the node merges the data, it publishes the colored point cloud and displays it in rviz.

fusion-lidar-camera-ROS 一、介绍 本仓库是一个ROS工作空间,其中ws_fusion_camera/src有一个工具包color_pc ws_fusion_camera │ README.md │ └───src │ └───package: c

hongyu wang 21 Nov 23, 2022
A generic and robust calibration toolbox for multi-camera systems

MC-Calib Toolbox described in the paper "MultiCamCalib: A Generic Calibration Toolbox for Multi-Camera Systems". Installation Requirements: Ceres, Boo

null 195 Nov 28, 2022
the implementations of 'A Flexible New Technique for Camera Calibration' and Bouguet's method

StereoCameraCalibration MonocularCameraCalibration/StereoCameraCalibration/StereoCameraRectification 1、Class "MonocularCameraCalibration" provides the

gtc1072 8 Nov 3, 2022
Harsh Badwaik 1 Dec 19, 2021
Windows 10 interface adjustment tool supports automatic switching of light and dark modes, automatic switching of themes and transparent setting of taskbar

win10_tools Windows 10 interface adjustment tool supports automatic switching of light and dark modes, automatic switching of themes and transparent s

Simon 1 Dec 3, 2021
calibrate a Livox LiDAR and a camera

Livox LiDAR-Camera Calibration This method is from the official method of Livox(https://github.com/Livox-SDK/livox_camera_lidar_calibration) It's just

null 9 Jul 23, 2022
A Gazebo based LiDAR-Camera Data Simulator.

Livox-LiDAR-Camera System Simulator A package to provide plug-in for Livox Series LiDAR. This work is inherited from EpsAvlc and LvFengchi's work: liv

zhijianglu 12 Nov 28, 2022
BAAF-Net - Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021)

Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021) This repository is for BAAF-Net introduce

null 89 Nov 8, 2022
Livox-Mapping - An all-in-one and ready-to-use LiDAR-inertial odometry system for Livox LiDAR

Livox-Mapping This repository implements an all-in-one and ready-to-use LiDAR-inertial odometry system for Livox LiDAR. The system is developed based

null 248 Nov 27, 2022
Visual-inertial-wheel fusion odometry, better performance in scenes with drastic changes in light

VIW-Fusion An visual-inertial-wheel fusion odometry VIW-Fusion is an optimization-based viusla-inertial-wheel fusion odometry, which is developed as a

庄庭达 248 Nov 29, 2022
A Robust LiDAR-Inertial Odometry for Livox LiDAR

LIO-Livox (A Robust LiDAR-Inertial Odometry for Livox LiDAR) This respository implements a robust LiDAR-inertial odometry system for Livox LiDAR. The

livox 347 Nov 22, 2022
Lidar-with-velocity - Lidar with Velocity: Motion Distortion Correction of Point Clouds from Oscillating Scanning Lidars

Lidar with Velocity A robust camera and Lidar fusion based velocity estimator to undistort the pointcloud. This repository is a barebones implementati

ISEE Research Group 160 Nov 29, 2022
Repository for building and operating REVOLVER: An automatic protein purification system for gravity columns. Developed at the University of Toronto.

REVOLVER: An automated protein purification system This repository contains all the hardware and firmware files to build and operate REVOLVER: an auto

Laboratory for Metabolic Systems Engineering 3 Jun 14, 2022
This project is used for lidar point cloud undistortion.

livox_cloud_undistortion This project is used for lidar point cloud undistortion. During the recording process, the lidar point cloud has naturally th

livox 70 Nov 18, 2022
Fast and Accurate Extrinsic Calibration for Multiple LiDARs and Cameras

Fast and Accurate Extrinsic Calibration for Multiple LiDARs and Cameras The pre-print version of our paper is available here. The pre-release code has

HKU-Mars-Lab 240 Nov 24, 2022
rlua -- High level bindings between Rust and Lua

rlua -- High level bindings between Rust and Lua

Amethyst Foundation 1.4k Nov 26, 2022
A real-time, direct and tightly-coupled LiDAR-Inertial SLAM for high velocities with spinning LiDARs

LIMO-Velo [Alpha] ?? [16 February 2022] ?? The project is on alpha stage, so be sure to open Issues and Discussions and give all the feedback you can!

Andreu Huguet 139 Nov 25, 2022
This repo contains source code of our paper presented in IROS2021 "Single-Shot is Enough: Panoramic Infrastructure Based Calibration of Multiple Cameras and 3D LiDARs"

Single-Shot is Enough: Panoramic Infrastructure Based Calibration of Multiple Cameras and 3D LiDARs Updates [2021/09/01] first commit, source code of

Alibaba 78 Nov 25, 2022