Python and C++ implementation of "MarkerPose: Robust real-time planar target tracking for accurate stereo pose estimation". Accepted at LXCV Workshop @ CVPR 2021.

Overview

MarkerPose: Robust Real-time Planar Target Tracking for Accurate Stereo Pose Estimation

This is a PyTorch and LibTorch implementation of MarkerPose: a robust, real-time pose estimation method based on a planar marker of three circles and a calibrated stereo vision system for high-accuracy pose estimation.

MarkerPose

MarkerPose method consists of three stages. In the first stage, marker points in a pixel-level accuracy, and their IDs are estimated with a SuperPoint-like network for both views. In the second stage, three square patches that contain each ellipse of the target are extracted centered in the rough 2D locations previously estimated. With EllipSegNet the contour of the ellipses is segmented for sub-pixel-level centroid estimation for the first and second view. Finally, in the last stage, with the sub-pixel matches of both views, triangulation is applied for 3D pose estimation. For more details see our paper.

robot_arms

Pose estimation example

To run the Python or C++ pose estimation examples using images of the marker attached to a robotic arm, you need first to clone this repository and download the dataset. This dataset contains the stereo calibration parameters, stereo images, and pretrained weights for SuperPoint and EllipSegNet.

  • Clone this repo: git clone https://github.com/jhacsonmeza/MarkerPose
  • Download the dataset here.
  • Move the dataset/ folder to the cloned repo folder: mv path/to/dataset/ MarkerPose/.

The folder structure into MarkerPose/ directory should be:

MarkerPose
    ├── C++
    ├── dataset
    ├── figures
    └── Python

To know how to run the pose estimation examples, see the Python/ folder for the PyTorch version, and the C++/ folder the LibTorch version. Furthermore, the code for training SuperPoint and EllipSegNet is also available in both versions.

MarkerPose for 3D freehand ultrasound

Freehand 3D ultrasound is a medical imaging technique that consists of tracking the pose of an ultrasound probe for mapping any ultrasound scan to 3D space. We reconstructed a cylindrical object which was submerged in water into another cylinder. The diameter of this inner cylinder was measured with the estimated point cloud. For more details about this experiment see our paper (Section 5.3). The following animation shows the stereo vision images with the estimated pose with MarkerPose for the acquired sequence. Furthermore, we have the ultrasound images mapped to 3D space with a 3D model of a probe.

robot_arms

The 3D representation was generated with OpenCV 3D Visualizer. The 3D model of the external cylindrical object where the probe was moved along, was generated with the structured light technique. For more details see our repo, where we propose to combine freehand ultrasound and structured light as a multimodal technique using MarkerPose for pose estimation.

Citation

If you find this code useful, please consider citing:

@inproceedings{meza2021markerpose,
    author    = {Meza, Jhacson and Romero, Lenny A. and Marrugo, Andres G.},
    title     = {MarkerPose: Robust Real-Time Planar Target Tracking for Accurate Stereo Pose Estimation},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2021},
    pages     = {1282-1290}
}
Issues
  • Training for randomly placed markers?

    Training for randomly placed markers?

    Hi, and thank you for making this code available, I have it running perfectly on your dataset.

    I would like to adapt it for use as a general marker detector, to be used to find the centre points of randomly placed markers in a space. Something like this:

    temp

    Where markers could be on any plane, and there could be any number of them. The camera is moving, so markers would come in and out of frame and the number of markers in frame would change dynamically. I would not need the classification step, just the sub pixel point detection.

    Do you have any tips for training the models for this use? I imagine the ellipsegnet can stay the same as it is, but i would need to re-train the superpoint network on different images, is that right?

    Thanks again!

    opened by antithing 3
  • Regarding the patches used for training the segmentation network

    Regarding the patches used for training the segmentation network

    Hello, Thanks for the amazing work! In your paper, you state: "For EllipSegNet training, we extract 120  120 patches from some of the images used for training the SuperPoinlike network, resulting in 11010 patches". Is this DB available? If not how can I create it? I am only interested in training EllipSegNet.

    imlist = sorted(glob.glob(os.path.join(root,'patch120','*')))
    masklist = sorted(glob.glob(os.path.join(root,'mask120','*')))
    
    

    Thanks.

    opened by ghost 1
Owner
Jhacson Meza
Computer vision and 3D reconstruction enthusiast. Master student. Mechatronic engineer.
Jhacson Meza
Python Inference Script is a Python package that enables developers to author machine learning workflows in Python and deploy without Python.

Python Inference Script(PyIS) Python Inference Script is a Python package that enables developers to author machine learning workflows in Python and d

Microsoft 10 Feb 23, 2022
Code and Data for our CVPR 2021 paper "Structured Scene Memory for Vision-Language Navigation"

SSM-VLN Code and Data for our CVPR 2021 paper "Structured Scene Memory for Vision-Language Navigation". Environment Installation Download Room-to-Room

hanqing 32 Jun 24, 2022
Official PyTorch Code of GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection (CVPR 2021)

GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Monocular 3D Object Detection GrooMeD-NMS: Grouped Mathematically Differentiable NMS for Mo

Abhinav Kumar 73 Jun 22, 2022
[CVPR 2021] NormalFusion: Real-Time Acquisition of Surface Normals for High-Resolution RGB-D Scanning

NormalFusion: Real-Time Acquisition of Surface Normals for High-Resolution RGB-D Scanning Project Page | Paper | Supplemental material #1 | Supplement

KAIST VCLAB 47 Jun 30, 2022
DeepI2P - Image-to-Point Cloud Registration via Deep Classification. CVPR 2021

#DeepI2P: Image-to-Point Cloud Registration via Deep Classification Summary Video PyTorch implementation for our CVPR 2021 paper DeepI2P. DeepI2P solv

Li Jiaxin 121 Jun 14, 2022
The repository contains our dataset and C++ implementation of the CVPR 2022 paper, Geometric Structure Preserving Warp for Natural Image Stitching.

Geometric Structure Preserving Warp for Natural Image Stitching This repository contains our dataset and C++ implementation of the CVPR 2022 paper, Ge

null 8 Jun 23, 2022
The 2nd solution (68.1 mIoU) for ICCV workshop challenge SensarUrban

SensatUrban 2nd Solution This repository is for the runner up solution for the ICCV 2021 workshop challenge SensatUrban. Getting Started Requirements

Benny 16 Jun 30, 2022
HybridPose: 6D Object Pose Estimation under Hybrid Representation (CVPR 2020)

HybridPose: 6D Object Pose Estimation under Hybrid Representations This repository contains authors' implementation of HybridPose: 6D Object Pose Esti

SONG, Chen 346 Jun 10, 2022
2021/3/30 ~ 2021/7/12 に行われる企画「競プロ典型 90 問」の問題・解説・ソースコードなどの資料をアップロードしています。

競プロ典型 90 問 日曜を除く毎朝 7:40 に競プロやアルゴリズムの教育的な問題を Twitter(@e869120)に投稿する企画です。 本企画は、2021 年 3 月 30 日から 7 月 12 日まで行われる予定です。 企画の目的 「競プロ典型 90 問」は、競プロ初級者から中上級者(レー

Masataka Yoneda 662 Jun 25, 2022
Instagram's experimental performance oriented greenfield implementation of Python.

Welcome to Skybison! Skybison is experimental performance-oriented greenfield implementation of Python 3.8. It contains a number of performance optimi

Meta Experimental 275 Jun 13, 2022
C++ Implementation of "An Equivariant Filter for Visual Inertial Odometry", ICRA 2021

EqF VIO (Equivariant Filter for Visual Inertial Odometry) This repository contains an implementation of an Equivariant Filter (EqF) for Visual Inertia

null 50 Jun 15, 2022
Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

eXtreme Gradient Boosting Community | Documentation | Resources | Contributors | Release Notes XGBoost is an optimized distributed gradient boosting l

Distributed (Deep) Machine Learning Community 22.9k Jun 24, 2022
High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

What is xLearn? xLearn is a high performance, easy-to-use, and scalable machine learning package that contains linear model (LR), factorization machin

Chao Ma 3k Jun 24, 2022
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

Website | Documentation | Tutorials | Installation | Release Notes CatBoost is a machine learning method based on gradient boosting over decision tree

CatBoost 6.6k Jul 1, 2022
In-situ data analyses and machine learning with OpenFOAM and Python

PythonFOAM: In-situ data analyses with OpenFOAM and Python Using Python modules for in-situ data analytics with OpenFOAM 8. NOTE that this is NOT PyFO

Argonne Leadership Computing Facility - ALCF 99 May 11, 2022
Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

Apache MXNet (incubating) for Deep Learning Apache MXNet is a deep learning framework designed for both efficiency and flexibility. It allows you to m

The Apache Software Foundation 20k Jul 3, 2022
Tensors and Dynamic neural networks in Python with strong GPU acceleration

PyTorch is a Python package that provides two high-level features: Tensor computation (like NumPy) with strong GPU acceleration Deep neural networks b

null 56.9k Jun 23, 2022