PointPillars MultiHead 40FPS - A REAL-TIME 3D detection network [Pointpillars] compiled by CUDA/TensorRT/C++.

Overview

English | 简体中文

PointPillars

High performance version of 3D object detection network -PointPillars, which can achieve the real-time processing (less than 1 ms / head)

  1. The inference part of PointPillars(pfe , backbone(multihead)) is optimized by tensorrt
  2. The pre- and post- processing are optimized by CUDA / C + recode.

Major Advance

Requirements (My Environment)

For *.onnx and *.trt engine file

  • Linux Ubuntu 18.04
  • OpenPCdet
  • ONNX IR version: 0.0.6
  • onnx2trt

For algorithm:

  • Linux Ubuntu 18.04
  • CMake 3.17
  • CUDA 10.2
  • TensorRT 7.1.3
  • yaml-cpp
  • google-test (not necessary)

For visualization

Usage

  1. clone thest two repositories, and make sure the dependences is complete

    mkdir workspace && cd workspace
    git clone https://github.com/hova88/PointPillars_MultiHead_40FPS.git --recursive && cd ..
    git clone https://github.com/hova88/OpenPCDet.git 
  2. generate engine file

    • 1.1 Pytorch model --> ONNX model : The specific conversion tutorial, i have put in the change log of hova88/OpenPCdet.

    • 1.2 ONNX model --> TensorRT model : after install the onnx2trt, things become very simple. Note that if you want to further improve the the inference speed, you must use half precision or mixed precision(like ,-d 16)

          onnx2trt cbgs_pp_multihead_pfe.onnx -o cbgs_pp_multihead_pfe.trt -b 1 -d 16 
          onnx2trt cbgs_pp_multihead_backbone.onnx -o cbgs_pp_multihead_backbone.trt -b 1 -d 16 
    • 1.3 engine file --> algorithm : Specified the path of engine files(*.onnx , *.trt) inbootstrap.yaml.

    • 1.4 Download the test pointcloud nuscenes_10sweeps_points.txt, and specified the path in bootstrap.yaml.

  3. Compiler

    cd PointPillars_MultiHead_40FPS
    mkdir build && cd build
    cmake .. && make -j8 && ./test/test_model
  4. Visualization

    cd PointPillars_MultiHead_40FPS/tools
    python viewer.py

Left figure shows the results of this repo, Right figure shows the official result of mmlab/OpenPCdet.

fig_method

Result

Use *.trt engine file on NVIDIA GeForce RTX 3080 Ti

with the ScoreThreshold = 0.1

 | ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄> 
 | ../model/cbgs_pp_multihead_pfe.trt >
 |_____________________> 
             (\__/) ||                 
             (•ㅅ•) ||                 
             /   づ                                                         
                                                                  
 | ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄> 
 | ../model/cbgs_pp_multihead_backbone.trt >
 |_____________________> 
             (\__/) ||                 ****
             (•ㅅ•) ||                 
             /   づ     
                                                                  
------------------------------------
Module        Time        
------------------------------------
Preprocess    0.571069 ms
Pfe           3.26203  ms
Scatter       0.384075 ms
Backbone      2.92882  ms
Postprocess   8.82032  ms
Summary       15.9707  ms
------------------------------------

with the ScoreThreshold = 0.4

 | ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄> 
 | ../model/cbgs_pp_multihead_pfe.trt >
 |_____________________> 
             (\__/) ||                 
             (•ㅅ•) ||                 
             /   づ                                                         
                                                                  
 | ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄> 
 | ../model/cbgs_pp_multihead_backbone.trt >
 |_____________________> 
             (\__/) ||                 ****
             (•ㅅ•) ||                 
             /   づ     
                                                                  
------------------------------------
Module        Time        
------------------------------------
Preprocess    0.337111 ms
Pfe           2.81834  ms
Scatter       0.161953 ms
Backbone      3.64112  ms
Postprocess   4.34731  ms
Summary       11.3101  ms
------------------------------------

Runtime logs

License

GNU General Public License v3.0 or later See COPYING to see the full text.

Issues
  • Cmake Error

    Cmake Error

    您好,我在执行 cmake .. 时,产生以下错误

    Building tests
    -- 
    
    -- BUILD SUMMARY
    --   CMAKE_GENERATOR      : Unix Makefiles
    --   Compiler ID          : GNU
    --   Build type           : Release
    --   Build shared libs    : ON
    --   Use double for kernel: OFF
    
    --   Build tests          : ON
    -- 
    
    -- Configuring done
    CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
    Please set them or make sure they are set and tested correctly in the CMake files:
    NVINFER
        linked by target "pointpillars" in directory /home/cavata/project/OpenPCDet/PointPillars_MultiHead_40FPS/pointpillars
        linked by target "test_model" in directory /home/cavata/project/OpenPCDet/PointPillars_MultiHead_40FPS/test
    
    -- Generating done
    CMake Generate step failed.  Build files cannot be regenerated correctly.
    

    请问您是否有相应的解决方案,十分感谢!

    opened by cagik 8
  • how to use this work on our own dataset

    how to use this work on our own dataset

    Our own dataset has been trained on openpcdet and I got the engine files. But I have trouble to understand the structure of this work. I will appreciate you if you could write a tutorial about how to run this work on our own dataset and then we could continue our research and cite your work on pointpillars. Thanks for your useful work!

    opened by FreemanGong 5
  • my data is bin file,and i have converted bin file to txt file;I have compiled program succesfully with TensorRT, but the test failed

    my data is bin file,and i have converted bin file to txt file;I have compiled program succesfully with TensorRT, but the test failed

    [==========] Running 1 test from 1 test suite. [----------] Global test environment set-up. [----------] 1 test from PointPillars [ RUN ] PointPillars.build_model ./models/custom_data_multihead_backbone.trt unknown file: Failure C++ exception with description "bad file: ./pointpillars/cfgs/pointpillar_sanyi_multi.yaml" thrown in the test body. [ FAILED ] PointPillars.build_model (1 ms) [----------] 1 test from PointPillars (1 ms total)

    [----------] Global test environment tear-down [==========] 1 test from 1 test suite ran. (1 ms total) [ PASSED ] 0 tests. [ FAILED ] 1 test, listed below: [ FAILED ] PointPillars.build_model

    1 FAILED TEST

    opened by qiuhui1991 5
  • ONNX model --> TensorRT model error

    ONNX model --> TensorRT model error

    hi, thanks for nice work. I got some error when I run onnx2trt cbgs_pp_multihead_backbone.onnx -o cbgs_pp_multihead_backbone.trt -b 1 -d 16

    (base) ➜  default git:(main) onnx2trt cbgs_pp_multihead_backbone.onnx -o cbgs_pp_multihead_backbone.trt -b 1 -d 16
    ----------------------------------------------------------------
    Input filename:   cbgs_pp_multihead_backbone.onnx
    ONNX IR version:  0.0.6
    Opset version:    10
    Producer name:    pytorch
    Producer version: 1.7
    Domain:
    Model version:    0
    Doc string:
    ----------------------------------------------------------------
    terminate called after throwing an instance of 'std::runtime_error'
      what():  Failed to create object
    [1]    19112 abort (core dumped)  onnx2trt cbgs_pp_multihead_backbone.onnx -o cbgs_pp_multihead_backbone.trt -b 1 -d
    

    my onnx2trt version

    (base) ➜  default git:(main) onnx2trt -V                                                                     
    Parser built against:
      ONNX IR version:  0.0.6
      TensorRT version: 7.1.3
    

    and I found some issue about this. But I'm not found solution. any suggestion about this? Crash when model with cast #406 run onnx2trt ERROR FAILED_ALLOCATION: std::bad_alloc #549

    opened by chasingw 5
  • If the lidar coordinate system changes, do I need to change the pre-processing code?

    If the lidar coordinate system changes, do I need to change the pre-processing code?

    the lidar coordinate system of nuscenes is different from that of kitti. If my point cloud coordinate system changes, does the pre-processing code need to be modified ?

    I also used openPCdet for training, because the number of categories changed, I modified some parameters of post-processing, but the output of the model was all wrong boxes. May I ask what needs to be modified in the pre-processed code?

    opened by rolan123 3
  • How to train my own dataset(with three categories),thanks!

    How to train my own dataset(with three categories),thanks!

    I also encounter some problems when try to utilize this project on my own datasets.The format of my datasets is the way Openpcdet needs and I have transfered it to kitti format.It is trained well on 3 categories(car,pedestrain,cyclist) just with little changes on "cfgs/kitti_models/pointpillar.yaml".But when I try to apply my own datasets with cbgs_pp_multihead.yaml then something goes wrong.I think the problem is the different between nuscences and kitti and the change of the number of categories. Can you give me some hint of how to change cbgs_pp_multihead.yaml to train my datasets(3 categories).Thanks!

    opened by HaoGe30 3
  • Cmake error

    Cmake error

    在cmake时候报错

    CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
    Please set them or make sure they are set and tested correctly in the CMake files:
    NVINFER
    
    opened by ycdhqzhiai 3
  • my project have four 'HEAD_CLS_NAME',and how to modify the code to run success in program,expecially the postprocess.cu;

    my project have four 'HEAD_CLS_NAME',and how to modify the code to run success in program,expecially the postprocess.cu;

    RPN_HEAD_CFGS: [ { 'HEAD_CLS_NAME': [ 'vehicle' ], }, { 'HEAD_CLS_NAME': [ 'car' ], }, { 'HEAD_CLS_NAME': [ 'pedestrian'], }, { 'HEAD_CLS_NAME': [ 'cyclist' ], }, ]

    opened by qiuhui1991 2
  • the inference in my computer is very slow, can you give me some suggestions?

    the inference in my computer is very slow, can you give me some suggestions?


    Module Time

    Preprocess 0.440277 ms Pfe 13.8402 ms Scatter 0.11535 ms Backbone 348.759 ms Postprocess 40.2281 ms Summary 403.392 ms

    num_objects----------------------------

    opened by qiuhui1991 2
  • kitti数据集格式mutihead检测结果错误

    kitti数据集格式mutihead检测结果错误

    我的模型是用kitti训练的,我将kitti配置文件改成muiltihead了,点云维度是4维,改了前处理和后处理部分参数,将num_gather_feature = 11改成10,kNumPointFeature = 5改成了4,类别是两种类别, dev_pfe_gather_feature_把第五行删掉了,最终预测结果全是错的或者为空。是点云维度原因吗?

    [==========] Running 1 test from 1 test suite. [----------] Global test environment set-up. [----------] 1 test from PointPillars [ RUN ] PointPillars.build_model ../model/robo_rpn/kitti_rpn_multihead_backbone.trt | ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄> | ../model/robo_rpn/kitti_rpn_multihead_pfe.trt > |_____________________> (_/) ||
    (•ㅅ•) ||
    /   づ
    WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.6.1 but loaded cuBLAS/cuBLAS LT 11.2.1 | ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄ ̄> | ../model/robo_rpn/kitti_rpn_multihead_backbone.trt > |_____________________> (_
    /) ||
    (•ㅅ•) ||
    /   づ
    WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.6.1 but loaded cuBLAS/cuBLAS LT 11.2.1 WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.6.1 but loaded cuBLAS/cuBLAS LT 11.2.1 WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.6.1 but loaded cuBLAS/cuBLAS LT 11.2.1 4 true1true------------------------------------ Module Time

    Preprocess 0.328729 ms Pfe 1.92755 ms Scatter 0.27791 ms Backbone 3.28076 ms Postprocess 1.63506 ms Summary 7.45741 ms

    4 true1true------------------------------------ Module Time

    Preprocess 0.326377 ms Pfe 1.91881 ms Scatter 0.03204 ms Backbone 3.17447 ms Postprocess 1.62952 ms Summary 7.08602 ms

    4 true1true------------------------------------ Module Time

    Preprocess 0.343313 ms Pfe 1.92314 ms Scatter 0.91915 ms Backbone 3.18731 ms Postprocess 1.60187 ms Summary 7.98153 ms

    4 true1true------------------------------------ Module Time

    Preprocess 0.328076 ms Pfe 1.91939 ms Scatter 0.032131 ms Backbone 3.17698 ms Postprocess 1.65143 ms Summary 7.11405 ms

    4 true1true------------------------------------ Module Time

    Preprocess 0.321695 ms Pfe 1.91993 ms Scatter 0.274737 ms Backbone 3.17974 ms Postprocess 1.6171 ms Summary 7.31876 ms

    4 true1true------------------------------------ Module Time

    Preprocess 0.31875 ms Pfe 1.9192 ms Scatter 0.031389 ms Backbone 3.1751 ms Postprocess 1.58972 ms Summary 7.03859 ms

    4 true1true------------------------------------ Module Time

    Preprocess 0.325528 ms Pfe 1.92111 ms Scatter 0.262098 ms Backbone 3.17563 ms Postprocess 1.6107 ms Summary 7.30026 ms

    4 true1true------------------------------------ Module Time

    Preprocess 0.329693 ms Pfe 1.91792 ms Scatter 0.030285 ms Backbone 3.17318 ms Postprocess 1.67297 ms Summary 7.1286 ms

    4 true1true------------------------------------ Module Time

    Preprocess 0.319577 ms Pfe 1.92314 ms Scatter 0.276512 ms Backbone 3.17424 ms Postprocess 1.60376 ms Summary 7.30224 ms

    4 true1true------------------------------------ Module Time

    Preprocess 0.321262 ms Pfe 1.91856 ms Scatter 0.031172 ms Backbone 3.16954 ms Postprocess 1.62139 ms Summary 7.06617 ms

    [ OK ] PointPillars.build_model (1574 ms) [----------] 1 test from PointPillars (1574 ms total)

    [----------] Global test environment tear-down [==========] 1 test from 1 test suite ran. (1574 ms total) [ PASSED ] 1 test.

    opened by elegentbamboo 1
  • unknown file: Failure C++ exception with description

    unknown file: Failure C++ exception with description "bad file: ../bootstrap.yaml" thrown in the test body.

    runtime log: [==========] Running 1 test from 1 test suite. [----------] Global test environment set-up. [----------] 1 test from PointPillars [ RUN ] PointPillars.build_model unknown file: Failure C++ exception with description "bad file: ../bootstrap.yaml" thrown in the test body. [ FAILED ] PointPillars.build_model (0 ms) [----------] 1 test from PointPillars (0 ms total)

    [----------] Global test environment tear-down [==========] 1 test from 1 test suite ran. (0 ms total) [ PASSED ] 0 tests. [ FAILED ] 1 test, listed below: [ FAILED ] PointPillars.build_model

    1 FAILED TEST

    opened by Leon-LiuY 1
  • error when doing cmake

    error when doing cmake

    Hi, there is an error occurred when I do cmake Error is as shown below.

    ============================================================================================ Building tests

    -- BUILD SUMMARY -- CMAKE_GENERATOR : Unix Makefiles -- Compiler ID : GNU -- Build type : Release -- Build shared libs : ON -- Use double for kernel: OFF -- Build tests : ON

    -- Configuring done CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: NVINFER linked by target "pointpillars" in directory /home/civeh/workspace/PointPillars_MultiHead_40FPS/pointpillars linked by target "test_model" in directory /home/civeh/workspace/PointPillars_MultiHead_40FPS/test NVONNXPARSERS linked by target "pointpillars" in directory /home/civeh/workspace/PointPillars_MultiHead_40FPS/pointpillars linked by target "test_model" in directory /home/civeh/workspace/PointPillars_MultiHead_40FPS/test

    -- Generating done CMake Generate step failed. Build files cannot be regenerated correctly.

    Please help

    opened by minho8849 1
  • about the performance in xavier

    about the performance in xavier

    I used the provided onnx and run it in xavier in MAXN mode. Running onnx model, takes about 130ms. Running trt model, takes about 130ms. I'm sure it had used GPU.

    这个速度 正常吗?

    opened by ideafold 4
  • Wrong result after the first time

    Wrong result after the first time

    Thank you for your great source code.. When I run the example more than one time, I received very random results. Can you explain the reason and give me some ideas to fix this bug?

    opened by vietanhdev 13
Owner
Yan haixu
You will ride eternal, shiny and chrome.
Yan haixu
Anomaly Detection on Dynamic (time-evolving) Graphs in Real-time and Streaming manner

Anomaly Detection on Dynamic (time-evolving) Graphs in Real-time and Streaming manner. Detecting intrusions (DoS and DDoS attacks), frauds, fake rating anomalies.

Stream-AD 681 Aug 3, 2022
Hardware-accelerated DNN model inference ROS2 packages using NVIDIA Triton/TensorRT for both Jetson and x86_64 with CUDA-capable GPU.

Isaac ROS DNN Inference Overview This repository provides two NVIDIA GPU-accelerated ROS2 nodes that perform deep learning inference using custom mode

NVIDIA Isaac ROS 42 Jul 18, 2022
Real time monaural source separation base on fully convolutional neural network operates on Time-frequency domain.

Real time monaural source separation base on fully convolutional neural network operates on Time-frequency domain.

James Fung 100 Aug 5, 2022
OpenVINO™ optimization for PointPillars*

OpenVINO™ optimization for PointPillars* There are 2 demonstrations in the repo. Demo of PointPillars Optimization - It demonstrates how to implement

Intel Corporation 11 Jul 27, 2022
(ROS) YOLO detection with TensorRT, utilizing tkDNN

tkDNN-ROS YOLO object detection with ROS and TensorRT using tkDNN Currently, only YOLO is supported. Comparison of performance and other YOLO implemen

EungChang-Mason-Lee 4 Jul 19, 2022
CUDA-accelerated Apriltag detection and pose estimation.

Isaac ROS Apriltag Overview This ROS2 node uses the NVIDIA GPU-accelerated AprilTags library to detect AprilTags in images and publishes their poses,

NVIDIA Isaac ROS 40 Aug 3, 2022
OpenPose: Real-time multi-person keypoint detection library for body, face, hands, and foot estimation

Build Type Linux MacOS Windows Build Status OpenPose has represented the first real-time multi-person system to jointly detect human body, hand, facia

null 24.7k Aug 13, 2022
An R3D network implemented with TensorRT

r3d_TensorRT An r3d network implemented with TensorRT8.x, The weight of the model comes from PyTorch. A description of the models in Pytroch can be fo

null 2 Nov 7, 2021
A GPU (CUDA) based Artificial Neural Network library

Updates - 05/10/2017: Added a new example The program "image_generator" is located in the "/src/examples" subdirectory and was submitted by Ben Bogart

Daniel Frenzel 91 Jun 13, 2022
International Business Machines 9 Jul 21, 2022
Port of the 2020 support library to Raspberry Pi for the VL53L3CX Time-of-Flight ranging sensor with advanced multi-object detection

Port of ST VL53L3CX (2020) driver library to Raspberry Pi This is a port of the support library to Raspberry Pi for the VL53L3CX Time-of-Flight rangin

Niall Douglas 4 Jul 27, 2022
NCNN implementation of Real-ESRGAN. Real-ESRGAN aims at developing Practical Algorithms for General Image Restoration.

Real-ESRGAN ncnn Vulkan This project is the ncnn implementation of Real-ESRGAN. Real-ESRGAN ncnn Vulkan heavily borrows from realsr-ncnn-vulkan. Many

Xintao 260 Aug 9, 2022
Implement yolov5 with Tensorrt C++ api, and integrate batchedNMSPlugin. A Python wrapper is also provided.

yolov5 Original codes from tensorrtx. I modified the yololayer and integrated batchedNMSPlugin. A yolov5s.wts is provided for fast demo. How to genera

weiwei zhou 41 Jul 6, 2022
TensorRT int8 量化部署 yolov5s 4.0 模型,实测3.3ms一帧!

tensorrt模型推理 git clone https://github.com/Wulingtian/yolov5_tensorrt_int8.git(求star) cd yolov5_tensorrt_int8 vim CMakeLists.txt 修改USER_DIR参数为自己的用户根目录

null 97 Jul 30, 2022
TensorRT implementation of RepVGG models from RepVGG: Making VGG-style ConvNets Great Again

RepVGG RepVGG models from "RepVGG: Making VGG-style ConvNets Great Again" https://arxiv.org/pdf/2101.03697.pdf For the Pytorch implementation, you can

weiwei zhou 67 Aug 4, 2022
Deep Learning API and Server in C++11 support for Caffe, Caffe2, PyTorch,TensorRT, Dlib, NCNN, Tensorflow, XGBoost and TSNE

Open Source Deep Learning Server & API DeepDetect (https://www.deepdetect.com/) is a machine learning API and server written in C++11. It makes state

JoliBrain 2.4k Aug 2, 2022
Simple samples for TensorRT programming

Introduction This is a collection of simplified TensorRT samples to get you started with TensorRT programming. Most of the samples are written in C++,

NVIDIA Corporation 479 Aug 10, 2022
Support Yolov4/Yolov3/Centernet/Classify/Unet. use darknet/libtorch/pytorch to onnx to tensorrt

ONNX-TensorRT Yolov4/Yolov3/CenterNet/Classify/Unet Implementation Yolov4/Yolov3 centernet INTRODUCTION you have the trained model file from the darkn

null 162 Jul 24, 2022