Support Yolov4/Yolov3/Centernet/Classify/Unet. use darknet/libtorch/pytorch to onnx to tensorrt

Overview

ONNX-TensorRT

Yolov4/Yolov3/CenterNet/Classify/Unet Implementation

Yolov4/Yolov3

centernet

INTRODUCTION

you have the trained model file from the darknet/libtorch/pytorch

  • yolov4 , yolov4-tiny
  • yolov3 , yolov3-tiny
  • centernet
  • classify(mnist\alexnet\resnet18\resnet34\resnet50\shufflenet_v2\mobilenet_v2)

Features

  • inequal net width and height

  • batch inference


    onnx-tensorrt batch inference : onnx re-export(batch:2)

  • support FP32,FP16,INT8

  • dynamic input size(tiny_tensorrt_dyn_onnx)

windows10

  • dependency : spdlog,onnx,onnx-tensorrt,protobuf-3.11.4,TensorRT 7.2.2.3 , cuda 11.1 , cudnn 8.0 , opencv3.4, vs2019

  • build:

    open MSVC tiny_tensorrt_onnx.sln file

    tiny_tensorrt_dyn_onnx:dynamic shape

    tiny_tensorrt_onnx: normal

  • build onnx-tensorrt

    step1: https://github.com/onnx/onnx-tensorrt.git

    step2: https://drive.google.com/drive/folders/1DndiqyCZ796p3-xXI3O4AMCIGcUWQ1q2?usp=sharing

    step3: builtin_op_importers.cpp replace onnx-tensorrt\builtin_op_importers.cpp

    step4: tortoiseGit->apply patch serial and choose 0001-Compile-onnx-tensorrt-by-MSVC-on-Windows.patch

    step5:build onnx.lib\onnx_proto.lib\nvonnxparser.dll\nvonnxparser_static.lib

Model and 3rdparty

model : https://drive.google.com/drive/folders/1KzBjmCOG9ghcq9L6-iqfz6QwBQq6Hl4_?usp=sharing

3rdparty:https://drive.google.com/drive/folders/1SddUgQ5kGlv6dDGPqnVWZxgCoBY85rM2?usp=sharing

API

struct Config
{
    std::string cfgFile = "configs/yolov3.cfg";

    std::string onnxModelpath = "configs/yolov3.onnx";

    std::string engineFile = "configs/yolov3.engine";

    std::string calibration_image_list_file = "configs/images/";

    std::vector<std::string> customOutput;

    int calibration_width = 0;

    int calibration_height = 0;
    
    int maxBatchSize = 1;

    int mode; //0,1,2

    //std::string calibration_image_list_file_txt = "configs/calibration_images.txt";
};

class YoloDectector
{
void init(Config config);
void detect(const std::vector<cv::Mat>& vec_image,
	std::vector<BatchResult>& vec_batch_result);
}

REFERENCE

https://github.com/onnx/onnx-tensorrt.git

https://github.com/NVIDIA/TensorRT/tree/master/samples/opensource/sampleDynamicReshape

https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps

https://github.com/enazoe/yolo-tensorrt.git

https://github.com/zerollzeng/tiny-tensorrt.git

Contact

Issues
  • how to run?

    how to run?

    my windows eviroment is: cuda 11.0 vs2019 how to run? when i run, it tip me: can not find the nvrtc64_111_0.dll

    which third_model should i to recompile them?

    opened by xinsuinizhuan 6
  • yolo5生成失败:

    yolo5生成失败:

    reading calib cache: E:\comm_Item\Item_done\onnx_tensorrt_pro\onnx_tensorrt_project-main\model\pytorch_onnx_tensorrt_yolov5\yolov5s.table TensorRT was linked against cuDNN 8.1.0 but loaded cuDNN 8.0.2 Detected 1 inputs and 7 output network tensors. TensorRT was linked against cuDNN 8.1.0 but loaded cuDNN 8.0.2 TensorRT was linked against cuDNN 8.1.0 but loaded cuDNN 8.0.2 Starting Calibration. dog.jpg 0 Calibrated batch 0 in 1.35348 seconds. person.jpg 1 Calibrated batch 1 in 1.3612 seconds. Post Processing Calibration data in 0.0005131 seconds. Calibration completed in 20.8498 seconds. reading calib cache: E:\comm_Item\Item_done\onnx_tensorrt_pro\onnx_tensorrt_project-main\model\pytorch_onnx_tensorrt_yolov5\yolov5s.table Writing Calibration Cache for calibrator: TRT-7203-MinMaxCalibration writing calib cache: E:\comm_Item\Item_done\onnx_tensorrt_pro\onnx_tensorrt_project-main\model\pytorch_onnx_tensorrt_yolov5\yolov5s.table size: 4711 TensorRT was linked against cuDNN 8.1.0 but loaded cuDNN 8.0.2 C:\source\rtSafe\cuda\cudaConvolutionRunner.cpp (483) - Cudnn Error in nvinfer1::rt::cuda::CudnnConvolutionRunner::executeConv: 8 (CUDNN_STATUS_EXECUTION_FAILED) C:\source\rtSafe\cuda\cudaConvolutionRunner.cpp (483) - Cudnn Error in nvinfer1::rt::cuda::CudnnConvolutionRunner::executeConv: 8 (CUDNN_STATUS_EXECUTION_FAILED) [2021-08-08 11:28:54.729] [info] serialize engine to E:\comm_Item\Item_done\onnx_tensorrt_pro\onnx_tensorrt_project-main\model\pytorch_onnx_tensorrt_yolov5\yolov5s_fp32_batch_1.engine [2021-08-08 11:28:54.730] [error] engine is empty, save engine failed [2021-08-08 11:28:54.731] [info] create execute context and malloc device memory... [2021-08-08 11:28:54.731] [info] init engine...

    opened by xinsuinizhuan 1
  • UNet training

    UNet training

    Hi,

    I trained a model with the public dataset but the result is strange. Could you please some tips for training.

    Thanks.

    I have one class and I set the classes param to 2

    unet_model = Unet(encoder_name="resnet50", encoder_weights="imagenet", decoder_channels=(256, 128, 64, 32, 16),
                      in_channels=3, classes=2)
    
    --width: 512
    --height: 512
    --epoch: 30
    --batchsize: 2
    

    dataset sample: 900 images ISIC_0000000 ISIC_0000000_Segmentation

    Result: image

    opened by sctrueew 0
  • yolov5-v5的yolov5x模型,在python版本测试结果和该项目tensorrt下跑的结果不一致问题?

    yolov5-v5的yolov5x模型,在python版本测试结果和该项目tensorrt下跑的结果不一致问题?

    如题,不知道作者是否遇到过,训练完成的yolov5x模型,在python版本正确率为98%,但是转换为tensorrt后经过测试,正确率只有90%左右,其中模型转换过程log如下: [09/27/2021-11:27:04] [I] Host Latency [09/27/2021-11:27:04] [I] min: 11.3848 ms (end to end 21.3677 ms) [09/27/2021-11:27:04] [I] max: 13.1256 ms (end to end 24.1753 ms) [09/27/2021-11:27:04] [I] mean: 11.67 ms (end to end 21.9034 ms) [09/27/2021-11:27:04] [I] median: 11.5836 ms (end to end 21.7285 ms) [09/27/2021-11:27:04] [I] percentile: 12.5283 ms at 99% (end to end 23.6667 ms at 99%) [09/27/2021-11:27:04] [I] throughput: 0 qps [09/27/2021-11:27:04] [I] walltime: 3.03151 s [09/27/2021-11:27:04] [I] Enqueue Time [09/27/2021-11:27:04] [I] min: 1.04535 ms [09/27/2021-11:27:04] [I] max: 4.6637 ms [09/27/2021-11:27:04] [I] median: 1.61969 ms [09/27/2021-11:27:04] [I] GPU Compute [09/27/2021-11:27:04] [I] min: 10.8311 ms [09/27/2021-11:27:04] [I] max: 12.5458 ms [09/27/2021-11:27:04] [I] mean: 11.0955 ms [09/27/2021-11:27:04] [I] median: 11.0142 ms [09/27/2021-11:27:04] [I] percentile: 11.9821 ms at 99% [09/27/2021-11:27:04] [I] total compute time: 3.01798 s &&&& PASSED TensorRT.trtexec # trtexec.exe --onnx=best.onnx --saveEngine=best.engine --fp16

    opened by zsffuture 6
  • yolov3-ocr.cfg does not have down_stride

    yolov3-ocr.cfg does not have down_stride

    Hi, I tried to transfer yolo3-spp pt file to onnx, and here is the error:

    Traceback (most recent call last): File "Libtorch_yolo_to_onnx.py", line 779, in main() File "Libtorch_yolo_to_onnx.py", line 771, in main model_def = builder.build_onnx_graph( File "Libtorch_yolo_to_onnx.py", line 353, in build_onnx_graph major_node_specs = self._make_onnx_node(layer_name, layer_dict) File "Libtorch_yolo_to_onnx.py", line 426, in _make_onnx_node node_creators[layer_type](layer_name, layer_dict) File "Libtorch_yolo_to_onnx.py", line 729, in _make_yolo_node down_stride = int(layer_dict['down_stride']) KeyError: 'down_stride'

    many thanks!

    -Scott

    opened by chenscottus 4
  • More about installation

    More about installation

    Hello,

    Thanks for the great work!!!

    Can one use ubuntu? or must it be on Windows?

    If yes, please can you provide more information for non windows users.

    Thanks once again

    opened by Source82 5
C++ trainable detection library based on libtorch (or pytorch c++). Yolov4 tiny provided now.

C++ Library with Neural Networks for Object Detection Based on LibTorch. ?? Libtorch Tutorials ?? Visit Libtorch Tutorials Project if you want to know

null 48 Aug 11, 2022
Deep Learning API and Server in C++11 support for Caffe, Caffe2, PyTorch,TensorRT, Dlib, NCNN, Tensorflow, XGBoost and TSNE

Open Source Deep Learning Server & API DeepDetect (https://www.deepdetect.com/) is a machine learning API and server written in C++11. It makes state

JoliBrain 2.4k Aug 2, 2022
YOLOv4 accelerated wtih TensorRT and multi-stream input using Deepstream

Deepstream 5.1 YOLOv4 App This Deepstream application showcases YOLOv4 running at high FPS throughput! P.S - Click the gif to watch the entire video!

Akash James 34 Jul 28, 2022
The MOT implement by Solov2+DeepSORT with C++ (Libtorch, TensorRT).

Tracking-Solov2-Deepsort This project implement the Multi-Object-Tracking(MOT) base on SOLOv2 and DeepSORT with C++。 The instance segmentation model S

ChenJianqu 32 Aug 12, 2022
The optical flow algorithm RAFT implemented with C++(Libtorch+TensorRT)

RAFT_CPP Attention/注意 There are some bug here,output the wrong result 代码存在bug,估计出来的光流值不准确,解决中 Quick Start 0.Export RAFT onnx model 首先加载训练完成的模型权重: pars

ChenJianqu 15 May 20, 2022
A c++ trainable semantic segmentation library based on libtorch (pytorch c++). Backbone: ResNet, ResNext. Architecture: FPN, U-Net, PAN, LinkNet, PSPNet, DeepLab-V3, DeepLab-V3+ by now.

中文 C++ library with Neural Networks for Image Segmentation based on LibTorch. The main features of this library are: High level API (just a line to cr

null 283 Aug 10, 2022
This is a code repository for pytorch c++ (or libtorch) tutorial.

LibtorchTutorials English version 环境 win10 visual sutdio 2017 或者Qt4.11.0 Libtorch 1.7 Opencv4.5 配置 libtorch+Visual Studio和libtorch+QT分别记录libtorch在VS和Q

null 358 Aug 3, 2022
LibtorchSegmentation - A c++ trainable semantic segmentation library based on libtorch (pytorch c++). Backbone: VGG, ResNet, ResNext. Architecture: FPN, U-Net, PAN, LinkNet, PSPNet, DeepLab-V3, DeepLab-V3+ by now.

English | 中文 C++ library with Neural Networks for Image Segmentation based on LibTorch. ⭐ Please give a star if this project helps you. ⭐ The main fea

null 281 Aug 6, 2022
Marking up images for use with Darknet.

What is DarkMark? DarkMark is a C++ GUI tool used to annotate images for use in neural networks. It was written specifically to be used with the Darkn

Stéphane Charette 77 Jul 22, 2022
STM32+OV2640+ESP8266+YOLOv3

qrDecoderOnSTM32 关键技术:STM32+OV2640+ESP8266+YOLOv3 + 卷积神经网络 + 多目标检测定位 项目背景及简述 1.1项目背景 当前人工智能技术正逐步融入传统行业,同时“互联网+”带来的线上销售日益挤压线下实体店面。实体店面,尤其是大型购物超市不得不融入“互

Wenjun 8 Jul 26, 2022
This repository is a tensorrt deployment of the onsets and frames model, which is implemented using pytorch.

Onsets and Frames TensorRT inference This repository is a tensorrt deployment of the onsets and frames model, which is implemented using pytorch (http

Xianke Wang 6 Jan 13, 2022
A lightweight, portable pure C99 onnx inference engine for embedded devices with hardware acceleration support.

Libonnx A lightweight, portable pure C99 onnx inference engine for embedded devices with hardware acceleration support. Getting Started The library's

xboot.org 411 Aug 8, 2022
License plate parsing using Darknet and YOLO

DarkPlate Note that DarkPlate by itself is not a complete software project. The intended purpose was to create a simple project showing how to use Dar

Stéphane Charette 27 Apr 11, 2022
DarkHelp - C++ wrapper library for Darknet

What is the DarkHelp C++ API? The DarkHelp C++ API is a wrapper to make it easier to use the Darknet neural network framework within a C++ application

Stéphane Charette 73 Jul 22, 2022
Pure C ONNX runtime with zero dependancies for embedded devices

?? cONNXr C ONNX Runtime A onnx runtime written in pure C99 with zero dependencies focused on embedded devices. Run inference on your machine learning

Alvaro 130 Aug 5, 2022
ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

ONNX Runtime is a cross-platform inference and training machine-learning accelerator compatible with deep learning frameworks, PyTorch and TensorFlow/Keras, as well as classical machine learning libraries such as scikit-learn, and more.

Microsoft 7.3k Aug 6, 2022
yolov5 onnx caffe

环境配置 ubuntu:18.04 cuda:10.0 cudnn:7.6.5 caffe: 1.0 OpenCV:3.4.2 Anaconda3:5.2.0 相关的安装包我已经放到百度云盘,可以从如下链接下载: https://pan.baidu.com/s/17bjiU4H5O36psGrHlF

null 53 Aug 10, 2022
Examples for using ONNX Runtime for machine learning inferencing.

Examples for using ONNX Runtime for machine learning inferencing.

Microsoft 269 Aug 4, 2022
YOLO v5 ONNX Runtime C++ inference code.

yolov5-onnxruntime C++ YOLO v5 ONNX Runtime inference code for object detection. Dependecies: OpenCV 4.5+ ONNXRuntime 1.7+ OS: Windows 10 or Ubuntu 20

null 72 Aug 4, 2022