An Out-of-the-Box TensorRT-based Framework for High Performance Inference with C++/Python Support

手写AI

Last update: Jan 5, 2023

Overview

Read this in other languages: English, 简体中文.

News:

Tutorial for exporting CenterNet from pytorch to tensorRT is released.

Tutorial Video

blibli : https://www.bilibili.com/video/BV1Xw411f7FW (Now only in Chinese. English is comming)
slides : http://zifuture.com:1556/fs/sxai/tensorRT.pptx (Now only in Chinese. English is comming)
tutorial folder: a good intro for beginner to get a general idea of our framework.(Chinese/English)

An Out-of-the-Box TensorRT-based Framework for High Performance Inference with C++/Python Support

C++ Interface: 3 lines of code is all you need to run a YoloX

commit(image).get(); // return vector
">
// create inference engine on gpu-0
//auto engine = Yolo::create_infer("yolov5m.fp32.trtmodel", Yolo::Type::V5, 0);
auto engine = Yolo::create_infer("yolox_m.fp32.trtmodel", Yolo::Type::X, 0);

// load image
auto image = cv::imread("1.jpg");

// do inference and get the result
auto box = engine->commit(image).get();  // return vector
      

Python Interface:

import trtpy

model     = models.resnet18(True).eval().to(device)
trt_model = tp.from_torch(model, input)
trt_out   = trt_model(input)

INTRO

High level interface for C++/Python.
Based on TensorRT8.0.
Simplify the implementation of custom plugin. And serialization and deserialization have been encapsulated for easier usage.
Simplify the compile of fp32, fp16 and int8 for facilitating the deployment with C++/Python in server or embeded device.
Models ready for use also with examples are RetinaFace, Scrfd, YoloV5, YoloX, Arcface, AlphaPose, CenterNet and DeepSORT(C++)

YoloX and YoloV5-series Model Test Report

app_yolo.cpp speed testing

Resolution (YoloV5P5, YoloX) = (640x640), (YoloV5P6) = (1280x1280)
max batch size = 16
preprocessing + inference + postprocessing
cuda10.2, cudnn8.2.2.26, TensorRT-8.0.1.6
RTX2080Ti
num of testing: take the average on the results of 100 times but excluding the first time for warmup
Testing log: [workspace/perf.result.std.log (workspace/perf.result.std.log)
code for testing: src/application/app_yolo.cpp
images for testing: 6 images in workspace/inference
- with resolution 810x1080，500x806，1024x684，550x676，1280x720，800x533 respetively
Testing method: load 6 images. Then do the inference on the 6 images, which will be repeated for 100 times. Note that each image should be preprocessed and postprocessed.

Model	Resolution	Type	Precision	Elapsed Time	FPS
yolox_x	640x640	YoloX	FP32	21.879	45.71
yolox_l	640x640	YoloX	FP32	12.308	81.25
yolox_m	640x640	YoloX	FP32	6.862	145.72
yolox_s	640x640	YoloX	FP32	3.088	323.81
yolox_x	640x640	YoloX	FP16	6.763	147.86
yolox_l	640x640	YoloX	FP16	3.933	254.25
yolox_m	640x640	YoloX	FP16	2.515	397.55
yolox_s	640x640	YoloX	FP16	1.362	734.48
yolox_x	640x640	YoloX	INT8	4.070	245.68
yolox_l	640x640	YoloX	INT8	2.444	409.21
yolox_m	640x640	YoloX	INT8	1.730	577.98
yolox_s	640x640	YoloX	INT8	1.060	943.15
yolov5x6	1280x1280	YoloV5_P6	FP32	68.022	14.70
yolov5l6	1280x1280	YoloV5_P6	FP32	37.931	26.36
yolov5m6	1280x1280	YoloV5_P6	FP32	20.127	49.69
yolov5s6	1280x1280	YoloV5_P6	FP32	8.715	114.75
yolov5x	640x640	YoloV5_P5	FP32	18.480	54.11
yolov5l	640x640	YoloV5_P5	FP32	10.110	98.91
yolov5m	640x640	YoloV5_P5	FP32	5.639	177.33
yolov5s	640x640	YoloV5_P5	FP32	2.578	387.92
yolov5x6	1280x1280	YoloV5_P6	FP16	20.877	47.90
yolov5l6	1280x1280	YoloV5_P6	FP16	10.960	91.24
yolov5m6	1280x1280	YoloV5_P6	FP16	7.236	138.20
yolov5s6	1280x1280	YoloV5_P6	FP16	3.851	259.68
yolov5x	640x640	YoloV5_P5	FP16	5.933	168.55
yolov5l	640x640	YoloV5_P5	FP16	3.450	289.86
yolov5m	640x640	YoloV5_P5	FP16	2.184	457.90
yolov5s	640x640	YoloV5_P5	FP16	1.307	765.10
yolov5x6	1280x1280	YoloV5_P6	INT8	12.207	81.92
yolov5l6	1280x1280	YoloV5_P6	INT8	7.221	138.49
yolov5m6	1280x1280	YoloV5_P6	INT8	5.248	190.55
yolov5s6	1280x1280	YoloV5_P6	INT8	3.149	317.54
yolov5x	640x640	YoloV5_P5	INT8	3.704	269.97
yolov5l	640x640	YoloV5_P5	INT8	2.255	443.53
yolov5m	640x640	YoloV5_P5	INT8	1.674	597.40
yolov5s	640x640	YoloV5_P5	INT8	1.143	874.91

app_yolo_fast.cpp speed testing. Never stop desiring for being faster

Highlight: 0.5 ms faster without any loss in precision compared with the above. Specifically, we remove the Focus and some transpose nodes etc, and implement them in CUDA kenerl function. But the rest remains the same.
Test log: workspace/perf.result.std.log
Code for testing: src/application/app_yolo_fast.cpp
Tips: you can do the modification while refering to the downloaded onnx. Any questions are welcomed through any kinds of contact.
Conclusion: the main idea of this work is to optimize the pre-and-post processing. If you go for yolox, yolov5 small version, the optimization might help you.

Model	Resolution	Type	Precision	Elapsed Time	FPS
yolox_x_fast	640x640	YoloX	FP32	21.598	46.30
yolox_l_fast	640x640	YoloX	FP32	12.199	81.97
yolox_m_fast	640x640	YoloX	FP32	6.819	146.65
yolox_s_fast	640x640	YoloX	FP32	2.979	335.73
yolox_x_fast	640x640	YoloX	FP16	6.764	147.84
yolox_l_fast	640x640	YoloX	FP16	3.866	258.64
yolox_m_fast	640x640	YoloX	FP16	2.386	419.16
yolox_s_fast	640x640	YoloX	FP16	1.259	794.36
yolox_x_fast	640x640	YoloX	INT8	3.918	255.26
yolox_l_fast	640x640	YoloX	INT8	2.292	436.38
yolox_m_fast	640x640	YoloX	INT8	1.589	629.49
yolox_s_fast	640x640	YoloX	INT8	0.954	1048.47
yolov5x6_fast	1280x1280	YoloV5_P6	FP32	67.075	14.91
yolov5l6_fast	1280x1280	YoloV5_P6	FP32	37.491	26.67
yolov5m6_fast	1280x1280	YoloV5_P6	FP32	19.422	51.49
yolov5s6_fast	1280x1280	YoloV5_P6	FP32	7.900	126.57
yolov5x_fast	640x640	YoloV5_P5	FP32	18.554	53.90
yolov5l_fast	640x640	YoloV5_P5	FP32	10.060	99.41
yolov5m_fast	640x640	YoloV5_P5	FP32	5.500	181.82
yolov5s_fast	640x640	YoloV5_P5	FP32	2.342	427.07
yolov5x6_fast	1280x1280	YoloV5_P6	FP16	20.538	48.69
yolov5l6_fast	1280x1280	YoloV5_P6	FP16	10.404	96.12
yolov5m6_fast	1280x1280	YoloV5_P6	FP16	6.577	152.06
yolov5s6_fast	1280x1280	YoloV5_P6	FP16	3.087	323.99
yolov5x_fast	640x640	YoloV5_P5	FP16	5.919	168.95
yolov5l_fast	640x640	YoloV5_P5	FP16	3.348	298.69
yolov5m_fast	640x640	YoloV5_P5	FP16	2.015	496.34
yolov5s_fast	640x640	YoloV5_P5	FP16	1.087	919.63
yolov5x6_fast	1280x1280	YoloV5_P6	INT8	11.236	89.00
yolov5l6_fast	1280x1280	YoloV5_P6	INT8	6.235	160.38
yolov5m6_fast	1280x1280	YoloV5_P6	INT8	4.311	231.97
yolov5s6_fast	1280x1280	YoloV5_P6	INT8	2.139	467.45
yolov5x_fast	640x640	YoloV5_P5	INT8	3.456	289.37
yolov5l_fast	640x640	YoloV5_P5	INT8	2.019	495.41
yolov5m_fast	640x640	YoloV5_P5	INT8	1.425	701.71
yolov5s_fast	640x640	YoloV5_P5	INT8	0.844	1185.47

Setup and Configuration

Linux

complied dependency (Out of the Box feature)
- link for download [lean-tensorRT8.0.1.6-protobuf3.11.4-cudnn8.2.2.tar.gz] (http://zifuture.com:1556/fs/25.shared/lean-tensorRT8.0.1.6-protobuf3.11.4-cudnn8.2.2.tar.gz)

VSCode (highly recommended!)
Configure your path for cudnn, cuda, tensorRT8.0 and protobuf.
Configure the compute capability matched with your nvidia graphics card in Makefile/CMakeLists.txt
- e.g. -gencode=arch=compute_75,code=sm_75. If you are using 3080Ti, that should be gencode=arch=compute_86,code=sm_86
- reference for the table for GPU Compute Capability: https://developer.nvidia.com/cuda-gpus#compute
Configure your library path in .vscode/c_cpp_properties.json
CUDA version: CUDA10.2
CUDNN version: cudnn8.2.2.26. Note that dev(.h file) and runtime(.so file) should be downloaded.
tensorRT version：tensorRT-8.0.1.6-cuda10.2
protobuf version（for onnx parser）：protobufv3.11.4
- if other version, refer to the ........
- link for download: https://github.com/protocolbuffers/protobuf/tree/v3.11.4
- download, compile and replace the path in Makefile/CMakeLists.txt with new path to protobuf3.11.4

CMake:
- mkdir build && cd build
- cmake ..
- make yolo -j8
Makefile:
- make yolo -j8

Linux: Compile for Python

compile and install
- Makefile：
  - set use_python := true in Makefile
- CMakeLists.txt:
  - set(HAS_PYTHON ON) in CMakeLists.txt
- Type in make pyinstall -j8
- Complied files are in python/trtpy/libtrtpyc.so

Windows

Please check the lean/README.md for the detailed dependency
In TensorRT.vcxproj, replace the with your own CUDA path
In TensorRT.vcxproj, replace the with your own CUDA path
In TensorRT.vcxproj, replace the compute_61,sm_61 with your compute capability.
- refer to the table in https://developer.nvidia.com/cuda-gpus#compute
Configure your dependency or download it to the foler /lean. Configure VC++ dir (include dir and refence)
Configure your env, debug->environment
Compile and run the example, where 3 options are available.

Windows: Compile for Python

Compile trtpyc.pyd. Choose python in visual studio to compile
Copy dll and execute 'python/copy_dll_to_trtpy.bat'
Execute the example in python dir by 'python test_yolov5.py'

if installation is needed, switch to target env(e.g. your conda env) then 'python setup.py install', which has to be followed by step 1 and step 2.
the compiled files are in python/trtpy/libtrtpyc.pyd

Other Protobuf Version

in onnx/make_pb.sh, replace the path protoc=/data/sxai/lean/protobuf3.11.4/bin/protoc in protoc with the protoc of your own version

#cd the path in terminal to /onnx
cd onnx

#execuete the command to make pb files
bash make_pb.sh

CMake:
- replace the set(PROTOBUF_DIR "/data/sxai/lean/protobuf3.11.4") in CMakeLists.txt with the same path of your protoc.

mkdir build && cd build
cmake ..
make yolo -j64

Makefile:
- replace the path lean_protobuf := /data/sxai/lean/protobuf3.11.4 in Makefile with the same path of protoc

make yolo -j64

TensorRT 7.x support

The default is tensorRT8.x

Replace onnx_parser_for_7.x/onnx_parser to src/tensorRT/onnx_parser
- bash onnx_parser/use_tensorrt_7.x.sh
Configure Makefile/CMakeLists.txt path to TensorRT7.x
Execute make yolo -j64

TensorRT 8.x support

The default is tensorRT8.x

Replace onnx_parser_for_8.x/onnx_parser to src/tensorRT/onnx_parser
- bash onnx_parser/use_tensorrt_8.x.sh
Configure Makefile/CMakeLists.txt path to TensorRT8.x
Execute make yolo -j64

Guide for Different Tasks/Model Support

YoloV5 Support

if pytorch >= 1.7, and the model is 5.0+, the model is suppored by the framework
if pytorch < 1.7 or yolov5(2.0, 3.0 or 4.0), minor modification should be done in opset.
if you want to achieve the inference with lower pytorch, dynamic batchsize and other advanced setting, please check our blog (http://zifuture.com:8090)(now in Chinese) and scan the QRcode via Wechat to join us.

Download yolov5

git clone [email protected]:ultralytics/yolov5.git

Modify the code for dynamic batchsize

# line 55 forward function in yolov5/models/yolo.py 
# bs, _, ny, nx = x[i].shape  # x(bs,255,20,20) to x(bs,3,20,20,85)
# x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()
# modified into:

bs, _, ny, nx = x[i].shape  # x(bs,255,20,20) to x(bs,3,20,20,85)
bs = -1
ny = int(ny)
nx = int(nx)
x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()

# line 70 in yolov5/models/yolo.py
#  z.append(y.view(bs, -1, self.no))
# modified into：
z.append(y.view(bs, self.na * ny * nx, self.no))

# line 52 in yolov5/export.py
# torch.onnx.export(dynamic_axes={'images': {0: 'batch', 2: 'height', 3: 'width'},  # shape(1,3,640,640)
#                                'output': {0: 'batch', 1: 'anchors'}  # shape(1,25200,85)  修改为
# modified into:
torch.onnx.export(dynamic_axes={'images': {0: 'batch'},  # shape(1,3,640,640)
                                'output': {0: 'batch'}  # shape(1,25200,85)

Export to onnx model

cd yolov5
python export.py --weights=yolov5s.pt --dynamic --include=onnx --opset=11

Copy the model and execute it

cp yolov5/yolov5s.onnx tensorRT_cpp/workspace/
cd tensorRT_cpp
make yolo -j32

YoloX Support

download from: https://github.com/Megvii-BaseDetection/YOLOX
If you don't want to export onnx by yourself, just make run in the repo of Megavii

Download YoloX

git clone [email protected]:Megvii-BaseDetection/YOLOX.git
cd YOLOX

Modify the code The modification ensures a successful int8 compilation and inference, otherwise Missing scale and zero-point for tensor (Unnamed Layer* 686) will be raised.

# line 206 forward fuction in yolox/models/yolo_head.py. Replace the commented code with the uncommented code
# self.hw = [x.shape[-2:] for x in outputs] 
self.hw = [list(map(int, x.shape[-2:])) for x in outputs]


# line 208 forward function in yolox/models/yolo_head.py. Replace the commented code with the uncommented code
# [batch, n_anchors_all, 85]
# outputs = torch.cat(
#     [x.flatten(start_dim=2) for x in outputs], dim=2
# ).permute(0, 2, 1)
proc_view = lambda x: x.view(-1, int(x.size(1)), int(x.size(2) * x.size(3)))
outputs = torch.cat(
    [proc_view(x) for x in outputs], dim=2
).permute(0, 2, 1)


# line 253 decode_output function in yolox/models/yolo_head.py Replace the commented code with the uncommented code
#outputs[..., :2] = (outputs[..., :2] + grids) * strides
#outputs[..., 2:4] = torch.exp(outputs[..., 2:4]) * strides
#return outputs
xy = (outputs[..., :2] + grids) * strides
wh = torch.exp(outputs[..., 2:4]) * strides
return torch.cat((xy, wh, outputs[..., 4:]), dim=-1)

# line 77 in tools/export_onnx.py
model.head.decode_in_inference = True

Export to onnx

# download model
wget https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_m.pth

# export
python tools/export_onnx.py -c yolox_m.pth -f exps/default/yolox_m.py --output-name=yolox_m.onnx --dynamic --no-onnxsim

Execute the command

cp YOLOX/yolox_m.onnx tensorRT_cpp/workspace/
cd tensorRT_cpp
make yolo -j32

Retinaface Support

https://github.com/biubug6/Pytorch_Retinaface

Download Pytorch_Retinaface Repo

git clone [email protected]:biubug6/Pytorch_Retinaface.git
cd Pytorch_Retinaface

Download model from the Training of README.md in https://github.com/biubug6/Pytorch_Retinaface#training .Then unzip it to the /weights . Here, we use mobilenet0.25_Final.pth
Modify the code

# line 24 in models/retinaface.py
# return out.view(out.shape[0], -1, 2) is modified into 
return out.view(-1, int(out.size(1) * out.size(2) * 2), 2)

# line 35 in models/retinaface.py
# return out.view(out.shape[0], -1, 4) is modified into
return out.view(-1, int(out.size(1) * out.size(2) * 2), 4)

# line 46 in models/retinaface.py
# return out.view(out.shape[0], -1, 10) is modified into
return out.view(-1, int(out.size(1) * out.size(2) * 2), 10)

# The following modification ensures the output of resize node is based on scale rather than shape such that dynamic batch can be achieved.
# line 89 in models/net.py
# up3 = F.interpolate(output3, size=[output2.size(2), output2.size(3)], mode="nearest") is modified into
up3 = F.interpolate(output3, scale_factor=2, mode="nearest")

# line 93 in models/net.py
# up2 = F.interpolate(output2, size=[output1.size(2), output1.size(3)], mode="nearest") is modified into
up2 = F.interpolate(output2, scale_factor=2, mode="nearest")

# The following code removes softmax (bug sometimes happens). At the same time, concatenate the output to simplify the decoding.
# line 123 in models/retinaface.py
# if self.phase == 'train':
#     output = (bbox_regressions, classifications, ldm_regressions)
# else:
#     output = (bbox_regressions, F.softmax(classifications, dim=-1), ldm_regressions)
# return output
# the above is modified into:
output = (bbox_regressions, classifications, ldm_regressions)
return torch.cat(output, dim=-1)

# set 'opset_version=11' to ensure a successful export
# torch_out = torch.onnx._export(net, inputs, output_onnx, export_params=True, verbose=False,
#     input_names=input_names, output_names=output_names)
# is modified into:
torch_out = torch.onnx._export(net, inputs, output_onnx, export_params=True, verbose=False, opset_version=11,
    input_names=input_names, output_names=output_names)

Export to onnx

python convert_to_onnx.py

Execute

cp FaceDetector.onnx ../tensorRT_cpp/workspace/mb_retinaface.onnx
cd ../tensorRT_cpp
make retinaface -j64

DBFace Support

https://github.com/dlunion/DBFace

make dbface -j64

Scrfd Support

https://github.com/deepinsight/insightface/tree/master/detection/scrfd
The know-how about exporting to onnx is comming. Before it is released, come and join us to disucss.

Arcface Support

https://github.com/deepinsight/insightface/tree/master/recognition/arcface_torch

commit(make_tuple(face, landmarks)).get(); cout << feature << endl; // 1x512 ">

auto arcface = Arcface::create_infer("arcface_iresnet50.fp32.trtmodel", 0);
auto feature = arcface->commit(make_tuple(face, landmarks)).get();
cout << feature << endl;  // 1x512

In the example of Face Recognition, workspace/face/library is the set of faces registered.
workspace/face/recognize is the set of face to be recognized.
the result is saved in workspace/face/result和workspace/face/library_draw

CenterNet Support

check the great details in tutorial/2.0

Bert Support(Chinese Classification)

https://github.com/649453932/Bert-Chinese-Text-Classification-Pytorch
make bert -j6

the INTRO to Interface

Python Interface：Get onnx and trtmodel from pytorch model more easily

Just one line of code to export onnx and trtmodel. And save them for usage in the future.

import trtpy

model = models.resnet18(True).eval()
trtpy.from_torch(
    model, 
    dummy_input, 
    max_batch_size=16, 
    onnx_save_file="test.onnx", 
    engine_save_file="engine.trtmodel"
)

Python Interface：TensorRT Inference

YoloX TensorRT Inference

import trtpy

yolo   = tp.Yolo(engine_file, type=tp.YoloType.X)   # engine_file is the trtmodel file
image  = cv2.imread("inference/car.jpg")
bboxes = yolo.commit(image).get()

Seamless Inference from Pytorch to TensorRT

import trtpy

model     = models.resnet18(True).eval().to(device) # pt model
trt_model = tp.from_torch(model, input)
trt_out   = trt_model(input)

C++ Interface：YoloX Inference

commit(image).get(); ">

// create infer engine on gpu 0
auto engine = Yolo::create_infer("yolox_m.fp32.trtmodel"， Yolo::Type::X, 0);

// load image
auto image = cv::imread("1.jpg");

// do inference and get the result
auto box = engine->commit(image).get();

C++ Interface：Comple Model in FP32/FP16

TRT::compile(
  TRT::Mode::FP32,   // compile model in fp32
  3,                          // max batch size
  "plugin.onnx",              // onnx file
  "plugin.fp32.trtmodel",     // save path
  {}                         //  redefine the shape of input when needed
);

For fp32 compilation, all you need is offering onnx file whose input shape is allowed to be redefined.

C++ Interface：Compile in int8

The in8 inference performs slightly worse than fp32 in precision(about -5% drop down), but stunningly faster. In the framework, we offer int8 inference

// define int8 calibration function to read data and handle it to tenor.
auto int8process = [](int current, int count, vector
    & images, shared_ptr
    
     & tensor){
    
     for(
     int i = 
     0; i < images.
     size(); ++i){
    
     // int8 compilation requires calibration. We read image data and set_norm_mat. Then the data will be transfered into the tensor.
        
     auto image = 
     cv::imread(images[i]);
        
     cv::resize(image, image, 
     cv::Size(
     640, 
     640));
        
     float mean[] = {
     0, 
     0, 
     0};
        
     float std[]  = {
     1, 
     1, 
     1};
        tensor->
     set_norm_mat(i, image, mean, std);
    }
};



     // Specify TRT::Mode as INT8

     auto model_file = 
     "yolov5m.int8.trtmodel";

     TRT::compile(
  TRT::Mode::INT8,            
     // INT8
  
     3,                          
     // max batch size
  
     "yolov5m.onnx",             
     // onnx
  model_file,                 
     // saved filename
  {},                         
     // redefine the input shape
  int8process,                
     // the recall function for calibration
  
     ".",                        
     // the dir where the image data is used for calibration
  
     ""                          
     // the dir where the data generated from calibration is saved(a.k.a where to load the calibration data.)
);

We integrate into only one int8process function to save otherwise a lot of issues that might happen in tensorRT official implementation.

C++ Interface：Inference

We introduce class Tensor for easier inference and data transfer between host to device. So that as a user, the details wouldn't be annoying.
class Engine is another facilitator.

print(); // load image auto image = imread("demo.jpg"); // get the model input and output node, which can be accessed by name or index auto input = engine->input(0); // or auto input = engine->input("images"); auto output = engine->output(0); // or auto output = engine->output("output"); // put the image into input tensor by calling set_norm_mat() float mean[] = {0, 0, 0}; float std[] = {1, 1, 1}; input->set_norm_mat(i, image, mean, std); // do the inference. Here sync(true) or async(false) is optional engine->forward(); // engine->forward(true or false) // get the outut_ptr, which can used to access the output float* output_ptr = output->cpu
(); ">
// load model and get a shared_ptr. get nullptr if fail to load.
auto engine = TRT::load_infer("yolov5m.fp32.trtmodel");

// print model info
engine->print();

// load image
auto image = imread("demo.jpg");

// get the model input and output node, which can be accessed by name or index
auto input = engine->input(0);   // or auto input = engine->input("images");
auto output = engine->output(0); // or auto output = engine->output("output");

// put the image into input tensor by calling set_norm_mat()
float mean[] = {0, 0, 0};
float std[]  = {1, 1, 1};
input->set_norm_mat(i, image, mean, std);

// do the inference. Here sync(true) or async(false) is optional
engine->forward(); // engine->forward(true or false)

// get the outut_ptr, which can used to access the output
float* output_ptr = output->cpu<float>();

C++ Interface：Plugin

You only need to define kernel function and inference process. The details of code(e.g the serialization, deserialization and injection of plugin etc) are under the hood.
Easy to implement a new plugin in FP32 and FP16. Refer to HSwish.cu for details.

template<>
__global__ void HSwishKernel(float* input, float* output, int edge) {

    KernelPositionBlock;
    float x = input[position];
    float a = x + 3;
    a = a < 0 ? 0 : (a >= 6 ? 6 : a);
    output[position] = x * a / 6;
}

int HSwish::enqueue(const std::vector
    & inputs, std::vector
    
     & outputs, 
     const std::vector
     
      & weights, 
      void* workspace, cudaStream_t stream) {

    
      int count = inputs[
      0].
      count();
    
      auto grid = 
      CUDATools::grid_dims(count);
    
      auto block = 
      CUDATools::block_dims(count);
    HSwishKernel <<
      
       0, stream >>> (inputs[
       0].
       ptr<
       float>(), outputs[
       0].
       ptr<
       float>(), count);
    
       return 
       0;
}



       RegisterPlugin(HSwish);

About Us

Our blog：http://www.zifuture.com/ (Now only in Chinese. English is comming)
Our video channel： https://space.bilibili.com/1413433465 (Now only in Chinese. English is comming)

Comments

no result in int8 mode

Hi, thanks for your awesome code share!

I test this repo on my Ubuntu 18 Linux with CUDA V11.2.152, TensorRT-8.0.0.3 and cudnn8.2.0.

I tested yolox_s model in FP32、FP16 and INT8 mode, but in INT8 mode, the network output has no results.

FP16 test code is below:

static void test_fp16(Yolo::Type type){

    TRT::set_device(0);
    INFO("===================== test %s fp16 ==================================", Yolo::type_name(type));

    const char* name = nullptr;
    if(type == Yolo::Type::V5){
        name = "yolov5m";
    }else if(type == Yolo::Type::X){
        name = "yolox_s";
    }

    if(not requires(name))
        return;

    string onnx_file = iLogger::format("%s.onnx", name);
    string model_file = iLogger::format("%s.fp16.trtmodel", name);
    int test_batch_size = 1;  // 当你需要修改batch大于1时，请注意你的模型是否修改（看readme.md代码修改部分），否则会有错误
    
    // 动态batch和静态batch，如果你想要弄清楚，请打开http://www.zifuture.com:8090/
    // 找到右边的二维码，扫码加好友后进群交流（免费哈，就是技术人员一起沟通）
    if(not iLogger::exists(model_file)){
        TRT::compile(
            TRT::TRTMode_FP16,   // 编译方式有，FP32、FP16、INT8
            {},                         // onnx时无效，caffe的输出节点标记
            test_batch_size,            // 指定编译的batch size
            onnx_file,                  // 需要编译的onnx文件
            model_file,                 // 储存的模型文件
            {},                         // 指定需要重定义的输入shape，这里可以对onnx的输入shape进行重定义
            false                       // 是否采用动态batch维度，true采用，false不采用，使用静态固定的batch size
        );
    }

    forward_engine(model_file, type);
}

Below is the output of the log:

[2021-09-01 19:38:45][info][app_yolo.cpp:240]:===================== test YoloX fp32 ==================================
[2021-09-01 19:38:45][info][trt_builder.cpp:473]:Compile FP32 Onnx Model 'yolox_s.onnx'.
[2021-09-01 19:38:45][info][trt_builder.cpp:602]:Input shape is 1 x 3 x 640 x 640
[2021-09-01 19:38:45][info][trt_builder.cpp:603]:Set max batch size = 1
[2021-09-01 19:38:45][info][trt_builder.cpp:604]:Set max workspace size = 1024.00 MB
[2021-09-01 19:38:45][info][trt_builder.cpp:605]:Dynamic batch dimension is false
[2021-09-01 19:38:45][info][trt_builder.cpp:608]:Network has 1 inputs:
[2021-09-01 19:38:45][info][trt_builder.cpp:614]:      0.[images] shape is 1 x 3 x 640 x 640
[2021-09-01 19:38:45][info][trt_builder.cpp:620]:Network has 1 outputs:
[2021-09-01 19:38:45][info][trt_builder.cpp:625]:      0.[output] shape is 1 x 8400 x 85
[2021-09-01 19:38:45][info][trt_builder.cpp:670]:Building engine...
[2021-09-01 19:38:45][warn][trt_builder.cpp:33]:NVInfer WARNING: Convolution + generic activation fusion is disable due to incompatible driver or nvrtc
[2021-09-01 19:38:46][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
[2021-09-01 19:38:46][warn][trt_builder.cpp:33]:NVInfer WARNING: Detected invalid timing cache, setup a local cache instead
[2021-09-01 19:39:18][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
[2021-09-01 19:39:18][info][trt_builder.cpp:690]:Build done 32689 ms !
[2021-09-01 19:39:18][warn][trt_builder.cpp:33]:NVInfer WARNING: The logger passed into createInferRuntime differs from one already assigned, 0x557f9ae330b0, logger not updated.

[2021-09-01 19:39:19][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
[2021-09-01 19:39:19][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
[2021-09-01 19:39:19][info][trt_infer.cpp:169]:Infer 0x7fe3f8000c40 detail
[2021-09-01 19:39:19][info][trt_infer.cpp:170]: Max Batch Size: 1
[2021-09-01 19:39:19][info][trt_infer.cpp:171]: Dynamic Batch Dimension: false
[2021-09-01 19:39:19][info][trt_infer.cpp:172]: Inputs: 1
[2021-09-01 19:39:19][info][trt_infer.cpp:176]:         0.images : shape {1 x 3 x 640 x 640}
[2021-09-01 19:39:19][info][trt_infer.cpp:179]: Outputs: 1
[2021-09-01 19:39:19][info][trt_infer.cpp:183]:         0.output : shape {1 x 8400 x 85}
[2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/car.jpg, 2 object, 10.10 ms
[2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/group.jpg, 1 object, 6.19 ms
[2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/zand.jpg, 2 object, 8.72 ms
[2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/zgjr.jpg, 3 object, 6.03 ms
[2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/gril.jpg, 1 object, 5.95 ms
[2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/yq.jpg, 1 object, 5.92 ms
[2021-09-01 19:39:19][info][yolo.cpp:214]:Engine destroy.
[2021-09-01 19:39:19][info][app_yolo.cpp:277]:===================== test YoloX fp16 ==================================
[2021-09-01 19:39:19][info][trt_builder.cpp:473]:Compile FP16 Onnx Model 'yolox_s.onnx'.
[2021-09-01 19:39:19][warn][trt_builder.cpp:483]:Platform not have fast fp16 support
[2021-09-01 19:39:19][info][trt_builder.cpp:602]:Input shape is 1 x 3 x 640 x 640
[2021-09-01 19:39:19][info][trt_builder.cpp:603]:Set max batch size = 1
[2021-09-01 19:39:19][info][trt_builder.cpp:604]:Set max workspace size = 1024.00 MB
[2021-09-01 19:39:19][info][trt_builder.cpp:605]:Dynamic batch dimension is false
[2021-09-01 19:39:19][info][trt_builder.cpp:608]:Network has 1 inputs:
[2021-09-01 19:39:19][info][trt_builder.cpp:614]:      0.[images] shape is 1 x 3 x 640 x 640
[2021-09-01 19:39:19][info][trt_builder.cpp:620]:Network has 1 outputs:
[2021-09-01 19:39:19][info][trt_builder.cpp:625]:      0.[output] shape is 1 x 8400 x 85
[2021-09-01 19:39:19][info][trt_builder.cpp:670]:Building engine...
[2021-09-01 19:39:19][warn][trt_builder.cpp:33]:NVInfer WARNING: Half2 support requested on hardware without native FP16 support, performance will be negatively affected.
[2021-09-01 19:39:19][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
[2021-09-01 19:39:19][warn][trt_builder.cpp:33]:NVInfer WARNING: Detected invalid timing cache, setup a local cache instead
[2021-09-01 19:39:47][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
[2021-09-01 19:39:47][info][trt_builder.cpp:690]:Build done 28282 ms !
[2021-09-01 19:39:47][warn][trt_builder.cpp:33]:NVInfer WARNING: The logger passed into createInferRuntime differs from one already assigned, 0x557f9ae330b0, logger not updated.

[2021-09-01 19:39:48][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
[2021-09-01 19:39:48][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
[2021-09-01 19:39:48][info][trt_infer.cpp:169]:Infer 0x7fe3f8015650 detail
[2021-09-01 19:39:48][info][trt_infer.cpp:170]: Max Batch Size: 1
[2021-09-01 19:39:48][info][trt_infer.cpp:171]: Dynamic Batch Dimension: false
[2021-09-01 19:39:48][info][trt_infer.cpp:172]: Inputs: 1
[2021-09-01 19:39:48][info][trt_infer.cpp:176]:         0.images : shape {1 x 3 x 640 x 640}
[2021-09-01 19:39:48][info][trt_infer.cpp:179]: Outputs: 1
[2021-09-01 19:39:48][info][trt_infer.cpp:183]:         0.output : shape {1 x 8400 x 85}
[2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/car.jpg, 2 object, 10.75 ms
[2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/group.jpg, 1 object, 6.38 ms
[2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/zand.jpg, 2 object, 9.12 ms
[2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/zgjr.jpg, 3 object, 6.10 ms
[2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/gril.jpg, 1 object, 6.07 ms
[2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/yq.jpg, 1 object, 6.01 ms
[2021-09-01 19:39:48][info][yolo.cpp:214]:Engine destroy.
[2021-09-01 19:39:48][info][app_yolo.cpp:190]:===================== test YoloX int8 ==================================
[2021-09-01 19:39:48][info][trt_builder.cpp:473]:Compile INT8 Onnx Model 'yolox_s.onnx'.
[2021-09-01 19:39:48][info][trt_builder.cpp:593]:Using image list[6 files]: inference
[2021-09-01 19:39:48][info][trt_builder.cpp:602]:Input shape is 1 x 3 x 640 x 640
[2021-09-01 19:39:48][info][trt_builder.cpp:603]:Set max batch size = 1
[2021-09-01 19:39:48][info][trt_builder.cpp:604]:Set max workspace size = 1024.00 MB
[2021-09-01 19:39:48][info][trt_builder.cpp:605]:Dynamic batch dimension is false
[2021-09-01 19:39:48][info][trt_builder.cpp:608]:Network has 1 inputs:
[2021-09-01 19:39:48][info][trt_builder.cpp:614]:      0.[images] shape is 1 x 3 x 640 x 640
[2021-09-01 19:39:48][info][trt_builder.cpp:620]:Network has 1 outputs:
[2021-09-01 19:39:48][info][trt_builder.cpp:625]:      0.[output] shape is 1 x 8400 x 85
[2021-09-01 19:39:48][info][trt_builder.cpp:670]:Building engine...
[2021-09-01 19:39:48][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
[2021-09-01 19:39:49][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
[2021-09-01 19:39:49][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
[2021-09-01 19:39:49][info][app_yolo.cpp:193]:Int8 1 / 6
[2021-09-01 19:39:49][info][app_yolo.cpp:193]:Int8 2 / 6
[2021-09-01 19:39:49][info][app_yolo.cpp:193]:Int8 3 / 6
[2021-09-01 19:39:49][info][app_yolo.cpp:193]:Int8 4 / 6
[2021-09-01 19:39:49][info][app_yolo.cpp:193]:Int8 5 / 6
[2021-09-01 19:39:50][info][app_yolo.cpp:193]:Int8 6 / 6
[2021-09-01 19:40:27][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
[2021-09-01 19:40:27][warn][trt_builder.cpp:33]:NVInfer WARNING: Detected invalid timing cache, setup a local cache instead
[2021-09-01 19:40:59][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
[2021-09-01 19:40:59][info][trt_builder.cpp:685]:No set entropyCalibratorFile, and entropyCalibrator will not save.
[2021-09-01 19:40:59][info][trt_builder.cpp:690]:Build done 70917 ms !
[2021-09-01 19:40:59][warn][trt_builder.cpp:33]:NVInfer WARNING: The logger passed into createInferRuntime differs from one already assigned, 0x557f9ae330b0, logger not updated.

[2021-09-01 19:40:59][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
[2021-09-01 19:40:59][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
[2021-09-01 19:40:59][info][trt_infer.cpp:169]:Infer 0x7fe3b4000c40 detail
[2021-09-01 19:40:59][info][trt_infer.cpp:170]: Max Batch Size: 1
[2021-09-01 19:40:59][info][trt_infer.cpp:171]: Dynamic Batch Dimension: false
[2021-09-01 19:40:59][info][trt_infer.cpp:172]: Inputs: 1
[2021-09-01 19:40:59][info][trt_infer.cpp:176]:         0.images : shape {1 x 3 x 640 x 640}
[2021-09-01 19:40:59][info][trt_infer.cpp:179]: Outputs: 1
[2021-09-01 19:40:59][info][trt_infer.cpp:183]:         0.output : shape {1 x 8400 x 85}
[2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/car.jpg, 0 object, 7.75 ms
[2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/group.jpg, 0 object, 4.04 ms
[2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/zand.jpg, 0 object, 6.04 ms
[2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/zgjr.jpg, 0 object, 3.74 ms
[2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/gril.jpg, 0 object, 3.72 ms
[2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/yq.jpg, 0 object, 3.74 ms
[2021-09-01 19:40:59][info][yolo.cpp:214]:Engine destroy.

Time-consuming in the FP16/FP32 modes is close, is it because the code I wrote has a problem?

Your sincerely!

opened by qixuxiang 14

AffineMatrix求解？

正常图像作缩放是不需要偏移的吧？ M = [ scale, 0, -scale * from.width * 0.5 + to.width * 0.5 0, scale, -scale * from.height * 0.5 + to.height * 0.5 0, 0, 1 ] -scale * from.width * 0.5 + to.width * 0.5这个偏移量抵消了，

但是第一次平移矩阵不是 P = [ 1, 0, -scale * from.width * 0.5 0, -1, scale * from.height * 0.5 0, 0, 1 ] 第二次平移矩阵不应该是： T = [ 1, 0, to.width * 0.5, 0, - 1, to.height * 0.5, 0, 0, 1 ]

？?? 参考的矩阵： https://www.cnblogs.com/xuanyuyt/p/7112876.html

opened by cqray1990 8
Environment Configuration Issues！

I have never used VS Code before. I want to learn from the video of the blogger, but there are always errors. Is there any solution？（I have installed Tensorrt on Windows）

opened by Havehandssook 6
arcface人脸特征提取不同人脸，计算的相似度都一样，是怎么回事呢？

余弦距离计算代码： def cosine_distance(matrix1, matrix2): matrix1_matrix2 = np.dot(matrix1, matrix2.transpose()) matrix1_norm = np.sqrt(np.multiply(matrix1, matrix1).sum(axis=1)) matrix1_norm = matrix1_norm[:, np.newaxis] matrix2_norm = np.sqrt(np.multiply(matrix2, matrix2).sum(axis=1)) matrix2_norm = matrix2_norm[:, np.newaxis] cosine_distance = np.divide(matrix1_matrix2, np.dot(matrix1_norm, matrix2_norm.transpose())) return cosine_distance

是不是没有进行归一化呢，归一化有没有好的函数推荐呢？

opened by seawater668 6

Error，arcface onnx 转tensorrt 错误

使用了本项目提供给的arcface_iresnet50.onnx 模型
错误如下：
While parsing node number 182 [BatchNormalization]:
ERROR: /home/Project/tensorRT_Pro-main/src/tensorRT/onnx_parser/onnx2trt_utils.cpp:1523 In function scaleHelper:
[8] Assertion failed: dims.nbDims == 4 || dims.nbDims == 5
[2021-10-04 16:39:25][error][trt_builder.cpp:517]:Can not parse OnnX file: arcface_iresnet50_iii.onnx

计算机环境如下：

tensorrt 7.2
cudnn8.1
cuda 11.2
protobufv3.11.4
gpu 3080  arch86

BatchNormalization_182 是模型的倒数第二层。

opened by create-li 6

preprocess engine->forward decode_kernel_invoker 线程问题

老师，您好，preprocess 在commits，是在主线程执行，而decode_kernel_invoker是在work 函数中和forward在一个线程中执行，为什么会这么设计呀？如果送入的图片存在高并发问题是否会导致主线程卡死在commits中，为什么不把preprocess过程和decode一样放在work线程中执行？

然后我有个想法是不是可以把preprocess 和decode 剥离出来，通过多线程实现， work只做forward的工作，这样是不是更高效？

opened by QZ1219 4
cmake编译失败

cmake能全部正常build，但是在make的时候正常编译到85%的时候报错了，以下是报错内容（我未修改报错文件内的代码）

[ 83%] Building CXX object CMakeFiles/pro.dir/src/application/app_yolo_fast.cpp.o [ 83%] Building CXX object CMakeFiles/pro.dir/src/application/app_alphapose.cpp.o [ 85%] Building CXX object CMakeFiles/pro.dir/src/application/app_python/interface.cpp.o In file included from /xxx/datav/projects/tensorRT_Pro/src/application/app_python/interface.cpp:6:0: /xxx/datav/projects/tensorRT_Pro/src/application/tools/pybind11.hpp:159:20: fatal error: Python.h: No such file or directory #include <Python.h> ^ compilation terminated. CMakeFiles/pro.dir/build.make:1142: recipe for target 'CMakeFiles/pro.dir/src/application/app_python/interface.cpp.o' failed make[3]: *** [CMakeFiles/pro.dir/src/application/app_python/interface.cpp.o] Error 1 make[3]: *** Waiting for unfinished jobs.... CMakeFiles/Makefile2:328: recipe for target 'CMakeFiles/pro.dir/all' failed make[2]: *** [CMakeFiles/pro.dir/all] Error 2 CMakeFiles/Makefile2:537: recipe for target 'CMakeFiles/yolo.dir/rule' failed make[1]: *** [CMakeFiles/yolo.dir/rule] Error 2 Makefile:300: recipe for target 'yolo' failed make: *** [yolo] Error 2

opened by sungh66 4

protobuf编译有问题

make报错

g++: error: google/protobuf/util/internal/.libs/proto_writer.o: No such file or directory
Makefile:2372: recipe for target 'libprotobuf.la' failed
make[2]: *** [libprotobuf.la] Error 1

是不是要先下载googletest，如果是的话，是不是哪个版本呢？ ubuntu18.04 jetson xavier nx

opened by DCC-lzhy 4

][trt_builder.cpp:36]:NVInfer: TensorRT was linked against cuBLAS/cuBLAS LT 11.6.3 but loaded cuBLAS/cuBLAS LT 11.3.1

warning, errors are showed above, it is annoying and weird to concern the compatibility issues among tensorrt version and cuda, and cuda toolkit versions. i cannot figure out the difference among them, any help will be approciated!!!

i have installed tensorrt !!!

dpkg -l | grep tensorrt ii nv-tensorrt-repo-ubuntu1804-cuda10.0-trt5.0.2.6-ga-20181009 1-1 amd64 nv-tensorrt repository configuration files ii nv-tensorrt-repo-ubuntu1804-cuda10.1-trt5.1.5.0-ga-20190427 1-1 amd64 nv-tensorrt repository configuration files ii nv-tensorrt-repo-ubuntu1804-cuda11.4-trt8.2.1.8-ga-20211117 1-1 amd64 nv-tensorrt repository configuration files ii tensorrt 8.2.1.8-1+cuda11.4 amd64

it should be 8.2.18 cuda11.4 as i am concerned.

and after i typed nvcc -V , the cuda version is nvcc as follows:

NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2020 NVIDIA Corporation Built on Mon_Nov_30_19:08:53_PST_2020 Cuda compilation tools, release 11.2, V11.2.67 Build cuda_11.2.r11.2/compiler.29373293_0

so which version of toolkit should i install??? currently, my cuda toolkit is v11.5.0 i think

opened by sainttelant 4
yolox master分支的模型能够正常导出到onnx但生成engine失败

[2021-10-26 14:29:01][info][app_yolo.cpp:121]:===================== test YoloX FP32 yolox_s ================================== [2021-10-26 14:29:01][info][trt_builder.cpp:471]:Compile FP32 Onnx Model 'yolox_s.onnx'. [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: INVALID_ARGUMENT: getPluginCreator could not find plugin ScatterND version 1 While parsing node number 565 [ScatterND]: ERROR: /home/work/tracking/tensorRT_Pro/src/tensorRT/onnx_parser/builtin_op_importers.cpp:4013 In function importFallbackPluginImporter: [8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?" [2021-10-26 14:29:02][error][trt_builder.cpp:517]:Can not parse OnnX file: yolox_s.onnx [2021-10-26 14:29:02][error][yolo.cpp:138]:Engine yolox_s.FP32.trtmodel load failed [2021-10-26 14:29:02][error][app_yolo.cpp:42]:Engine is nullptr [100%] Built target yolo

opened by deep-practice 4

对输入输出的内存申请为啥要加个size_matrix 仿射矩阵的大小，而且对输出affine_matrix_device的内存分配是sizeof(job.additional.d2i)

        uint8_t* gpu_workspace        = (uint8_t*)workspace->gpu(size_matrix + size_image);
        float*   affine_matrix_device = (float*)gpu_workspace;
        uint8_t* image_device         = size_matrix + gpu_workspace;

        uint8_t* cpu_workspace        = (uint8_t*)workspace->cpu(size_matrix + size_image);
        float* affine_matrix_host     = (float*)cpu_workspace;
        uint8_t* image_host           = size_matrix + cpu_workspace;
       memcpy(image_host, image.data, size_image);
        memcpy(affine_matrix_host, job.additional.d2i, sizeof(job.additional.d2i));
        checkCudaRuntime(cudaMemcpyAsync(image_device, image_host, size_image, cudaMemcpyHostToDevice, preprocess_stream));
        checkCudaRuntime(cudaMemcpyAsync(affine_matrix_device, affine_matrix_host, sizeof(job.additional.d2i), cudaMemcpyHostToDevice, preprocess_stream));

opened by cqray1990 3

请问下是否支持arm

CUDA Runtime error cudaSetDevice(device_id) # initialization error, code = cudaErrorInitializationError [ 3 ] in file /src/tensorRT/infer/trt_infer.cpp:472 请问下代码是否支持这个是在jetson nx编译通过后，转engine是报错，麻烦大佬帮忙看看

opened by Waynepoo 0

rtsp视频流解码报错

报错如下，这个错误影响解码吗

[h264 @ 0x7f918c007980] concealing 1466 DC, 1466 AC, 1466 MV errors in I frame
[rtsp @ 0x7f918c0021c0] RTP Xiph packet settings (0,1,0) is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
Decode Error occurred for picture 0

opened by ZJU-lishuang 0

onnx转trt报错

Hi [2022-12-08 17:36:09][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:09][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:09][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::`anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: g:\chengc\project\tensorrt-onnx-fasterrcnn-fpn-roialign-master\tensorrt_code\src\tensorrt\onnx_parser\modelimporter.cpp:736: While parsing node number 364 [Add -> "1963"]: [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: g:\chengc\project\tensorrt-onnx-fasterrcnn-fpn-roialign-master\tensorrt_code\src\tensorrt\onnx_parser\modelimporter.cpp:737: --- Begin node --- [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: g:\chengc\project\tensorrt-onnx-fasterrcnn-fpn-roialign-master\tensorrt_code\src\tensorrt\onnx_parser\modelimporter.cpp:738: input: "1935" input: "1940" output: "1963" name: "Add_1095" op_type: "Add"

[2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: g:\chengc\project\tensorrt-onnx-fasterrcnn-fpn-roialign-master\tensorrt_code\src\tensorrt\onnx_parser\modelimporter.cpp:739: --- End node --- [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: g:\chengc\project\tensorrt-onnx-fasterrcnn-fpn-roialign-master\tensorrt_code\src\tensorrt\onnx_parser\modelimporter.cpp:742: ERROR: g:\chengc\project\tensorrt-onnx-fasterrcnn-fpn-roialign-master\tensorrt_code\src\tensorrt\onnx_parser\onnx2trt_utils.cpp:959 In function elementwiseHelper: [8] Assertion failed: tensor_ptr->getDimensions().nbDims == maxNbDims && "Failed to broadcast tensors elementwise!"

opened by Chengcheng1998727 1

Releases(v1.0)

v1.0(Sep 14, 2021)

第一个版本固定下来
Source code(tar.gz)
Source code(zip)

An Out-of-the-Box TensorRT-based Framework for High Performance Inference with C++/Python Support

Related tags

Overview

News:

Tutorial Video

An Out-of-the-Box TensorRT-based Framework for High Performance Inference with C++/Python Support

INTRO

YoloX and YoloV5-series Model Test Report

Setup and Configuration

Guide for Different Tasks/Model Support

the INTRO to Interface

About Us

Comments

Releases(v1.0)

v1.0(Sep 14, 2021)

Owner

手写AI

Python Inference Script is a Python package that enables developers to author machine learning workflows in Python and deploy without Python.

TensorRT is a C++ library for high performance inference on NVIDIA GPUs and deep learning accelerators.

ncnn is a high-performance neural network inference framework optimized for the mobile platform

Hardware-accelerated DNN model inference ROS2 packages using NVIDIA Triton/TensorRT for both Jetson and x86_64 with CUDA-capable GPU.

Helper Class for Deep Learning Inference Frameworks: TensorFlow Lite, TensorRT, OpenCV, ncnn, MNN, SNPE, Arm NN, NNAbla

Forward - A library for high performance deep learning inference on NVIDIA GPUs

A library for high performance deep learning inference on NVIDIA GPUs.

Edge ML Library - High-performance Compute Library for On-device Machine Learning Inference

PPLNN is a high-performance deep-learning inference engine for efficient AI inferencing.

Benchmark framework of 3D integrated CIM accelerators for popular DNN inference, support both monolithic and heterogeneous 3D integration

Experimental and Comparative Performance Measurements of High Performance Computing Based on OpenMP and MPI

Implement yolov5 with Tensorrt C++ api, and integrate batchedNMSPlugin. A Python wrapper is also provided.

Deep Learning API and Server in C++11 support for Caffe, Caffe2, PyTorch,TensorRT, Dlib, NCNN, Tensorflow, XGBoost and TSNE

Support Yolov4/Yolov3/Centernet/Classify/Unet. use darknet/libtorch/pytorch to onnx to tensorrt

A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.

APFS module for linux, with experimental write support (out-of-tree repository)

A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

KSAI Lite is a deep learning inference framework of kingsoft, based on tensorflow lite