An Out-of-the-Box TensorRT-based Framework for High Performance Inference with C++/Python Support

Overview

Read this in other languages: English, 简体中文.

News:

Tutorial for exporting CenterNet from pytorch to tensorRT is released.

Tutorial Video

An Out-of-the-Box TensorRT-based Framework for High Performance Inference with C++/Python Support

  • C++ Interface: 3 lines of code is all you need to run a YoloX

    commit(image).get(); // return vector ">
    // create inference engine on gpu-0
    //auto engine = Yolo::create_infer("yolov5m.fp32.trtmodel", Yolo::Type::V5, 0);
    auto engine = Yolo::create_infer("yolox_m.fp32.trtmodel", Yolo::Type::X, 0);
    
    // load image
    auto image = cv::imread("1.jpg");
    
    // do inference and get the result
    auto box = engine->commit(image).get();  // return vector
          
  • Python Interface:

    import trtpy
    
    model     = models.resnet18(True).eval().to(device)
    trt_model = tp.from_torch(model, input)
    trt_out   = trt_model(input)

INTRO

  1. High level interface for C++/Python.
  2. Based on TensorRT8.0.
  3. Simplify the implementation of custom plugin. And serialization and deserialization have been encapsulated for easier usage.
  4. Simplify the compile of fp32, fp16 and int8 for facilitating the deployment with C++/Python in server or embeded device.
  5. Models ready for use also with examples are RetinaFace, Scrfd, YoloV5, YoloX, Arcface, AlphaPose, CenterNet and DeepSORT(C++)

YoloX and YoloV5-series Model Test Report

app_yolo.cpp speed testing
  1. Resolution (YoloV5P5, YoloX) = (640x640), (YoloV5P6) = (1280x1280)
  2. max batch size = 16
  3. preprocessing + inference + postprocessing
  4. cuda10.2, cudnn8.2.2.26, TensorRT-8.0.1.6
  5. RTX2080Ti
  6. num of testing: take the average on the results of 100 times but excluding the first time for warmup
  7. Testing log: [workspace/perf.result.std.log (workspace/perf.result.std.log)
  8. code for testing: src/application/app_yolo.cpp
  9. images for testing: 6 images in workspace/inference
    • with resolution 810x1080,500x806,1024x684,550x676,1280x720,800x533 respetively
  10. Testing method: load 6 images. Then do the inference on the 6 images, which will be repeated for 100 times. Note that each image should be preprocessed and postprocessed.

Model Resolution Type Precision Elapsed Time FPS
yolox_x 640x640 YoloX FP32 21.879 45.71
yolox_l 640x640 YoloX FP32 12.308 81.25
yolox_m 640x640 YoloX FP32 6.862 145.72
yolox_s 640x640 YoloX FP32 3.088 323.81
yolox_x 640x640 YoloX FP16 6.763 147.86
yolox_l 640x640 YoloX FP16 3.933 254.25
yolox_m 640x640 YoloX FP16 2.515 397.55
yolox_s 640x640 YoloX FP16 1.362 734.48
yolox_x 640x640 YoloX INT8 4.070 245.68
yolox_l 640x640 YoloX INT8 2.444 409.21
yolox_m 640x640 YoloX INT8 1.730 577.98
yolox_s 640x640 YoloX INT8 1.060 943.15
yolov5x6 1280x1280 YoloV5_P6 FP32 68.022 14.70
yolov5l6 1280x1280 YoloV5_P6 FP32 37.931 26.36
yolov5m6 1280x1280 YoloV5_P6 FP32 20.127 49.69
yolov5s6 1280x1280 YoloV5_P6 FP32 8.715 114.75
yolov5x 640x640 YoloV5_P5 FP32 18.480 54.11
yolov5l 640x640 YoloV5_P5 FP32 10.110 98.91
yolov5m 640x640 YoloV5_P5 FP32 5.639 177.33
yolov5s 640x640 YoloV5_P5 FP32 2.578 387.92
yolov5x6 1280x1280 YoloV5_P6 FP16 20.877 47.90
yolov5l6 1280x1280 YoloV5_P6 FP16 10.960 91.24
yolov5m6 1280x1280 YoloV5_P6 FP16 7.236 138.20
yolov5s6 1280x1280 YoloV5_P6 FP16 3.851 259.68
yolov5x 640x640 YoloV5_P5 FP16 5.933 168.55
yolov5l 640x640 YoloV5_P5 FP16 3.450 289.86
yolov5m 640x640 YoloV5_P5 FP16 2.184 457.90
yolov5s 640x640 YoloV5_P5 FP16 1.307 765.10
yolov5x6 1280x1280 YoloV5_P6 INT8 12.207 81.92
yolov5l6 1280x1280 YoloV5_P6 INT8 7.221 138.49
yolov5m6 1280x1280 YoloV5_P6 INT8 5.248 190.55
yolov5s6 1280x1280 YoloV5_P6 INT8 3.149 317.54
yolov5x 640x640 YoloV5_P5 INT8 3.704 269.97
yolov5l 640x640 YoloV5_P5 INT8 2.255 443.53
yolov5m 640x640 YoloV5_P5 INT8 1.674 597.40
yolov5s 640x640 YoloV5_P5 INT8 1.143 874.91
app_yolo_fast.cpp speed testing. Never stop desiring for being faster
  • Highlight: 0.5 ms faster without any loss in precision compared with the above. Specifically, we remove the Focus and some transpose nodes etc, and implement them in CUDA kenerl function. But the rest remains the same.
  • Test log: workspace/perf.result.std.log
  • Code for testing: src/application/app_yolo_fast.cpp
  • Tips: you can do the modification while refering to the downloaded onnx. Any questions are welcomed through any kinds of contact.
  • Conclusion: the main idea of this work is to optimize the pre-and-post processing. If you go for yolox, yolov5 small version, the optimization might help you.
Model Resolution Type Precision Elapsed Time FPS
yolox_x_fast 640x640 YoloX FP32 21.598 46.30
yolox_l_fast 640x640 YoloX FP32 12.199 81.97
yolox_m_fast 640x640 YoloX FP32 6.819 146.65
yolox_s_fast 640x640 YoloX FP32 2.979 335.73
yolox_x_fast 640x640 YoloX FP16 6.764 147.84
yolox_l_fast 640x640 YoloX FP16 3.866 258.64
yolox_m_fast 640x640 YoloX FP16 2.386 419.16
yolox_s_fast 640x640 YoloX FP16 1.259 794.36
yolox_x_fast 640x640 YoloX INT8 3.918 255.26
yolox_l_fast 640x640 YoloX INT8 2.292 436.38
yolox_m_fast 640x640 YoloX INT8 1.589 629.49
yolox_s_fast 640x640 YoloX INT8 0.954 1048.47
yolov5x6_fast 1280x1280 YoloV5_P6 FP32 67.075 14.91
yolov5l6_fast 1280x1280 YoloV5_P6 FP32 37.491 26.67
yolov5m6_fast 1280x1280 YoloV5_P6 FP32 19.422 51.49
yolov5s6_fast 1280x1280 YoloV5_P6 FP32 7.900 126.57
yolov5x_fast 640x640 YoloV5_P5 FP32 18.554 53.90
yolov5l_fast 640x640 YoloV5_P5 FP32 10.060 99.41
yolov5m_fast 640x640 YoloV5_P5 FP32 5.500 181.82
yolov5s_fast 640x640 YoloV5_P5 FP32 2.342 427.07
yolov5x6_fast 1280x1280 YoloV5_P6 FP16 20.538 48.69
yolov5l6_fast 1280x1280 YoloV5_P6 FP16 10.404 96.12
yolov5m6_fast 1280x1280 YoloV5_P6 FP16 6.577 152.06
yolov5s6_fast 1280x1280 YoloV5_P6 FP16 3.087 323.99
yolov5x_fast 640x640 YoloV5_P5 FP16 5.919 168.95
yolov5l_fast 640x640 YoloV5_P5 FP16 3.348 298.69
yolov5m_fast 640x640 YoloV5_P5 FP16 2.015 496.34
yolov5s_fast 640x640 YoloV5_P5 FP16 1.087 919.63
yolov5x6_fast 1280x1280 YoloV5_P6 INT8 11.236 89.00
yolov5l6_fast 1280x1280 YoloV5_P6 INT8 6.235 160.38
yolov5m6_fast 1280x1280 YoloV5_P6 INT8 4.311 231.97
yolov5s6_fast 1280x1280 YoloV5_P6 INT8 2.139 467.45
yolov5x_fast 640x640 YoloV5_P5 INT8 3.456 289.37
yolov5l_fast 640x640 YoloV5_P5 INT8 2.019 495.41
yolov5m_fast 640x640 YoloV5_P5 INT8 1.425 701.71
yolov5s_fast 640x640 YoloV5_P5 INT8 0.844 1185.47

Setup and Configuration

Linux
  1. VSCode (highly recommended!)
  2. Configure your path for cudnn, cuda, tensorRT8.0 and protobuf.
  3. Configure the compute capability matched with your nvidia graphics card in Makefile/CMakeLists.txt
  4. Configure your library path in .vscode/c_cpp_properties.json
  5. CUDA version: CUDA10.2
  6. CUDNN version: cudnn8.2.2.26. Note that dev(.h file) and runtime(.so file) should be downloaded.
  7. tensorRT version:tensorRT-8.0.1.6-cuda10.2
  8. protobuf version(for onnx parser):protobufv3.11.4
  • CMake:
    • mkdir build && cd build
    • cmake ..
    • make yolo -j8
  • Makefile:
    • make yolo -j8
Linux: Compile for Python
  • compile and install
    • Makefile:
      • set use_python := true in Makefile
    • CMakeLists.txt:
      • set(HAS_PYTHON ON) in CMakeLists.txt
    • Type in make pyinstall -j8
    • Complied files are in python/trtpy/libtrtpyc.so
Windows
  1. Please check the lean/README.md for the detailed dependency

  2. In TensorRT.vcxproj, replace the with your own CUDA path

  3. In TensorRT.vcxproj, replace the with your own CUDA path

  4. In TensorRT.vcxproj, replace the compute_61,sm_61 with your compute capability.

  5. Configure your dependency or download it to the foler /lean. Configure VC++ dir (include dir and refence)

  6. Configure your env, debug->environment

  7. Compile and run the example, where 3 options are available.

Windows: Compile for Python
  1. Compile trtpyc.pyd. Choose python in visual studio to compile
  2. Copy dll and execute 'python/copy_dll_to_trtpy.bat'
  3. Execute the example in python dir by 'python test_yolov5.py'
  • if installation is needed, switch to target env(e.g. your conda env) then 'python setup.py install', which has to be followed by step 1 and step 2.
  • the compiled files are in python/trtpy/libtrtpyc.pyd
Other Protobuf Version
  • in onnx/make_pb.sh, replace the path protoc=/data/sxai/lean/protobuf3.11.4/bin/protoc in protoc with the protoc of your own version
#cd the path in terminal to /onnx
cd onnx

#execuete the command to make pb files
bash make_pb.sh
  • CMake:
    • replace the set(PROTOBUF_DIR "/data/sxai/lean/protobuf3.11.4") in CMakeLists.txt with the same path of your protoc.
mkdir build && cd build
cmake ..
make yolo -j64
  • Makefile:
    • replace the path lean_protobuf := /data/sxai/lean/protobuf3.11.4 in Makefile with the same path of protoc
make yolo -j64
TensorRT 7.x support
  • The default is tensorRT8.x
  1. Replace onnx_parser_for_7.x/onnx_parser to src/tensorRT/onnx_parser
    • bash onnx_parser/use_tensorrt_7.x.sh
  2. Configure Makefile/CMakeLists.txt path to TensorRT7.x
  3. Execute make yolo -j64
TensorRT 8.x support
  • The default is tensorRT8.x
  1. Replace onnx_parser_for_8.x/onnx_parser to src/tensorRT/onnx_parser
    • bash onnx_parser/use_tensorrt_8.x.sh
  2. Configure Makefile/CMakeLists.txt path to TensorRT8.x
  3. Execute make yolo -j64

Guide for Different Tasks/Model Support

YoloV5 Support
  • if pytorch >= 1.7, and the model is 5.0+, the model is suppored by the framework
  • if pytorch < 1.7 or yolov5(2.0, 3.0 or 4.0), minor modification should be done in opset.
  • if you want to achieve the inference with lower pytorch, dynamic batchsize and other advanced setting, please check our blog (http://zifuture.com:8090)(now in Chinese) and scan the QRcode via Wechat to join us.
  1. Download yolov5
git clone [email protected]:ultralytics/yolov5.git
  1. Modify the code for dynamic batchsize
# line 55 forward function in yolov5/models/yolo.py 
# bs, _, ny, nx = x[i].shape  # x(bs,255,20,20) to x(bs,3,20,20,85)
# x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()
# modified into:

bs, _, ny, nx = x[i].shape  # x(bs,255,20,20) to x(bs,3,20,20,85)
bs = -1
ny = int(ny)
nx = int(nx)
x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()

# line 70 in yolov5/models/yolo.py
#  z.append(y.view(bs, -1, self.no))
# modified into:
z.append(y.view(bs, self.na * ny * nx, self.no))

# line 52 in yolov5/export.py
# torch.onnx.export(dynamic_axes={'images': {0: 'batch', 2: 'height', 3: 'width'},  # shape(1,3,640,640)
#                                'output': {0: 'batch', 1: 'anchors'}  # shape(1,25200,85)  修改为
# modified into:
torch.onnx.export(dynamic_axes={'images': {0: 'batch'},  # shape(1,3,640,640)
                                'output': {0: 'batch'}  # shape(1,25200,85) 
  1. Export to onnx model
cd yolov5
python export.py --weights=yolov5s.pt --dynamic --include=onnx --opset=11
  1. Copy the model and execute it
cp yolov5/yolov5s.onnx tensorRT_cpp/workspace/
cd tensorRT_cpp
make yolo -j32
YoloX Support
  1. Download YoloX
git clone [email protected]:Megvii-BaseDetection/YOLOX.git
cd YOLOX
  1. Modify the code The modification ensures a successful int8 compilation and inference, otherwise Missing scale and zero-point for tensor (Unnamed Layer* 686) will be raised.
# line 206 forward fuction in yolox/models/yolo_head.py. Replace the commented code with the uncommented code
# self.hw = [x.shape[-2:] for x in outputs] 
self.hw = [list(map(int, x.shape[-2:])) for x in outputs]


# line 208 forward function in yolox/models/yolo_head.py. Replace the commented code with the uncommented code
# [batch, n_anchors_all, 85]
# outputs = torch.cat(
#     [x.flatten(start_dim=2) for x in outputs], dim=2
# ).permute(0, 2, 1)
proc_view = lambda x: x.view(-1, int(x.size(1)), int(x.size(2) * x.size(3)))
outputs = torch.cat(
    [proc_view(x) for x in outputs], dim=2
).permute(0, 2, 1)


# line 253 decode_output function in yolox/models/yolo_head.py Replace the commented code with the uncommented code
#outputs[..., :2] = (outputs[..., :2] + grids) * strides
#outputs[..., 2:4] = torch.exp(outputs[..., 2:4]) * strides
#return outputs
xy = (outputs[..., :2] + grids) * strides
wh = torch.exp(outputs[..., 2:4]) * strides
return torch.cat((xy, wh, outputs[..., 4:]), dim=-1)

# line 77 in tools/export_onnx.py
model.head.decode_in_inference = True
  1. Export to onnx
# download model
wget https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_m.pth

# export
python tools/export_onnx.py -c yolox_m.pth -f exps/default/yolox_m.py --output-name=yolox_m.onnx --dynamic --no-onnxsim
  1. Execute the command
cp YOLOX/yolox_m.onnx tensorRT_cpp/workspace/
cd tensorRT_cpp
make yolo -j32
Retinaface Support
  1. Download Pytorch_Retinaface Repo
git clone [email protected]:biubug6/Pytorch_Retinaface.git
cd Pytorch_Retinaface
  1. Download model from the Training of README.md in https://github.com/biubug6/Pytorch_Retinaface#training .Then unzip it to the /weights . Here, we use mobilenet0.25_Final.pth

  2. Modify the code

# line 24 in models/retinaface.py
# return out.view(out.shape[0], -1, 2) is modified into 
return out.view(-1, int(out.size(1) * out.size(2) * 2), 2)

# line 35 in models/retinaface.py
# return out.view(out.shape[0], -1, 4) is modified into
return out.view(-1, int(out.size(1) * out.size(2) * 2), 4)

# line 46 in models/retinaface.py
# return out.view(out.shape[0], -1, 10) is modified into
return out.view(-1, int(out.size(1) * out.size(2) * 2), 10)

# The following modification ensures the output of resize node is based on scale rather than shape such that dynamic batch can be achieved.
# line 89 in models/net.py
# up3 = F.interpolate(output3, size=[output2.size(2), output2.size(3)], mode="nearest") is modified into
up3 = F.interpolate(output3, scale_factor=2, mode="nearest")

# line 93 in models/net.py
# up2 = F.interpolate(output2, size=[output1.size(2), output1.size(3)], mode="nearest") is modified into
up2 = F.interpolate(output2, scale_factor=2, mode="nearest")

# The following code removes softmax (bug sometimes happens). At the same time, concatenate the output to simplify the decoding.
# line 123 in models/retinaface.py
# if self.phase == 'train':
#     output = (bbox_regressions, classifications, ldm_regressions)
# else:
#     output = (bbox_regressions, F.softmax(classifications, dim=-1), ldm_regressions)
# return output
# the above is modified into:
output = (bbox_regressions, classifications, ldm_regressions)
return torch.cat(output, dim=-1)

# set 'opset_version=11' to ensure a successful export
# torch_out = torch.onnx._export(net, inputs, output_onnx, export_params=True, verbose=False,
#     input_names=input_names, output_names=output_names)
# is modified into:
torch_out = torch.onnx._export(net, inputs, output_onnx, export_params=True, verbose=False, opset_version=11,
    input_names=input_names, output_names=output_names)


  1. Export to onnx
python convert_to_onnx.py
  1. Execute
cp FaceDetector.onnx ../tensorRT_cpp/workspace/mb_retinaface.onnx
cd ../tensorRT_cpp
make retinaface -j64
DBFace Support
make dbface -j64
Scrfd Support
Arcface Support
commit(make_tuple(face, landmarks)).get(); cout << feature << endl; // 1x512 ">
auto arcface = Arcface::create_infer("arcface_iresnet50.fp32.trtmodel", 0);
auto feature = arcface->commit(make_tuple(face, landmarks)).get();
cout << feature << endl;  // 1x512
  • In the example of Face Recognition, workspace/face/library is the set of faces registered.
  • workspace/face/recognize is the set of face to be recognized.
  • the result is saved in workspace/face/resultworkspace/face/library_draw
CenterNet Support

check the great details in tutorial/2.0

Bert Support(Chinese Classification)

the INTRO to Interface

Python Interface:Get onnx and trtmodel from pytorch model more easily
  • Just one line of code to export onnx and trtmodel. And save them for usage in the future.
import trtpy

model = models.resnet18(True).eval()
trtpy.from_torch(
    model, 
    dummy_input, 
    max_batch_size=16, 
    onnx_save_file="test.onnx", 
    engine_save_file="engine.trtmodel"
)
Python Interface:TensorRT Inference
  • YoloX TensorRT Inference
import trtpy

yolo   = tp.Yolo(engine_file, type=tp.YoloType.X)   # engine_file is the trtmodel file
image  = cv2.imread("inference/car.jpg")
bboxes = yolo.commit(image).get()
  • Seamless Inference from Pytorch to TensorRT
import trtpy

model     = models.resnet18(True).eval().to(device) # pt model
trt_model = tp.from_torch(model, input)
trt_out   = trt_model(input)
C++ Interface:YoloX Inference
commit(image).get(); ">
// create infer engine on gpu 0
auto engine = Yolo::create_infer("yolox_m.fp32.trtmodel", Yolo::Type::X, 0);

// load image
auto image = cv::imread("1.jpg");

// do inference and get the result
auto box = engine->commit(image).get();
C++ Interface:Comple Model in FP32/FP16
TRT::compile(
  TRT::Mode::FP32,   // compile model in fp32
  3,                          // max batch size
  "plugin.onnx",              // onnx file
  "plugin.fp32.trtmodel",     // save path
  {}                         //  redefine the shape of input when needed
);
  • For fp32 compilation, all you need is offering onnx file whose input shape is allowed to be redefined.
C++ Interface:Compile in int8
  • The in8 inference performs slightly worse than fp32 in precision(about -5% drop down), but stunningly faster. In the framework, we offer int8 inference
// define int8 calibration function to read data and handle it to tenor.
auto int8process = [](int current, int count, vector
    & images, shared_ptr
    
     & tensor){
    
     for(
     int i = 
     0; i < images.
     size(); ++i){
    
     // int8 compilation requires calibration. We read image data and set_norm_mat. Then the data will be transfered into the tensor.
        
     auto image = 
     cv::imread(images[i]);
        
     cv::resize(image, image, 
     cv::Size(
     640, 
     640));
        
     float mean[] = {
     0, 
     0, 
     0};
        
     float std[]  = {
     1, 
     1, 
     1};
        tensor->
     set_norm_mat(i, image, mean, std);
    }
};



     // Specify TRT::Mode as INT8

     auto model_file = 
     "yolov5m.int8.trtmodel";

     TRT::compile(
  TRT::Mode::INT8,            
     // INT8
  
     3,                          
     // max batch size
  
     "yolov5m.onnx",             
     // onnx
  model_file,                 
     // saved filename
  {},                         
     // redefine the input shape
  int8process,                
     // the recall function for calibration
  
     ".",                        
     // the dir where the image data is used for calibration
  
     ""                          
     // the dir where the data generated from calibration is saved(a.k.a where to load the calibration data.)
);
    
   
  • We integrate into only one int8process function to save otherwise a lot of issues that might happen in tensorRT official implementation.
C++ Interface:Inference
  • We introduce class Tensor for easier inference and data transfer between host to device. So that as a user, the details wouldn't be annoying.

  • class Engine is another facilitator.

print(); // load image auto image = imread("demo.jpg"); // get the model input and output node, which can be accessed by name or index auto input = engine->input(0); // or auto input = engine->input("images"); auto output = engine->output(0); // or auto output = engine->output("output"); // put the image into input tensor by calling set_norm_mat() float mean[] = {0, 0, 0}; float std[] = {1, 1, 1}; input->set_norm_mat(i, image, mean, std); // do the inference. Here sync(true) or async(false) is optional engine->forward(); // engine->forward(true or false) // get the outut_ptr, which can used to access the output float* output_ptr = output->cpu (); ">
// load model and get a shared_ptr. get nullptr if fail to load.
auto engine = TRT::load_infer("yolov5m.fp32.trtmodel");

// print model info
engine->print();

// load image
auto image = imread("demo.jpg");

// get the model input and output node, which can be accessed by name or index
auto input = engine->input(0);   // or auto input = engine->input("images");
auto output = engine->output(0); // or auto output = engine->output("output");

// put the image into input tensor by calling set_norm_mat()
float mean[] = {0, 0, 0};
float std[]  = {1, 1, 1};
input->set_norm_mat(i, image, mean, std);

// do the inference. Here sync(true) or async(false) is optional
engine->forward(); // engine->forward(true or false)

// get the outut_ptr, which can used to access the output
float* output_ptr = output->cpu<float>();
C++ Interface:Plugin
  • You only need to define kernel function and inference process. The details of code(e.g the serialization, deserialization and injection of plugin etc) are under the hood.
  • Easy to implement a new plugin in FP32 and FP16. Refer to HSwish.cu for details.
template<>
__global__ void HSwishKernel(float* input, float* output, int edge) {

    KernelPositionBlock;
    float x = input[position];
    float a = x + 3;
    a = a < 0 ? 0 : (a >= 6 ? 6 : a);
    output[position] = x * a / 6;
}

int HSwish::enqueue(const std::vector
    & inputs, std::vector
    
     & outputs, 
     const std::vector
     
      & weights, 
      void* workspace, cudaStream_t stream) {

    
      int count = inputs[
      0].
      count();
    
      auto grid = 
      CUDATools::grid_dims(count);
    
      auto block = 
      CUDATools::block_dims(count);
    HSwishKernel <<
      
       0, stream >>> (inputs[
       0].
       ptr<
       float>(), outputs[
       0].
       ptr<
       float>(), count);
    
       return 
       0;
}



       RegisterPlugin(HSwish);
      
     
    
   

About Us

Comments
  • no result in int8 mode

    no result in int8 mode

    Hi, thanks for your awesome code share!

    I test this repo on my Ubuntu 18 Linux with CUDA V11.2.152, TensorRT-8.0.0.3 and cudnn8.2.0.

    I tested yolox_s model in FP32、FP16 and INT8 mode, but in INT8 mode, the network output has no results.

    FP16 test code is below:

    static void test_fp16(Yolo::Type type){
    
        TRT::set_device(0);
        INFO("===================== test %s fp16 ==================================", Yolo::type_name(type));
    
        const char* name = nullptr;
        if(type == Yolo::Type::V5){
            name = "yolov5m";
        }else if(type == Yolo::Type::X){
            name = "yolox_s";
        }
    
        if(not requires(name))
            return;
    
        string onnx_file = iLogger::format("%s.onnx", name);
        string model_file = iLogger::format("%s.fp16.trtmodel", name);
        int test_batch_size = 1;  // 当你需要修改batch大于1时,请注意你的模型是否修改(看readme.md代码修改部分),否则会有错误
        
        // 动态batch和静态batch,如果你想要弄清楚,请打开http://www.zifuture.com:8090/
        // 找到右边的二维码,扫码加好友后进群交流(免费哈,就是技术人员一起沟通)
        if(not iLogger::exists(model_file)){
            TRT::compile(
                TRT::TRTMode_FP16,   // 编译方式有,FP32、FP16、INT8
                {},                         // onnx时无效,caffe的输出节点标记
                test_batch_size,            // 指定编译的batch size
                onnx_file,                  // 需要编译的onnx文件
                model_file,                 // 储存的模型文件
                {},                         // 指定需要重定义的输入shape,这里可以对onnx的输入shape进行重定义
                false                       // 是否采用动态batch维度,true采用,false不采用,使用静态固定的batch size
            );
        }
    
        forward_engine(model_file, type);
    }
    

    Below is the output of the log:

    [2021-09-01 19:38:45][info][app_yolo.cpp:240]:===================== test YoloX fp32 ==================================
    [2021-09-01 19:38:45][info][trt_builder.cpp:473]:Compile FP32 Onnx Model 'yolox_s.onnx'.
    [2021-09-01 19:38:45][info][trt_builder.cpp:602]:Input shape is 1 x 3 x 640 x 640
    [2021-09-01 19:38:45][info][trt_builder.cpp:603]:Set max batch size = 1
    [2021-09-01 19:38:45][info][trt_builder.cpp:604]:Set max workspace size = 1024.00 MB
    [2021-09-01 19:38:45][info][trt_builder.cpp:605]:Dynamic batch dimension is false
    [2021-09-01 19:38:45][info][trt_builder.cpp:608]:Network has 1 inputs:
    [2021-09-01 19:38:45][info][trt_builder.cpp:614]:      0.[images] shape is 1 x 3 x 640 x 640
    [2021-09-01 19:38:45][info][trt_builder.cpp:620]:Network has 1 outputs:
    [2021-09-01 19:38:45][info][trt_builder.cpp:625]:      0.[output] shape is 1 x 8400 x 85
    [2021-09-01 19:38:45][info][trt_builder.cpp:670]:Building engine...
    [2021-09-01 19:38:45][warn][trt_builder.cpp:33]:NVInfer WARNING: Convolution + generic activation fusion is disable due to incompatible driver or nvrtc
    [2021-09-01 19:38:46][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:38:46][warn][trt_builder.cpp:33]:NVInfer WARNING: Detected invalid timing cache, setup a local cache instead
    [2021-09-01 19:39:18][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:18][info][trt_builder.cpp:690]:Build done 32689 ms !
    [2021-09-01 19:39:18][warn][trt_builder.cpp:33]:NVInfer WARNING: The logger passed into createInferRuntime differs from one already assigned, 0x557f9ae330b0, logger not updated.
    
    [2021-09-01 19:39:19][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:19][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:19][info][trt_infer.cpp:169]:Infer 0x7fe3f8000c40 detail
    [2021-09-01 19:39:19][info][trt_infer.cpp:170]: Max Batch Size: 1
    [2021-09-01 19:39:19][info][trt_infer.cpp:171]: Dynamic Batch Dimension: false
    [2021-09-01 19:39:19][info][trt_infer.cpp:172]: Inputs: 1
    [2021-09-01 19:39:19][info][trt_infer.cpp:176]:         0.images : shape {1 x 3 x 640 x 640}
    [2021-09-01 19:39:19][info][trt_infer.cpp:179]: Outputs: 1
    [2021-09-01 19:39:19][info][trt_infer.cpp:183]:         0.output : shape {1 x 8400 x 85}
    [2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/car.jpg, 2 object, 10.10 ms
    [2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/group.jpg, 1 object, 6.19 ms
    [2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/zand.jpg, 2 object, 8.72 ms
    [2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/zgjr.jpg, 3 object, 6.03 ms
    [2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/gril.jpg, 1 object, 5.95 ms
    [2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/yq.jpg, 1 object, 5.92 ms
    [2021-09-01 19:39:19][info][yolo.cpp:214]:Engine destroy.
    [2021-09-01 19:39:19][info][app_yolo.cpp:277]:===================== test YoloX fp16 ==================================
    [2021-09-01 19:39:19][info][trt_builder.cpp:473]:Compile FP16 Onnx Model 'yolox_s.onnx'.
    [2021-09-01 19:39:19][warn][trt_builder.cpp:483]:Platform not have fast fp16 support
    [2021-09-01 19:39:19][info][trt_builder.cpp:602]:Input shape is 1 x 3 x 640 x 640
    [2021-09-01 19:39:19][info][trt_builder.cpp:603]:Set max batch size = 1
    [2021-09-01 19:39:19][info][trt_builder.cpp:604]:Set max workspace size = 1024.00 MB
    [2021-09-01 19:39:19][info][trt_builder.cpp:605]:Dynamic batch dimension is false
    [2021-09-01 19:39:19][info][trt_builder.cpp:608]:Network has 1 inputs:
    [2021-09-01 19:39:19][info][trt_builder.cpp:614]:      0.[images] shape is 1 x 3 x 640 x 640
    [2021-09-01 19:39:19][info][trt_builder.cpp:620]:Network has 1 outputs:
    [2021-09-01 19:39:19][info][trt_builder.cpp:625]:      0.[output] shape is 1 x 8400 x 85
    [2021-09-01 19:39:19][info][trt_builder.cpp:670]:Building engine...
    [2021-09-01 19:39:19][warn][trt_builder.cpp:33]:NVInfer WARNING: Half2 support requested on hardware without native FP16 support, performance will be negatively affected.
    [2021-09-01 19:39:19][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:19][warn][trt_builder.cpp:33]:NVInfer WARNING: Detected invalid timing cache, setup a local cache instead
    [2021-09-01 19:39:47][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:47][info][trt_builder.cpp:690]:Build done 28282 ms !
    [2021-09-01 19:39:47][warn][trt_builder.cpp:33]:NVInfer WARNING: The logger passed into createInferRuntime differs from one already assigned, 0x557f9ae330b0, logger not updated.
    
    [2021-09-01 19:39:48][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:48][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:48][info][trt_infer.cpp:169]:Infer 0x7fe3f8015650 detail
    [2021-09-01 19:39:48][info][trt_infer.cpp:170]: Max Batch Size: 1
    [2021-09-01 19:39:48][info][trt_infer.cpp:171]: Dynamic Batch Dimension: false
    [2021-09-01 19:39:48][info][trt_infer.cpp:172]: Inputs: 1
    [2021-09-01 19:39:48][info][trt_infer.cpp:176]:         0.images : shape {1 x 3 x 640 x 640}
    [2021-09-01 19:39:48][info][trt_infer.cpp:179]: Outputs: 1
    [2021-09-01 19:39:48][info][trt_infer.cpp:183]:         0.output : shape {1 x 8400 x 85}
    [2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/car.jpg, 2 object, 10.75 ms
    [2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/group.jpg, 1 object, 6.38 ms
    [2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/zand.jpg, 2 object, 9.12 ms
    [2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/zgjr.jpg, 3 object, 6.10 ms
    [2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/gril.jpg, 1 object, 6.07 ms
    [2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/yq.jpg, 1 object, 6.01 ms
    [2021-09-01 19:39:48][info][yolo.cpp:214]:Engine destroy.
    [2021-09-01 19:39:48][info][app_yolo.cpp:190]:===================== test YoloX int8 ==================================
    [2021-09-01 19:39:48][info][trt_builder.cpp:473]:Compile INT8 Onnx Model 'yolox_s.onnx'.
    [2021-09-01 19:39:48][info][trt_builder.cpp:593]:Using image list[6 files]: inference
    [2021-09-01 19:39:48][info][trt_builder.cpp:602]:Input shape is 1 x 3 x 640 x 640
    [2021-09-01 19:39:48][info][trt_builder.cpp:603]:Set max batch size = 1
    [2021-09-01 19:39:48][info][trt_builder.cpp:604]:Set max workspace size = 1024.00 MB
    [2021-09-01 19:39:48][info][trt_builder.cpp:605]:Dynamic batch dimension is false
    [2021-09-01 19:39:48][info][trt_builder.cpp:608]:Network has 1 inputs:
    [2021-09-01 19:39:48][info][trt_builder.cpp:614]:      0.[images] shape is 1 x 3 x 640 x 640
    [2021-09-01 19:39:48][info][trt_builder.cpp:620]:Network has 1 outputs:
    [2021-09-01 19:39:48][info][trt_builder.cpp:625]:      0.[output] shape is 1 x 8400 x 85
    [2021-09-01 19:39:48][info][trt_builder.cpp:670]:Building engine...
    [2021-09-01 19:39:48][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:49][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:49][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:49][info][app_yolo.cpp:193]:Int8 1 / 6
    [2021-09-01 19:39:49][info][app_yolo.cpp:193]:Int8 2 / 6
    [2021-09-01 19:39:49][info][app_yolo.cpp:193]:Int8 3 / 6
    [2021-09-01 19:39:49][info][app_yolo.cpp:193]:Int8 4 / 6
    [2021-09-01 19:39:49][info][app_yolo.cpp:193]:Int8 5 / 6
    [2021-09-01 19:39:50][info][app_yolo.cpp:193]:Int8 6 / 6
    [2021-09-01 19:40:27][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:40:27][warn][trt_builder.cpp:33]:NVInfer WARNING: Detected invalid timing cache, setup a local cache instead
    [2021-09-01 19:40:59][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:40:59][info][trt_builder.cpp:685]:No set entropyCalibratorFile, and entropyCalibrator will not save.
    [2021-09-01 19:40:59][info][trt_builder.cpp:690]:Build done 70917 ms !
    [2021-09-01 19:40:59][warn][trt_builder.cpp:33]:NVInfer WARNING: The logger passed into createInferRuntime differs from one already assigned, 0x557f9ae330b0, logger not updated.
    
    [2021-09-01 19:40:59][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:40:59][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:40:59][info][trt_infer.cpp:169]:Infer 0x7fe3b4000c40 detail
    [2021-09-01 19:40:59][info][trt_infer.cpp:170]: Max Batch Size: 1
    [2021-09-01 19:40:59][info][trt_infer.cpp:171]: Dynamic Batch Dimension: false
    [2021-09-01 19:40:59][info][trt_infer.cpp:172]: Inputs: 1
    [2021-09-01 19:40:59][info][trt_infer.cpp:176]:         0.images : shape {1 x 3 x 640 x 640}
    [2021-09-01 19:40:59][info][trt_infer.cpp:179]: Outputs: 1
    [2021-09-01 19:40:59][info][trt_infer.cpp:183]:         0.output : shape {1 x 8400 x 85}
    [2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/car.jpg, 0 object, 7.75 ms
    [2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/group.jpg, 0 object, 4.04 ms
    [2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/zand.jpg, 0 object, 6.04 ms
    [2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/zgjr.jpg, 0 object, 3.74 ms
    [2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/gril.jpg, 0 object, 3.72 ms
    [2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/yq.jpg, 0 object, 3.74 ms
    [2021-09-01 19:40:59][info][yolo.cpp:214]:Engine destroy.
    

    Time-consuming in the FP16/FP32 modes is close, is it because the code I wrote has a problem?

    Your sincerely!

    opened by qixuxiang 14
  • AffineMatrix求解?

    AffineMatrix求解?

    正常图像作缩放是不需要偏移的吧? M = [ scale, 0, -scale * from.width * 0.5 + to.width * 0.5 0, scale, -scale * from.height * 0.5 + to.height * 0.5 0, 0, 1 ] -scale * from.width * 0.5 + to.width * 0.5这个偏移量抵消了,

    但是第一次平移矩阵不是 P = [ 1, 0, -scale * from.width * 0.5 0, -1, scale * from.height * 0.5 0, 0, 1 ] 第二次平移矩阵不应该是: T = [ 1, 0, to.width * 0.5, 0, - 1, to.height * 0.5, 0, 0, 1 ]

    ??? 参考的矩阵: https://www.cnblogs.com/xuanyuyt/p/7112876.html

    opened by cqray1990 8
  • Environment Configuration Issues!

    Environment Configuration Issues!

    QQ截图20211119174416 QQ截图20211119173519 I have never used VS Code before. I want to learn from the video of the blogger, but there are always errors. Is there any solution?(I have installed Tensorrt on Windows)

    opened by Havehandssook 6
  • arcface人脸特征提取不同人脸,计算的相似度都一样,是怎么回事呢?

    arcface人脸特征提取不同人脸,计算的相似度都一样,是怎么回事呢?

    余弦距离计算代码: def cosine_distance(matrix1, matrix2): matrix1_matrix2 = np.dot(matrix1, matrix2.transpose()) matrix1_norm = np.sqrt(np.multiply(matrix1, matrix1).sum(axis=1)) matrix1_norm = matrix1_norm[:, np.newaxis] matrix2_norm = np.sqrt(np.multiply(matrix2, matrix2).sum(axis=1)) matrix2_norm = matrix2_norm[:, np.newaxis] cosine_distance = np.divide(matrix1_matrix2, np.dot(matrix1_norm, matrix2_norm.transpose())) return cosine_distance

    是不是没有进行归一化呢,归一化有没有好的函数推荐呢?

    opened by seawater668 6
  • Error,arcface onnx 转tensorrt 错误

    Error,arcface onnx 转tensorrt 错误

    使用了本项目提供给的arcface_iresnet50.onnx 模型
    错误如下:
    While parsing node number 182 [BatchNormalization]:
    ERROR: /home/Project/tensorRT_Pro-main/src/tensorRT/onnx_parser/onnx2trt_utils.cpp:1523 In function scaleHelper:
    [8] Assertion failed: dims.nbDims == 4 || dims.nbDims == 5
    [2021-10-04 16:39:25][error][trt_builder.cpp:517]:Can not parse OnnX file: arcface_iresnet50_iii.onnx
    

    计算机环境如下:

    tensorrt 7.2
    cudnn8.1
    cuda 11.2
    protobufv3.11.4
    gpu 3080  arch86
    

    BatchNormalization_182 是模型的倒数第二层。

    opened by create-li 6
  • preprocess  engine->forward  decode_kernel_invoker  线程问题

    preprocess engine->forward decode_kernel_invoker 线程问题

    老师,您好,preprocess 在commits,是在主线程执行,而decode_kernel_invoker是在work 函数中和forward在一个线程中执行,为什么会这么设计呀? 如果送入的图片存在高并发问题是否会导致主线程卡死在commits中,为什么不把preprocess过程和decode一样放在work线程中执行?

    然后我有个想法是不是可以把preprocess 和decode 剥离出来,通过多线程实现, work只做forward的工作,这样是不是更高效?

    opened by QZ1219 4
  • cmake编译失败

    cmake编译失败

    cmake能全部正常build,但是在make的时候正常编译到85%的时候报错了,以下是报错内容(我未修改报错文件内的代码)

    [ 83%] Building CXX object CMakeFiles/pro.dir/src/application/app_yolo_fast.cpp.o [ 83%] Building CXX object CMakeFiles/pro.dir/src/application/app_alphapose.cpp.o [ 85%] Building CXX object CMakeFiles/pro.dir/src/application/app_python/interface.cpp.o In file included from /xxx/datav/projects/tensorRT_Pro/src/application/app_python/interface.cpp:6:0: /xxx/datav/projects/tensorRT_Pro/src/application/tools/pybind11.hpp:159:20: fatal error: Python.h: No such file or directory #include <Python.h> ^ compilation terminated. CMakeFiles/pro.dir/build.make:1142: recipe for target 'CMakeFiles/pro.dir/src/application/app_python/interface.cpp.o' failed make[3]: *** [CMakeFiles/pro.dir/src/application/app_python/interface.cpp.o] Error 1 make[3]: *** Waiting for unfinished jobs.... CMakeFiles/Makefile2:328: recipe for target 'CMakeFiles/pro.dir/all' failed make[2]: *** [CMakeFiles/pro.dir/all] Error 2 CMakeFiles/Makefile2:537: recipe for target 'CMakeFiles/yolo.dir/rule' failed make[1]: *** [CMakeFiles/yolo.dir/rule] Error 2 Makefile:300: recipe for target 'yolo' failed make: *** [yolo] Error 2

    opened by sungh66 4
  • protobuf编译有问题

    protobuf编译有问题

    make报错

    g++: error: google/protobuf/util/internal/.libs/proto_writer.o: No such file or directory
    Makefile:2372: recipe for target 'libprotobuf.la' failed
    make[2]: *** [libprotobuf.la] Error 1
    

    是不是要先下载googletest,如果是的话,是不是哪个版本呢? ubuntu18.04 jetson xavier nx

    opened by DCC-lzhy 4
  • ][trt_builder.cpp:36]:NVInfer: TensorRT was linked against cuBLAS/cuBLAS LT 11.6.3 but loaded cuBLAS/cuBLAS LT 11.3.1

    ][trt_builder.cpp:36]:NVInfer: TensorRT was linked against cuBLAS/cuBLAS LT 11.6.3 but loaded cuBLAS/cuBLAS LT 11.3.1

    warning, errors are showed above, it is annoying and weird to concern the compatibility issues among tensorrt version and cuda, and cuda toolkit versions. i cannot figure out the difference among them, any help will be approciated!!!

    i have installed tensorrt !!!

    dpkg -l | grep tensorrt ii nv-tensorrt-repo-ubuntu1804-cuda10.0-trt5.0.2.6-ga-20181009 1-1 amd64 nv-tensorrt repository configuration files ii nv-tensorrt-repo-ubuntu1804-cuda10.1-trt5.1.5.0-ga-20190427 1-1 amd64 nv-tensorrt repository configuration files ii nv-tensorrt-repo-ubuntu1804-cuda11.4-trt8.2.1.8-ga-20211117 1-1 amd64 nv-tensorrt repository configuration files ii tensorrt 8.2.1.8-1+cuda11.4 amd64

    it should be 8.2.18 cuda11.4 as i am concerned.

    and after i typed nvcc -V , the cuda version is nvcc as follows:

    NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2020 NVIDIA Corporation Built on Mon_Nov_30_19:08:53_PST_2020 Cuda compilation tools, release 11.2, V11.2.67 Build cuda_11.2.r11.2/compiler.29373293_0

    so which version of toolkit should i install??? currently, my cuda toolkit is v11.5.0 i think

    opened by sainttelant 4
  • yolox master分支的模型能够正常导出到onnx但生成engine失败

    yolox master分支的模型能够正常导出到onnx但生成engine失败

    [2021-10-26 14:29:01][info][app_yolo.cpp:121]:===================== test YoloX FP32 yolox_s ================================== [2021-10-26 14:29:01][info][trt_builder.cpp:471]:Compile FP32 Onnx Model 'yolox_s.onnx'. [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: INVALID_ARGUMENT: getPluginCreator could not find plugin ScatterND version 1 While parsing node number 565 [ScatterND]: ERROR: /home/work/tracking/tensorRT_Pro/src/tensorRT/onnx_parser/builtin_op_importers.cpp:4013 In function importFallbackPluginImporter: [8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?" [2021-10-26 14:29:02][error][trt_builder.cpp:517]:Can not parse OnnX file: yolox_s.onnx [2021-10-26 14:29:02][error][yolo.cpp:138]:Engine yolox_s.FP32.trtmodel load failed [2021-10-26 14:29:02][error][app_yolo.cpp:42]:Engine is nullptr [100%] Built target yolo

    opened by deep-practice 4
  • 对输入输出的内存申请为啥要加个size_matrix 仿射矩阵的大小,而且对输出affine_matrix_device的内存分配是sizeof(job.additional.d2i)

    对输入输出的内存申请为啥要加个size_matrix 仿射矩阵的大小,而且对输出affine_matrix_device的内存分配是sizeof(job.additional.d2i)

            uint8_t* gpu_workspace        = (uint8_t*)workspace->gpu(size_matrix + size_image);
            float*   affine_matrix_device = (float*)gpu_workspace;
            uint8_t* image_device         = size_matrix + gpu_workspace;
    
            uint8_t* cpu_workspace        = (uint8_t*)workspace->cpu(size_matrix + size_image);
            float* affine_matrix_host     = (float*)cpu_workspace;
            uint8_t* image_host           = size_matrix + cpu_workspace;
           memcpy(image_host, image.data, size_image);
            memcpy(affine_matrix_host, job.additional.d2i, sizeof(job.additional.d2i));
            checkCudaRuntime(cudaMemcpyAsync(image_device, image_host, size_image, cudaMemcpyHostToDevice, preprocess_stream));
            checkCudaRuntime(cudaMemcpyAsync(affine_matrix_device, affine_matrix_host, sizeof(job.additional.d2i), cudaMemcpyHostToDevice, preprocess_stream));
    
    opened by cqray1990 3
  • 请问下是否支持arm

    请问下是否支持arm

    CUDA Runtime error cudaSetDevice(device_id) # initialization error, code = cudaErrorInitializationError [ 3 ] in file /src/tensorRT/infer/trt_infer.cpp:472 请问下代码是否支持 这个是在jetson nx编译通过后,转engine是报错,麻烦大佬帮忙看看

    opened by Waynepoo 0
  • rtsp视频流解码报错

    rtsp视频流解码报错

    报错如下,这个错误影响解码吗

    [h264 @ 0x7f918c007980] concealing 1466 DC, 1466 AC, 1466 MV errors in I frame
    [rtsp @ 0x7f918c0021c0] RTP Xiph packet settings (0,1,0) is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
    Decode Error occurred for picture 0
    
    opened by ZJU-lishuang 0
  • onnx转trt报错

    onnx转trt报错

    Hi [2022-12-08 17:36:09][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:09][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:09][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: 9: [graphShapeAnalyzer.cpp::nvinfer1::builder::`anonymous-namespace'::ShapeNodeRemover::throwIfError::1306] Error Code 9: Internal Error (Reshape_1087: reshape changes volume ) [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: g:\chengc\project\tensorrt-onnx-fasterrcnn-fpn-roialign-master\tensorrt_code\src\tensorrt\onnx_parser\modelimporter.cpp:736: While parsing node number 364 [Add -> "1963"]: [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: g:\chengc\project\tensorrt-onnx-fasterrcnn-fpn-roialign-master\tensorrt_code\src\tensorrt\onnx_parser\modelimporter.cpp:737: --- Begin node --- [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: g:\chengc\project\tensorrt-onnx-fasterrcnn-fpn-roialign-master\tensorrt_code\src\tensorrt\onnx_parser\modelimporter.cpp:738: input: "1935" input: "1940" output: "1963" name: "Add_1095" op_type: "Add"

    [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: g:\chengc\project\tensorrt-onnx-fasterrcnn-fpn-roialign-master\tensorrt_code\src\tensorrt\onnx_parser\modelimporter.cpp:739: --- End node --- [2022-12-08 17:36:10][error][trt_builder.cpp:30]:NVInfer: g:\chengc\project\tensorrt-onnx-fasterrcnn-fpn-roialign-master\tensorrt_code\src\tensorrt\onnx_parser\modelimporter.cpp:742: ERROR: g:\chengc\project\tensorrt-onnx-fasterrcnn-fpn-roialign-master\tensorrt_code\src\tensorrt\onnx_parser\onnx2trt_utils.cpp:959 In function elementwiseHelper: [8] Assertion failed: tensor_ptr->getDimensions().nbDims == maxNbDims && "Failed to broadcast tensors elementwise!"

    opened by Chengcheng1998727 1
Releases(v1.0)
Owner
手写AI
手写AI
Python Inference Script is a Python package that enables developers to author machine learning workflows in Python and deploy without Python.

Python Inference Script(PyIS) Python Inference Script is a Python package that enables developers to author machine learning workflows in Python and d

Microsoft 13 Nov 4, 2022
TensorRT is a C++ library for high performance inference on NVIDIA GPUs and deep learning accelerators.

TensorRT Open Source Software This repository contains the Open Source Software (OSS) components of NVIDIA TensorRT. Included are the sources for Tens

NVIDIA Corporation 6.4k Jan 4, 2023
ncnn is a high-performance neural network inference framework optimized for the mobile platform

ncnn ncnn is a high-performance neural network inference computing framework optimized for mobile platforms. ncnn is deeply considerate about deployme

Tencent 16.2k Jan 5, 2023
Hardware-accelerated DNN model inference ROS2 packages using NVIDIA Triton/TensorRT for both Jetson and x86_64 with CUDA-capable GPU.

Isaac ROS DNN Inference Overview This repository provides two NVIDIA GPU-accelerated ROS2 nodes that perform deep learning inference using custom mode

NVIDIA Isaac ROS 62 Dec 14, 2022
Helper Class for Deep Learning Inference Frameworks: TensorFlow Lite, TensorRT, OpenCV, ncnn, MNN, SNPE, Arm NN, NNAbla

InferenceHelper This is a helper class for deep learning frameworks especially for inference This class provides an interface to use various deep lear

iwatake 192 Dec 26, 2022
Forward - A library for high performance deep learning inference on NVIDIA GPUs

a library for high performance deep learning inference on NVIDIA GPUs.

Tencent 123 Mar 17, 2021
A library for high performance deep learning inference on NVIDIA GPUs.

Forward - A library for high performance deep learning inference on NVIDIA GPUs Forward - A library for high performance deep learning inference on NV

Tencent 509 Dec 17, 2022
Edge ML Library - High-performance Compute Library for On-device Machine Learning Inference

Edge ML Library (EMLL) offers optimized basic routines like general matrix multiplications (GEMM) and quantizations, to speed up machine learning (ML) inference on ARM-based devices. EMLL supports fp32, fp16 and int8 data types. EMLL accelerates on-device NMT, ASR and OCR engines of Youdao, Inc.

NetEase Youdao 179 Dec 20, 2022
PPLNN is a high-performance deep-learning inference engine for efficient AI inferencing.

PPLNN, which is short for "PPLNN is a Primitive Library for Neural Network", is a high-performance deep-learning inference engine for efficient AI inferencing.

null 939 Dec 29, 2022
Benchmark framework of 3D integrated CIM accelerators for popular DNN inference, support both monolithic and heterogeneous 3D integration

3D+NeuroSim V1.0 The DNN+NeuroSim framework was developed by Prof. Shimeng Yu's group (Georgia Institute of Technology). The model is made publicly av

NeuroSim 11 Dec 15, 2022
Experimental and Comparative Performance Measurements of High Performance Computing Based on OpenMP and MPI

High-Performance-Computing-Experiments Experimental and Comparative Performance Measurements of High Performance Computing Based on OpenMP and MPI 实验结

Jiang Lu 1 Nov 27, 2021
Implement yolov5 with Tensorrt C++ api, and integrate batchedNMSPlugin. A Python wrapper is also provided.

yolov5 Original codes from tensorrtx. I modified the yololayer and integrated batchedNMSPlugin. A yolov5s.wts is provided for fast demo. How to genera

weiwei zhou 46 Dec 6, 2022
Deep Learning API and Server in C++11 support for Caffe, Caffe2, PyTorch,TensorRT, Dlib, NCNN, Tensorflow, XGBoost and TSNE

Open Source Deep Learning Server & API DeepDetect (https://www.deepdetect.com/) is a machine learning API and server written in C++11. It makes state

JoliBrain 2.4k Dec 30, 2022
Support Yolov4/Yolov3/Centernet/Classify/Unet. use darknet/libtorch/pytorch to onnx to tensorrt

ONNX-TensorRT Yolov4/Yolov3/CenterNet/Classify/Unet Implementation Yolov4/Yolov3 centernet INTRODUCTION you have the trained model file from the darkn

null 172 Dec 29, 2022
A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.

Light Gradient Boosting Machine LightGBM is a gradient boosting framework that uses tree based learning algorithms. It is designed to be distributed a

Microsoft 14.5k Jan 5, 2023
APFS module for linux, with experimental write support (out-of-tree repository)

Apple File System ================= The Apple File System (APFS) is the copy-on-write filesystem currently used on all Apple devices. This module pro

APFS for Linux 260 Jan 4, 2023
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

Website | Documentation | Tutorials | Installation | Release Notes CatBoost is a machine learning method based on gradient boosting over decision tree

CatBoost 6.9k Dec 31, 2022
High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

What is xLearn? xLearn is a high performance, easy-to-use, and scalable machine learning package that contains linear model (LR), factorization machin

Chao Ma 3k Dec 23, 2022
KSAI Lite is a deep learning inference framework of kingsoft, based on tensorflow lite

KSAI Lite English | 简体中文 KSAI Lite是一个轻量级、灵活性强、高性能且易于扩展的深度学习推理框架,底层基于tensorflow lite,定位支持包括移动端、嵌入式以及服务器端在内的多硬件平台。 当前KSAI Lite已经应用在金山office内部业务中,并逐步支持金山

null 80 Dec 27, 2022