Mmdeploy - OpenMMLab Model Deployment Framework

Overview

docs badge codecov license issue resolution open issues

Introduction

English | 简体中文

MMDeploy is an open-source deep learning model deployment toolset. It is a part of the OpenMMLab project.

Major features

  • Fully support OpenMMLab models

    We provide a unified model deployment toolbox for the codebases in OpenMMLab. The supported codebases are listed as below, and more will be added in the future

    • MMClassification
    • MMDetection
    • MMSegmentation
    • MMEditing
    • MMOCR
  • Multiple inference backends are available

    Models can be exported and run in different backends. The following ones are supported, and more will be taken into consideration

    • ONNX Runtime
    • TensorRT
    • PPLNN
    • ncnn
    • OpenVINO
  • Efficient and highly scalable SDK Framework by C/C++

    All kinds of modules in SDK can be extensible, such as Transform for image processing, Net for Neural Network inference, Module for postprocessing and so on

License

This project is released under the Apache 2.0 license.

Installation

Please refer to build.md for installation.

Getting Started

Please see getting_started.md for the basic usage of MMDeploy. We also provide other tutorials for:

Please refer to FAQ for frequently asked questions.

Benchmark and model zoo

Results and supported model list are available in the benchmark and model list.

Contributing

We appreciate all contributions to improve MMDeploy. Please refer to CONTRIBUTING.md for the contributing guideline.

Acknowledgement

We would like to sincerely thank the following teams for their contributions to MMDeploy:

Citation

If you find this project useful in your research, please consider cite:

@misc{=mmdeploy,
    title={OpenMMLab's Model Deployment Toolbox.},
    author={MMDeploy Contributors},
    howpublished = {\url{https://github.com/open-mmlab/mmdeploy}},
    year={2021}
}

Projects in OpenMMLab

  • MMCV: OpenMMLab foundational library for computer vision.
  • MIM: MIM Installs OpenMMLab Packages.
  • MMClassification: OpenMMLab image classification toolbox and benchmark.
  • MMDetection: OpenMMLab detection toolbox and benchmark.
  • MMDetection3D: OpenMMLab's next-generation platform for general 3D object detection.
  • MMSegmentation: OpenMMLab semantic segmentation toolbox and benchmark.
  • MMAction2: OpenMMLab's next-generation action understanding toolbox and benchmark.
  • MMTracking: OpenMMLab video perception toolbox and benchmark.
  • MMPose: OpenMMLab pose estimation toolbox and benchmark.
  • MMEditing: OpenMMLab image and video editing toolbox.
  • MMOCR: A Comprehensive Toolbox for Text Detection, Recognition and Understanding.
  • MMGeneration: OpenMMLab image and video generative models toolbox.
  • MMFlow: OpenMMLab optical flow toolbox and benchmark.
  • MMFewShot: OpenMMLab FewShot Learning Toolbox and Benchmark.
  • MMHuman3D: OpenMMLab Human Pose and Shape Estimation Toolbox and Benchmark.
  • MMSelfSup: OpenMMLab self-supervised learning Toolbox and Benchmark.
  • MMRazor: OpenMMLab Model Compression Toolbox and Benchmark.
  • MMDeploy: OpenMMLab Model Deployment Framework.
Issues
  • Tensor size mismatch

    Tensor size mismatch

    Hello, I am trying to convert a CascadeNet pth model into onnx and getting the same error:

    [email protected]:~/workspace/mmdeploy# python ./tools/deploy.py configs/mmdet/detection/detection_onnxruntime_dynamic.py cascade_mask_rcnn_hrnetv2p_w32_20e.py General.Model.table.detection.v2.pth demo.png [2022-05-19 16:21:23.549] [mmdeploy] [info] Register 'DirectoryModel' 2022-05-19 16:21:23,618 - mmdeploy - INFO - torch2onnx start. [2022-05-19 16:21:24.866] [mmdeploy] [info] Register 'DirectoryModel' /opt/conda/lib/python3.7/site-packages/mmdet/models/builder.py:53: UserWarning: train_cfg and test_cfg is deprecated, please specify them in model 'please specify them in model', UserWarning) load checkpoint from local path: General.Model.table.detection.v2.pth /opt/conda/lib/python3.7/site-packages/mmdet/datasets/utils.py:69: UserWarning: "ImageToTensor" pipeline is replaced by "DefaultFormatBundle" for batch inference. It is recommended to manually replace it in the test data pipeline in your config file. 'data pipeline in your config file.', UserWarning) 2022-05-19 16:21:32,430 - mmdeploy - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future. 2022-05-19:16:21:32,mmdeploy WARNING [utils.py:92] DeprecationWarning: get_onnx_config will be deprecated in the future. /root/workspace/mmdeploy/mmdeploy/core/optimizers/function_marker.py:158: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! ys_shape = tuple(int(s) for s in ys.shape) /opt/conda/lib/python3.7/site-packages/torch/nn/functional.py:3455: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details. "See the documentation of nn.Upsample for details.".format(mode) /opt/conda/lib/python3.7/site-packages/mmdet/models/dense_heads/anchor_head.py:123: UserWarning: DeprecationWarning: anchor_generator is deprecated, please use "prior_generator" instead warnings.warn('DeprecationWarning: anchor_generator is deprecated, ' /opt/conda/lib/python3.7/site-packages/mmdet/core/anchor/anchor_generator.py:333: UserWarning: grid_anchors would be deprecated soon. Please use grid_priors warnings.warn('grid_anchors would be deprecated soon. ' /opt/conda/lib/python3.7/site-packages/mmdet/core/anchor/anchor_generator.py:370: UserWarning: single_level_grid_anchors would be deprecated soon. Please use single_level_grid_priors 'single_level_grid_anchors would be deprecated soon. ' /root/workspace/mmdeploy/mmdeploy/codebase/mmdet/models/dense_heads/rpn_head.py:78: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! assert cls_score.size()[-2:] == bbox_pred.size()[-2:] /root/workspace/mmdeploy/mmdeploy/pytorch/functions/topk.py:28: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect. k = torch.tensor(k, device=input.device, dtype=torch.long) /root/workspace/mmdeploy/mmdeploy/pytorch/functions/topk.py:33: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! return ctx.origin_func(input, k, dim=dim, largest=largest, sorted=sorted) /opt/conda/lib/python3.7/site-packages/mmdet/core/bbox/coder/legacy_delta_xywh_bbox_coder.py:77: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! assert pred_bboxes.size(0) == bboxes.size(0) 2022-05-19:16:21:35,root ERROR [utils.py:43] The size of tensor a (4) must match the size of tensor b (4432) at non-singleton dimension 2 Traceback (most recent call last): File "/root/workspace/mmdeploy/mmdeploy/utils/utils.py", line 38, in target_wrapper result = target(*args, **kwargs) File "/root/workspace/mmdeploy/mmdeploy/apis/pytorch2onnx.py", line 113, in torch2onnx output_file=output_file) File "/root/workspace/mmdeploy/mmdeploy/apis/pytorch2onnx.py", line 55, in torch2onnx_impl verbose=verbose) File "/opt/conda/lib/python3.7/site-packages/torch/onnx/init.py", line 276, in export custom_opsets, enable_onnx_checker, use_external_data_format) File "/opt/conda/lib/python3.7/site-packages/torch/onnx/utils.py", line 94, in export use_external_data_format=use_external_data_format) File "/opt/conda/lib/python3.7/site-packages/torch/onnx/utils.py", line 698, in _export dynamic_axes=dynamic_axes) File "/opt/conda/lib/python3.7/site-packages/torch/onnx/utils.py", line 456, in _model_to_graph use_new_jit_passes) File "/opt/conda/lib/python3.7/site-packages/torch/onnx/utils.py", line 417, in _create_jit_graph graph, torch_out = _trace_and_get_graph_from_model(model, args) File "/opt/conda/lib/python3.7/site-packages/torch/onnx/utils.py", line 377, in _trace_and_get_graph_from_model torch.jit._get_trace_graph(model, args, strict=False, _force_outplace=False, _return_inputs_states=True) File "/opt/conda/lib/python3.7/site-packages/torch/jit/_trace.py", line 1139, in _get_trace_graph outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs) File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/opt/conda/lib/python3.7/site-packages/torch/jit/_trace.py", line 130, in forward self._force_outplace, File "/opt/conda/lib/python3.7/site-packages/torch/jit/_trace.py", line 116, in wrapper outs.append(self.inner(*trace_inputs)) File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 887, in _call_impl result = self._slow_forward(*input, **kwargs) File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 860, in _slow_forward result = self.forward(*input, **kwargs) File "/root/workspace/mmdeploy/mmdeploy/core/rewriters/rewriter_utils.py", line 371, in wrapper return self.func(self, *args, **kwargs) File "/root/workspace/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/base.py", line 69, in base_detector__forward return __forward_impl(ctx, self, img, img_metas=img_metas, **kwargs) File "/root/workspace/mmdeploy/mmdeploy/core/optimizers/function_marker.py", line 261, in g rets = f(*args, **kwargs) File "/root/workspace/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/base.py", line 28, in __forward_impl return self.simple_test(img, img_metas, **kwargs) File "/root/workspace/mmdeploy/mmdeploy/core/rewriters/rewriter_utils.py", line 371, in wrapper return self.func(self, *args, **kwargs) File "/root/workspace/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/two_stage.py", line 58, in two_stage_detector__simple_test proposals, _ = self.rpn_head.simple_test_rpn(x, img_metas) File "/opt/conda/lib/python3.7/site-packages/mmdet/models/dense_heads/dense_test_mixins.py", line 130, in simple_test_rpn proposal_list = self.get_bboxes(*rpn_outs, img_metas=img_metas) File "/root/workspace/mmdeploy/mmdeploy/core/rewriters/rewriter_utils.py", line 371, in wrapper return self.func(self, *args, **kwargs) File "/root/workspace/mmdeploy/mmdeploy/codebase/mmdet/models/dense_heads/rpn_head.py", line 122, in rpn_head__get_bboxes max_shape=img_metas[0]['img_shape']) File "/opt/conda/lib/python3.7/site-packages/mmdet/core/bbox/coder/legacy_delta_xywh_bbox_coder.py", line 79, in decode self.stds, max_shape, wh_ratio_clip) File "/opt/conda/lib/python3.7/site-packages/mmcv/utils/parrots_jit.py", line 22, in wrapper_inner return func(*args, **kargs) File "/opt/conda/lib/python3.7/site-packages/mmdet/core/bbox/coder/legacy_delta_xywh_bbox_coder.py", line 181, in legacy_delta2bbox denorm_deltas = deltas * stds + means RuntimeError: The size of tensor a (4) must match the size of tensor b (4432) at non-singleton dimension 2 2022-05-19 16:21:36,282 - mmdeploy - ERROR - torch2onnx failed.

    Does anyone know what the problem might be? Thanks in advance!

    opened by saffie91 21
  • MMdeploy build & install script + prerequisites

    MMdeploy build & install script + prerequisites

    Hello,

    Over the last couple of months, I have been working on a bash script for building and installing MMDeploy and prerequisites to ease the install procedure when we use MMDetection, MMDeploy tools. Maybe this script can serve as a starting point for an easier install procedure when building and installing MMDeploy from source.

    Naturally, this script is a work in progress. There are probably parts that can be cleaned up - and maybe some dependencies are not needed to install anymore.

    Let me know if there are any suggestions and ideas on how to integrate this script with the MMDeploy

    Note:

    1. Only tested with "standard" python virtual environment, as I do not use Conda
    2. Only tested on linux
    3. Only supports TensorRT build at this point.
    4. The script should work on both x86_64 and aarch64 (jetson) platforms.
    opened by tehkillerbee 21
  • Inference error

    Inference error

    Hi: I have converted faster-rcnn model downloaded from mmdetection zoo to trt engine sucessfully, but when I run inference_model the error happened: [2022-04-22 07:27:52.715] [mmdeploy] [info] [model.cpp:95] Register 'DirectoryModel' 2022-04-22 07:27:57,889 - mmdeploy - INFO - Successfully loaded tensorrt plugins from /root/workspace/mmdeploy/build/lib/libmmdeploy_tensorrt_ops.so 2022-04-22 07:27:57,889 - mmdeploy - INFO - Successfully loaded tensorrt plugins from /root/workspace/mmdeploy/build/lib/libmmdeploy_tensorrt_ops.so /opt/conda/lib/python3.8/site-packages/mmdet-2.22.0-py3.8.egg/mmdet/datasets/utils.py:66: UserWarning: "ImageToTensor" pipeline is replaced by "DefaultFormatBundle" for batch inference. It is recomm ended to manually replace it in the test data pipeline in your config file. warnings.warn( #assertion/root/workspace/mmdeploy/csrc/backend_ops/tensorrt/batched_nms/trt_batched_nms.cpp,98 Aborted (core dumped)

    Could you please tell me why it happend and how to deal with it? Thank you.

    TensorRT 
    opened by wulouzhu 21
  • JetsonTX2+Jetpack4.6, deploy error.

    JetsonTX2+Jetpack4.6, deploy error.

    env:

    `022-03-05 23:29:30,973 - mmdeploy - INFO -

    2022-03-05 23:29:30,973 - mmdeploy - INFO - Environmental information 2022-03-05 23:29:37,230 - mmdeploy - INFO - sys.platform: linux 2022-03-05 23:29:37,230 - mmdeploy - INFO - Python: 3.6.7 | packaged by conda-forge | (default, Jan 7 2019, 02:49:06) [GCC 7.3.0] 2022-03-05 23:29:37,231 - mmdeploy - INFO - CUDA available: True 2022-03-05 23:29:37,231 - mmdeploy - INFO - GPU 0: NVIDIA Tegra X2 2022-03-05 23:29:37,231 - mmdeploy - INFO - CUDA_HOME: /usr/local/cuda-10.2 2022-03-05 23:29:37,231 - mmdeploy - INFO - NVCC: Build cuda_10.2_r440.TC440_70.29663091_0 2022-03-05 23:29:37,231 - mmdeploy - INFO - GCC: gcc (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04) 7.5.0 2022-03-05 23:29:37,231 - mmdeploy - INFO - PyTorch: 1.8.0 2022-03-05 23:29:37,231 - mmdeploy - INFO - PyTorch compiling details: PyTorch built with:

    • GCC 7.5
    • C++ Version: 201402
    • OpenMP 201511 (a.k.a. OpenMP 4.5)
    • NNPACK is enabled
    • CPU capability usage: NO AVX
    • CUDA Runtime 10.2
    • NVCC architecture flags: -gencode;arch=compute_53,code=sm_53;-gencode;arch=compute_62,code=sm_62;-gencode;arch=compute_72,code=sm_72
    • CuDNN 8.2.1
      • Built with CuDNN 8.0
    • Build settings: BLAS_INFO=open, BUILD_TYPE=Release, CUDA_VERSION=10.2, CUDNN_VERSION=8.0.0, CXX_COMPILER=/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_XNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -DMISSING_ARM_VST1 -DMISSING_ARM_VLD1 -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=open, TORCH_VERSION=1.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EIGEN_FOR_BLAS=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=OFF, USE_MKLDNN=OFF, USE_MPI=ON, USE_NCCL=0, USE_NNPACK=ON, USE_OPENMP=ON,

    2022-03-05 23:29:37,232 - mmdeploy - INFO - TorchVision: 0.9.0 2022-03-05 23:29:37,232 - mmdeploy - INFO - OpenCV: 4.5.5 2022-03-05 23:29:37,232 - mmdeploy - INFO - MMCV: 1.4.6 2022-03-05 23:29:37,232 - mmdeploy - INFO - MMCV Compiler: GCC 7.5 2022-03-05 23:29:37,232 - mmdeploy - INFO - MMCV CUDA Compiler: 10.2 2022-03-05 23:29:37,232 - mmdeploy - INFO - MMDeployment: 0.3.0+34879e6 2022-03-05 23:29:37,232 - mmdeploy - INFO -

    2022-03-05 23:29:37,232 - mmdeploy - INFO - Backend information 2022-03-05 23:29:38,109 - mmdeploy - INFO - onnxruntime: None ops_is_avaliable : False 2022-03-05 23:29:38,111 - mmdeploy - INFO - tensorrt: 8.0.1.6 ops_is_avaliable : True 2022-03-05 23:29:38,116 - mmdeploy - INFO - ncnn: None ops_is_avaliable : False 2022-03-05 23:29:38,120 - mmdeploy - INFO - pplnn_is_avaliable: False 2022-03-05 23:29:38,123 - mmdeploy - INFO - openvino_is_avaliable: False 2022-03-05 23:29:38,123 - mmdeploy - INFO -

    2022-03-05 23:29:38,124 - mmdeploy - INFO - Codebase information 2022-03-05 23:29:38,125 - mmdeploy - INFO - mmcls: None 2022-03-05 23:29:38,128 - mmdeploy - INFO - mmdet: 2.22.0 2022-03-05 23:29:38,129 - mmdeploy - INFO - mmedit: None 2022-03-05 23:29:38,130 - mmdeploy - INFO - mmocr: None 2022-03-05 23:29:38,131 - mmdeploy - INFO - mmseg: None`

    error:

    python ./tools/deploy.py configs/mmdet/detection/detection_tensorrt_dynamic-320x320-1344x1344.py ../mmdetection/configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py ../mmdetection/checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth ../mmdetection/demo/demo.jpg --work-dir test-work/ --show --device cuda:0
    `2022-03-05 23:26:46,769 - mmdeploy - INFO - torch2onnx start. load checkpoint from local path: ../mmdetection/checkpoints/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth /home/jetson/archiconda3/envs/mm/lib/python3.6/site-packages/mmdet/datasets/utils.py:70: UserWarning: "ImageToTensor" pipeline is replaced by "DefaultFormatBundle" for batch inference. It is recommended to manually replace it in the test data pipeline in your config file.

    `data pipeline in your config file.', UserWarning) /home/jetson/archiconda3/envs/mm/lib/python3.6/site-packages/mmdeploy-0.3.0-py3.6.egg/mmdeploy/core/optimizers/function_marker.py:158: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! ys_shape = tuple(int(s) for s in ys.shape) /home/jetson/archiconda3/envs/mm/lib/python3.6/site-packages/mmdet/models/dense_heads/anchor_head.py:123: UserWarning: DeprecationWarning: anchor_generator is deprecated, please use "prior_generator" instead warnings.warn('DeprecationWarning: anchor_generator is deprecated, ' /home/jetson/archiconda3/envs/mm/lib/python3.6/site-packages/mmdet/core/anchor/anchor_generator.py:333: UserWarning: grid_anchors would be deprecated soon. Please use grid_priors warnings.warn('grid_anchors would be deprecated soon. ' /home/jetson/archiconda3/envs/mm/lib/python3.6/site-packages/mmdet/core/anchor/anchor_generator.py:370: UserWarning: single_level_grid_anchors would be deprecated soon. Please use single_level_grid_priors 'single_level_grid_anchors would be deprecated soon. ' /home/jetson/archiconda3/envs/mm/lib/python3.6/site-packages/mmdeploy-0.3.0-py3.6.egg/mmdeploy/codebase/mmdet/models/dense_heads/rpn_head.py:77: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! assert cls_score.size()[-2:] == bbox_pred.size()[-2:] /home/jetson/archiconda3/envs/mm/lib/python3.6/site-packages/mmdeploy-0.3.0-py3.6.egg/mmdeploy/pytorch/functions/topk.py:54: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! if k > size: /home/jetson/archiconda3/envs/mm/lib/python3.6/site-packages/mmdeploy-0.3.0-py3.6.egg/mmdeploy/codebase/mmdet/core/bbox/delta_xywh_bbox_coder.py:39: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! assert pred_bboxes.size(0) == bboxes.size(0) /home/jetson/archiconda3/envs/mm/lib/python3.6/site-packages/mmdeploy-0.3.0-py3.6.egg/mmdeploy/codebase/mmdet/core/bbox/delta_xywh_bbox_coder.py:41: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! assert pred_bboxes.size(1) == bboxes.size(1) /home/jetson/archiconda3/envs/mm/lib/python3.6/site-packages/mmdeploy-0.3.0-py3.6.egg/mmdeploy/codebase/mmdet/core/post_processing/bbox_nms.py:260: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! dets, labels = TRTBatchedNMSop.apply(boxes, scores, int(scores.shape[-1]), /home/jetson/archiconda3/envs/mm/lib/python3.6/site-packages/mmdeploy-0.3.0-py3.6.egg/mmdeploy/mmcv/ops/nms.py:177: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! out_boxes = min(num_boxes, after_topk) /home/jetson/archiconda3/envs/mm/lib/python3.6/site-packages/mmdeploy-0.3.0-py3.6.egg/mmdeploy/mmcv/ops/nms.py:181: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! (batch_size, out_boxes)).to(scores.device) /home/jetson/archiconda3/envs/mm/lib/python3.6/site-packages/torch/onnx/symbolic_opset9.py:2607: UserWarning: Exporting aten::index operator of advanced indexing in opset 11 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results. "If indices include negative values, the exported graph will produce incorrect results.") 2022-03-05 23:28:15,099 - mmdeploy - INFO - torch2onnx success. 2022-03-05 23:28:15,675 - mmdeploy - INFO - onnx2tensorrt of test-work/end2end.onnx start. 2022-03-05 23:28:21,674 - mmdeploy - WARNING - Could not load the library of tensorrt plugins. Because the file does not exist: [TensorRT] INFO: [MemUsageChange] Init CUDA: CPU +240, GPU +0, now: CPU 309, GPU 5307 (MiB) [TensorRT] WARNING: onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32. [TensorRT] WARNING: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped [TensorRT] WARNING: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped [TensorRT] WARNING: onnx2trt_utils.cpp:390: One or more weights outside the range of INT32 was clamped [TensorRT] INFO: No importer registered for op: TRTBatchedNMS. Attempting to import as plugin. [TensorRT] INFO: Searching for plugin: TRTBatchedNMS, plugin_version: 1, plugin_namespace: [TensorRT] ERROR: 3: getPluginCreator could not find plugin: TRTBatchedNMS version: 1 2022-03-05:23:28:33,root ERROR [utils.py:43] Failed to parse onnx, In node 1143 (importFallbackPluginImporter): UNSUPPORTED_NODE: Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"

    Traceback (most recent call last): File "/home/jetson/archiconda3/envs/mm/lib/python3.6/site-packages/mmdeploy-0.3.0-py3.6.egg/mmdeploy/utils/utils.py", line 38, in target_wrapper result = target(*args, **kwargs) File "/home/jetson/archiconda3/envs/mm/lib/python3.6/site-packages/mmdeploy-0.3.0-py3.6.egg/mmdeploy/backend/tensorrt/onnx2tensorrt.py", line 72, in onnx2tensorrt device_id=device_id) File "/home/jetson/archiconda3/envs/mm/lib/python3.6/site-packages/mmdeploy-0.3.0-py3.6.egg/mmdeploy/backend/tensorrt/utils.py", line 76, in create_trt_engine raise RuntimeError(f'Failed to parse onnx, {error_msgs}') RuntimeError: Failed to parse onnx, In node 1143 (importFallbackPluginImporter): UNSUPPORTED_NODE: Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"

    2022-03-05 23:28:34,549 - mmdeploy - ERROR - onnx2tensorrt of test-work/end2end.onnx failed. `

    opened by Richard-mei 20
  • I get an error when trying to deploy yolox to tensorrt.

    I get an error when trying to deploy yolox to tensorrt.

    I tried to deploy the yolox tiny model to TensorRT by executing the following command, but an error occurs during the conversion stage from ONNX to TensorRT.

    python {mmdeployDIR}/tools/deploy.py \
    {mmdeployDIR}/configs/mmdet/detection/detection_tensorrt_dynamic-160x160-608x608.py \
    {mmdetecitonDIR}/configs/yolox/mmdetection_custom_dataset.py \
    {mmdetecitonDIR}/checkpoints/yolox_tiny_8x8_300e_coco_20211124_171234-b4047906.pth \
    {mmdetecitonDIR}/demo/demo.jpg --work-dir deploy_test/ --show --device cuda --dump-info
    

    The execution environment is as follows

    Python: 3.6.9
    GPU: Tesla T4
    NVCC: Cuda compilation tools, release 11.1, V11.1.105
    PyTorch: 1.10.1+cu111
    TorchVision: 0.11.2+cu111
    OpenCV: 4.5.4-dev
    MMCV: 1.4.8
    MMCV Compiler: GCC 7.5
    MMCV CUDA Compiler: 11.1
    MMDetection: 2.23.0+
    

    The output when the command is executed is as follows

    2022-05-09 10:15:15,164 - mmdeploy - INFO - torch2onnx start.
    load checkpoint from local path: {mmdetecitonDIR}//checkpoints/yolox_tiny_8x8_300e_coco_20211124_171234-b4047906.pth
    2022-05-09 10:15:58,128 - mmdeploy - WARNING - DeprecationWarning: get_onnx_config will be deprecated in the future. 
    2022-05-09:10:15:58,mmdeploy WARNING  [utils.py:92] DeprecationWarning: get_onnx_config will be deprecated in the future. 
    {mmdeployDIR}/core/optimizers/function_marker.py:158: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
      ys_shape = tuple(int(s) for s in ys.shape)
    /anaconda/envs/azureml_py36/lib/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2157.)
      return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
    {mmdeployDIR}/mmdeploy/codebase/mmdet/core/post_processing/bbox_nms.py:260: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
      dets, labels = TRTBatchedNMSop.apply(boxes, scores, int(scores.shape[-1]),
    {mmdeployDIR}/mmdeploy/mmcv/ops/nms.py:177: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
      out_boxes = min(num_boxes, after_topk)
    {mmdeployDIR}/mmdeploy/mmcv/ops/nms.py:181: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
      (batch_size, out_boxes)).to(scores.device)
    WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
    WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
    WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
    WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
    WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
    WARNING: The shape inference of mmdeploy::TRTBatchedNMS type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
    2022-05-09 10:16:05,280 - mmdeploy - INFO - torch2onnx success.
    2022-05-09 10:16:05,541 - mmdeploy - INFO - onnx2tensorrt of deploy_test/end2end.onnx start.
    2022-05-09 10:16:19,570 - mmdeploy - INFO - Successfully loaded tensorrt plugins from {mmdeployDIR}/build/lib/libmmdeploy_tensorrt_ops.so
    2022-05-09:10:16:19,mmdeploy INFO     [init_plugins.py:32] Successfully loaded tensorrt plugins from {mmdeployDIR}/build/lib/libmmdeploy_tensorrt_ops.so
    [05/09/2022-10:16:19] [TRT] [I] [MemUsageChange] Init CUDA: CPU +315, GPU +0, now: CPU 406, GPU 242 (MiB)
    [05/09/2022-10:16:20] [TRT] [I] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 423 MiB, GPU 242 MiB
    [05/09/2022-10:16:20] [TRT] [I] [MemUsageSnapshot] End constructing builder kernel library: CPU 651 MiB, GPU 316 MiB
    [05/09/2022-10:16:21] [TRT] [W] onnx2trt_utils.cpp:365: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
    [05/09/2022-10:16:21] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
    [05/09/2022-10:16:21] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
    [05/09/2022-10:16:21] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
    [05/09/2022-10:16:21] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
    [05/09/2022-10:16:21] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
    [05/09/2022-10:16:21] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
    [05/09/2022-10:16:21] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
    [05/09/2022-10:16:21] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
    [05/09/2022-10:16:21] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
    [05/09/2022-10:16:21] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
    [05/09/2022-10:16:21] [TRT] [W] onnx2trt_utils.cpp:391: One or more weights outside the range of INT32 was clamped
    [05/09/2022-10:16:21] [TRT] [I] No importer registered for op: TRTBatchedNMS. Attempting to import as plugin.
    [05/09/2022-10:16:21] [TRT] [I] Searching for plugin: TRTBatchedNMS, plugin_version: 1, plugin_namespace: 
    [05/09/2022-10:16:21] [TRT] [I] Successfully created plugin: TRTBatchedNMS
    [05/09/2022-10:16:24] [TRT] [E] 4: [graphShapeAnalyzer.cpp::processCheck::593] Error Code 4: Internal Error (Concat_355: dimensions not compatible for concatenation)
    2022-05-09:10:16:24,root ERROR    [utils.py:43] Failed to create TensorRT engine
    Traceback (most recent call last):
      File "{mmdeployDIR}/mmdeploy/utils/utils.py", line 38, in target_wrapper
        result = target(*args, **kwargs)
      File "{mmdeployDIR}/mmdeploy/backend/tensorrt/onnx2tensorrt.py", line 83, in onnx2tensorrt
        device_id=device_id)
      File "{mmdeployDIR}/mmdeploy/backend/tensorrt/utils.py", line 116, in create_trt_engine
    
    opened by NJIMAMTO 17
  • image_segmentation.cpp call onnxruntime-cuda thrown an abnormal

    image_segmentation.cpp call onnxruntime-cuda thrown an abnormal

    win10 mmdeploy image_segmentation.cpp call onnxruntime-static-cuda and onnxruntime-dynamic-cuda thrown an abnormal call onnxruntime-static-cpu and onnxruntime-dynamic-cpu is ok call tensorrt-static-cuda is ok call tensorrt-dynamic-cuda error

    1649925571(1) 1649925671(1) 1649925777(1) 1649925905(1)

    opened by joryoker 17
  • [WIP] Propsal to MMDeploy's build document in zh_cn

    [WIP] Propsal to MMDeploy's build document in zh_cn

    Motivation

    • 社区反馈 MMDeploy 文档不够详实,可读性不好。
    • 随着支持的平台、设备和功能的增多,也需要对当前的文档,尤其是build.md、get_started文档,做出必要的调整。使得文档的结构更清晰,内容更完整,可读性更好
    • 编译安装命令整理为脚本,放在代码库中管理

    Modification

    以下为 MMDeploy 安装文档说明书草案。欢迎大家多提宝贵意见


    下载 MMDeploy

    git clone -b master [email protected]:open-mmlab/mmdeploy.git MMDeploy
    cd MMDeploy
    export MMDEPLOY_DIR=$(pwd)
    git submodule update --init --recursive
    

    提示:

    • 如果由于网络等原因导致拉取仓库子模块失败,可以尝试通过如下指令手动再次安装子模块:
      git clone [email protected]:NVIDIA/cub.git third_party/cub
      cd third_party/cub
      git checkout c3cceac115
       # 返回至 third_party 目录, 克隆 pybind11
      cd ..
      git clone [email protected]:pybind/pybind11.git pybind11
      cd pybind11
      git checkout 70a58c5
    

    Linux-x86_64 下构建方式

    在 linux-x86_64 下,MMDeploy 提供 3 种构建包的方式,分别是Dockerfile方式、源码安装、和直接使用预编译包。

    1. Dockerfile 方式(推荐)

    MMDeploy 分别提供了 cpu 和 cuda 平台下的 dockerfile。其中,ONNXRuntime、ncnn 和 OpenVINO 等推理引擎安装在 cpu docker镜像中,而 TensorRT 将会安装在 cuda docker镜像中。 用户可以根据文档 使用docker构建MMDeploy 的详细步骤,构建 MMDeploy

    2. 源码安装

    安装编译工具链

    名称 版本 安装脚本
    gcc >= 7
    bash build_scripts/linux-x86_64/gcc.sh
    
    cmake >= 3.14.0
    bash build_scripts/linux-x86_64/cmake.sh
    

    安装依赖包

    MMDeploy Converter 专用依赖

    依赖项名称 安装方法
    conda 通过 conda 创建并激活 Python 环境
    conda create -n mmdeploy python=3.7 -y
    conda activate mmdeploy
    
    pytorch 要求版本是 torch>=1.8.0, 可查看官网获取更详细的安装教程。
    # CUDA 11.1
    conda install pytorch==1.8.0 torchvision==0.9.0 cudatoolkit=11.1 -c pytorch -c conda-forge
    
    mmcv-full 更多安装方式可查看教程
    export cu_version=cu111 # cuda 11.1
    export torch_version=torch1.8.0
    pip install mmcv-full==1.4.0 -f https://download.openmmlab.com/mmcv/dist/${cu_version}/${torch_version}/index.html
    
    OpenMMLab Codebase 比如安装mmdetection
    pip install mmdet==2.19.0
    

    MMDeploy SDK 专用依赖

    执行下述安装脚本,默认把依赖安装到 mmdeploy/.deps下

    依赖项名称 安装方法
    spdlog
    # 在该脚本中,1)下载依赖包v1.9.2;2)编译&安装;3)设置环境变量SPDLOG_DIR
    bash build_scripts/linux-x86_64/spdlog.sh <destination_path>
    
    OpenCV
    # 在该脚本中,1)下载依赖包v4,5.5;2)编译&安装;3)设置环境变量OPENCV_DIR
    bash build_scripts/linux-x86_64/opencv.sh <destination_path>
    
    ppl.cv **当推理平台为 cuda 时,需要安装该依赖项**
    # 在该脚本中,1)下载依赖包v0.6.1;2)编译&安装;3)设置环境变量PPLCV_DIR
    bash build_scripts/linux-x86_64/ppl.cv.sh <destination_path>
    

    MMDeploy 推理引擎依赖

    MMDeploy Converter & SDK 共享推理引擎。用户可以参考下表,选择并安装感兴趣的推理引擎

    推理引擎 依赖项名称 安装方法
    TensorRT TensorRT 1. 从 NVIDIA 官网下载二进制包
    2. 执行安装脚本
     # 1) 解压包; 2) 安装whl文件; 3) 写环境变量 TENSORRT_DIR
     bash build_scripts/tensorrt.sh ${package_path} ${destination_path}
    
    cudnn 1. 从 NVIDIA 官网下载二进制包
    2. 执行安装脚本
    # 1) 解压包;2)安装whl文件;3)写环境变量CUDNN_DIR
    bash build_scripts/cudnn.sh ${package_path} ${destination_path}
    
    pplnn ppl.nn
    # 1)下载pplnn;2)编译&安装;3)写环境变量PPLNN_DIR
    bash build_scripts/pplnn.sh
    
    ONNXRuntime ONNXRuntime
      # 1)下载onnxruntime;2)安装;3)写环境变量ONNXRUNTIME_DIR
    bash build_scripts/onnxruntime.sh
    
    OpenVINO OpenVINO
    bash build_scripts/openvino.sh
    
    ncnn ncnn
      # 1)下载ncnn;2)编译&安装;3)写环境变量NCNN_DIR
    bash build_scripts/ncnn.sh
    

    编译 MMDeploy

    编译安装 Converter

    1. 编译自定义算子

    bash build_scripts/linux_x86-64/build_custom_ops.sh -DMMDEPLOY_TARGET_BACKENDS=ort 
    
    • TensorRT 自定义算子
    bash build_scripts/linux_x86-64/build_custom_ops.sh -DMMDEPLOY_TARGET_BACKENDS=trt
    
    • ncnn 自定义算子
    bash build_scripts/linux_x86-64/build_custom_ops.sh -DMMDEPLOY_TARGET_BACKENDS=ncnn
    

    2. 安装

    cd ${MMDEPLOY_DIR} # 切换至项目根目录
    pip install -e .
    

    编译 SDK

    1. 设置目标设备

    cpu 是 SDK 目标设备的默认选项。你也可以通过MMDEPLOY_TARGET_DEVICES传入其他设备名称。SDK 支持 "cpu" 和 "cuda" 两种设备。当有多个设备时,设备名称之间使用分号隔开。 比如,-DMMDEPLOY_TARGET_DEVICES="cpu;cuda"。 如果你的开发环境中有多个cuda版本,则需要通过-DCUDA_TOOLKIT_ROOT_DIR=/path/of/cuda来明确使用的版本。

    2. 设置推理后端

    默认情况下,SDK不设置任何后端, 因为它与应用场景高度相关。你可以通过设置MMDEPLOY_TARGET_BACKENDS激活感兴趣的推理后端。已支持的推理后端有‘ort’, 'trt', 'ncnn', 'pplnn' 和 'openvino' 当选择多个时, 中间使用分号隔开。比如,-DMMDEPLOY_TARGET_BACKENDS="trt;ort;pplnn;ncnn;openvino"

    3. 设置后处理组件

    需要通过MMDEPLOY_CODEBASES设置SDK后处理组件,才能加载OpenMMLab算法仓库的后处理功能。已支持的算法仓库有'mmcls','mmdet','mmedit','mmseg'和'mmocr'。 如果选择多个codebase,中间使用分号隔开。比如,-DMMDEPLOY_CODEBASES=mmcls;mmdet。也可以通过-DMMDEPLOY_CODEBASES=all方式,加载所有codebase。

    • 汇总以上
    # 使用cpu设备和ONNXRuntime推理
    bash build_scripts/linux-x86_64/mmdeploy_sdk.sh -DMMDEPLOY_TARGET_DEVICES=cpu -DMMDEPLOY_TARGET_BACKENDS=ort -DMMDEPLOY_CODEBASES=all
    
    # 使用cuda设备和TensorRT推理
    bash build_scripts/linux-x86_64/mmdeploy_sdk.sh -DMMDEPLOY_TARGET_DEVICES=cuda -DMMDEPLOY_TARGET_BACKENDS=trt -DMMDEPLOY_CODEBASES=all
    
    

    3. 预编译包

    提供 linux-x86_64/cpu/mmdeploy_onnxruntime.tar.gz,linux-x86_64/cuda/mmdeploy_cu111_trt8.2.3.tar.gz

    NVIDIA Jetson 下构建方式

    1. 源码安装

    安装 Jetpack

    安装依赖

    MMDeploy Converter专用依赖

    MMDeploy SDK 专用依赖

    MMDeploy 推理引擎依赖

    编译安装 MMDeploy

    Windows 下构建方式

    1. 源码安装

    安装编译工具链

    安装依赖

    MMDeploy Converter专用依赖

    MMDeploy SDK 专用依赖

    MMDeploy 推理引擎依赖

    编译安装 MMDeploy

    2. 预编译包

    Android 下构建方式

    1. 源码安装

    安装编译工具链

    安装依赖

    MMDeploy Converter专用依赖

    MMDeploy SDK 专用依赖

    MMDeploy 推理引擎依赖

    编译安装 MMDeploy

    2. 预编译包

    community discussion 
    opened by lvhan028 17
  • MMDeploy exported models on Nvidia Jetson devices

    MMDeploy exported models on Nvidia Jetson devices

    MMDeploy looks good and I had a few questions relating to deploying the output model on Nvidia EDGE device that I could not find on Readthedocs and Github pages. Apologies in advance if the questions are very basic in nature:

    1. For Nvidia deployment, we will be selecting the TensorRT deployment option. So once I convert the model and get the SDK using my Mac, will I still need to install PyTorch on the Nvidia device that I want to use for inference? In other words, My Nvidia Device has the JetPack already installed, which includes TensorRT. So, can I simply 1) run the converted model, 2) need SDK with the converted model or 3) Need SDK, converted model and PyTorch + other installations suggested by MMPose?

    2. I intend to use the model for pose detection using MMPose. Will there be any additional dependencies?

    Update: I am planning to install the MMDeploy on my Mac and then get the updated model and SDK for my inference device which is Nvidia

    opened by RS00001 17
  • Mask R-CNN speed up

    Mask R-CNN speed up

    I was measured inference time for Mask R-CNN in MMDetection and in TensorRT after deployment. With tensorRT python API I use obtained .engine file to inference, and inference time dropped from 0.16s to 0.13s. Is it normal that the boost is so low? I use GTX 1050 ti and deployed with fp32 and 640x480 resolution. And one more question: deployed model outputs each mask as a 28x28 array. I resize it to fill bounding box of the object, but the mask is still coarse. In MMDetection output masks already have shape of input image and they are much more precise than after deployment. Do I something wrong?

    opened by IvDmNe 16
  • RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

    RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

    Hi, I am using mmdeploy to convert maskrcnn pth model to tensorrt, and i got this error: RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

    my command: python tools/deploy.py \ configs/mmdet/instance-seg/instance-seg_tensorrt_dynamic-320x320-1344x1344.py \ /home/mmlab/mmdetection-2.20.0/project/mrcnn_r18_fpn_160e/mrcnn_r18_fpn_160e.py \ /home/mmlab/mmdetection-2.20.0/project/mrcnn_r18_fpn_160e/epoch_150.pth \ /home/mmlab/mmdetection-2.20.0/demo/demo.jpg \ --work-dir work_dir \ --device cuda:0

    my env: gcc g++ 7.5.0 cuda 10.2 cudnn 7.6.5 tensorrt 7.2.3 torch 1.8.0

    opened by Baiyixuan1113 16
  • [Refactor][tools] Add prebuild tools.

    [Refactor][tools] Add prebuild tools.

    This PR provides tools to build prebuilt tools. Some other files are also modified to support the pre-build packaging. Has been tested on linux-x64, both PC and jetson. TesnorRT on both platform and ONNXRuntime on PC. Usage:

    python ${MMDEPLOY_DIR}/tools/package_tools/mmdeploy_builder.py \
        ${MMDEPLOY_DIR}/tools/package_tools/configs/linux_x86_trt.yaml \
        ${MMDEPLOY_DIR}
    

    You might need to edit your own config file to build a different package. Windows package can also be built, but it has not been tested.

    opened by grimoire 15
  • Convertion SwinTransformer to ONNX, NotImplementedError: [Adaptive pool]:input size not accessible

    Convertion SwinTransformer to ONNX, NotImplementedError: [Adaptive pool]:input size not accessible

    export DEPLOY_CFG_PATH="/aihub/doing/mmdeploy/configs/mmseg/segmentation_onnxruntime_static-512x512.py"
    export MODEL_CFG_PATH="/aihub/doing/mmsegmentation/work_dirs112/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K.py"
    export MODEL_CHECKPOINT_PATH="/aihub/doing/mmsegmentation/work_dirs112/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K/best_mIoU_iter_15096.pth"
    export INPUT_IMG="/aihub/1654077548404731.png"
    export TEST_IMG="/aihub/1654077548404731.png"
    export WORK_DIR="/aihub/doing/mmsegmentation/work_dirs112/upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K"
    export CALIB_DATA_CFG="/aihub/da"
    
    python ./tools/deploy.py \
        ${DEPLOY_CFG_PATH} \
        ${MODEL_CFG_PATH} \
        ${MODEL_CHECKPOINT_PATH} \
        ${INPUT_IMG} \
        --test-img ${TEST_IMG} \
        --work-dir ${WORK_DIR} \
        --calib-dataset-cfg ${CALIB_DATA_CFG} \
        --log-level INFO \
        --show \
        --dump-info
    

    With the script above to export swin tiny, Error occured as follows:

    Traceback (most recent call last):
      File "/home/lx/sw/anaconda3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
        self.run()
      File "/home/lx/sw/anaconda3/lib/python3.8/multiprocessing/process.py", line 108, in run
        self._target(*self._args, **self._kwargs)
      File "/home/lx/.local/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 105, in __call__
        ret = func(*args, **kwargs)
      File "/home/lx/.local/lib/python3.8/site-packages/mmdeploy/apis/pytorch2onnx.py", line 86, in torch2onnx
        export(
      File "/home/lx/.local/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 354, in _wrap
        return self.call_function(func_name_, *args, **kwargs)
      File "/home/lx/.local/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 324, in call_function
        return self.call_function_local(func_name, *args, **kwargs)
      File "/home/lx/.local/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 273, in call_function_local
        return pipe_caller(*args, **kwargs)
      File "/home/lx/.local/lib/python3.8/site-packages/mmdeploy/apis/core/pipeline_manager.py", line 105, in __call__
        ret = func(*args, **kwargs)
      File "/home/lx/.local/lib/python3.8/site-packages/mmdeploy/apis/onnx/export.py", line 121, in export
        torch.onnx.export(
      File "/home/lx/.local/lib/python3.8/site-packages/torch/onnx/__init__.py", line 305, in export
        return utils.export(model, args, f, export_params, verbose, training,
      File "/home/lx/.local/lib/python3.8/site-packages/torch/onnx/utils.py", line 118, in export
        _export(model, args, f, export_params, verbose, training, input_names, output_names,
      File "/home/lx/.local/lib/python3.8/site-packages/torch/onnx/utils.py", line 719, in _export
        _model_to_graph(model, args, verbose, input_names,
      File "/home/lx/.local/lib/python3.8/site-packages/mmdeploy/core/rewriters/rewriter_utils.py", line 371, in wrapper
        return self.func(self, *args, **kwargs)
      File "/home/lx/.local/lib/python3.8/site-packages/mmdeploy/apis/onnx/optimizer.py", line 10, in model_to_graph__custom_optimizer
        graph, params_dict, torch_out = ctx.origin_func(*args, **kwargs)
      File "/home/lx/.local/lib/python3.8/site-packages/torch/onnx/utils.py", line 503, in _model_to_graph
        graph = _optimize_graph(graph, operator_export_type,
      File "/home/lx/.local/lib/python3.8/site-packages/torch/onnx/utils.py", line 232, in _optimize_graph
        graph = torch._C._jit_pass_onnx(graph, operator_export_type)
      File "/home/lx/.local/lib/python3.8/site-packages/torch/onnx/__init__.py", line 354, in _run_symbolic_function
        return utils._run_symbolic_function(*args, **kwargs)
      File "/home/lx/.local/lib/python3.8/site-packages/torch/onnx/utils.py", line 1061, in _run_symbolic_function
        return symbolic_fn(g, *inputs, **attrs)
      File "/home/lx/.local/lib/python3.8/site-packages/mmdeploy/core/rewriters/rewriter_utils.py", line 360, in __call__
        return self.func(self, *args, **kwargs)
      File "/home/lx/.local/lib/python3.8/site-packages/mmdeploy/pytorch/ops/adaptive_avg_pool.py", line 72, in adaptive_avg_pool2d__default
        return adaptive_avg_pool2d(*args)
      File "/home/lx/.local/lib/python3.8/site-packages/torch/onnx/symbolic_helper.py", line 172, in wrapper
        return fn(g, *args, **kwargs)
      File "/home/lx/.local/lib/python3.8/site-packages/mmdeploy/pytorch/ops/adaptive_avg_pool.py", line 22, in symbolic_fn
        raise NotImplementedError(
    NotImplementedError: [Adaptive pool]:input size not accessible
    

    The input parameter in the symbolic_fn is of type torch._C.Value and input.isCompleteTensor() is False,

        if not input.isCompleteTensor():
            if output_size == [1] * len(output_size):
                return g.op('GlobalMaxPool', input), None
            raise NotImplementedError(
                '[Adaptive pool]:input size not accessible')
    

    Have Anyone Ecounted the Same Problem? Any help would be great appreciated!

    opened by lx-r 0
  • NotImplementedError: SOLOv2 does not support ONNX EXPORT

    NotImplementedError: SOLOv2 does not support ONNX EXPORT

    1 i am trying to convert SOLOv2 model from Pytorch to Onnx with the following command.

    python tools/deployment/pytorch2onnx.py /home/xxd/dyg/open-mmlab/mmdetection/configs/solov2/solov2_r50_fpn_1x_coco.py /home/xxd/dyg/open-mmlab/mmdetection/work_dirs/solov2_r50_fpn_1x_coco/epoch_5.pth --output-file /home/xxd/dyg/open-mmlab/mmdetection/work_dirs/solov2_r50_fpn_1x_coco/epoch_5.onnx --input-img /media/xxd/Data2/datasets/voc/test_img/2008_002378.jpg --test-img /media/xxd/Data2/datasets/voc/test_img/2008_002378.jpg --shape 640 480 --show --verify

    2 then i got the following error information.

    tools/deployment/pytorch2onnx.py:297: UserWarning: Arguments like --mean, --std, --dataset would be parsed directly from config file and are deprecated and will be removed in future releases. warnings.warn('Arguments like --mean, --std, --dataset would be
    /home/xxd/anaconda3/envs/openmmlab2/lib/python3.8/site-packages/mmcv/onnx/symbolic.py:481: UserWarning: DeprecationWarning: This function will be deprecated in future. Welcome to use the unified model deployment toolbox MMDeploy: https://github.com/open-mmlab/mmdeploy warnings.warn(msg) load checkpoint from local path: /home/xxd/dyg/open-mmlab/mmdetection/work_dirs/solov2_r50_fpn_1x_coco/epoch_5.pth Traceback (most recent call last): File "tools/deployment/pytorch2onnx.py", line 333, in pytorch2onnx( File "tools/deployment/pytorch2onnx.py", line 89, in pytorch2onnx torch.onnx.export( File "/home/xxd/anaconda3/envs/openmmlab2/lib/python3.8/site-packages/torch/onnx/init.py", line 316, in export return utils.export(model, args, f, export_params, verbose, training, File "/home/xxd/anaconda3/envs/openmmlab2/lib/python3.8/site-packages/torch/onnx/utils.py", line 107, in export _export(model, args, f, export_params, verbose, training, input_names, output_names, File "/home/xxd/anaconda3/envs/openmmlab2/lib/python3.8/site-packages/torch/onnx/utils.py", line 724, in _export _model_to_graph(model, args, verbose, input_names, File "/home/xxd/anaconda3/envs/openmmlab2/lib/python3.8/site-packages/torch/onnx/utils.py", line 493, in _model_to_graph graph, params, torch_out, module = _create_jit_graph(model, args) File "/home/xxd/anaconda3/envs/openmmlab2/lib/python3.8/site-packages/torch/onnx/utils.py", line 437, in _create_jit_graph graph, torch_out = _trace_and_get_graph_from_model(model, args) File "/home/xxd/anaconda3/envs/openmmlab2/lib/python3.8/site-packages/torch/onnx/utils.py", line 388, in _trace_and_get_graph_from_model torch.jit._get_trace_graph(model, args, strict=False, _force_outplace=False, _return_inputs_states=True) File "/home/xxd/anaconda3/envs/openmmlab2/lib/python3.8/site-packages/torch/jit/_trace.py", line 1166, in _get_trace_graph outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs) File "/home/xxd/anaconda3/envs/openmmlab2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/xxd/anaconda3/envs/openmmlab2/lib/python3.8/site-packages/torch/jit/_trace.py", line 127, in forward graph, out = torch._C._create_graph_by_tracing( File "/home/xxd/anaconda3/envs/openmmlab2/lib/python3.8/site-packages/torch/jit/_trace.py", line 118, in wrapper outs.append(self.inner(*trace_inputs)) File "/home/xxd/anaconda3/envs/openmmlab2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/home/xxd/anaconda3/envs/openmmlab2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1090, in _slow_forward result = self.forward(*input, **kwargs) File "/home/xxd/anaconda3/envs/openmmlab2/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 110, in new_func return old_func(*args, **kwargs) File "/home/xxd/dyg/open-mmlab/mmdetection/mmdet/models/detectors/base.py", line 169, in forward return self.onnx_export(img[0], img_metas[0]) File "/home/xxd/dyg/open-mmlab/mmdetection/mmdet/models/detectors/base.py", line 359, in onnx_export raise NotImplementedError(f'{self.class.name} does ' NotImplementedError: SOLOv2 does not support ONNX EXPORT

    opened by dygcjlu 1
  • Deploy models via torchblade

    Deploy models via torchblade

    Describe the feature

    BladeDISC is an end-to-end DynamIc Shape Compiler project for machine learning workloads. It can be used for deployment as an alternative to TensorRT.

    Motivation

    • Target hardware includes also AMD GPU.
    • ".. compared to static optimizing compilers (i.e., XLA and TensorRT), DISC shows comparable or even better performance."

    Related resources

    https://github.com/alibaba/BladeDISC/blob/main/docs/developers/bladedisc_torch_overview.md

    Additional context

    • Models need to be serialized via TorchScript first
    • Inference performance table on various models https://github.com/facebookresearch/detectron2/issues/4083
    opened by user41pp 0
  • mmdeploy 2022 开源夏令营

    mmdeploy 2022 开源夏令营

    夏令营只有学生能参加 夏令营只有学生能参加 夏令营只有学生能参加

    背景

    • GLCC 的经费确实有限
    • 参加报名的学生确实都很优秀

    OpenMMLab mmdeploy repo 单独准备了一些题目和礼品,作为开源 GLCC 夏令营的补充。 有兴趣的同学,请直接评论自己的 微信 账号,附加题目名字作为认领方式。

    题目

    一、改善 arm Linux 部署体验

    【难度】⭐⭐⭐ 【描述】jetson nano 有 arm CPU, mmdeploy 团队提供 arm 设备登录方式。希望参考已有的 android ncnn 部署文档,写一份 arm linux 部署说明;遇到不合理的点直接修改,一起 PR 到 mmdeploy 【学习技能】 1. cmake 2. Linux 常用命令

    二、优化 ncnn int8 模型部署

    【最多可 7 位同学同时选择】 【难度】⭐⭐⭐⭐ 【描述】mmdeploy 目前支持 ncnn int8。但很多模型都没有验证。希望验证 mmdet/mmpose/mmtrack 等 repo 模型量化后的精度。如遇精度问题需给出掉点原因。测试结果 PR 到 mmdeploy;掉点结果需要写出知乎小软文。 【学习技能】 1. ncnn int8 原理 2. 对应 repo 下模型论文

    三、优化 trt int8 量化模型部署

    【难度】⭐⭐⭐⭐ 【描述】mmdeploy 目前未接入 trt int8 量化。希望参考接入 ncnn int8 方法,新增 trt 量化接口,PR 到 mmdeploy 【学习技能】 1. nvidia trt 量化原理 2. ppq 用法

    四、改进 CI 覆盖

    【听起来不难,实际上巨麻烦】 【难度】⭐⭐⭐⭐ 【描述】mmdeploy githuh action 目前没有覆盖完整的 end2end 转换。希望 CI 增加 cls/det/track 等 repo 到对应硬件的模型测试 【学习技能】 1. mm 系列大部分 repo 用法 2. Github Action 开发和调试

    五、新增 snpe backend

    【需要 qcom 芯片手机才能调试】 【难度】⭐⭐⭐⭐⭐ 【描述】参考 docs 中如何接入新的 backend 文档,支持高通推理框架 snpe,完成 fp32 模型推理。PR 到 mmdeploy 【学习技能】 1. 学习 snpe 3. 学习 cmake 4. 学习 gcc-arm 交叉编译

    六、新增 golang 接口

    【难度】⭐⭐⭐⭐⭐ 【描述】参考 mmdeploy 已支持的 c api 接口,新增 golang API。 【学习技能】 1. Golang 2. mmdeploy SDK

    七、新增 Rust 接口

    【难度】⭐⭐⭐⭐⭐⭐ 【描述】参考 mmdeploy 已支持的 c api 接口,新增 Rust API,并发布到 crates.io 上。最终可通过 cargo 直接集成 Rust mmdeploy。可以参考 rust-ncnn/mgelite-rs 等 rust 系列 repo 【学习技能】 1. 学习 Rust 2. mmdeploy SDK 3. 上传 crates.io,以及如何单独维护一个小型 repo

    GLCC 夏令营 的差异

    GLCC 的经费来自 CCFxGitlink,我们需要保证项目可在 DDL 内完成,所以 GLCC 倾向已经具备对应技能的同学。

    mmdeploy 的经费来自 OpenMMLab,【学习技能】指 “现在可能还不会,未来 3 个月即将学会”。

    opened by tpoisonooo 10
Releases(v0.5.0)
  • v0.5.0(Jun 9, 2022)

    Highlight

    • Provide prebuilt packages since v0.5.0
    • Decouple pytorch2onnx and onnx2backends
    • Support text detection models PANet, PSENet and DBNet, with CUDA accelerated postprocessing in SDK
    • Support MMRotate

    Features

    • Add prebuild tools (#545, #347)
    • Experimental executor support in SDK (#497)
    • Support ViT on ncnn (#477, #403)
    • Support LiteHRNet on ncnn (#316)
    • Support more text detection models PANet, PSENet and DBNet, with CUDA accelerated postprocessing in SDK (#446, #526, #534)
    • Add C# API for SDK (#388, #535)
    • Support ncnn quantization (#476)
    • Support RepPoints on TensorRT (#457)
    • Support MMRotate on ONNX Runtime and TensorRT (#277, #312, #422, #450, #428, #473)
    • Support MMRazor (#220, #467)

    Improvements

    • Remove spdlog manual installation but still keep it as an option (#423, #544) Users can turn on the following option to use the external spdlog
    cmake .. -DMMDEPLOY_SPDLOG_EXTERNAL=ON
    
    • Add SDK python demos (#554)
    • Add ONNX passes support (#390)
    • Decouple pytorch2onnx and onnx2backends (#529, #540)
    • Add scripts and configs to test metrics of deployed model with all inference backend (#425, #302, #551, #542)
    • Support MDCN and DeformConv TensorRT FP16 (#503, #468)
    • Add interactive build script for Linux and NVIDIA platform (#399)
    • Optimize global average pooling when exporting ONNX (#478)
    • Refactor onnx2ncnn, add test cases and simplify code (#436)
    • Remove expand operation from mmdet rewrite (#371)

    Bug fixes

    • Update CMake scripts to fix building problems (#544, #553)
    • Make ONNXRuntime wrapper work both for cpu and cuda execution (#438, #532)
    • Fix PSPNet-TorchScript conversion error (#538)
    • Resolve the incompatible issue when upgrading MMPose from v0.25.0 to v0.26.0 (#518, #527)
    • Fix mismatched device issue when testing Mask R-CNN deployed model (#511)
    • Remove redundant resize in mmseg EncoderDecoder rewrite (#480)
    • Fix display bugs on headless devices (#451)
    • Fix MMDet3D pillarencode deployment failure (#331)
    • Make the latest spdlog compatible (#423)
    • Fix CI (#462, #447, #440, #426, #441)
    • Fix a bug that causes exporting to onnx failed with static shape and batch size > 1 (#501)
    • Make --work-dir default to$pwd in tools/deploy.py (#483)

    Documentation

    • Fix user document errors, reorganize them, update REAME and rewrite the GET_STARTED chapters (#418, #482, #509, #531, #547, #543)
    • Rewrite the get_started for Jetson platforms (#484, #449, #415, #381)
    • Fix APIs rendering failure in readthedocs (#443)
    • Remove '' in API docstring (#495)
    • More tutorials in Chinese are checked in - Tutorial 05: ONNX Model Editing and Tutorial 04: onnx custom op (#508, #517)

    Contributors

    @sanjaypavo @PeterH0323 @tehkillerbee @zytx121 @triple-Mu @zhiqwang @gyf304 @lakshanthad @Dchaoqun @zhouzaida @NagatoYuki0943 @VVsssssk @irexyc @RunningLeon @hanrui1sensetime @lzhangzz @grimoire @tpoisonooo @AllentDan @SingleZombie

    Source code(tar.gz)
    Source code(zip)
    mmdeploy-0.5.0-linux-x86_64-cuda10.2-tensorrt8.2.3.0.tar.gz(95.71 MB)
    mmdeploy-0.5.0-linux-x86_64-cuda11.1-tensorrt8.2.3.0.tar.gz(105.07 MB)
    mmdeploy-0.5.0-linux-x86_64-onnxruntime1.8.1.tar.gz(49.96 MB)
    mmdeploy-0.5.0-windows-amd64-cuda10.2-tensorrt8.2.3.0.zip(173.17 MB)
    mmdeploy-0.5.0-windows-amd64-cuda11.1-tensorrt8.2.3.0.zip(219.82 MB)
    mmdeploy-0.5.0-windows-amd64-onnxruntime1.8.1.zip(69.20 MB)
  • v0.4.1(Apr 29, 2022)

    Improvements

    • Add IPython notebook tutorial (#234)
    • Support detecting TensorRT from CUDA_TOOLKIT_ROOT_DIR (#357)
    • Build onnxruntime backend in GPU dockerfile (#366)
    • Add CircleCI workflow for linting (#348)
    • Support saving results when testing the deployed model of MMEdit (#336)
    • Support GPU postprocessing for instance segmentation (#276)

    Bug fixes

    • Make empty bounding box list allowed in text recognizer and pose detector C API (#310, #396)
    • Fix the logic of extracting model name from config (#394)
    • Fix feature test for std::source_location (#416)
    • Add missing codegen for sm_53 to support Jetson Nano (#407)
    • Fix crash caused by accessing the wrong tensor in segmentor C API (#363)
    • Fix reading mat type from the wrong image in a batch (#362)
    • Fix missing binary flag when saving temp OpenVINO model (#353)
    • Fix Windows build for pose demo (#307)

    Documents

    • Refine documents by fixing typos, correcting build commands, and removing redundant doc tree (#352, #360, #378, #398)
    • Add a tutorial about torch2onnx in Chinese (#365)

    Contributors

    @irexyc @VVsssssk @AllentDan @lzhangzz @PeterH0323 @RunningLeon @zly19540609 @triple-Mu @grimoire @hanrui1sensetime @SingleZombie @Adenialzz @tpoisonooo @lvhan028 @xizi

    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(Apr 1, 2022)

    Features

    • Support MMPose model inference in SDK: HRNet, LiteHRNet and MSPN
    • Support MMDetection3D: PointPillars and CenterPoint(pillar)
    • Support Andoid platform so as to benefit the development of android apps
    • Support fcn_unet deployment with dynamic shape
    • Support TorchScript

    Improvements

    • Optimize TRTMultiLevelRoiAlign plugin
    • Remove RoiAlign plugin for ONNXRuntime
    • Add DCN TensorRT plugin
    • Update pad logic in detection heads
    • Refactor the rewriter module of Model Converter
    • Suppress CMAKE_CUDA_ARCHITECTURES warnings
    • Update cmake scripts to ensure that the thirdparty packages are relocatable

    Bug fixes

    • Fix the crash on the headless installation
    • Correct the deployment configs for MMSegmentation
    • Optimize prepocess module and fix the potential use-after-free issue
    • Resolve the compatibility with torch 1.11
    • Fix the errors when deploying yolox model
    • Fix the errors occurred during docker build

    Documents

    • Reorganize the build documents. Add more details about how to build MMDeploy on Linx, Windows and Android platforms
    • Publish two chapters about the knowledge of model deployment
    • Update the supported model list, including MMSegmentation,MMPose and MMDetection3D
    • Translate the tutorial of "How to support new backends" into Chinese
    • Update the FAQ

    Contributors

    @irexyc @lvhan028 @RunningLeon @hanrui1sensetime @AllentDan @grimoire @lzhangzz @SemyonBevzuk @VVsssssk @SingleZombie @raykindle @yydc-0 @haofanwang @LJoson @PeterH0323

    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Feb 28, 2022)

    Features

    • Support for windows platform.(#106)
    • Support mmpose codebase.(#94)
    • Support GFL model from mmdetection.(#124)
    • Support export hardsigmoid in torch<=1.8.(#169)

    Improvements

    • Support mmocr v0.4+.(#115)
    • Upgrade isort in pre-commit config.(#141)
    • Opimize delta2bboxes.(#152)

    Bug fixes

    • Fix onnxruntime wrapper for gpu inference. (#123)
    • Fix ci.(#144)
    • Fix tests for OpenVINO with python 3.6. (#125)
    • Added TensorRT version check. (#133)
    • Fix a type error when computing scale_factor in rewriting interpolate.(#185)

    Documents

    • Add Chinese documents How_to_support_new_model.md and How_to_write_config.md (#147,#137)

    Contributors

    A total of 19 developers contributed to this release.

    @grimoire @RunningLeon @AllentDan @lvhan028 @hhaAndroid @SingleZombie @lzhangzz @hanrui1sensetime @Vvsssssk @SemyonBevzuk @ypwhs @TheSeriousProgrammer @matrixgame2018 @tehkillerbee @uniyushu @haofanwang @ypwhs @zhouzaida @q3394101

    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Jan 28, 2022)

    Features

    • Support Nvidia Jetson deployment. (Nano, TX2, Xavier)
    • Add Python interface for SDK inference. (#27)
    • Support yolox on ncnn. (#29)
    • Support segmentation model UNet. (#77)
    • Add docker files. (#67)

    Improvements

    • Add coverage report, CI to GitHub repository. (#16, #34, #35)
    • Refactor the config utilities. (#12, #36)
    • Remove redundant copy operation when converting model. (#61)
    • Simplify single batch NMS. (#99)

    Documents

    • Now our English and Chinese documents are available on readthedocs: English 简体中文
    • Benchmark and tutorial for Nvidia Jetson Nano. (#71)
    • Fix docstring, links in documents. (#18, #32, #60, #84)
    • More documents for TensorRT and OpenVINO. (#96, #102)

    Bug fixes

    • Avoid outputting empty tensor in NMS for ONNX Runtime. (#42)
    • Fix TensorRT 7 SSD. (#49)
    • Fix mmseg dynamic shape. (#57)
    • Fix bugs about pplnn. (#40, #74)

    Contributors

    A total of 14 developers contributed to this release.

    @grimoire @RunningLeon @AllentDan @SemyonBevzuk @lvhan028 @hhaAndroid @Stephenfang51 @SingleZombie @lzhangzz @hanrui1sensetime @VVsssssk @zhiqwang @tehkillerbee @Echo-minn

    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Dec 27, 2021)

    Major Features

    • Fully support OpenMMLab models

      We provide a unified model deployment toolbox for the codebases in OpenMMLab. The supported codebases are listed as below, and more will be added in the future

      • [x] MMClassification (== 0.19.0)
      • [x] MMDetection (== 2.19.0)
      • [x] MMSegmentation (== 0.19.0)
      • [x] MMEditing (== 0.11.0)
      • [x] MMOCR (== 0.3.0)
    • Multiple inference backends are available

      Models can be exported and run in different backends. The following ones are supported, and more will be taken into consideration

      • [x] ONNX Runtime (>= 1.8.0)
      • [x] TensorRT (>= 7.2)
      • [x] PPLNN (== 0.3.0)
      • [x] ncnn (== 20211208)
      • [x] OpenVINO (2021 4 LTS)
    • Efficient and highly scalable SDK Framework by C/C++

      All kinds of modules in SDK can be extensible, such as Transform for image processing, Net for Neural Network inference, Module for postprocessing and so on.

    Contributors

    A total of 11 developers contributed to this release.

    @grimoire @lvhan028 @AllentDan @VVsssssk @SemyonBevzuk @lzhangzz @RunningLeon @SingleZombie @del-zhenwu @zhouzaida @hanrui1sensetime

    Source code(tar.gz)
    Source code(zip)
model infer framework with multithreads based on PaddleX

model_infer_multiThreads (最近更新:2021-10-28 增加了原生的所有api接口,支持clas/det/seg/mask) 该repo基于PaddleX模型推理动态链接库的接口代码进行修改,支持多线程并行访问。大部分代码均来自paddleX的model_infer.cp

Liang Su 14 Apr 27, 2022
High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

What is xLearn? xLearn is a high performance, easy-to-use, and scalable machine learning package that contains linear model (LR), factorization machin

Chao Ma 3k Jun 24, 2022
Deploying Deep Learning Models in C++: BERT Language Model

This repository show the code to deploy a deep learning model serialized and running in C++ backend.

null 42 Mar 24, 2022
Object Based Generic Perception Object Model

This model is a highly parameterizable generic perception sensor and tracking model. It can be parameterized as a Lidar or a Radar. The model is based on object lists and all modeling is performed on object level.

TU Darmstadt - FZD 5 Jun 11, 2022
This is a small example project, that showcases the possibility of using a surrogate model to estimate the drag coefficient of arbitrary triangles.

flowAroundTriangles This is a small example project, that showcases the possibility of using a surrogate model to estimate the drag coefficient of arb

null 5 Jun 23, 2022
MozoLM: A language model (LM) serving library

A language model serving library, with middleware functionality including mixing of probabilities from disparate base language model types and tokenizations along with RPC client/server interactions.

Google Research 35 Jun 19, 2022
HackySAC is a C++ header only library for model estimation using RANSAC.

HackySAC HackySAC is a C++ header only library for model estimation using RANSAC. Available under the MIT license. Examples Minimal working example fo

Jonathan Broere 1 Oct 10, 2021
Operating system model using an assembler RISC-V RV32I instruction set.(development)

General Information Operating system model using an assembler RISC-V RV32I instruction set.(development) С++ Standard - c++17 gcc 9.3.0(Linux,unicode)

Alex Green 1 Dec 21, 2021
Hardware-accelerated DNN model inference ROS2 packages using NVIDIA Triton/TensorRT for both Jetson and x86_64 with CUDA-capable GPU.

Isaac ROS DNN Inference Overview This repository provides two NVIDIA GPU-accelerated ROS2 nodes that perform deep learning inference using custom mode

NVIDIA Isaac ROS 36 Jun 20, 2022
It is a simple AI model running at Art-Pi.

Garbage Classification Device at Embedded AI Summer School 2021 of Nanjing IC Training Base Based on garbage datasets online, apply TensorFlow to get

MJ.XU 5 May 15, 2022
A lightweight 2D Pose model can be deployed on Linux/Window/Android, supports CPU/GPU inference acceleration, and can be detected in real time on ordinary mobile phones.

A lightweight 2D Pose model can be deployed on Linux/Window/Android, supports CPU/GPU inference acceleration, and can be detected in real time on ordinary mobile phones.

JinquanPan 43 Jun 21, 2022
A GKR-based zero-knowledge proof protocol for CNN model inference.

zkCNN Introduction This is the implementation of this paper, which is a GKR-based zero-knowledge proof for CNN reference, containing some common CNN m

null 26 Jun 21, 2022
Movenet cpp deploy; model transformed from tensorflow

MoveNet-PaddleLite Adapted from PaddleDetection; Movenet cpp deploy based on PaddleLite; Movenet model transformed from tensorflow; 简介 Movenet是近年的优秀开源

null 9 May 28, 2022
Pedestrian simulator powered by the social force model

Pedestrian simulator powered by the social force model

Social Robotics Lab, University of Freiburg 282 Jun 22, 2022
A Tool for Verifying Neural Networks using SMT-Based Model Checking

Project Title QNNVerifier Description A Tool for Verifying Neural Networks using SMT-Based Model Checking. Using Frama-C and ESBMC as the backends. Yo

null 2 Dec 11, 2021
CoDi is a cellular automaton model for spiking neural networks

CoDi CoDi is a cellular automaton (CA) model for spiking neural networks (SNNs). CoDi is an acronym for Collect and Distribute, referring to the signa

Jett LaRue 6 May 5, 2022
Deploy ultralytics Yolov5 pretained model with C++ language

Introdution Deploy ultralytics Yolov5 pretained model with C++ language ; Env GCC 7.5 Opencv 4.5.4 Get ONNX Model go to yolov5 release page download y

Xee 32 Jun 11, 2022
power-grid-model is a Python library for steady-state distribution power system analysis

Power Grid Model power-grid-model is a Python library for steady-state distribution power system analysis. The core of the library is written in C++.

Alliander Open Source 34 Jun 9, 2022