Forward - A library for high performance deep learning inference on NVIDIA GPUs
- Forward - A library for high performance deep learning inference on NVIDIA GPUs
Forward is a library for high performance deep learning inference on NVIDIA GPUs. It provides a well-designed scheme that directly parse Tensorflow/PyTorch/Keras models to high-performance engine based on TensorRT. Compared to TensorRT, it is easy-to-use and easy-to-expand. So far, Forward supports not only mainstream deep learning models in CV, NLP and Recommend fields, but also some advanced models such as BERT, GAN, FaceSwap, StyleTransfer.
- Utilize TensorRT API and customized operators for high-performance deep learning inference.
- Support not only mainstream deep learning models in CV, NLP and Recommend fields, but also advanced models such as BERT, GAN, FaceSwap, StyleTransfer.
- Support FLOAT/HALF/INT8 infer modes.
- Easy to use: Load directly Tensorflow(.pb)/PyTorch(.pth)/Keras(.h5) models and then do inference with TensorRT.
- Easy to expand: Register customized layers refer to add_support_op.md.
- Provide C++ and Python interfaces.
- NVIDIA CUDA >= 10.0, CuDNN >= 7 (Recommended version: CUDA 10.2 )
- TensorRT >= 220.127.116.11, (Recommended version: TensorRT-18.104.22.168)
- CMake >= 3.10.1
- GCC >= 5.4.0, ld >= 2.26.1
- (Pytorch) pytorch == 1.3.1
- (Tensorflow) TensorFlow == 1.15.0 (download Tensorflow 1.15.0 and unzip it to
- (Keras) HDF 5
Build with CMake
Generate Makefiles or VS project (Windows) and build. Forward can be built for different framework, such as Fwd-Torch, Fwd-Python-Torch, Fwd-Tf, Fwd-Python-Tf, Fwd-Keras, Fwd-Python-Keras, which controlled by CMake options. For example, Fwd-Python-Tf is built as below.
mkdir build cd build cmake .. \ -DTensorRT_ROOT=/path/to/TensorRT \ -DENABLE_LOGGING=ON \ -DENABLE_PROFILING=ON \ -DENABLE_DYNAMIC_BATCH=ON \ -DBUILD_PTYHON_LIB=ON \ -DENABLE_TORCH=OFF \ -DENABLE_TENSORFLOW=ON \ -DENABLE_KERAS=OFF \ make -j
CMake build arguments
TensorRT_ROOT[Required]: Path to the TensorRT installation directory containing libraries
- More CMake options refer to CMake Options
When the project is built, unit_test can be used to verify the project is successfully built.
cd build/bin ./unit_test --gtest_filter=TestTfNodes.*
Refer to Demo for using Forward-Cpp in Linux
Refer to Demo for using Forward-Python
Notice: The name of INPUT in models can be viewed by model viewers, such as Netron.
Models & Operators
Any form of contribution is welcome. The above contributors have been officially released by Tencent.
We very much welcome developers to contribute to Tencent's open source, and we will also give them incentives to acknowledge and thank them. Here we provide an official description of Tencent's open source contribution. Specific contribution rules for each project are formulated by the project team. Developers can choose the appropriate project and participate according to the corresponding rules. The Tencent Project Management Committee will report regularly to qualified contributors and awards will be issued by the official contact.