Deploying Deep Learning Models in C++: BERT Language Model

Last update: Nov 14, 2022

Overview

Deploying Deep Learning Models in C++: BERT Language Model

This repository show the code to deploy a deep learning model serialized and running in C++ backend.

For the sake of comparison, the same model was loaded using PyTorch JIT and ONNX Runtime.

The C++ server can be used as a code template to deploy another models.

Results

The test consist of make 100 async calls to the server. The client measure the time taken to get the response of the 100 requests. To reduce variability it is performed 10x, i.e, 1000 requests.

The results was obtained in a Macbook Pro with a i7 6C-12T CPU.
The CPU fans was set the max to reduce any possible thermal-throttling.
Before start each test the temps was lowered down around 50 C.

Model Details

Model: BERT Large uncased pretrained from hugginface.
Input: Text with [MASK] token to be predicted
Output: Tensor in shape (seq_len, 30522)

Python server details

FastAPI web-app server running by hypercorn with 6 workers (adjust this by the number of threads you have).

server-python/app_pt.py: Load the model using PyTorch load_model(), the standard way of load a PyTorch model.
server-python/app_jit.py: Load the serialized model using PyTorch JIT runtime.
server-python/app_onnx.py: Load the serialized model using ONNX runtime.

C++ server details

To deploy the model in C++ it was used the same serialized model used in JIT runtime. The model is loaded using libtorch C++ API. To deploy as a http-server, it is used crow. The server is started in multithreaded mode.

server-cpp/server.cpp: Code to generate the c++ application that load the model and start the http server on port 8000.

Deploy a model in C++ is not as straightforward as it is in Python. Check the libs folder for more details about the codes used.

Client details

Client uses async and multi-thread to perform as many requests as possible in parallel. See the test_api.py

TO-DO

Experiment on CUDA device.
Experiment another models like BERT.
Exprimento quantized models.

References

You might also like...

High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

What is xLearn? xLearn is a high performance, easy-to-use, and scalable machine learning package that contains linear model (LR), factorization machin

3k Dec 23, 2022

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

Apache MXNet (incubating) for Deep Learning Apache MXNet is a deep learning framework designed for both efficiency and flexibility. It allows you to m

20.2k Dec 31, 2022

Microsoft Cognitive Toolkit (CNTK), an open source deep-learning toolkit

CNTK Chat Windows build status Linux build status The Microsoft Cognitive Toolkit (https://cntk.ai) is a unified deep learning toolkit that describes

17.3k Dec 23, 2022

header only, dependency-free deep learning framework in C++14

The project may be abandoned since the maintainer(s) are just looking to move on. In the case anyone is interested in continuing the project, let us k

5.6k Dec 31, 2022

LibDEEP BSD-3-ClauseLibDEEP - Deep learning library. BSD-3-Clause

LibDEEP LibDEEP is a deep learning library developed in C language for the development of artificial intelligence-based techniques. Please visit our W

22 Dec 8, 2022

Caffe: a fast open framework for deep learning.

Caffe Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research (BAIR)/The Berke

33k Dec 30, 2022

Deep Learning API and Server in C++11 support for Caffe, Caffe2, PyTorch,TensorRT, Dlib, NCNN, Tensorflow, XGBoost and TSNE

Open Source Deep Learning Server & API DeepDetect (https://www.deepdetect.com/) is a machine learning API and server written in C++11. It makes state

2.4k Dec 30, 2022

Forward - A library for high performance deep learning inference on NVIDIA GPUs

a library for high performance deep learning inference on NVIDIA GPUs.

123 Mar 17, 2021

A library for high performance deep learning inference on NVIDIA GPUs.

Forward - A library for high performance deep learning inference on NVIDIA GPUs Forward - A library for high performance deep learning inference on NV

509 Dec 17, 2022

Deploying Deep Learning Models in C++: BERT Language Model

Related tags

Overview

Deploying Deep Learning Models in C++: BERT Language Model

Results

Model Details

Python server details

C++ server details

Client details

TO-DO

References

You might also like...

High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

Microsoft Cognitive Toolkit (CNTK), an open source deep-learning toolkit

header only, dependency-free deep learning framework in C++14

LibDEEP BSD-3-ClauseLibDEEP - Deep learning library. BSD-3-Clause

Caffe: a fast open framework for deep learning.

Deep Learning API and Server in C++11 support for Caffe, Caffe2, PyTorch,TensorRT, Dlib, NCNN, Tensorflow, XGBoost and TSNE

Forward - A library for high performance deep learning inference on NVIDIA GPUs

A library for high performance deep learning inference on NVIDIA GPUs.

Owner

Deep Scalable Sparse Tensor Network Engine (DSSTNE) is an Amazon developed library for building Deep Learning (DL) machine learning (ML) models

tutorial on how to train deep learning models with c++ and dlib.

Transformer related optimization, including BERT, GPT

a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.

A library for creating Artificial Neural Networks, for use in Machine Learning and Deep Learning algorithms.

Deep Learning in C Programming Language. Provides an easy way to create and train ANNs.

Triton - a language and compiler for writing highly efficient custom Deep-Learning primitives.

Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.

A flexible, high-performance serving system for machine learning models

Turi Create simplifies the development of custom machine learning models.