Deploying Deep Learning Models in C++: BERT Language Model

Overview

Deploying Deep Learning Models in C++: BERT Language Model

This repository show the code to deploy a deep learning model serialized and running in C++ backend.

For the sake of comparison, the same model was loaded using PyTorch JIT and ONNX Runtime.

The C++ server can be used as a code template to deploy another models.

Results

The test consist of make 100 async calls to the server. The client measure the time taken to get the response of the 100 requests. To reduce variability it is performed 10x, i.e, 1000 requests.

  • The results was obtained in a Macbook Pro with a i7 6C-12T CPU.
  • The CPU fans was set the max to reduce any possible thermal-throttling.
  • Before start each test the temps was lowered down around 50 C.

Model Details

  • Model: BERT Large uncased pretrained from hugginface.
  • Input: Text with [MASK] token to be predicted
  • Output: Tensor in shape (seq_len, 30522)

Python server details

FastAPI web-app server running by hypercorn with 6 workers (adjust this by the number of threads you have).

C++ server details

To deploy the model in C++ it was used the same serialized model used in JIT runtime. The model is loaded using libtorch C++ API. To deploy as a http-server, it is used crow. The server is started in multithreaded mode.

  • server-cpp/server.cpp: Code to generate the c++ application that load the model and start the http server on port 8000.

Deploy a model in C++ is not as straightforward as it is in Python. Check the libs folder for more details about the codes used.

Client details

Client uses async and multi-thread to perform as many requests as possible in parallel. See the test_api.py

TO-DO

  • Experiment on CUDA device.
  • Experiment another models like BERT.
  • Exprimento quantized models.

References

You might also like...
High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.
High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

What is xLearn? xLearn is a high performance, easy-to-use, and scalable machine learning package that contains linear model (LR), factorization machin

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more
Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

Apache MXNet (incubating) for Deep Learning Apache MXNet is a deep learning framework designed for both efficiency and flexibility. It allows you to m

Microsoft Cognitive Toolkit (CNTK), an open source deep-learning toolkit

CNTK Chat Windows build status Linux build status The Microsoft Cognitive Toolkit (https://cntk.ai) is a unified deep learning toolkit that describes

header only, dependency-free deep learning framework in C++14
header only, dependency-free deep learning framework in C++14

The project may be abandoned since the maintainer(s) are just looking to move on. In the case anyone is interested in continuing the project, let us k

LibDEEP BSD-3-ClauseLibDEEP - Deep learning library. BSD-3-Clause

LibDEEP LibDEEP is a deep learning library developed in C language for the development of artificial intelligence-based techniques. Please visit our W

Caffe: a fast open framework for deep learning.

Caffe Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research (BAIR)/The Berke

Deep Learning API and Server in C++11 support for Caffe, Caffe2, PyTorch,TensorRT, Dlib, NCNN, Tensorflow, XGBoost and TSNE

Open Source Deep Learning Server & API DeepDetect (https://www.deepdetect.com/) is a machine learning API and server written in C++11. It makes state

 Forward - A library for high performance deep learning inference on NVIDIA GPUs
Forward - A library for high performance deep learning inference on NVIDIA GPUs

a library for high performance deep learning inference on NVIDIA GPUs.

A library for high performance deep learning inference on NVIDIA GPUs.
A library for high performance deep learning inference on NVIDIA GPUs.

Forward - A library for high performance deep learning inference on NVIDIA GPUs Forward - A library for high performance deep learning inference on NV

Owner
null
Deep Scalable Sparse Tensor Network Engine (DSSTNE) is an Amazon developed library for building Deep Learning (DL) machine learning (ML) models

Amazon DSSTNE: Deep Scalable Sparse Tensor Network Engine DSSTNE (pronounced "Destiny") is an open source software library for training and deploying

Amazon Archives 4.4k Sep 26, 2022
tutorial on how to train deep learning models with c++ and dlib.

Dlib Deep Learning tutorial on how to train deep learning models with c++ and dlib. usage git clone https://github.com/davisking/dlib.git mkdir build

Abdolkarim Saeedi 1 Dec 21, 2021
Transformer related optimization, including BERT, GPT

FasterTransformer This repository provides a script and recipe to run the highly optimized transformer-based encoder and decoder component, and it is

NVIDIA Corporation 1.4k Sep 27, 2022
a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.

a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.

Tencent 1.2k Oct 2, 2022
A library for creating Artificial Neural Networks, for use in Machine Learning and Deep Learning algorithms.

iNeural A library for creating Artificial Neural Networks, for use in Machine Learning and Deep Learning algorithms. What is a Neural Network? Work on

Fatih Küçükkarakurt 5 Apr 5, 2022
Deep Learning in C Programming Language. Provides an easy way to create and train ANNs.

cDNN is a Deep Learning Library written in C Programming Language. cDNN provides functions that can be used to create Artificial Neural Networks (ANN)

Vishal R 11 Apr 18, 2022
Triton - a language and compiler for writing highly efficient custom Deep-Learning primitives.

Triton - a language and compiler for writing highly efficient custom Deep-Learning primitives.

OpenAI 4.1k Oct 2, 2022
Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.

This is the Vowpal Wabbit fast online learning code. Why Vowpal Wabbit? Vowpal Wabbit is a machine learning system which pushes the frontier of machin

Vowpal Wabbit 8k Sep 28, 2022
A flexible, high-performance serving system for machine learning models

XGBoost Serving This is a fork of TensorFlow Serving, extended with the support for XGBoost, alphaFM and alphaFM_softmax frameworks. For more informat

iQIYI 125 Sep 22, 2022
Turi Create simplifies the development of custom machine learning models.

Quick Links: Installation | Documentation Turi Create Turi Create simplifies the development of custom machine learning models. You don't have to be a

Apple 10.8k Oct 5, 2022