Coqui Inference Engine
Coqui Inference Engine is a library for efficiently deploying speech models.
This project is at an early proof-of-concept stage. Collaboration on design and implementation is very welcome. Join our Gitter channel by clicking the badge above!
This project is the successor to the STT "native client", containing the core inference logic for deploying Coqui STT models (and eventually Coqui TTS and other models too).
Coqui Inference Engine aims to be:
- Fast: streaming inference with low latency on small devices (phones, IoT)
- Easy to use: simple, stable, well-documented API
- Available: easy to expose to different programming languages, available in standard package managers
- Extensible: able to handle different model types, architectures, and formats
Build instructions
For the build you'll need to install CMake >= 3.10.
Currently you'll also have to build onnxruntime yourself and place the built files manually before building the inference engine, following the steps below:
$ # Clone the Coqui Inference Engine repo
$ git clone https://github.com/coqui-ai/inference-engine
$ # Clone onnxruntime repo
$ git clone --recursive https://github.com/microsoft/onnxruntime/
$ cd onnxruntime
$ # Build it
$ ./build.sh --config Debug --parallel
$ # Copy built files for inference engine build
$ cp -R build/*/*/libonnxruntime* ../inference-engine/onnxruntime/lib
Now, we're ready to build the inference engine:
$ cd ../inference-engine
$ # Create build dir
$ mkdir build
$ cd build
$ # Prepare build
$ cmake -DCMAKE_BUILD_TYPE=Debug ..
$ # Build
$ make -j
You should now be able to run the test client by running ./main.
$ ./main --model ../output_graph.onnx --audio ../test-audio.wav
Exporting a test model from Coqui STT
You can use the experimental-inference-engine-export branch of Coqui STT to export an STT checkpoint in the format expected by the inference engine.
$ git clone --branch experimental-inference-engine-export https://github.com/coqui-ai/STT
$ cd STT
$ python -m pip install -e .
$ cd native_client/ctcdecode
$ make bindings
$ python -m pip install --force-reinstall dist/*.whl
After the steps above, you can then follow the documentation for exporting a model, and include the --export_onnx true flag. You should then get an output_graph.onnx file exported which can be read by the inference engine.
33 Sep 2, 2022
9 Dec 14, 2022
13 Nov 24, 2022
34 Aug 2, 2022
276 Dec 28, 2022
71 Nov 20, 2022
2 May 24, 2022
7 Jul 9, 2022
122 Nov 28, 2022
1 Jul 17, 2022
13 Jul 24, 2022
267 Dec 23, 2022
1.2k Jan 8, 2023
5 Sep 10, 2022
540 Dec 22, 2022
198 Jan 5, 2023
1 Oct 25, 2021
3 Jan 6, 2022
262 Jan 2, 2023