A C++ standalone library for machine learning

Overview

Flashlight: Fast, Flexible Machine Learning in C++

Quickstart | Installation | Documentation

CircleCI Documentation Status Docker Image Build Status Join the chat at https://gitter.im/flashlight-ml/community

Docker Image for CUDA backend Docker Image for CPU backend

Install CUDA backend with vcpkg Install CPU backend with vcpkg

Flashlight is a fast, flexible machine learning library written entirely in C++ from the Facebook AI Research Speech team and the creators of Torch and Deep Speech. Its core features include:

  • Just-in-time kernel compilation with modern C++ with the ArrayFire tensor library.
  • CUDA and CPU backends for GPU and CPU training.
  • An emphasis on efficiency and scale.

Native support in C++ and simple extensibility makes Flashlight a powerful research framework that's hackable to its core and enables fast iteration on new experimental setups and algorithms without sacrificing performance. In a single repository, Flashlight provides apps for research across multiple domains:

Project Layout

Flashlight is broken down into a few parts:

  • flashlight/lib contains kernels and standalone utilities for sequence losses, beam search decoding, text processing, and more.
  • flashlight/fl is the core neural network library using the ArrayFire tensor library.
  • flashlight/app are applications of the core library to machine learning across domains.
  • flashlight/ext are extensions on top of Flashlight and ArrayFire that are useful across apps.

Quickstart

First, build and install Flashlight and link it to your own project.

Sequential forms a sequence of Flashlight Modules for chaining computation.

Implementing a simple convnet is easy.
#include <flashlight/fl/flashlight.h>

Sequential model;

model.add(View(af::dim4(IM_DIM, IM_DIM, 1, -1)));
model.add(Conv2D(
    1 /* input channels */,
    32 /* output channels */,
    5 /* kernel width */,
    5 /* kernel height */,
    1 /* stride x */,
    1 /* stride y */,
    PaddingMode::SAME; /* padding mode */,
    PaddingMode::SAME; /* padding mode */));
model.add(ReLU());
model.add(Pool2D(
    2 /* kernel width */,
    2 /* kernel height */,
    2 /* stride x */,
    2 /* stride y */));
model.add(Conv2D(32, 64, 5, 5, 1, 1, PaddingMode::SAME;, PaddingMode::SAME;));
model.add(ReLU());
model.add(Pool2D(2, 2, 2, 2));
model.add(View(af::dim4(7 * 7 * 64, -1)));
model.add(Linear(7 * 7 * 64, 1024));
model.add(ReLU());
model.add(Dropout(0.5));
model.add(Linear(1024, 10));
model.add(LogSoftmax());

Performing forward and backward computation is straightforwards:

auto output = model.forward(input);
auto loss = categoricalCrossEntropy(output, target);
loss.backward();

See the MNIST example for a full tutorial including a training loop and dataset abstractions.

Variable is the base Flashlight tensor that operates on ArrayFire arrays. Tape-based Automatic differentiation in Flashlight is simple and works as you'd expect.

Autograd Example
auto A = Variable(af::randu(1000, 1000), true /* calcGrad */);
auto B = 2.0 * A;
auto C = 1.0 + B;
auto D = log(C);
D.backward(); // populates A.grad() along with gradients for B, C, and D.

Building and Installing

Install with vcpkg | With Docker | From Source | From Source with vcpkg | Build Your Project with Flashlight

Requirements

At minimum, compilation requires:

  • A C++ compiler with good C++14 support (e.g. gcc/g++ >= 5)
  • CMake — version 3.10 or later, and make
  • A Linux-based operating system.

See the full dependency list for more details if building from source.

Instructions for building/installing Python bindings can be found here.

Flashlight Build Setups

Flashlight can be broken down into several components as described above. Each component can be incrementally built by specifying the correct build options.

There are two ways to work with Flashlight:

  1. As an installed library that you link to with your own project. This is best for building standalone applications dependent on Flashlight.
  2. With in-source development where the Flashlight project source is changed and rebuilt. This is best if customizing/hacking the core framework or the Flashlight-provided app binaries.

Flashlight can be built in one of two ways:

  1. With vcpkg, a C++ package manager.
  2. From source by installing dependencies as needed.

Installing Flashlight with vcpkg

Library Installation with vcpkg

Flashlight is most-easily built and installed with vcpkg. Both the CUDA and CPU backends are supported with vcpkg. For either backend, first install Intel MKL. For the CUDA backend, install CUDA >= 9.2, cuDNN, and NCCL. Then, after installing vcpkg, install the libraries and core with:

./vcpkg install flashlight-cuda # CUDA backend, OR
./vcpkg install flashlight-cpu  # CPU backend

To install Flashlight apps, check the features available for installation by running ./vcpkg search flashlight-cuda or ./vcpkg search flashlight-cpu. Each app is a "feature": for example, ./vcpkg install flashlight-cuda[asr] installs the ASR app with the CUDA backend.

Below is the currently-supported list of features (for each of flashlight-cuda and flashlight-cpu):

flashlight-{cuda/cpu}[lib]      # Flashlight libraries
flashlight-{cuda/cpu}[nn]       # Flashlight neural net library
flashlight-{cuda/cpu}[asr]      # Flashlight speech recognition app
flashlight-{cuda/cpu}[lm]       # Flashlight language modeling app
flashlight-{cuda/cpu}[imgclass] # Flashlight image classification app

Flashlight app binaries are also built for the selected features and are installed into the vcpkg install tree's tools directory.

Integrating Flashlight into your own project with is simple using vcpkg's CMake toolchain integration.

From-Source Build with vcpkg

First, install the dependencies for your backend of choice using vcpkg (click to expand the below):

Installing CUDA Backend Dependencies with vcpkg

To build the Flashlight CUDA backend from source using dependencies installed with vcpkg, install CUDA >= 9.2, cuDNN, NCCL, and Intel MKL, then build the rest of the dependencies for the CUDA backend based on which Flashlight features you'd like to build:

./vcpkg install \
    cuda intel-mkl fftw3 cub kenlm                \ # if building flashlight libraries
    arrayfire[cuda] cudnn nccl openmpi cereal stb \ # if building the flashlight neural net library
    gflags glog                                   \ # if building any flashlight apps
    libsndfile                                    \ # if building the flashlight asr app
    gtest                                           # optional, if building tests
Installing CPU Backend Dependencies with vcpkg

To build the Flashlight CPU backend from source using dependencies installed with vcpkg, install Intel MKL, then build the rest of the dependencies for the CPU backend based on which Flashlight features you'd like to build:

./vcpkg install \
    intel-mkl fftw3 kenlm                              \ # for flashlight libraries
    arrayfire[cpu] gloo[mpi] openmpi onednn cereal stb \ # for the flashlight neural net library
    gflags glog                                        \ # for any flashlight apps
    libsndfile                                         \ # for the flashlight asr app
    gtest                                                # optional, for tests
Build Using the vcpkg Toolchain File

To build Flashlight from source with these dependencies, clone the repository:

git clone https://github.com/facebookresearch/flashlight.git && cd flashlight
mkdir -p build && cd build

Then, build from source using vcpkg's CMake toolchain:

cmake .. \
    -DCMAKE_BUILD_TYPE=Release
    -DFL_BACKEND=CUDA
    -DCMAKE_TOOLCHAIN_FILE=[path to your vcpkg clone]/scripts/buildsystems/vcpkg.cmake
make -j$(nproc)
make install -j$(nproc) # only if you want to install Flashlight for external use

To build a subset of Flashlight's features, see the build options below.

Building from Source

To build from source, first install the below dependencies. Most are available with your system's local package manager.

Some dependencies marked below are downloaded and installed automatically if not found on the local system. FL_BUILD_STANDALONE determines this behavior — if disabled, dependencies won't be downloaded and built when building Flashlight.

Once all dependencies are installed, clone the repository:

git clone https://github.com/facebookresearch/flashlight.git && cd flashlight
mkdir -p build && cd build

Then build all Flashlight components with:

cmake .. -DCMAKE_BUILD_TYPE=Release -DFL_BACKEND=[backend] [...build options]
make -j$(nproc)
make install

Setting the MKLROOT environment variable (export MKLROOT=/opt/intel/mkl on most Linux-based systems) can help CMake find Intel MKL if not initially found.

To build a smaller subset of Flashlight features/apps, see the build options below for a complete list of options.

To install Flashlight in a custom directory, use CMake's CMAKE_INSTALL_PREFIX argument. Flashlight libraries can be built as shared libraries using CMake's BUILD_SHARED_LIBS argument.

Flashlight uses modern CMake and IMPORTED targets for most dependencies. If a dependency isn't found, passing -D<package>_DIR to your cmake command or exporting <package>_DIR as an environment variable equal to the path to <package>Config.cmake can help locate dependencies on your system. See the documentation for more details. If CMake is failing to locate a package, check to see if a corresponding issue has already been created before creating your own.

Dependencies

Dependencies marked with * are automatically downloaded and built from source if not found on the system. Setting FL_BUILD_STANDALONE to OFF disables this behavior.

Dependencies marked with ^ are required if building with distributed training enabled (FL_BUILD_DISTRIBUTED — see the build options below). Distributed training is required for all apps.

Dependencies marked with are installable via vcpkg. See the instructions for installing those dependencies above for doing a Flashlight from-source build.

Component Backend Dependencies
libraries CUDA CUDA >= 9.2, CUB*† (if CUDA < 11)
CPU A BLAS library (Intel MKL >= 2018, OpenBLAS†, etc)
core Any ArrayFire >= 3.7.3†, an MPI library^(OpenMPI†, etc),  cereal*† >= 1.3.0, stb*†
CUDA CUDA >= 9.2, NCCL^, cuDNN
CPU oneDNN† >= 2.0, gloo (with MPI)*^†
app: all Any Google Glog†, Gflags
app: asr Any libsndfile*† >= 10.0.28, a BLAS library (Intel MKL >= 2018, OpenBLAS†, etc)
app: imgclass Any -
app: lm Any -
tests Any Google Test (gtest, with gmock)*† >= 1.10.0

Build Options

The Flashlight CMake build accepts the following build options (prefixed with -D when running CMake from the command line):

Name Options Default Value Description
FL_BACKEND CUDA, CPU, OPENCL CUDA Backend with which to build all components.
FL_BUILD_STANDALONE ON, OFF ON Downloads/builds some dependencies if not found.
FL_BUILD_LIBRARIES ON, OFF ON Build the Flashlight libraries.
FL_BUILD_CORE ON, OFF ON Build the Flashlight neural net library.
FL_BUILD_DISTRIBUTED ON, OFF ON Build with distributed training; required for apps.
FL_BUILD_CONTRIB ON, OFF ON Build contrib APIs subject to breaking changes.
FL_BUILD_APPS ON, OFF ON Build apps (see below).
FL_BUILD_APP_ASR ON, OFF ON Build the automatic speech recognition app.
FL_BUILD_APP_IMGCLASS ON, OFF ON Build the image classification app.
FL_BUILD_APP_LM ON, OFF ON Build the language modeling app.
FL_BUILD_APP_ASR_TOOLS ON, OFF ON Build automatic speech recognition app tools.
FL_BUILD_TESTS ON, OFF ON Build tests.
FL_BUILD_EXAMPLES ON, OFF ON Build examples.
FL_BUILD_EXPERIMENTAL ON, OFF OFF Build experimental components.
CMAKE_BUILD_TYPE See docs. Debug See the CMake documentation.
CMAKE_INSTALL_PREFIX [Directory] See docs. See the CMake documentation.

Building Your Own Project with Flashlight

Flashlight is most-easily linked to using CMake. Flashlight exports the following CMake targets when installed:

  • flashlight::fl-libraries — contains flashlight libraries headers and symbols.
  • flashlight::flashlight — contains flashlight libraries as well as the flashlight core autograd and neural network library.
  • flashlight::flashlight-app-asr — contains the automatic speech recognition app along with the flashlight core and flashlight libraries.
  • flashlight::flashlight-app-imgclass — contains the image classification app along with the flashlight core and flashlight libraries.
  • flashlight::flashlight-app-lm — contains the language modeling app along with the flashlight core and flashlight libraries.

Given a simple project.cpp file that includes and links to Flashlight:

#include <iostream>

#include <arrayfire.h>
#include <flashlight/fl/flashlight.h>

int main() {
 fl::Variable v(af::constant(1, 1), true);
 auto result = v + 10;
 std::cout << "Hello World!" << std::endl;
 af::print("Array value is ", result.array()); // 11.000
 return 0;
}

The following CMake configuration links Flashlight and sets include directories:

cmake_minimum_required(VERSION 3.10)
set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

add_executable(myProject project.cpp)

find_package(flashlight CONFIG REQUIRED)
target_link_libraries(myProject PRIVATE flashlight::flashlight)

With a vcpkg Flashlight Installation

If you installed Flashlight with vcpkg, the above CMake configuration for myProject can be built by running:

cd project && mkdir build && cd build
cmake .. \
  -DCMAKE_TOOLCHAIN_FILE=[path to vcpkg clone]/scripts/buildsystems/vcpkg.cmake \
  -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

With a From-Source Flashlight Installation

If using a from-source installation of Flashlight, Flashlight will be found automatically by CMake:

cd project && mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

If Flashlight is installed in a custom location using a CMAKE_INSTALL_PREFIX, passing -Dflashlight_DIR=[install prefix]/share/flashlight/cmake as an argument to your cmake command can help CMake find Flashlight.

Building and Running Flashlight with Docker

Flashlight and its dependencies can also be built with the provided Dockerfiles — see the accompanying Docker documentation for more information.

Contributing and Contact

Contact: [email protected], [email protected], [email protected]m, [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]

Flashlight is being very actively developed. See CONTRIBUTING for more on how to help out.

Acknowledgments

Some of Flashlight's code is derived from arrayfire-ml.

License

Flashlight is under a BSD license. See LICENSE for more information.

Issues
  • Build errors for CPU backend

    Build errors for CPU backend

    While trying to build for CPU backend, I get a list of build error starting with these:-

    [ 61%] Linking CXX executable MemoryFrameworkTest CMakeFiles/MemoryFrameworkTest.dir/memory/MemoryFrameworkTest.cpp.o: In function (anonymous namespace)::MockTestMemoryManager::alloc(bool, unsigned int, long long*, unsigned int)': MemoryFrameworkTest.cpp:(.text+0x5ca): undefined reference totesting::internal::UntypedFunctionMockerBase::SetOwnerAndName(void const*, char const*)' MemoryFrameworkTest.cpp:(.text+0x5eb): undefined reference to testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith(void*)' CMakeFiles/MemoryFrameworkTest.dir/memory/MemoryFrameworkTest.cpp.o: In function(anonymous namespace)::MockTestMemoryManager::allocated(void*)': MemoryFrameworkTest.cpp:(.text+0x6dc): undefined reference to testing::internal::UntypedFunctionMockerBase::SetOwnerAndName(void const*, char const*)' MemoryFrameworkTest.cpp:(.text+0x6ee): undefined reference totesting::internal::UntypedFunctionMockerBase::UntypedInvokeWith(void*)' CMakeFiles/MemoryFrameworkTest.dir/memory/MemoryFrameworkTest.cpp.o: In function (anonymous namespace)::MockTestMemoryManager::jitTreeExceedsMemoryPressure(unsigned long)': MemoryFrameworkTest.cpp:(.text+0x7dc): undefined reference totesting::internal::UntypedFunctionMockerBase::SetOwnerAndName(void const*, char const*)' MemoryFrameworkTest.cpp:(.text+0x7ee): undefined reference to testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith(void*)' CMakeFiles/MemoryFrameworkTest.dir/memory/MemoryFrameworkTest.cpp.o: In function(anonymous namespace)::MockTestMemoryManager::getMemoryPressure()': MemoryFrameworkTest.cpp:(.text+0x8d8): undefined reference to testing::internal::UntypedFunctionMockerBase::SetOwnerAndName(void const*, char const*)' MemoryFrameworkTest.cpp:(.text+0x8e5): undefined reference totesting::internal::UntypedFunctionMockerBase::UntypedInvokeWith(void*)'

    opened by mitulsaha 58
  • Object Detection App using Detection Transformers.

    Object Detection App using Detection Transformers.

    CLA Signed 
    opened by padentomasello 41
  • Flashlight for Computer Vision

    Flashlight for Computer Vision

    CLA Signed 
    opened by padentomasello 39
  • [sfx] time stretch

    [sfx] time stretch

    Add time stretch using sox's algorithm but without libsox threading issues. The algorithm is almost verbatim copied to make review simpler. Changes to original code applied only where required.

    CLA Signed 
    opened by avidov 35
  • getting no error while cmake, but it stuck on make

    getting no error while cmake, but it stuck on make

    getting no error while cmake, but it stuck on make. plz help

    cmake .. -DCMAKE_BUILD_TYPE=Release -DFLASHLIGHT_BACKEND=CPU

    -- ArrayFire found (include: /opt/arrayfire/include, library: ArrayFire::afcuda) -- Could NOT find cereal (missing: cereal_INCLUDE_DIRS) -- cereal NOT found. Will download from source -- Checking for [mkl_gf_lp64 - mkl_gnu_thread - mkl_core - iomp5 - pthread - m] -- Library mkl_gf_lp64: /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so -- Library mkl_gnu_thread: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library iomp5: not found -- Checking for [mkl_gf_lp64 - mkl_intel_thread - mkl_core - iomp5 - pthread - m] -- Library mkl_gf_lp64: /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so -- Library mkl_intel_thread: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library iomp5: not found -- Checking for [mkl_gf - mkl_gnu_thread - mkl_core - iomp5 - pthread - m] -- Library mkl_gf: not found -- Checking for [mkl_gf - mkl_intel_thread - mkl_core - iomp5 - pthread - m] -- Library mkl_gf: not found -- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - iomp5 - pthread - m] -- Library mkl_intel_lp64: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so -- Library mkl_gnu_thread: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library iomp5: not found -- Checking for [mkl_intel_lp64 - mkl_intel_thread - mkl_core - iomp5 - pthread - m] -- Library mkl_intel_lp64: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so -- Library mkl_intel_thread: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library iomp5: not found -- Checking for [mkl_intel - mkl_gnu_thread - mkl_core - iomp5 - pthread - m] -- Library mkl_intel: not found -- Checking for [mkl_intel - mkl_intel_thread - mkl_core - iomp5 - pthread - m] -- Library mkl_intel: not found -- Checking for [mkl_gf_lp64 - mkl_gnu_thread - mkl_core - pthread - m] -- Library mkl_gf_lp64: /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so -- Library mkl_gnu_thread: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so -- Library m: /usr/lib/x86_64-linux-gnu/libm.so -- Checking for [mkl_gf_lp64 - mkl_intel_thread - mkl_core - pthread - m] -- Library mkl_gf_lp64: /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so -- Library mkl_intel_thread: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so -- Library m: /usr/lib/x86_64-linux-gnu/libm.so -- Checking for [mkl_gf - mkl_gnu_thread - mkl_core - pthread - m] -- Library mkl_gf: not found -- Checking for [mkl_gf - mkl_intel_thread - mkl_core - pthread - m] -- Library mkl_gf: not found -- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - pthread - m] -- Library mkl_intel_lp64: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so -- Library mkl_gnu_thread: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so -- Library m: /usr/lib/x86_64-linux-gnu/libm.so -- Checking for [mkl_intel_lp64 - mkl_intel_thread - mkl_core - pthread - m] -- Library mkl_intel_lp64: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so -- Library mkl_intel_thread: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so -- Library m: /usr/lib/x86_64-linux-gnu/libm.so -- Checking for [mkl_intel - mkl_gnu_thread - mkl_core - pthread - m] -- Library mkl_intel: not found -- Checking for [mkl_intel - mkl_intel_thread - mkl_core - pthread - m] -- Library mkl_intel: not found -- Checking for [mkl_gf_lp64 - mkl_sequential - mkl_core - m] -- Library mkl_gf_lp64: /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so -- Library mkl_sequential: /opt/intel/mkl/lib/intel64/libmkl_sequential.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library m: /usr/lib/x86_64-linux-gnu/libm.so -- MKL found -- A library with BLAS API found. -- MKLDNN headers found in /usr/local/include -- Using MKLDNN library found in /usr/local/lib/libmkldnn.so -- Using MKL with MKL-DNN -- MKLDNN found -- Will build flashlight contrib assets. -- Gloo found -- NCCL not found -- MPI_CXX found -- MPI_CXX compile flags: -pthread -- MPI_CXX include path: /usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include/usr/lib/x86_64-linux-gnu/openmpi/include -- MPI_CXX LINK flags path: -pthread -- MPI_CXX libraries: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so/usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so -- MPI_C found -- MPI_C compile flags: -pthread -- MPI_C include path: /usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include/usr/lib/x86_64-linux-gnu/openmpi/include -- MPI_C LINK flags path: -pthread -- MPI_C libraries: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so -- gtest found: (include: /usr/include, lib: /usr/lib/libgtest.a;/usr/lib/libgtest_main.a -- Configuring done -- Generating done -- Build files have been written to: /home/test/Desktop/asr/wav2latter/flashlight/build

    make -j 4

    [ 14%] Linking CXX executable DatasetUtilsTest [ 14%] Building CXX object tests/CMakeFiles/ContribSerializationTest.dir/__/flashlight/contrib/modules/Transformer.cpp.o [ 14%] Built target DatasetUtilsTest [ 14%] Linking CXX executable ContribSerializationTest [ 14%] Built target ContribSerializationTest Makefile:140: recipe for target 'all' failed make: *** [all] Error 2

    opened by shubhamrc49 31
  • Conformer Module

    Conformer Module

    Hi, do you have an implementation of the Conformer Block in Flashlight you can share?

    These are my rough notes for what conformer seems to be: https://gist.github.com/lunixbochs/4d8a8c0ab9be45469337b82363bc2105

    This is roughly what I have so far, based on TDSBlock: https://gist.github.com/lunixbochs/207eff6e78b29e26712cee6fca42c400

    Here's my janky conformer W2L arch based on TDS:

    V -1 NFEAT 1 0
    SAUG 80 27 2 10 0.05 2
    PD 0 5 3
    C2 1 15 10 1 2 1 0 0
    R
    DO 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    CONFORMER 15 144 4 80 32 20 20 0.1
    V 1200 -1 1 0
    L 1200 NLABEL
    V NLABEL 0 -1 1
    
    CONFORMER is:
        return std::make_shared<w2l::ConformerBlock>(
            channels, encoderDim, attentionHeads, width, kernelSize,
            ffnInnerDim, mhsaInnerDim, dropout, rPad, lNormIncludeTime);
    

    Whitepaper is here: https://arxiv.org/pdf/2005.08100.pdf I believe this is a tensorflow implementation: https://github.com/TensorSpeech/TensorFlowASR/blob/main/tensorflow_asr/models/conformer.py#L209

    I haven't worked with Keras enough to get a sense for the correct activation shapes through this, my dimensions are definitely not quite right, and I'm kind of guessing on the precise architecture. I'm happy to fiddle with this more but if you have any advice for major things I may be doing wrong that would be appreciated. Right now I know for sure my shape going into the transformer is wrong, but I don't know how exactly. And I probably did the depth-wise convolution wrong.

    enhancement question 
    opened by lunixbochs 30
  • Update warpctc library

    Update warpctc library

    Original PR: [!215]

    Summary

    Brings the latest updates of warpctc and uses warpctc CMakeLists.txt submit in this PR. It works but I think it will take time to be approved.

    CLA Signed 
    opened by alealv 27
  • Add compiling support for CUDA v11 and cuDNN v8

    Add compiling support for CUDA v11 and cuDNN v8

    Original Issue: [#198] & [#213] & [#147]

    Summary

    • I updated CMakeList.txt of third party library (warp-ctc)[https://github.com/baidu-research/warp-ctc]. The latest CMakeLists.txt handles conditional compilation for architecture compute_30 depending on CUDA version (CUDAv11 deprecates this architecture).
    • The same is done for main CMakeLists.txt file
    • The deprecation policy of CUDA has changed. Therefore, the condition of cudnnSetRNNDescriptor used here needs to also include until version 8000 of cuDNN.
    • Finally, with CUDA v11 Nvidia cub is included in CUDA toolkit, but because flashlight downloads and uses it's own version, it collides with Thrust version. Hence, THRUST_IGNORE_CUB_VERSION_CHECK must be define in if using CUDA v11
    CLA Signed 
    opened by alealv 23
  • Why long audio get poor performance?

    Why long audio get poor performance?

    Question

    I found that decode long audio is much worse than split long audio into multi parts. I use librispeech as dataset and test with decode_transformer_s2s_ngram.cfg. e.g.

    LibriSpeech/test-clean/1995/1836/1995-1836-0004.flac is an audio with 33.91 seconds, its groundTruth text

    as she awaited her guests she surveyed the table with both satisfaction and disquietude for her social functions were few tonight there were she checked them off on her fingers sir james creighton the rich english manufacturer and lady creighton mister and missus vanderpool mister harry cresswell and his sister john taylor and his sister and mister charles smith whom the evening papers mentioned as likely to be united states senator from new jersey a selection of guests that had been determined unknown to the hostess by the meeting of cotton interests earlier in the day
    

    The decode result is

    as she awaited her guest she surveyed the table with both satisfaction and disquietude for her social functions were few to night there were she checked them off on her fingers sir james clinton the rich english manufacturer and lady horton mister and missus vanderpole mister harry cresswell and his sister john taylor and his sister and mister charles smith whom the evening papers mentioned as likely to be united states senator from new jersey a selection of guests that had been determined unknown to the hostess and mister charles smith whom the evening papers mentioned as likely to be united states senator from
    

    Note the ending part evening papers mentioned as likely to be united states senator from is far from ground truth determined unknown to the hostess by the meeting of cotton interests earlier in the day

    But if I break this long audio into 2 parts as following

    // first part ground truth
    as she awaited her guests she surveyed the table with both satisfaction and disquietude for her social functions were few tonight there were she checked them off on her fingers sir james creighton the rich english manufacturer and lady creighton mister and missus vanderpool
    
    //latter part ground truth
    mister harry cresswell and his sister john taylor and his sister and mister charles smith whom the evening papers mentioned as likely to be united states senator from new jersey a selection of guests that had been determined unknown to the hostess by the meeting of cotton interests earlier in the day
    

    The decode result is better as following

    //first part decode
    as she awaited her guest she surveyed the table with both satisfaction and disquietude for her social functions were few to night there were she checked them off on her fingers sir james clinton the rich english manufacturer and lady horton mister and missus vanderpole
    
    //latter part decode
    mister harry creswell and his sister john taylor and his sister and mister charles smith whom the evening papers mentioned as likely to be united states senator from new jersey a selection of guests that had been determined unknown to the hostess by the meeting of cotton interests earlier in the day
    

    For my result,

    | | WER | LER | |--------------------|-------|-------| | Decode whole audio | 22.9% | 15.3% | | Decode first part | 15.1% | 8.9% | | Decode latter part | 15.5% | 10.1 |

    Is this behavior OK for decoding?

    question 
    opened by zjuturtle 19
  • Automatic Mixed Precision (AMP) support in flashlight

    Automatic Mixed Precision (AMP) support in flashlight

    Summary: Implements Automatic Mixed Precision (AMP) support in Flashlight. All appropriate modules will support half precision for their computations and use mixed precision (a combination of half precision and full precision) to strike a balance between accuracy and speedup. It also enables Tensor Cores when applicable to further accelerate the training.

    CLA Signed 
    opened by mtmd 17
  • Segmentation fault while performing Convolution2D

    Segmentation fault while performing Convolution2D

    Bug Description

    Segmentation Fault running Test Code - Mnist while calling Convolution 2D.

    Reproduction Steps

    1. Downloaded the requisite file from http://yann.lecun.com/exdb/mnist/
    2. Ran the Mnist.cpp - print is happening just before Convolution2D is called
    3. Same issue with self written code

    Platform and Hardware

    [Please list your operating system, [GPU] hardware, compiler, and other details if relevant] RHEL8.0, -1- INTEL: Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz, 34296 MB Intel(R) FPGA Emulation Platform for OpenCL(TM): Intel(R) FPGA Emulation Device, 34296 MB ArrayFire v3.8.0 (OpenCL, 64-bit Linux, build d99887a)

    I had installed flashlight using vcpkg. Does that install latest version or I have to install fresh version?

    bug 
    opened by avi-champ 1
  • colab error: FL_BUILD_CORE Required to build FL_BUILD_APP_ASR

    colab error: FL_BUILD_CORE Required to build FL_BUILD_APP_ASR

    Bug Description

    New to flashlight, decided to try the asr colab

    Reproduction Steps

    1. Go to URL: : https://colab.research.google.com/github/flashlight/flashlight/blob/master/flashlight/app/asr/tutorial/notebooks/InferenceAndAlignmentCTC.ipynb
    2. set backend: CPU
    3. run step 1 (no errors)
    4. run step 2

    Results:

    env: MKLROOT=/opt/intel/mkl
    env: ArrayFire_DIR=/opt/arrayfire/share/ArrayFire/cmake
    env: DNNL_DIR=/opt/dnnl/dnnl_lnx_2.0.0_cpu_iomp/lib/cmake/dnnl
    -- The CXX compiler identification is GNU 7.5.0
    -- The C compiler identification is GNU 7.5.0
    -- Check for working CXX compiler: /usr/bin/c++
    -- Check for working CXX compiler: /usr/bin/c++ -- works
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Detecting CXX compile features
    -- Detecting CXX compile features - done
    -- Check for working C compiler: /usr/bin/cc
    -- Check for working C compiler: /usr/bin/cc -- works
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Detecting C compile features
    -- Detecting C compile features - done
    -- Performing Test COMPILER_SUPPORTS_RDYNAMIC
    -- Performing Test COMPILER_SUPPORTS_RDYNAMIC - Success
    -- -rdynamic supported.
    CMake Error at cmake/InternalUtils.cmake:126 (message):
      FL_BUILD_CORE Required to build FL_BUILD_APP_ASR
    Call Stack (most recent call first):
      flashlight/app/CMakeLists.txt:11 (fl_dependent_option)
      CMakeLists.txt:247 (include)
    
    
    -- Configuring incomplete, errors occurred!
    See also "/content/flashlight/build/CMakeFiles/CMakeOutput.log".
    
    bug 
    opened by ultrasaurus 3
  • (uncommitted/untracked changes)

    (uncommitted/untracked changes)

    Differential Revision: D32724654

    CLA Signed fb-exported 
    opened by jacobkahn 1
  • autograd tensor extension

    autograd tensor extension

    Differential Revision: D32515803

    CLA Signed fb-exported 
    opened by jacobkahn 1
  • [Changing Hyper Parameters of Conformer Model]

    [Changing Hyper Parameters of Conformer Model]

    Question

    I want to train conformer model from scratch but after tweaking some hyperparameters (to make the model smaller) i.e. reducing the convolutional layers etc.

    In the ASR finetuning tutorial, it uses the model architecture file which I don't find in repository. How can I train a new model architecture or change an existing one before training?

    Additional Context

    For example in ASR finetuning tutorial Link:

    https://github.com/EnricoASchmitz/flashlight_2/tree/722132844f1c4ecb92713eea8322ea1d5e9eff73/flashlight/app/asr/tutorial

    It directly downloads the predefined model parameters and architecture files. !wget -nv --continue -o /dev/null https://dl.fbaipublicfiles.com/wav2letter/rasr/tutorial/am_transformer_ctc_stride3_letters_70Mparams.bin -O model.bin # acoustic model !wget -nv --continue -o /dev/null https://dl.fbaipublicfiles.com/wav2letter/rasr/tutorial/am_transformer_ctc_stride3_letters_70Mparams.arch -O arch.txt # model architecture file

    Is there any way to modify these files? Or is there any other approach which I should follow? Thanks.

    question 
    opened by harisgulzar1 1
  • Add TensorExtension API

    Add TensorExtension API

    Differential Revision: D32159630

    CLA Signed fb-exported 
    opened by jacobkahn 1
  • missing ` in readme

    missing ` in readme

    typo: ` missing in readme.

    CLA Signed 
    opened by vincentqb 3
  • Remove vcpkg build from ci

    Remove vcpkg build from ci

    IMPORTANT: Please do not create a Pull Request without creating an issue first. Changes must be discussed.

    Original Issue: [corresponding issue on Github]

    Note: You can add closes #[issue number] to automatically close the issue that this PR resolves when it is merged.

    Summary

    [Explain the details for this change and the problem that the pull request solves]

    Test Plan (required)

    [steps by which you tested that your fix resolves the issue. These might include specific commands and configurations]

    CLA Signed 
    opened by padentomasello 1
  • Lower number of processes in make for cpu build

    Lower number of processes in make for cpu build

    IMPORTANT: Please do not create a Pull Request without creating an issue first. Changes must be discussed.

    Original Issue: [corresponding issue on Github]

    Note: You can add closes #[issue number] to automatically close the issue that this PR resolves when it is merged.

    Summary

    [Explain the details for this change and the problem that the pull request solves]

    Test Plan (required)

    [steps by which you tested that your fix resolves the issue. These might include specific commands and configurations]

    CLA Signed 
    opened by padentomasello 1
  • Add lib common when building sound affects

    Add lib common when building sound affects

    IMPORTANT: Please do not create a Pull Request without creating an issue first. Changes must be discussed.

    Original Issue: [corresponding issue on Github]

    Note: You can add closes #[issue number] to automatically close the issue that this PR resolves when it is merged.

    Summary

    [Explain the details for this change and the problem that the pull request solves]

    Test Plan (required)

    [steps by which you tested that your fix resolves the issue. These might include specific commands and configurations]

    CLA Signed 
    opened by padentomasello 3
Releases(v0.3.1)
  • v0.3.1(Oct 15, 2021)

    v0.3.1 is a patch release that includes fixes since v0.3 and the initial fl::Tensor API.

    New Features

    • Introduce the fl::Tensor API. See flashlight/fl/tensor for more.
      • Flashlight will be adapted to use the API in a forthcoming release.
    • A distributed stub backend to enable building apps without a distributed backend for the time being (will be disentangled in a later release) (#614)
    • A benchmark app to use for end to end performance benchmarking of common models
    • C++17 minimum requirement
    • Introduce DynamicScaler to abstract away repeated logic in mixed precision training
    • Support for automatic mixed precision training with DETR, ResNet, and ViT
    • Codecov

    Fixes

    • Change behavior of comformer skip connections (#587)
    • Fixes to typing in the advanced index kernel
    • Removes eval() in the backward pass
    • Serialization support for audio augmentations
    • Rewrite of the SinusoidalPositionEmbedding operator to avoid concatenation

    Thanks to external contributors for their contributions, including: @lunixbochs @tlikhomanenko

    Source code(tar.gz)
    Source code(zip)
  • v0.3(Apr 16, 2021)

    First stable release post-consolidation. Separates Flashlight into four parts:

    • flashlight/lib contains kernels and standalone utilities for sequence losses, beam search decoding, text processing, and more.
    • flashlight/fl is the core neural network library using the ArrayFire tensor library.
    • flashlight/app are applications of the core library to machine learning across domains.
    • flashlight/ext are extensions on top of Flashlight and ArrayFire that are useful across apps.

    Major Features

    • Automatic mixed precision training (AMP) -- typed tensor and autograd operators
    • Framework for building custom memory managers on top of ArrayFire (docs)
    • OneDNN as a backend for primitive operations on the CPU
    • New dataset abstractions in core (flashlight/fl/dataset)
    • Application libraries
    • Audio augmentation library (ASR)
    • Tools for training models using iterative pseudo-labeling (IPL) (ASR)
    • [early] OpenCL support with both RoCM and Intel

    Build Changes/Improvements

    • C++ 17 support -- gcc 7/clang 6 required.
    • Support for vcpkg via FL_BUILD_STANDALONE
    • Consolidation of wav2letter and app-based build selection
    • CMake 3.10 minimum, better support for shared objects
    • First class support for CUDA and Halide kernels
    • Improved support for downloading not-found dependencies (Gloo, KenLM, libsndfile)
    • Improved support for dependency management for downstream projects using Flashlight's installed CMake config (cmake/flashlightConfig.cmake.in)
    • Supporting padding in transformer/multihead attention
    • SpecAugment for raw waves (implemented vie low-pass filter)
    • Conformer Implementation
    • Improve autograd for indexing operator (support repeated indices)
    • Improve python bindings build, supporting setup.py install
    • A lot of docs.

    Improvements/new features in wav2letter (flashlight/app/asr)

    • Fixed padding issues in s2s models: pre-training window, encoder attention, encoder-decoder attention
    • Refactor s2s codebase
    • Fixes to memory allocations for s2s beam-search decoder (less memory, no OOM issues)
    • Fixes to beam-search decoder to support non-empty surround
    • Fixes to dataset pipeline + dynamic batching support
    Source code(tar.gz)
    Source code(zip)
  • v0.2(Dec 28, 2020)

  • v0.1(Dec 22, 2018)

    Going Native with Flashlight

    To get started, visit the documentation.

    Credits

    Thanks to our contributors:

    Vineel Pratap, Awni Hannun, Jacob Kahn, Qiantong Xu, Jeff Cai, Gabriel Synnaeve, Vitaliy Liptchinsky, Ronan Collobert, Ann Lee, Jay Mahadeokar and Tatiana Likhomanenko

    Source code(tar.gz)
    Source code(zip)
Edge ML Library - High-performance Compute Library for On-device Machine Learning Inference

Edge ML Library (EMLL) offers optimized basic routines like general matrix multiplications (GEMM) and quantizations, to speed up machine learning (ML) inference on ARM-based devices. EMLL supports fp32, fp16 and int8 data types. EMLL accelerates on-device NMT, ASR and OCR engines of Youdao, Inc.

NetEase Youdao 166 Dec 2, 2021
A lightweight C++ machine learning library for embedded electronics and robotics.

Fido Fido is an lightweight, highly modular C++ machine learning library for embedded electronics and robotics. Fido is especially suited for robotic

The Fido Project 410 Dec 7, 2021
mlpack: a scalable C++ machine learning library --

a fast, flexible machine learning library Home | Documentation | Doxygen | Community | Help | IRC Chat Download: current stable version (3.4.2) mlpack

mlpack 3.8k Dec 4, 2021
A toolkit for making real world machine learning and data analysis applications in C++

dlib C++ library Dlib is a modern C++ toolkit containing machine learning algorithms and tools for creating complex software in C++ to solve real worl

Davis E. King 10.7k Dec 3, 2021
null 5.3k Dec 6, 2021
Machine Learning Framework for Operating Systems - Brings ML to Linux kernel

Machine Learning Framework for Operating Systems - Brings ML to Linux kernel

File systems and Storage Lab (FSL) 154 Dec 3, 2021
Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow

eXtreme Gradient Boosting Community | Documentation | Resources | Contributors | Release Notes XGBoost is an optimized distributed gradient boosting l

Distributed (Deep) Machine Learning Community 21.9k Dec 2, 2021
RNNLIB is a recurrent neural network library for sequence learning problems. Forked from Alex Graves work http://sourceforge.net/projects/rnnl/

Origin The original RNNLIB is hosted at http://sourceforge.net/projects/rnnl while this "fork" is created to repeat results for the online handwriting

Sergey Zyrianov 865 Nov 24, 2021
Samsung Washing Machine replacing OS control unit

hacksung Samsung Washing Machine WS1702 replacing OS control unit More info at https://www.hackster.io/roni-bandini/dead-washing-machine-returns-to-li

null 21 Nov 27, 2021
Caffe: a fast open framework for deep learning.

Caffe Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research (BAIR)/The Berke

Berkeley Vision and Learning Center 32.1k Dec 4, 2021
Distributed (Deep) Machine Learning Community 677 Nov 23, 2021
Frog is an integration of memory-based natural language processing (NLP) modules developed for Dutch. All NLP modules are based on Timbl, the Tilburg memory-based learning software package.

Frog - A Tagger-Lemmatizer-Morphological-Analyzer-Dependency-Parser for Dutch Copyright 2006-2020 Ko van der Sloot, Maarten van Gompel, Antal van den

Language Machines 66 Nov 28, 2021
C-based/Cached/Core Computer Vision Library, A Modern Computer Vision Library

Build Status Travis CI VM: Linux x64: Raspberry Pi 3: Jetson TX2: Backstory I set to build ccv with a minimalism inspiration. That was back in 2010, o

Liu Liu 6.8k Dec 2, 2021
libsvm websitelibsvm - A simple, easy-to-use, efficient library for Support Vector Machines. [BSD-3-Clause] website

Libsvm is a simple, easy-to-use, and efficient software for SVM classification and regression. It solves C-SVM classification, nu-SVM classification,

Chih-Jen Lin 4.1k Dec 7, 2021
Open Source Computer Vision Library

OpenCV: Open Source Computer Vision Library Resources Homepage: https://opencv.org Courses: https://opencv.org/courses Docs: https://docs.opencv.org/m

OpenCV 58.2k Nov 30, 2021
oneAPI Data Analytics Library (oneDAL)

Intel® oneAPI Data Analytics Library Installation | Documentation | Support | Examples | Samples | How to Contribute Intel® oneAPI Data Analytics Libr

oneAPI-SRC 451 Nov 30, 2021
A C library for product recommendations/suggestions using collaborative filtering (CF)

Recommender A C library for product recommendations/suggestions using collaborative filtering (CF). Recommender analyzes the feedback of some users (i

Ghassen Hamrouni 241 Nov 13, 2021
An open library of computer vision algorithms

VLFeat -- Vision Lab Features Library Version 0.9.21 The VLFeat open source library implements popular computer vision algorithms specialising in imag

VLFeat.org 1.5k Nov 29, 2021