Flashlight is a C++ standalone library for machine learning

Last update: Jan 8, 2023

Overview

Quickstart | Installation | Documentation

Flashlight is a fast, flexible machine learning library written entirely in C++ from the Facebook AI Research Speech team and the creators of Torch and Deep Speech. Its core features include:

Just-in-time kernel compilation with modern C++ with the ArrayFire tensor library.
CUDA and CPU backends for GPU and CPU training.
An emphasis on efficiency and scale.

Native support in C++ and simple extensibility makes Flashlight a powerful research framework that's hackable to its core and enables fast iteration on new experimental setups and algorithms without sacrificing performance. In a single repository, Flashlight provides apps for research across multiple domains:

Automatic speech recognition (the wav2letter project) — Documentation | Tutorial
Image classification
Object detection
Language modeling

Project Layout

Flashlight is broken down into a few parts:

flashlight/lib contains kernels and standalone utilities for sequence losses, beam search decoding, text processing, and more.
flashlight/fl is the core neural network library using the ArrayFire tensor library.
flashlight/app are applications of the core library to machine learning across domains.
flashlight/ext are extensions on top of Flashlight and ArrayFire that are useful across apps.

Quickstart

First, build and install Flashlight and link it to your own project.

Sequential forms a sequence of Flashlight Modules for chaining computation.

Implementing a simple convnet is easy.

#include <flashlight/fl/flashlight.h>

Sequential model;

model.add(View(af::dim4(IM_DIM, IM_DIM, 1, -1)));
model.add(Conv2D(
    1 /* input channels */,
    32 /* output channels */,
    5 /* kernel width */,
    5 /* kernel height */,
    1 /* stride x */,
    1 /* stride y */,
    PaddingMode::SAME; /* padding mode */,
    PaddingMode::SAME; /* padding mode */));
model.add(ReLU());
model.add(Pool2D(
    2 /* kernel width */,
    2 /* kernel height */,
    2 /* stride x */,
    2 /* stride y */));
model.add(Conv2D(32, 64, 5, 5, 1, 1, PaddingMode::SAME;, PaddingMode::SAME;));
model.add(ReLU());
model.add(Pool2D(2, 2, 2, 2));
model.add(View(af::dim4(7 * 7 * 64, -1)));
model.add(Linear(7 * 7 * 64, 1024));
model.add(ReLU());
model.add(Dropout(0.5));
model.add(Linear(1024, 10));
model.add(LogSoftmax());

Performing forward and backward computation is straightforwards:

auto output = model.forward(input);
auto loss = categoricalCrossEntropy(output, target);
loss.backward();

See the MNIST example for a full tutorial including a training loop and dataset abstractions.

Variable is the base Flashlight tensor that operates on ArrayFire arrays. Tape-based Automatic differentiation in Flashlight is simple and works as you'd expect.

Autograd Example

auto A = Variable(af::randu(1000, 1000), true /* calcGrad */);
auto B = 2.0 * A;
auto C = 1.0 + B;
auto D = log(C);
D.backward(); // populates A.grad() along with gradients for B, C, and D.

Building and Installing

Install with vcpkg | With Docker | From Source | From Source with vcpkg | Build Your Project with Flashlight

Requirements

At minimum, compilation requires:

A C++ compiler with good C++17 support (e.g. gcc/g++ >= 7)
CMake — version 3.10 or later, and make
A Linux-based operating system.

See the full dependency list for more details if building from source.

Instructions for building/installing Python bindings can be found here.

Flashlight Build Setups

Flashlight can be broken down into several components as described above. Each component can be incrementally built by specifying the correct build options.

There are two ways to work with Flashlight:

As an installed library that you link to with your own project. This is best for building standalone applications dependent on Flashlight.
With in-source development where the Flashlight project source is changed and rebuilt. This is best if customizing/hacking the core framework or the Flashlight-provided app binaries.

Flashlight can be built in one of two ways:

With vcpkg, a C++ package manager.
From source by installing dependencies as needed.

Installing Flashlight with `vcpkg`

Library Installation with `vcpkg`

Flashlight is most-easily built and installed with vcpkg. Both the CUDA and CPU backends are supported with vcpkg. For either backend, first install Intel MKL. For the CUDA backend, install CUDA >= 9.2, cuDNN, and NCCL. Then, after installing vcpkg, install the libraries and core with:

./vcpkg/vcpkg install flashlight-cuda # CUDA backend, OR
./vcpkg/vcpkg install flashlight-cpu  # CPU backend

To install Flashlight apps, check the features available for installation by running ./vcpkg search flashlight-cuda or ./vcpkg search flashlight-cpu. Each app is a "feature": for example, ./vcpkg install flashlight-cuda[asr] installs the ASR app with the CUDA backend.

Below is the currently-supported list of features (for each of flashlight-cuda and flashlight-cpu):

flashlight-{cuda/cpu}[lib]      # Flashlight libraries
flashlight-{cuda/cpu}[nn]       # Flashlight neural net library
flashlight-{cuda/cpu}[asr]      # Flashlight speech recognition app
flashlight-{cuda/cpu}[lm]       # Flashlight language modeling app
flashlight-{cuda/cpu}[imgclass] # Flashlight image classification app

Flashlight app binaries are also built for the selected features and are installed into the vcpkg install tree's tools directory.

Integrating Flashlight into your own project with is simple using vcpkg's CMake toolchain integration.

From-Source Build with `vcpkg`

First, install the dependencies for your backend of choice using vcpkg (click to expand the below):

Installing CUDA Backend Dependencies with vcpkg

To build the Flashlight CUDA backend from source using dependencies installed with vcpkg, install CUDA >= 9.2, cuDNN, NCCL, and Intel MKL, then build the rest of the dependencies for the CUDA backend based on which Flashlight features you'd like to build:

./vcpkg install \
    cuda intel-mkl fftw3 cub kenlm                \ # if building flashlight libraries
    arrayfire[cuda] cudnn nccl openmpi cereal stb \ # if building the flashlight neural net library
    gflags glog                                   \ # if building any flashlight apps
    libsndfile                                    \ # if building the flashlight asr app
    gtest                                           # optional, if building tests

Installing CPU Backend Dependencies with vcpkg

To build the Flashlight CPU backend from source using dependencies installed with vcpkg, install Intel MKL, then build the rest of the dependencies for the CPU backend based on which Flashlight features you'd like to build:

./vcpkg install \
    intel-mkl fftw3 kenlm                              \ # for flashlight libraries
    arrayfire[cpu] gloo[mpi] openmpi onednn cereal stb \ # for the flashlight neural net library
    gflags glog                                        \ # for any flashlight apps
    libsndfile                                         \ # for the flashlight asr app
    gtest                                                # optional, for tests

Build Using the `vcpkg` Toolchain File

To build Flashlight from source with these dependencies, clone the repository:

git clone https://github.com/flashlight/flashlight.git && cd flashlight
mkdir -p build && cd build

Then, build from source using vcpkg's CMake toolchain:

cmake .. \
    -DCMAKE_BUILD_TYPE=Release
    -DFL_BACKEND=CUDA
    -DCMAKE_TOOLCHAIN_FILE=[path to your vcpkg clone]/scripts/buildsystems/vcpkg.cmake
make -j$(nproc)
make install -j$(nproc) # only if you want to install Flashlight for external use

To build a subset of Flashlight's features, see the build options below.

Building from Source

To build from source, first install the below dependencies. Most are available with your system's local package manager.

Some dependencies marked below are downloaded and installed automatically if not found on the local system. FL_BUILD_STANDALONE determines this behavior — if disabled, dependencies won't be downloaded and built when building Flashlight.

Once all dependencies are installed, clone the repository:

git clone https://github.com/flashlight/flashlight.git && cd flashlight
mkdir -p build && cd build

Then build all Flashlight components with:

cmake .. -DCMAKE_BUILD_TYPE=Release -DFL_BACKEND=[backend] [...build options]
make -j$(nproc)
make install

Setting the MKLROOT environment variable (export MKLROOT=/opt/intel/oneapi/mkl/latest or export MKLROOT=/opt/intel/mkl on most Linux-based systems) can help CMake find Intel MKL if not initially found.

To build a smaller subset of Flashlight features/apps, see the build options below for a complete list of options.

To install Flashlight in a custom directory, use CMake's CMAKE_INSTALL_PREFIX argument. Flashlight libraries can be built as shared libraries using CMake's BUILD_SHARED_LIBS argument.

Flashlight uses modern CMake and IMPORTED targets for most dependencies. If a dependency isn't found, passing -D_DIR to your cmake command or exporting _DIR as an environment variable equal to the path to Config.cmake can help locate dependencies on your system. See the documentation for more details. If CMake is failing to locate a package, check to see if a corresponding issue has already been created before creating your own.

Dependencies

Dependencies marked with * are automatically downloaded and built from source if not found on the system. Setting FL_BUILD_STANDALONE to OFF disables this behavior.

Dependencies marked with ^ are required if building with distributed training enabled (FL_BUILD_DISTRIBUTED — see the build options below). Distributed training is required for all apps.

Dependencies marked with † are installable via vcpkg. See the instructions for installing those dependencies above for doing a Flashlight from-source build.

Component	Backend	Dependencies
libraries	CUDA	CUDA >= 9.2, CUB*† (if CUDA < 11)
libraries	CPU	A BLAS library (Intel MKL >= 2018, OpenBLAS†, etc)
core	Any	ArrayFire >= 3.7.3†, an MPI library^(OpenMPI†, etc), cereal† >= 1.3.0, stb†
	CUDA	CUDA >= 9.2, NCCL^, cuDNN
	CPU	oneDNN† >= 2.0, gloo (with MPI)*^†
app: all	Any	Google Glog†, Gflags†
app: asr	Any	libsndfile*† >= 10.0.28, a BLAS library (Intel MKL >= 2018, OpenBLAS†, etc)
app: imgclass	Any	-
app: objdet	Any	-
app: lm	Any	-
tests	Any	Google Test (gtest, with gmock)*† >= 1.10.0

Build Options

The Flashlight CMake build accepts the following build options (prefixed with -D when running CMake from the command line):

Name	Options	Default Value	Description
FL_BACKEND	CUDA, CPU, OPENCL	CUDA	Backend with which to build all components.
FL_BUILD_STANDALONE	ON, OFF	ON	Downloads/builds some dependencies if not found.
FL_BUILD_LIBRARIES	ON, OFF	ON	Build the Flashlight libraries.
FL_BUILD_CORE	ON, OFF	ON	Build the Flashlight neural net library.
FL_BUILD_DISTRIBUTED	ON, OFF	ON	Build with distributed training; required for apps.
FL_BUILD_CONTRIB	ON, OFF	ON	Build contrib APIs subject to breaking changes.
FL_BUILD_ALL_APPS	ON, OFF	OFF	Defines default value for every app (see below).
FL_BUILD_APP_ASR	ON, OFF	FL_BUILD_ALL_APPS	Build the automatic speech recognition app.
FL_BUILD_APP_IMGCLASS	ON, OFF	FL_BUILD_ALL_APPS	Build the image classification app.
FL_BUILD_APP_OBJDET	ON, OFF	FL_BUILD_ALL_APPS	Build automatic speech recognition app tools.
FL_BUILD_APP_LM	ON, OFF	FL_BUILD_ALL_APPS	Build the language modeling app.
FL_BUILD_APP_ASR_TOOLS	ON, OFF	FL_BUILD_APP_ASR	Build automatic speech recognition app tools.
FL_BUILD_TESTS	ON, OFF	ON	Build tests.
FL_BUILD_EXAMPLES	ON, OFF	ON	Build examples.
FL_BUILD_EXPERIMENTAL	ON, OFF	OFF	Build experimental components.
CMAKE_BUILD_TYPE	See docs.	Debug	See the CMake documentation.
CMAKE_INSTALL_PREFIX	[Directory]	See docs.	See the CMake documentation.

Building Your Own Project with Flashlight

Flashlight is most-easily linked to using CMake. Flashlight exports the following CMake targets when installed:

flashlight::fl-libraries — contains flashlight libraries headers and symbols.
flashlight::flashlight — contains flashlight libraries as well as the flashlight core autograd and neural network library.
flashlight::flashlight-app-asr — contains the automatic speech recognition app along with the flashlight core and flashlight libraries.
flashlight::flashlight-app-imgclass — contains the image classification app along with the flashlight core and flashlight libraries.
flashlight::flashlight-app-objdet — contains the object detection app along with the flashlight core and flashlight libraries.
flashlight::flashlight-app-lm — contains the language modeling app along with the flashlight core and flashlight libraries.

Given a simple project.cpp file that includes and links to Flashlight:

#include <iostream>

#include <arrayfire.h>
#include <flashlight/fl/flashlight.h>

int main() {
 fl::Variable v(af::constant(1, 1), true);
 auto result = v + 10;
 std::cout << "Hello World!" << std::endl;
 af::print("Array value is ", result.array()); // 11.000
 return 0;
}

The following CMake configuration links Flashlight and sets include directories:

cmake_minimum_required(VERSION 3.10)
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

add_executable(myProject project.cpp)

find_package(flashlight CONFIG REQUIRED)
target_link_libraries(myProject PRIVATE flashlight::flashlight)

With a `vcpkg` Flashlight Installation

If you installed Flashlight with vcpkg, the above CMake configuration for myProject can be built by running:

cd project && mkdir build && cd build
cmake .. \
  -DCMAKE_TOOLCHAIN_FILE=[path to vcpkg clone]/scripts/buildsystems/vcpkg.cmake \
  -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

With a From-Source Flashlight Installation

If using a from-source installation of Flashlight, Flashlight will be found automatically by CMake:

cd project && mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

If Flashlight is installed in a custom location using a CMAKE_INSTALL_PREFIX, passing -Dflashlight_DIR=[install prefix]/share/flashlight/cmake as an argument to your cmake command can help CMake find Flashlight.

Building and Running Flashlight with Docker

Flashlight and its dependencies can also be built with the provided Dockerfiles — see the accompanying Docker documentation for more information.

Contributing and Contact

Contact: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]

Flashlight is being very actively developed. See CONTRIBUTING for more on how to help out.

Acknowledgments

Some of Flashlight's code is derived from arrayfire-ml.

License

Flashlight is under a BSD license. See LICENSE for more information.

Comments

Build errors for CPU backend

While trying to build for CPU backend, I get a list of build error starting with these:-

[ 61%] Linking CXX executable MemoryFrameworkTest CMakeFiles/MemoryFrameworkTest.dir/memory/MemoryFrameworkTest.cpp.o: In function (anonymous namespace)::MockTestMemoryManager::alloc(bool, unsigned int, long long*, unsigned int)': MemoryFrameworkTest.cpp:(.text+0x5ca): undefined reference totesting::internal::UntypedFunctionMockerBase::SetOwnerAndName(void const*, char const*)' MemoryFrameworkTest.cpp:(.text+0x5eb): undefined reference to testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith(void*)' CMakeFiles/MemoryFrameworkTest.dir/memory/MemoryFrameworkTest.cpp.o: In function(anonymous namespace)::MockTestMemoryManager::allocated(void*)': MemoryFrameworkTest.cpp:(.text+0x6dc): undefined reference to testing::internal::UntypedFunctionMockerBase::SetOwnerAndName(void const*, char const*)' MemoryFrameworkTest.cpp:(.text+0x6ee): undefined reference totesting::internal::UntypedFunctionMockerBase::UntypedInvokeWith(void*)' CMakeFiles/MemoryFrameworkTest.dir/memory/MemoryFrameworkTest.cpp.o: In function (anonymous namespace)::MockTestMemoryManager::jitTreeExceedsMemoryPressure(unsigned long)': MemoryFrameworkTest.cpp:(.text+0x7dc): undefined reference totesting::internal::UntypedFunctionMockerBase::SetOwnerAndName(void const*, char const*)' MemoryFrameworkTest.cpp:(.text+0x7ee): undefined reference to testing::internal::UntypedFunctionMockerBase::UntypedInvokeWith(void*)' CMakeFiles/MemoryFrameworkTest.dir/memory/MemoryFrameworkTest.cpp.o: In function(anonymous namespace)::MockTestMemoryManager::getMemoryPressure()': MemoryFrameworkTest.cpp:(.text+0x8d8): undefined reference to testing::internal::UntypedFunctionMockerBase::SetOwnerAndName(void const*, char const*)' MemoryFrameworkTest.cpp:(.text+0x8e5): undefined reference totesting::internal::UntypedFunctionMockerBase::UntypedInvokeWith(void*)'

opened by mitulsaha 58
[gh-actions] Action for running lightweight oneDNN benchmarks

See title — action stub/starting point. Builds on Ubuntu 20.04.

We'll ship this lib to downstream projects pulling in libflashlight on install.

Test Plan: CI
CLA Signed

opened by jacobkahn 41

NameError: name 'CriterionType' is not defined

This is the error I get:

NameError                                 Traceback (most recent call last)
/tmp/ipykernel_18106/1529090364.py in <module>
     31 
     32 upstream = 'wav2vec2_hug_base_960'
---> 33 runner = Runner(args, config)

~/Desktop/ASR/s3prl/s3prl/downstream/runner.py in __init__(self, args, config)
     50         self.upstream = self._get_upstream()
     51         self.featurizer = self._get_featurizer()
---> 52         self.downstream = self._get_downstream()
     53         self.all_entries = [self.upstream, self.featurizer, self.downstream]
     54 

~/Desktop/ASR/s3prl/s3prl/downstream/runner.py in _get_downstream(self)
    131             upstream_rate = self.featurizer.model.downsample_rate,
    132             **self.config,
--> 133             **vars(self.args)
    134         ).to(self.args.device)
    135 

~/Desktop/ASR/s3prl/s3prl/downstream/

I reviewed this issue but I did not help me https://github.com/flashlight/flashlight/issues/416#issue-783305697

when I run this import flashlight --> does not raises any error
when I run this from flashlight.lib.text.decoder import CriterionType --> it raises this: ModuleNotFoundError: No module named 'flashlight.lib.text'
I can successfully run examples in the flashlight without any error. so this command python flashlight/bindings/python/example/criterion_example.py does NOT raise any error
And I have kenlm in this address /usr/local/share/kenlm

@jacobkahn I tried to address the questions @tlikhomanenko had asked in that issue in order to make it easier.

I really really appreciate it if you could give me some hints where should I check.

bug

opened by benam2 36

[sfx] time stretch

Add time stretch using sox's algorithm but without libsox threading issues. The algorithm is almost verbatim copied to make review simpler. Changes to original code applied only where required.
CLA Signed

opened by avidov 35
getting no error while cmake, but it stuck on make

getting no error while cmake, but it stuck on make. plz help

cmake .. -DCMAKE_BUILD_TYPE=Release -DFLASHLIGHT_BACKEND=CPU

-- ArrayFire found (include: /opt/arrayfire/include, library: ArrayFire::afcuda) -- Could NOT find cereal (missing: cereal_INCLUDE_DIRS) -- cereal NOT found. Will download from source -- Checking for [mkl_gf_lp64 - mkl_gnu_thread - mkl_core - iomp5 - pthread - m] -- Library mkl_gf_lp64: /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so -- Library mkl_gnu_thread: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library iomp5: not found -- Checking for [mkl_gf_lp64 - mkl_intel_thread - mkl_core - iomp5 - pthread - m] -- Library mkl_gf_lp64: /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so -- Library mkl_intel_thread: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library iomp5: not found -- Checking for [mkl_gf - mkl_gnu_thread - mkl_core - iomp5 - pthread - m] -- Library mkl_gf: not found -- Checking for [mkl_gf - mkl_intel_thread - mkl_core - iomp5 - pthread - m] -- Library mkl_gf: not found -- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - iomp5 - pthread - m] -- Library mkl_intel_lp64: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so -- Library mkl_gnu_thread: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library iomp5: not found -- Checking for [mkl_intel_lp64 - mkl_intel_thread - mkl_core - iomp5 - pthread - m] -- Library mkl_intel_lp64: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so -- Library mkl_intel_thread: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library iomp5: not found -- Checking for [mkl_intel - mkl_gnu_thread - mkl_core - iomp5 - pthread - m] -- Library mkl_intel: not found -- Checking for [mkl_intel - mkl_intel_thread - mkl_core - iomp5 - pthread - m] -- Library mkl_intel: not found -- Checking for [mkl_gf_lp64 - mkl_gnu_thread - mkl_core - pthread - m] -- Library mkl_gf_lp64: /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so -- Library mkl_gnu_thread: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so -- Library m: /usr/lib/x86_64-linux-gnu/libm.so -- Checking for [mkl_gf_lp64 - mkl_intel_thread - mkl_core - pthread - m] -- Library mkl_gf_lp64: /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so -- Library mkl_intel_thread: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so -- Library m: /usr/lib/x86_64-linux-gnu/libm.so -- Checking for [mkl_gf - mkl_gnu_thread - mkl_core - pthread - m] -- Library mkl_gf: not found -- Checking for [mkl_gf - mkl_intel_thread - mkl_core - pthread - m] -- Library mkl_gf: not found -- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - pthread - m] -- Library mkl_intel_lp64: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so -- Library mkl_gnu_thread: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so -- Library m: /usr/lib/x86_64-linux-gnu/libm.so -- Checking for [mkl_intel_lp64 - mkl_intel_thread - mkl_core - pthread - m] -- Library mkl_intel_lp64: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so -- Library mkl_intel_thread: /opt/intel/mkl/lib/intel64/libmkl_intel_thread.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so -- Library m: /usr/lib/x86_64-linux-gnu/libm.so -- Checking for [mkl_intel - mkl_gnu_thread - mkl_core - pthread - m] -- Library mkl_intel: not found -- Checking for [mkl_intel - mkl_intel_thread - mkl_core - pthread - m] -- Library mkl_intel: not found -- Checking for [mkl_gf_lp64 - mkl_sequential - mkl_core - m] -- Library mkl_gf_lp64: /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so -- Library mkl_sequential: /opt/intel/mkl/lib/intel64/libmkl_sequential.so -- Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so -- Library m: /usr/lib/x86_64-linux-gnu/libm.so -- MKL found -- A library with BLAS API found. -- MKLDNN headers found in /usr/local/include -- Using MKLDNN library found in /usr/local/lib/libmkldnn.so -- Using MKL with MKL-DNN -- MKLDNN found -- Will build flashlight contrib assets. -- Gloo found -- NCCL not found -- MPI_CXX found -- MPI_CXX compile flags: -pthread -- MPI_CXX include path: /usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include/usr/lib/x86_64-linux-gnu/openmpi/include -- MPI_CXX LINK flags path: -pthread -- MPI_CXX libraries: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so/usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so -- MPI_C found -- MPI_C compile flags: -pthread -- MPI_C include path: /usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/opal/mca/event/libevent2022/libevent/include/usr/lib/x86_64-linux-gnu/openmpi/include -- MPI_C LINK flags path: -pthread -- MPI_C libraries: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so -- gtest found: (include: /usr/include, lib: /usr/lib/libgtest.a;/usr/lib/libgtest_main.a -- Configuring done -- Generating done -- Build files have been written to: /home/test/Desktop/asr/wav2latter/flashlight/build

make -j 4

[ 14%] Linking CXX executable DatasetUtilsTest [ 14%] Building CXX object tests/CMakeFiles/ContribSerializationTest.dir/__/flashlight/contrib/modules/Transformer.cpp.o [ 14%] Built target DatasetUtilsTest [ 14%] Linking CXX executable ContribSerializationTest [ 14%] Built target ContribSerializationTest Makefile:140: recipe for target 'all' failed make: *** [all] Error 2

opened by shubhamrc49 31
Conformer Module
Hi, do you have an implementation of the Conformer Block in Flashlight you can share?

These are my rough notes for what conformer seems to be: https://gist.github.com/lunixbochs/4d8a8c0ab9be45469337b82363bc2105

This is roughly what I have so far, based on TDSBlock: https://gist.github.com/lunixbochs/207eff6e78b29e26712cee6fca42c400

Here's my janky conformer W2L arch based on TDS:

V -1 NFEAT 1 0 SAUG 80 27 2 10 0.05 2 PD 0 5 3 C2 1 15 10 1 2 1 0 0 R DO 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 CONFORMER 15 144 4 80 32 20 20 0.1 V 1200 -1 1 0 L 1200 NLABEL V NLABEL 0 -1 1 CONFORMER is: return std::make_shared<w2l::ConformerBlock>( channels, encoderDim, attentionHeads, width, kernelSize, ffnInnerDim, mhsaInnerDim, dropout, rPad, lNormIncludeTime);

Whitepaper is here: https://arxiv.org/pdf/2005.08100.pdf I believe this is a tensorflow implementation: https://github.com/TensorSpeech/TensorFlowASR/blob/main/tensorflow_asr/models/conformer.py#L209

I haven't worked with Keras enough to get a sense for the correct activation shapes through this, my dimensions are definitely not quite right, and I'm kind of guessing on the precise architecture. I'm happy to fiddle with this more but if you have any advice for major things I may be doing wrong that would be appreciated. Right now I know for sure my shape going into the transformer is wrong, but I don't know how exactly. And I probably did the depth-wise convolution wrong.
enhancement question
opened by lunixbochs 30
Update warpctc library

Original PR: [!215]

Summary

Brings the latest updates of warpctc and uses warpctc CMakeLists.txt submit in this PR. It works but I think it will take time to be approved.
CLA Signed

opened by alealv 27
Add compiling support for CUDA v11 and cuDNN v8
Original Issue: [#198] & [#213] & [#147]

Summary

I updated CMakeList.txt of third party library (warp-ctc)[https://github.com/baidu-research/warp-ctc]. The latest CMakeLists.txt handles conditional compilation for architecture compute_30 depending on CUDA version (CUDAv11 deprecates this architecture).

The same is done for main CMakeLists.txt file

The deprecation policy of CUDA has changed. Therefore, the condition of cudnnSetRNNDescriptor used here needs to also include until version 8000 of cuDNN.

Finally, with CUDA v11 Nvidia cub is included in CUDA toolkit, but because flashlight downloads and uses it's own version, it collides with Thrust version. Hence, THRUST_IGNORE_CUB_VERSION_CHECK must be define in if using CUDA v11

CLA Signed
opened by alealv 23
[Error when importing Flashlight] ImportError: libfl-libraries.so.0: undefined symbol: _ZN2lm5ngram6ConfigC1Ev

Question

I want to fine-tune Wav2vec with my own data. I got error when running this command:

python3.7 fairseq/train.py './labelled_manifest' --save-dir './model_finetuning_wav2vec' --wer-args '("./labelled_manifest/lm.bin","./labelled_manifest/lexicon.txt",2,-1)' --post-process letter --valid-subset valid --no-epoch-checkpoints --best-checkpoint-metric wer --num-workers 128 --max-update 400000 --sentence-avg --task audio_pretraining --arch wav2vec_ctc --w2v-path './w2v2_pre_train_model/wav2vec_small.pt' --labels ltr --apply-mask --mask-selection static --mask-other 0 --mask-length 10 --mask-prob 0.5 --layerdrop 0.1 --mask-channel-selection static --mask-channel-other 0 --mask-channel-length 64 --mask-channel-prob 0.5 --zero-infinity --feature-grad-mult 0.0 --freeze-finetune-updates 10000 --validate-after-updates 10000 --optimizer adam --adam-betas '(0.9, 0.98)' --adam-eps 1e-08 --lr 2e-05 --lr-scheduler tri_stage --warmup-steps 8000 --hold-steps 32000 --decay-steps 40000 --final-lr-scale 0.05 --final-dropout 0.0 --dropout 0.0 --activation-dropout 0.1 --criterion ctc --attention-dropout 0.0 --max-tokens 1280000 --seed 2337 --log-format json --log-interval 500 --ddp-backend no_c10d

The error is the following: NameError: name 'CriterionType' is not defined

I have successfully installed flashlight according to these steps.

I know that an issue has been opened with same error #468 but it doesn't fix my problem.

Additional Context

Im on Ubuntu 18.04 python 3.7.4 Cuda 11.2
question

opened by Kamilbentounes 22

Augment dataset as we train, augment datasets on disk, and use library to add sound effects

Summary

commit 942fc5d1805d13265d6488d8e8b235031716a798 (HEAD -> noise5, gilad/noise5) Author: Gilad Avidov [email protected] Date: Tue Sep 22 01:42:45 2020 -0700

[sfx] add real-time dataset augmentation to Train.cpp

The input dataset is augmented in the inner working of the dataset class hierarchy. The dataset class hierarchy reads sound files, transforms them to frequency domain using STFT and extracts using filterband. The augmentation is applied in the time domain. It takes place before frequency domain transformation. Train.cpp reads datasets for training and validation. A newly added set of flags supports augmenting both datasets.

Here is an example for a training command that applies sound effects to the training dataset. The sound effects are:
Reverberation with random number of echos, random absorption coefficient and random distance to the reflective objects.
Additive noise with random 2 noise clips applied with random SNR chosen in the range of 10..25.
Normalization of the augmented sound to ensure the signal is still in valid range after applying the sound effects.
The sound effects are applied over the GPU as a backend.

Example config file: sfx_config_noise_reverb_norm.json

{
    "soundEffectChain": [
        {
            "type_": "AdditiveNoise",
            "noiseConfig_": {
                "maxTimeRatio_": 1.0,
                "minSnr_": 10.0,
                "maxSnr_": 30.0,
                "nClipsPerUtteranceMin_": 4,
                "nClipsPerUtteranceMax_": 10,
                "listFilePath_": "/private/home/wesbz/experiments/noise-unbalanced-16kHz-mono-train.lst",
                "randomNoiseWithReplacement_": true,
                "randomSeed_": 1234
            }
        },
        {
            "type_": "Reverberation",
            "reverbConfig_": {
                "absorptionCoefficientMin_": 0.1,
                "absorptionCoefficientMax_": 0.5,
                "distanceToWallInMetersMin_": 1.0,
                "distanceToWallInMetersMax_": 10.0,
                "numEchosMin_": 0,
                "numEchosMax_": 10,
                "jitter_": 0.1,
                "randomSeed_": 1234567
            }
        },
        {
            "type_": "Normalize"
        }
    ]
}

Set the flag --sfx_config_filename to point to the config file

Train train --flagsfile=${FLAGS_FILE}  --sfx_config_filename=sfx_config_noise_reverb_norm.json

commit bf6e9b285d3acd194339961476e0fbfea20b86d8 Author: Gilad Avidov [email protected] Date: Tue Sep 22 01:01:12 2020 -0700

[sfx] add dataset augmentation tool.

ApplySoundEffect is an executable for creating datasets by augmenting existing ones. It applies a sound effect chain to sounds from list files. ApplySoundEffect runs as follows:
reads a list of list files
reads the sound files from these list files
augment the sound files
save the augmented files with the same path but prefixed on a different root dir.
add the augmented files to a consolidated list file of the augmented files.

Example for creating new dataset called librispeech-clean-other-noise1-snr-10-20-clips-3 by augmenting librispeech dev-clean and dev-other with noise from a dataset at /dataset/noise1.lst.

Example config file: sfx_config_noise.json
{
    "soundEffectChain": [
        {
            "type_": "AdditiveNoise",
            "noiseConfig_": {
                "maxTimeRatio_": 1.0,
                "minSnr_": 10.0,
                "maxSnr_": 20.0,
                "nClipsPerUtteranceMin_": 0,
                "nClipsPerUtteranceMax_": 3,
                "listFilePath_": "/private/home/wesbz/experiments/noise-unbalanced-16kHz-mono-train.lst",
                "randomNoiseWithReplacement_": true,
                "randomSeed_": 1234
            }
        },
    ]
}

Set the flag --sfx_config_filename to point to the config file

ApplySoundEffect \
  --input_rootdir=/dataset/librispeech/ \
  --input_listfiles=dev-clean.lst,dev-other.lst \
  --output_rootdir=/home/$USER/ \
  --output_listfile="librispeech-clean-other-noise1-snr-10-20-clips-3.lst" \
  --sfx_config_filename=sfx_config_noise.json

commit 956e12ff24216af9dd82db58998a4b872188850b Author: Gilad Avidov [email protected] Date: Sun Sep 20 23:20:13 2020 -0700

[sfx] experimental cpu/gpu sound effect augmentation library

Add experimental sound effect library for high performance (support both cpu/gpu) speech augmentation.

Currently implemented effects:
 - additive noise
 - reverberation
 - amplification.
 -normalization

Sound effect objects are instantiated, configured and then chained.
The sound effect chain is applied to the input set one at a time.
Threading safe.

        auto soundEffectChain = std::make_shared<SoundEffectChain>();

        // Add reverberation sound effect.
        soundEffectChain->add(std::make_shared<Reverberation>(
            std::make_shared<Reverberation::ReverbEchoRirGenerator>(reverbConf)));

        // Add additive noise sound effect.
        soundEffectChain->add(std::make_shared<AdditiveNoise>(
           std::make_shared<AdditiveNoise::RandomNoiseGenerator>(noiseConf)));

        // Add sound normalization (scale down when out of range) sound effect.
        soundEffectChain->add(std::make_shared<Normalize>());

        // Apply the sound effect chain on sound data.
        std::function<void(std::vector<float>*)> augment = trainDsSoundEffect->asStdFunction();
            augment(sound)

Add the library at fl::app::asr::sfx namespace and
app/asr/experimental/soundeffect directory.

Quip: https://fb.quip.com/UobCAPwM6e0t

Test Plan (required)

[steps by which you tested that your fix resolves the issue. These might include specific commands and configurations]

CLA Signed

opened by avidov 20

Why long audio get poor performance?

Question

I found that decode long audio is much worse than split long audio into multi parts. I use librispeech as dataset and test with decode_transformer_s2s_ngram.cfg. e.g.

LibriSpeech/test-clean/1995/1836/1995-1836-0004.flac is an audio with 33.91 seconds, its groundTruth text

as she awaited her guests she surveyed the table with both satisfaction and disquietude for her social functions were few tonight there were she checked them off on her fingers sir james creighton the rich english manufacturer and lady creighton mister and missus vanderpool mister harry cresswell and his sister john taylor and his sister and mister charles smith whom the evening papers mentioned as likely to be united states senator from new jersey a selection of guests that had been determined unknown to the hostess by the meeting of cotton interests earlier in the day

The decode result is

as she awaited her guest she surveyed the table with both satisfaction and disquietude for her social functions were few to night there were she checked them off on her fingers sir james clinton the rich english manufacturer and lady horton mister and missus vanderpole mister harry cresswell and his sister john taylor and his sister and mister charles smith whom the evening papers mentioned as likely to be united states senator from new jersey a selection of guests that had been determined unknown to the hostess and mister charles smith whom the evening papers mentioned as likely to be united states senator from

Note the ending part evening papers mentioned as likely to be united states senator from is far from ground truth determined unknown to the hostess by the meeting of cotton interests earlier in the day

But if I break this long audio into 2 parts as following

// first part ground truth
as she awaited her guests she surveyed the table with both satisfaction and disquietude for her social functions were few tonight there were she checked them off on her fingers sir james creighton the rich english manufacturer and lady creighton mister and missus vanderpool

//latter part ground truth
mister harry cresswell and his sister john taylor and his sister and mister charles smith whom the evening papers mentioned as likely to be united states senator from new jersey a selection of guests that had been determined unknown to the hostess by the meeting of cotton interests earlier in the day

The decode result is better as following

//first part decode
as she awaited her guest she surveyed the table with both satisfaction and disquietude for her social functions were few to night there were she checked them off on her fingers sir james clinton the rich english manufacturer and lady horton mister and missus vanderpole

//latter part decode
mister harry creswell and his sister john taylor and his sister and mister charles smith whom the evening papers mentioned as likely to be united states senator from new jersey a selection of guests that had been determined unknown to the hostess by the meeting of cotton interests earlier in the day

For my result,

| | WER | LER | |--------------------|-------|-------| | Decode whole audio | 22.9% | 15.3% | | Decode first part | 15.1% | 8.9% | | Decode latter part | 15.5% | 10.1 |

Is this behavior OK for decoding?

question

opened by zjuturtle 19

Unable To Run cmake after installing packages

Bug Description

[A clear, concise description of the bug]

After installing all the packages with vcpkg, I'm running:

cmake .. \
    -DCMAKE_BUILD_TYPE=Release \
    -DFL_BUILD_ARRAYFIRE=ON \
    -DCMAKE_TOOLCHAIN_FILE=/flashlight/vcpkg/scripts/buildsystems/vcpkg.cmake

And here's my error output:

-- The CXX compiler identification is GNU 9.4.0
-- The C compiler identification is GNU 9.4.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test COMPILER_SUPPORTS_RDYNAMIC
-- Performing Test COMPILER_SUPPORTS_RDYNAMIC - Success
-- -rdynamic supported.
-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - not found
-- Looking for CL_VERSION_2_1
-- Looking for CL_VERSION_2_1 - not found
-- Looking for CL_VERSION_2_0
-- Looking for CL_VERSION_2_0 - not found
-- Looking for CL_VERSION_1_2
-- Looking for CL_VERSION_1_2 - not found
-- Looking for CL_VERSION_1_1
-- Looking for CL_VERSION_1_1 - not found
-- Looking for CL_VERSION_1_0
-- Looking for CL_VERSION_1_0 - not found
-- Could NOT find OpenCL (missing: OpenCL_LIBRARY OpenCL_INCLUDE_DIR)
-- Found GTest: /flashlight/vcpkg/installed/x64-linux/share/gtest/GTestConfig.cmake (found suitable version "1.12.1", minimum required is "1.10.0")
-- gtest found: (include: , lib: GTest::gtest;GTest::gtest_main
-- Found gtest and gmock on system.
-- Found Threads: TRUE
-- Will build flashlight core and extensions.
-- Could NOT find cereal (missing: cereal_INCLUDE_DIRS)
-- Found cereal (include: )
-- cereal NOT found. Will download from source
-- Will build flashlight contrib assets.
-- ArrayFire found (include: /flashlight/vcpkg/installed/x64-linux/include, library: ArrayFire::afcpu)
-- oneDNN found
-- MKL_THREADING = OMP
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of void*
-- Check size of void* - done
-- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]
--   Library mkl_intel_lp64: /opt/intel/mkl/lib/intel64/libmkl_intel_lp64.so
--   Library mkl_gnu_thread: /opt/intel/mkl/lib/intel64/libmkl_gnu_thread.so
--   Library mkl_core: /opt/intel/mkl/lib/intel64/libmkl_core.so
--   Library gomp: -fopenmp
--   Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so
--   Library m: /usr/lib/x86_64-linux-gnu/libm.so
--   Library dl: /usr/lib/x86_64-linux-gnu/libdl.so
-- Looking for cblas_sgemm
-- Looking for cblas_sgemm - found
-- MKL library found
-- Looking for C++ include filesystem
-- Looking for C++ include filesystem - found
-- Performing Test CXX_FILESYSTEM_NO_LINK_NEEDED
-- Performing Test CXX_FILESYSTEM_NO_LINK_NEEDED - Success
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
-- CUDNN libname: libcudnn.so;libcudnn.dylib;cudnn64
-- Could NOT find CUDNN (missing: CUDNN_LIBRARY) (Required is at least version "7.1")
-- Found MPI_C: /flashlight/vcpkg/installed/x64-linux/lib/libmpi.so (found version "3.1")
-- Found MPI_CXX: /flashlight/vcpkg/installed/x64-linux/lib/libmpi.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1")
-- MPI_VERSION found:  3.1
-- MPI_CXX found
-- MPI_CXX compile flags: -pthread
-- MPI_CXX include path: /flashlight/vcpkg/installed/x64-linux/include
-- MPI_CXX LINK flags path: -Wl,-rpath -Wl,/flashlight/vcpkg/installed/x64-linux/lib -Wl,--enable-new-dtags -pthread
-- MPI_CXX libraries: /flashlight/vcpkg/installed/x64-linux/lib/libmpi.so
-- MPI_C found
-- MPI_C compile flags: -pthread
-- MPI_C include path: /flashlight/vcpkg/installed/x64-linux/include
-- MPI_C LINK flags path: -Wl,-rpath -Wl,/flashlight/vcpkg/installed/x64-linux/lib -Wl,--enable-new-dtags -pthread
-- MPI_C libraries: /flashlight/vcpkg/installed/x64-linux/lib/libmpi.so
-- using distributed stub
-- Configuring done
CMake Error in CMakeLists.txt:
  Target "flashlight" INTERFACE_INCLUDE_DIRECTORIES property contains path:

    "/flashlight/vcpkg/installed/x64-linux/include"

  which is prefixed in the source directory.


CMake Error in CMakeLists.txt:
  Target "flashlight" INTERFACE_INCLUDE_DIRECTORIES property contains path:

    "/flashlight/vcpkg/installed/x64-linux/include"

  which is prefixed in the source directory.


-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    FL_BUILD_ARRAYFIRE


CMake Generate step failed.  Build files cannot be regenerated correctly.

What's the best way to resolve this issue?

Platform and Hardware

[Please list your operating system, [GPU] hardware, compiler, and other details if relevant]

I'm running this on a custom docker image on my M1 apple laptop. The docker image is based off:

FROM mcr.microsoft.com/devcontainers/cpp:0-ubuntu-20.04

I'm trying to build this for flashlight[CPU].

I'm running the code on this commit.

Additional Context

[Add any additional information here]

I ran both:

./vcpkg/vcpkg install flashlight-cpu

and:

./vcpkg install \
    intel-mkl fftw3 kenlm                              \ # for flashlight libraries
    arrayfire[cpu] gloo[mpi] openmpi onednn cereal stb \ # for the flashlight neural net library
    gflags glog                                        \ # for the flashlight runtime pkg (any flashlight apps using it)
    libsndfile                                         \ # for the flashlight speech pkg
    gtest                                                # optional, for tests

to install the packages.

I'll be happy to provide any more details if needed!

bug

opened by JinLi711 0

Bump certifi from 2020.12.5 to 2022.12.7 in /flashlight/app/objdet/scripts
Bumps certifi from 2020.12.5 to 2022.12.7.

Commits

9e9e840 2022.12.07

b81bdb2 2022.09.24

939a28f 2022.09.14

aca828a 2022.06.15.2

de0eae1 Only use importlib.resources's new files() / Traversable API on Python ≥3.11 ...

b8eb5e9 2022.06.15.1

47fb7ab Fix deprecation warning on Python 3.11 (#199)

b0b48e0 fixes #198 -- update link in license

9d514b4 2022.06.15

4151e88 Add py.typed to MANIFEST.in to package in sdist (#196)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

CLA Signed dependencies
opened by dependabot[bot] 5
Cover all ops in JIT frontend
Summary

As title. This enables

applying JIT to almost any existing application

capturing maximal graph (which will become immensely helpful with the upcoming GraphVizPrinter & performance profiling).

The changes are quite simple -- just apply CustomNode to inject evaluation logic for all the remaining ops, and shape inference for certain ops.

There are small TODOs I dropped and minor changes I plan to send out in a few upcoming PRs, but they are pretty niche, and shouldn't affect overall applicability of the JIT.

Test Plan (required)

I manually used fl::setDefaultTensorType<JitTensor<DefaultTensorType_t>> in all existing tensor tests, e.g., TensorBaseTest to make sure things are correct (except for about 5% due to the TODOs I mentioned above).
CLA Signed oncall: jit
opened by StrongerXi 3
Enhance Binary Ops in JIT IR & frontend
Depends on #1055

Summary

Cover more binary ops in both IR and frontend (JitTensor)

Support broadcasting for inputs with different ranks

Test Plan (required)

New unit tests are added

make JitTensorTest
CLA Signed oncall: jit
opened by StrongerXi 1
Bump pillow from 9.0.1 to 9.3.0 in /flashlight/app/objdet/scripts
Bumps pillow from 9.0.1 to 9.3.0.

Release notes

Sourced from pillow's releases.

9.3.0

https://pillow.readthedocs.io/en/stable/releasenotes/9.3.0.html

Changes

Initialize libtiff buffer when saving #6699 [@radarhere]

Limit SAMPLESPERPIXEL to avoid runtime DOS #6700 [@wiredfool]

Inline fname2char to fix memory leak #6329 [@nulano]

Fix memory leaks related to text features #6330 [@nulano]

Use double quotes for version check on old CPython on Windows #6695 [@hugovk]

GHA: replace deprecated set-output command with GITHUB_OUTPUT file #6697 [@nulano]

Remove backup implementation of Round for Windows platforms #6693 [@cgohlke]

Upload fribidi.dll to GitHub Actions #6532 [@nulano]

Fixed set_variation_by_name offset #6445 [@radarhere]

Windows build improvements #6562 [@nulano]

Fix malloc in _imagingft.c:font_setvaraxes #6690 [@cgohlke]

Only use ASCII characters in C source file #6691 [@cgohlke]

Release Python GIL when converting images using matrix operations #6418 [@hmaarrfk]

Added ExifTags enums #6630 [@radarhere]

Do not modify previous frame when calculating delta in PNG #6683 [@radarhere]

Added support for reading BMP images with RLE4 compression #6674 [@npjg]

Decode JPEG compressed BLP1 data in original mode #6678 [@radarhere]

pylint warnings #6659 [@marksmayo]

Added GPS TIFF tag info #6661 [@radarhere]

Added conversion between RGB/RGBA/RGBX and LAB #6647 [@radarhere]

Do not attempt normalization if mode is already normal #6644 [@radarhere]

Fixed seeking to an L frame in a GIF #6576 [@radarhere]

Consider all frames when selecting mode for PNG save_all #6610 [@radarhere]

Don't reassign crc on ChunkStream close #6627 [@radarhere]

Raise a warning if NumPy failed to raise an error during conversion #6594 [@radarhere]

Only read a maximum of 100 bytes at a time in IMT header #6623 [@radarhere]

Show all frames in ImageShow #6611 [@radarhere]

Allow FLI palette chunk to not be first #6626 [@radarhere]

If first GIF frame has transparency for RGB_ALWAYS loading strategy, use RGBA mode #6592 [@radarhere]

Round box position to integer when pasting embedded color #6517 [@radarhere]

Removed EXIF prefix when saving WebP #6582 [@radarhere]

Pad IM palette to 768 bytes when saving #6579 [@radarhere]

Added DDS BC6H reading #6449 [@ShadelessFox]

Added support for opening WhiteIsZero 16-bit integer TIFF images #6642 [@JayWiz]

Raise an error when allocating translucent color to RGB palette #6654 [@jsbueno]

Moved mode check outside of loops #6650 [@radarhere]

Added reading of TIFF child images #6569 [@radarhere]

Improved ImageOps palette handling #6596 [@PososikTeam]

Defer parsing of palette into colors #6567 [@radarhere]

Apply transparency to P images in ImageTk.PhotoImage #6559 [@radarhere]

Use rounding in ImageOps contain() and pad() #6522 [@bibinhashley]

Fixed GIF remapping to palette with duplicate entries #6548 [@radarhere]

Allow remap_palette() to return an image with less than 256 palette entries #6543 [@radarhere]

Corrected BMP and TGA palette size when saving #6500 [@radarhere]

... (truncated)

Changelog

Sourced from pillow's changelog.

9.3.0 (2022-10-29)

Limit SAMPLESPERPIXEL to avoid runtime DOS #6700 [wiredfool]

Initialize libtiff buffer when saving #6699 [radarhere]

Inline fname2char to fix memory leak #6329 [nulano]

Fix memory leaks related to text features #6330 [nulano]

Use double quotes for version check on old CPython on Windows #6695 [hugovk]

Remove backup implementation of Round for Windows platforms #6693 [cgohlke]

Fixed set_variation_by_name offset #6445 [radarhere]

Fix malloc in _imagingft.c:font_setvaraxes #6690 [cgohlke]

Release Python GIL when converting images using matrix operations #6418 [hmaarrfk]

Added ExifTags enums #6630 [radarhere]

Do not modify previous frame when calculating delta in PNG #6683 [radarhere]

Added support for reading BMP images with RLE4 compression #6674 [npjg, radarhere]

Decode JPEG compressed BLP1 data in original mode #6678 [radarhere]

Added GPS TIFF tag info #6661 [radarhere]

Added conversion between RGB/RGBA/RGBX and LAB #6647 [radarhere]

Do not attempt normalization if mode is already normal #6644 [radarhere]

... (truncated)

Commits

d594f4c Update CHANGES.rst [ci skip]

909dc64 9.3.0 version bump

1a51ce7 Merge pull request #6699 from hugovk/security-libtiff_buffer

2444cdd Merge pull request #6700 from hugovk/security-samples_per_pixel-sec

744f455 Added release notes

0846bfa Add to release notes

799a6a0 Fix linting

00b25fd Hide UserWarning in logs

05b175e Tighter test case

13f2c5a Prevent DOS with large SAMPLESPERPIXEL in Tiff IFD

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

CLA Signed dependencies
opened by dependabot[bot] 2
Swap to CMake FetchContent and cleanup modules

See title. CMake FetchContent solves a bunch of install-related issues and makes reasoning about configurable downstream dependencies easier.

Test plan: CI + local builds without the required dependencies and with FL_BUILD_STANDALONE set to ON
CLA Signed

opened by jacobkahn 1

Releases(v0.3.2)

v0.3.2(Mar 19, 2022)
This is a patch release that will be the last release before breaking changes to Flashlight given the Flashlight Tensor framework (fl::Tensor). This and other patch releases to v0.3.x will be updated as needed in the 0.3 branch, which will diverge from main.

Fixes in this release include:

Build fixes, removing empty target issues for install (#834)

Updates to Google Mock syntax to require a minimum version of 1.10.

An updated CITATION file for citing FL

Updates to CircleCI per breaking changes

Improvements to CMake BLAS lib locating (#806)

Fix gradient computation for batched matrix multiply (#760)

Refactoring Flashlight components into a pkg-based layout (#713)

This release contains improvements to the Flashlight Tensor API (flashlight/fl/tensor) including:

A roll operator for Tensor

Fixes to static initialization for ArrayFireBackend state

Fixes to the concatenate operation with the ArrayFireBackend

Better Tensor and Shape stringification functions (#808)

TensorBackend::isDataTypeSupported()

TensorBackend::getDeviceCount()

Adds a sorting operation that returns both values and indices (#809)

Better protection for OOB indexing on a Shape (#810)

The TensorExtension API for autograd and vision primitives (#762)

Scalar exponentiation by a tensor (#774)

Tensor broadcasting for binary ops (#775)

A scalar (0D) tensor type

A flip operator (#766)

Rounding and cumsum operators

Reductions that return host-side primitives now return Tensors (#755)

Source code(tar.gz)
Source code(zip)
v0.3.1(Oct 15, 2021)
v0.3.1 is a patch release that includes fixes since v0.3 and the initial fl::Tensor API.

New Features

Introduce the fl::Tensor API. See flashlight/fl/tensor for more.

Flashlight will be adapted to use the API in a forthcoming release.

A distributed stub backend to enable building apps without a distributed backend for the time being (will be disentangled in a later release) (#614)

A benchmark app to use for end to end performance benchmarking of common models

C++17 minimum requirement

Introduce DynamicScaler to abstract away repeated logic in mixed precision training

Support for automatic mixed precision training with DETR, ResNet, and ViT

Codecov

Fixes

Change behavior of comformer skip connections (#587)

Fixes to typing in the advanced index kernel

Removes eval() in the backward pass

Serialization support for audio augmentations

Rewrite of the SinusoidalPositionEmbedding operator to avoid concatenation

Thanks to external contributors for their contributions, including: @lunixbochs @tlikhomanenko
Source code(tar.gz)
Source code(zip)
v0.3(Apr 16, 2021)
First stable release post-consolidation. Separates Flashlight into four parts:

flashlight/lib contains kernels and standalone utilities for sequence losses, beam search decoding, text processing, and more.

flashlight/fl is the core neural network library using the ArrayFire tensor library.

flashlight/app are applications of the core library to machine learning across domains.

flashlight/ext are extensions on top of Flashlight and ArrayFire that are useful across apps.

Major Features

Automatic mixed precision training (AMP) -- typed tensor and autograd operators

Framework for building custom memory managers on top of ArrayFire (docs)

OneDNN as a backend for primitive operations on the CPU

New dataset abstractions in core (flashlight/fl/dataset)

Application libraries

Speech recognition (formerly the wav2letter project)

Language modeling (autoregressive and masked/BERT-style LMs)

Image classification (Resnet, VIT)

Object detection (DETR)

Audio augmentation library (ASR)

Tools for training models using iterative pseudo-labeling (IPL) (ASR)

[early] OpenCL support with both RoCM and Intel

Build Changes/Improvements

C++ 17 support -- gcc 7/clang 6 required.

Support for vcpkg via FL_BUILD_STANDALONE

Consolidation of wav2letter and app-based build selection

CMake 3.10 minimum, better support for shared objects

First class support for CUDA and Halide kernels

Improved support for downloading not-found dependencies (Gloo, KenLM, libsndfile)

Improved support for dependency management for downstream projects using Flashlight's installed CMake config (cmake/flashlightConfig.cmake.in)

Supporting padding in transformer/multihead attention

SpecAugment for raw waves (implemented vie low-pass filter)

Conformer Implementation

Improve autograd for indexing operator (support repeated indices)

Improve python bindings build, supporting setup.py install

A lot of docs.

Improvements/new features in wav2letter (flashlight/app/asr)

Fixed padding issues in s2s models: pre-training window, encoder attention, encoder-decoder attention

Refactor s2s codebase

Fixes to memory allocations for s2s beam-search decoder (less memory, no OOM issues)

Fixes to beam-search decoder to support non-empty surround

Fixes to dataset pipeline + dynamic batching support

Source code(tar.gz)
Source code(zip)
v0.2(Dec 28, 2020)

This is an old release and is the final release before the wav2letter was consolidated into Flashlight. See the ASR application in Flashlight for the new wav2letter source.

The core Flashlight library has been moved to flashlight/fl.
Source code(tar.gz)
Source code(zip)
v0.1(Dec 22, 2018)

Going Native with Flashlight

To get started, visit the documentation.

Credits

Thanks to our contributors:

Vineel Pratap, Awni Hannun, Jacob Kahn, Qiantong Xu, Jeff Cai, Gabriel Synnaeve, Vitaliy Liptchinsky, Ronan Collobert, Ann Lee, Jay Mahadeokar and Tatiana Likhomanenko
Source code(tar.gz)
Source code(zip)