RNNLIB is a recurrent neural network library for sequence learning problems. Forked from Alex Graves work http://sourceforge.net/projects/rnnl/

Overview

Origin

The original RNNLIB is hosted at http://sourceforge.net/projects/rnnl while this "fork" is created to repeat results for the online handwriting prediction and synthesis reported in http://arxiv.org/abs/1308.0850. The later by now is Alex Graves's classic paper on LSTM networks showing of what RNN can learn about the structure present in the sequential input.

Building

Building rnnlib requires the following:

  • C++11 compiler
  • fortran for building OpenBLAS
  • cmake
  • libcurl
  • automake
  • libtool
  • texinfo

In addition, the following python packages are needed for the auxiliary scripts in the 'utils' directory:

  • SciPy
  • PyLab
  • PIL

And this package is needed to create and manipulate netcdf data files with python, and to run the experiments in the 'examples' directory:

  • ScientificPython (NOT Scipy)

To build RNNLIB do

$ cmake -DCMAKE_BUILD_TYPE=Release . $ cmake --build .

Cmake run creates the binary files 'rnnlib', 'rnnsynth' and 'gradient_check' in the current directory.

It is recommended that you add the directory containing the 'rnnlib' binary to your path, as otherwise the tools in the 'utilities' directory will not work.

Project files for the integrated development environments can be generated by cmake. Run cmake --help to get list of supported IDEs.

Handwriting synthesis

Step in to examples/online_prediction and go through few steps below to prepare the training data, train the model and eventually plot the results of the synthesis

Downloading online handwriting dataset

Start by registering and downloading pen strokes data from http://www.iam.unibe.ch/~fkiwww/iamondb/data/lineStrokes-all.tar.gz Text lables for strokes can be found here http://www.iam.unibe.ch/~fkiwww/iamondb/data/ascii-all.tar.gz Then unzip ./lineStrokes and ./ascii under examples/online_prediction. Data format in the downloaded files can not be used as is and requires further preprocessing to convert pen coordinates to offsets from previous point and merge them into the single file of netcdf format.

Preparing the training data

Run ./build_netcdf.sh to split dataset to training and validation sets. The same script does all necessary preprocessing including normalisation of the input and makes corresponding online.nc and online_validation.nc files for use with rnnlib .

Each point in the input sequences from online.nc consists of three numbers: the x and y offset from the previous point, and the binary end-of-stroke feature.

Gradient check

To gain some confidence that the build is fine run the gradient check:

gradient_check --autosave=false check_synth2.config

Training

The training goes in two steps. First it is done without weights regularization and then repeated again with adaptive weight noise (MDL in rnnlib terms) from the best network recorded by step one. Training with MDL from the beginning will have too slow convergence rate.

Step 1

rnnlib --verbose=false synth1d.config

Where synth1d.config is 1st step configuration file that defines network topology: 3 LSTM hidden layers of 400 cells, 20 gaussian mixtures as output layer, 10 mixtures for character warping window layer Somewhere between training epoch 10-15 it will find optimal solution and will do "early stopping" w/o improvement for 20 epoch. "Early" here takes 3 days on Intel Sandybridge CPU. Normally training can be stopped as long as loss starts rising up for 2-3 consequent epochs. The best solution found is stored in [email protected]_loss.save file

Step 2

Best loss error from step 1 is expected to be around -1080 nats and it can be further improved (ca. 10%) by using weights regularisation. Loss error goes up and down during the training unlike in Step 1. Therefore one must be more patient to declare early stopping and wait for 20 epochs with loss worse then the best result so far. Rnnlib has implementation of MDL regulariser which is used in this step. The command line is as following:

rnnlib --mdl=true --mdlOptimiser=rmsprop from_step1.best_loss.save

Synthesis

Handwriting synthesis is done by rnnsynth binary using network parameters obtained by step 2:

rnnsynth from_step2.best_loss.save

The character sequence is given to stdin and output is written to stdout. The output sequence is the same as input where each data point has x,y offsets and end-of-stroke flag.

Plotting the results

Rnnsynth output is the sequence of x,y offsets and end-of-stroke flags. To visualise it one can use show_pen.m Octave script:

octave:>show_pen('/tmp/trace1')

Where /tmp/trace1 contains stdout from rnnsynth.

Rnnlib configuration file

Configuration options are exlained in http://sourceforge.net/p/rnnl/wiki/Home/. Since then there are few things added:

  • lstm1d as hiddenType layer type - optimised LSTM layer when input dimension is 1d
  • rmsprop optimizer type
  • mixtures=N where N is number of gaussians in the output layer
  • charWindowSize=N where N is the number of gaussians in the character window layer
  • skipConnections=true|false - whether to add skip connections; default is true

Contact

Please create github issues to discuss the problems

Comments
  • Constants in show_pen.m

    Constants in show_pen.m

    I am trying to adapt this program to run with my own sample sets. While doing so, I found that the outputs of the neural network don't correspond directly to relative coordinates, but instead need to have some constants multiplied and added to them, labelled muXY and devXY in show_pen.m.

    Tweaking these constants, I have not found what their exact significance is, but only that if the ratio between them changes too much, the output becomes distorted.

    I assume that these constants are dependent on the dataset. I have not found any references to them in the source code, nor in the original paper published by Alex Graves. What are these constants and how would I calculate them for my own dataset?

    opened by Symaxion 3
  • change bias

    change bias

    How can I prime the trained rnn when running rnnsynth and change the bias? Searched for days, tried a lot via config file or command line - no luck so far ... anybody? thx so much!

    opened by rollasoul 1
  • online_prediction train error

    online_prediction train error

    I don't know what happened and how to handle it. When I run this command, I got the result as follow: @szcom Is it right? writing to log file [email protected] task = prediction

    network: task = prediction 13MultilayerNet

    5 layers: 10InputLayer "input" 1D (+) size 3 (R) 11Lstm1dLayerI4TanhS0_8LogisticE "hidden_0_0" 1D (+) inputSize 1600 outputSize 400 source "input" 1200 peeps (R) 11Lstm1dLayerI4TanhS0_8LogisticE "hidden_1_0" 1D (+) inputSize 1600 outputSize 400 source "hidden_0_0" 1200 peeps (R) 11Lstm1dLayerI4TanhS0_8LogisticE "hidden_2_0" 1D (+) inputSize 1600 outputSize 400 source "hidden_1_0" 1200 peeps 18MixtureOutputLayer "output" 1D (+) size 121 source "hidden_2_0"

    20 connections: "bias_to_output" (121 wts) "hidden_2_0_to_output" (48400 wts) "hidden_0_0_to_output" (48400 wts) "hidden_1_0_to_output" (48400 wts) "bias_to_charwindow_0" (30 wts) "hidden_0_0_to_charwindow_0" (12000 wts) "bias_to_hidden_0_0" (1600 wts) "hidden_0_0_to_hidden_0_0_delay_-1" (640000 wts) "input_to_hidden_0_0" (4800 wts) "charwindow_0_to_hidden_0_0_delay_-1" (92800 wts) "bias_to_hidden_1_0" (1600 wts) "hidden_1_0_to_hidden_1_0_delay_-1" (640000 wts) "hidden_0_0_to_hidden_1_0" (640000 wts) "input_to_hidden_1_0" (4800 wts) "charwindow_0_to_hidden_1_0" (92800 wts) "bias_to_hidden_2_0" (1600 wts) "hidden_2_0_to_hidden_2_0_delay_-1" (640000 wts) "hidden_1_0_to_hidden_2_0" (640000 wts) "input_to_hidden_2_0" (4800 wts) "charwindow_0_to_hidden_2_0" (92800 wts)

    bidirectional = false symmetry = false 3658551 weights

    setting random seed to 1006135989

    3658551 uninitialised weights randomised uniformly in [-0.1,0.1] trainer: epoch = 0 savename = [email protected] batchLearn = false seqsPerWeightUpdate = 1 maxTestsNoBest = 20

    training data: numSequences = 10748 numTimesteps = 6765601 avg timesteps/seq = 629.475 1 filenames online.nc inputSize = 3 outputSize = 0 numDims = 1 task = prediction shuffled = true

    validation data: numSequences = 1438 numTimesteps = 885964 avg timesteps/seq = 616.108 1 filenames online_validation.nc inputSize = 3 outputSize = 0 numDims = 1 task = prediction shuffled = false


    rmsprop learnRate = 0.0001 momentum = 0.9

    autosave filename [email protected] best save filename root [email protected]

    training... loading sequences from 0 to 10748

    epoch 0 took 7 hours 13 minutes 27 seconds (2.41971 secs/seq, 260.145 its/sec, 3709.15 MwtOps/sec)

    train errors (running): loss -753.446

    validation errors: loading sequences from 0 to 1438 loss -866.95

    saving to [email protected] best network (loss) saving to [email protected]_loss.save

    epoch 1 took 6 hours 22 minutes 42 seconds (2.1364 secs/seq, 294.643 its/sec, 4201.02 MwtOps/sec)

    train errors (running): loss -914.131

    validation errors:

    opened by ghost 0
  • How to create Labels ?

    How to create Labels ?

    Hi everyone, I wonder how to create labels ? I noticed in the arabic offline there is a .tru file per image including a couple of fields. I don't understand what they are. I went over the Python code generating the binary files. I got some ideas but I really don't get what this is "wordTargetStrings" ? In .tru file there is a field or keyword called "ZIP" and wordTargetStrings is just getting the values in front of that. I appreciate if someone can clarify that for me. Cheers, Saman

    opened by samansarraf 0
  • "Cmake --build." Is not completed

    I followed the manual to include the library and build it. However, an error occurs during the build process. Is there a better way to deal with it?

    In file included from /Users/home/rnnlib/src/Val.cpp:22: In file included from /Users/home/rnnlib/src/MultilayerNet.hpp:21: In file included from /Users/home/rnnlib/src/Mdrnn.hpp:30: /Users/home/rnnlib/src/NetcdfDataset.hpp:26:10: fatal error: 'netcdfcpp.h' file not found #include "netcdfcpp.h" ^~~~~~~~~~~~~ 2 warnings and 1 error generated. make[2]: *** [CMakeFiles/rnnval.dir/src/Val.cpp.o] Error 1 make[1]: *** [CMakeFiles/rnnval.dir/all] Error 2 make: *** [all] Error 2

    opened by shibryo 0
  • The demo is not working anymore?

    The demo is not working anymore?

    Hi can someone please fix this! As this is very handy to learn writing or studiying the science behind writing. Please! Or is there something similar like this?

    opened by alexfangraves 0
  • Fix Example Dataset Generation

    Fix Example Dataset Generation

    The dataset generation was broken because ScientificPython was not working as expected anymore. Fortunately the same tools are also provided by SciPy and netCDF4, so I just replaced the old library.

    Additionally, I did some refactoring/restructuring and fixed what seemed to be a memory leak.

    opened by hendraet 0
  • Binaries and network parameters files for online_prediction

    Binaries and network parameters files for online_prediction

    Hello,

    I am trying to use rnnsynth for a project. I eventually managed to compile the binaries on OSX but having trouble to use the preparation script (build_netcdf.sh) and with training the model.

    Would it be possible to share the "from_step2.best_loss.save" file (generated from step 2) so users can generate plots and position output from rnnsynth, at least for the online_prediction example?

    Also, considering that it is quite hard to build the binaries. It would greatly help further users to distribute a binary package for a standard Linux.

    Thank you for your further help, it will be greatly appreciated!

    Kevin Donnot

    opened by KevinDonnot 0
  • Generating sequence

    Generating sequence

    Hi, I'm trying to generate the sequence for "test" and then visualise it using show_pen. but the samples that I get are too few to represent the word test. Note: I trained my own model following the described steps.

    After running the rnnsynth, I got this result : test sentence:test generating samples from network Sample -0.196839 -0.00302385 0 Sample 0.0456886 0.205597 0 Sample 0.244101 2.64111 0 Sample -0.111819 -0.0829409 0 Sample 0.117248 -0.937839 1 Sample 1.0191 -2.28362 1 Sample -0.224143 -0.0129589 0 Sample -0.500786 -0.014552 0 Sample -0.122401 0.0879939 0 Sample 0.0576331 0.384149 0 End of sentence

    opened by HamzaouiManal 0
Owner
Sergey Zyrianov
Sergey Zyrianov
C++ NN šŸ§  A simple Neural Network library written in C++

C++ NN ?? A simple Neural Network library written in C++ Installation ??

Rohith 8 Dec 13, 2022
Convolutional Neural Networks

Darknet Darknet is an open source neural network framework written in C and CUDA. It is fast, easy to install, and supports CPU and GPU computation. D

Joseph Redmon 23.7k Jan 9, 2023
Edge ML Library - High-performance Compute Library for On-device Machine Learning Inference

Edge ML Library (EMLL) offers optimized basic routines like general matrix multiplications (GEMM) and quantizations, to speed up machine learning (ML) inference on ARM-based devices. EMLL supports fp32, fp16 and int8 data types. EMLL accelerates on-device NMT, ASR and OCR engines of Youdao, Inc.

NetEase Youdao 180 Jan 7, 2023
BLLIP reranking parser (also known as Charniak-Johnson parser, Charniak parser, Brown reranking parser) See http://pypi.python.org/pypi/bllipparser/ for Python module.

BLLIP Reranking Parser Copyright Mark Johnson, Eugene Charniak, 24th November 2005 --- August 2006 We request acknowledgement in any publications that

Brown Laboratory for Linguistic Information Processing 218 Dec 17, 2022
A lightweight C++ machine learning library for embedded electronics and robotics.

Fido Fido is an lightweight, highly modular C++ machine learning library for embedded electronics and robotics. Fido is especially suited for robotic

The Fido Project 413 Dec 17, 2022
A C++ standalone library for machine learning

Flashlight: Fast, Flexible Machine Learning in C++ Quickstart | Installation | Documentation Flashlight is a fast, flexible machine learning library w

Facebook Research 4.7k Jan 8, 2023
mlpack: a scalable C++ machine learning library --

a fast, flexible machine learning library Home | Documentation | Doxygen | Community | Help | IRC Chat Download: current stable version (3.4.2) mlpack

mlpack 4.2k Dec 30, 2022
Flashlight is a C++ standalone library for machine learning

Flashlight is a fast, flexible machine learning library written entirely in C++ from the Facebook AI Research Speech team and the creators of Torch and Deep Speech.

null 4.7k Jan 8, 2023
ML++ - A library created to revitalize C++ as a machine learning front end

ML++ Machine learning is a vast and exiciting discipline, garnering attention from specialists of many fields. Unfortunately, for C++ programmers and

marc 1k Dec 31, 2022
Caffe: a fast open framework for deep learning.

Caffe Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research (BAIR)/The Berke

Berkeley Vision and Learning Center 33k Jan 1, 2023
A toolkit for making real world machine learning and data analysis applications in C++

dlib C++ library Dlib is a modern C++ toolkit containing machine learning algorithms and tools for creating complex software in C++ to solve real worl

Davis E. King 11.6k Dec 31, 2022
Distributed (Deep) Machine Learning Community 682 Dec 28, 2022
Frog is an integration of memory-based natural language processing (NLP) modules developed for Dutch. All NLP modules are based on Timbl, the Tilburg memory-based learning software package.

Frog - A Tagger-Lemmatizer-Morphological-Analyzer-Dependency-Parser for Dutch Copyright 2006-2020 Ko van der Sloot, Maarten van Gompel, Antal van den

Language Machines 70 Dec 14, 2022
null 5.7k Jan 4, 2023
Machine Learning Framework for Operating Systems - Brings ML to Linux kernel

Machine Learning Framework for Operating Systems - Brings ML to Linux kernel

File systems and Storage Lab (FSL) 186 Nov 24, 2022
C-based/Cached/Core Computer Vision Library, A Modern Computer Vision Library

Build Status Travis CI VM: Linux x64: Raspberry Pi 3: Jetson TX2: Backstory I set to build ccv with a minimalism inspiration. That was back in 2010, o

Liu Liu 6.9k Jan 6, 2023
libsvm websitelibsvm - A simple, easy-to-use, efficient library for Support Vector Machines. [BSD-3-Clause] website

Libsvm is a simple, easy-to-use, and efficient software for SVM classification and regression. It solves C-SVM classification, nu-SVM classification,

Chih-Jen Lin 4.3k Jan 2, 2023
Open Source Computer Vision Library

OpenCV: Open Source Computer Vision Library Resources Homepage: https://opencv.org Courses: https://opencv.org/courses Docs: https://docs.opencv.org/m

OpenCV 65.6k Jan 1, 2023
oneAPI Data Analytics Library (oneDAL)

IntelĀ® oneAPI Data Analytics Library Installation | Documentation | Support | Examples | Samples | How to Contribute IntelĀ® oneAPI Data Analytics Libr

oneAPI-SRC 534 Dec 30, 2022