header only, dependency-free deep learning framework in C++14

Hi, this PR changes parallel_for function to use ThreadPool library written by Jakob Progsch, Václav Zeman. https://github.com/progschj/ThreadPool

This will improve execution speed on Linux.

Task list:

• [x] Basic functions and pass the tests on core quantization utilities.
• [x] Quantized convolution layer and pass the tests for q_conv.
• [x] Quantized deconvolution layer and pass the tests for q_deconv.
• [x] Quantized fully connected layer and pass the tests for q_fully_connected.
• [x] Quantized bias inside other kernels.
• [x] Ensure acceptable accuracy on typical examples.
• [x] Add low precision gemm from Google TF as a kernel for matmul.
• [x] Resolve unnecessary quantize-dequantize procedure.
• [x] Add backward propagation for quantization.

Method:

The method is totally the same as described in Pete's blog for quantization and my basic codes are modified from TensorFlow quantization module.

Reproducing the problem described in https://github.com/tiny-dnn/tiny-dnn/issues/540 No fix yet. As my test shown, the problem exists only with batch size >10.

First commit for device abstraction:

• The convolution class has been abstracted
• Added a PlantUML in order to show the "big picture" of the current implementation.
• Removed test warnings

I've rerun clang-format on current code base. All test are passed at my PC.

Command to run formatting is find -iname *.h -o -iname *.cpp | xargs clang-format-4.0 -style=file -i

I suppose to add it as a git hook or something like that.

@naibaf7 @nyanp @bhack @mtamburrano I open this ticket for LibDNN integration discussion.

@naibaf7 What's the current status of LibDNN standalone? Recently, the initial backend architecture was merged. Please, give it a shot and feel free to comment about design or any consideration you think that will be convenient for LibDNN or others optimizations. Thx !

We've decided to move tiny-cnn to an organization account to accelarate its development (discussed in #226).

Since it is clear that we are expanding the scope of tiny-cnn from convolutional networks to general networks, the project name tiny-Cnn seems a bit inaccurate now. I want to change the project name to more appropriate one (if we agreed), at the timing of the transferring repository.

In #226 we have these 3 proposals:

• tiny-dnn (convolutioanl net -> deep net)
• hornet (loose acronyms of Header Only Network)
• tiny-cnn (of course we can keep its name)

Whichever we take, naming of project doesn't affect the library API except for its namespace, and hyperlinks and folks, pull requests will be correctly redirected to the new repository.

Please feel free to give me your feedback if you have suggestions for the naming! We want to decide the name and move to a new account until around 7/25.

Well, as usual, I made a mess with GIT, and apparently I can't easily push to @edgarriba's original PR.

Today I reworked a bit @edgarriba's #400, I changed a bit the interface, added the lazy-allocation and lazy-movement of memory around, renamed accessor to host_ptr and host_at (to clarify they work on host memory), and implemented the generic functions for binary and unary host operations, element-wise and scalar, so it's now easier to implement functions such as add, sub, mul, div, exp, sqrt, ...

I commented out linspace, if someone has a strong opinion about that, we can still get it back in.

Since global average pooling layer calculates average of all activations per channel, we pick up contigious 8 floats and keep on performing vertical sum channelwise. At the end a net sum is accumulated by horizontal sum. This is repeated for all channels of a layer.

Current code falls back to internal backend if nnpack or other unsupported backend is chosen.

I open this ticket to discuss about modeling tensors in a classes as discussed in https://github.com/tiny-dnn/tiny-dnn/issues/235#issuecomment-239196739

@pansk Proposed to have different structures depending on the data nature:

In an email thread with @nyanp we were discussing about having a structure that represents i/o tensors and another which represents parameters (weights, biases, convolution coefficients, ...). The first structure was supposed to "automatically" move between the CPU and the GPU when needed (e.g., when interleaving CPU and GPU layers, something which is probably still inefficient, but should be allowed for prototyping new, complex layers), while the second was conceived for full-time resident on the GPU (for GPU backends) and was supposed to be downloaded/uploaded only for serialization/deserialization purposes (probably manually).

Add the Tensor structure:

• data hold by std::vector<> with 64bytes alignment
• three different data accessors: t.ptr<float_t>(), t.at<float_t>() and t[idx]
• basic reshape() and resize() routines
• basic toDevice() and fromDevice() routines
• implement element-wise add, sub, mul, div

Any time I try to use a deconv layer in a network im hit with the

terminate called after throwing an instance of 'std::bad_function_call' what(): bad_function_call Aborted (coredumped)``

I try the exact same setup with fully_connected_layers and it works just fine. Perhaps a bug?

I did a simple regression (MLP) with my 2155 data sets. The training seemed successfully completed, however, when I got "get_loss" with mse, "d" seems not to be divided by the total data sets which is 2155.

input_data:2155 lines x 6 input elements output_data:2155 lines x 1 output element

double loss = net.get_loss<tiny_dnn::mse>(input_data, target_data) std::cout<<"mse="<< loss <<std::endl;

---- loss_function.h ---- class mse { public: static float_t f(const vec_t &y, const vec_t &t) { assert(y.size() == t.size()); float_t d{0.0};

for (size_t i = 0; i < y.size(); ++i) d += (y[i] - t[i]) * (y[i] - t[i]);
**[Ichi]: this calculation is right. I confirmed ("predicted value" - "target value")^2 with Excel spared sheet.**

return d / static_cast<float_t>(y.size());
**[Ichi]: divided by one??? When I outputted "y.size()" with std::cout, it was "1".  I'm not a skillful C++ programmer. I might make a mistake.**

}....

I am pretty new to using C++ for Deep Neural Network? Could someone help with how to install this library? I have downloaded the zip, extracted it but I can't seem to include the tiny_dnn.h file.

I am using the Dev C++ editor. Could someone tell me how to add this particular library to the additional library of Dev C++?

Not used very often, so I can see this oversight.

I had a 20x1 data sample, and I used a 3x1 average pooling layer with 1x1 stride. The result should be an 18x1 output, but instead received an error from pooling_size_mismatch. In this case, average_pooling_layer wanted the width and height of the data to be a multiple of the pooling window, which it is not in my case.

However, due to the stride, the size was fine.