==========================

An API for neural networks implemented in C++ with template meta-programming. Perhaps the first of its kind.

==========================

All the necessary components are in header files (yay templates!) So just add what you need from the include folder.

==========================

Our brains work by a large web of connected neurons, or simple binary states. These neurons are connected by synapses, which have a strength associated with them. When a neuron fires, it's signal is sent through all of it's connecting synapses to other neurons to determine their value. When we learn, our brain adjusts the strengths of the associated synapses to limit the amount of activated neurons.

A neural network is a machine learning algorithm based off of the brain. Within a network, there are layers. Each of these layers has a number of neurons, which take on floating point values, and weights, symbolic of synapses, attached to the neurons in the next layer. These networks then run in a way similar to our brains, given an input, all neurons are fed forward to the next layer by summing the value of the neurons times the weights connecting two neurons. Commonly, a bias is an addition to the network which is used as a simple shift to neurons. The bias is added to the sum of the weights times the neurons to produce the output of the neuron, which is then commonly ran through a continuous activation function, such as a sigmoid, to bound the value of the neuron as well as give the network a differentiable property.

Weights can be connected between neurons in different ways. Most common are full connectivity layers and shared weight layers. Full connectivity layers have weights going from every input neuron to every output neuron, so every neuron in the layers are connected to every neuron in the layers above. Shared weights are a way of forming similar connections between different neurons by a common weight pattern. A common implementation of this is convolutional layers.

Convolutional layers make use of mathematical convolution, an operation used to produce feature maps, or highlights from an image. Convolution is formally defined as the sum of all values in the domains of two functions which are multiplied by one another. In real life cases, this is commonly discrete, and is most easily understood in images. Image convolution involves iterating a mask over an image to produce an output, where the output pixel values are equivelant to the sum of the mask multiplied by neighboring pixels in the input when anchored at the center of the mask. This operation draws features from the image, such as edges or curves, and is associated with the way our visual cortex processes imagery.

Networks learn through different algorithms, although the two implemented here are the up-down or wake-sleep algorithm and vanilla backpropagation. Backpropagation is an algorithm which computes the derivatives of the error with respect to the weights, and adjusts the weights in order to find a minimum in the error function. This is a way of approximating the actual error signal of every neuron, so a small step size is often used to prevent divergence. The wake-sleep or up-down algorithm trains the network without knowledge of data in an encoder-decoder format. The layers in the network are fed forward, backwards, and forwards again, before a difference is calculated to adjust the weights.

=================================

This API is based off of template meta-programming to optimize efficiency. Therefore, much of this API is based on the assumption that a network architecture will be defined at compile time rather than runtime. This has caused most of the class to become static, therefore you may want to typedef NeuralNet<...> Net; in your source file for clarity. More details on accessing a NeuralNet can be found in it's section.

Note that because this is a template based approach, then almost all errors will be indescript compiler errors. Generally it is because a particular layer does not "connect" to the next.

===============================

===============================

These macros are used to signify layer types, optimization methods, loss functions, and activation functions. They are prefixed with MTNN_FUNC_* for activation functions, MTNN_LAYER_* for layers, MTNN_OPT_* for optimization methods, and MTNN_COST_* for cost functions. Their name should explain their use. The available layers can be found below.

Available activation functions are linear (y = x), sigmoid (y = 1/(1 + exp(-x)), bipolar sigmoid (y = 2/(1 + exp(-x)) - 1), tanh (y = tanh), and rectified linear (y = max(0, x)).

Available loss functions are quadratic, cross entropy, log likelihood, and custom targets.

Available optimization methods are vanilla backprop (with momentum, l2 weight decay, etc. as desired), Adam, and Adagrad.

===============================

This class is a simple matrix implementation, with some extra methods that can be used in situations outside of this neural network.

This table contains methods used only in the source code of the network

Can be initialized with initialization lists, so brace initializers may create some problems.

===============================

This class is a slightly more advanced wrapper of just a std::vector<Matrix2D<T, r, c>(f), with basic initialization functions.

Can be initialized with initialization lists, so brace initializers may create some problems.

===============================

There is no Layer class, but all of the "*Layer" classes are implemented similarily. Note that only members that are used will be used as this API uses implicit static initialization.

Use the index parameter to create different instances if using the same type of layer multiple times (eg. if using a InputLayer taking 1 input on multiple networks, add a distinct index to prevent them from modifying each other's data)

===============================

Basic fully connected perceptron layer.

===============================

Basic convolutional layer, masks or kernels must be square (but not odd!).

With padding, then output is same size. Otherwise output is reduced.

===============================

Basic LSTM layer (uses tanh activation). STILL IN DEVELOPMENT, WON'T WORK WITH THREADS.

max_t_store states how many time steps to perform bptt on.

===================================

Basic maxpooling layer. Maxpool is performed on each feature map independently.

=====================================

Basic softmax layer. This will compute derivatives for any cost function, not just log-likelihood. Softmax is performed on each feature map independently.

=====================================

Basic batch normalization layer. Gamma and beta are in weights and biases.

If using, then batch learning and the respective overloads must be used.

=====================================

Basic input layer just to signify the beginning of the network. Required

=====================================

Basic output layer just to signify the end of the network. Required

===============================

This is the class that encapsulates all of the rest. Has all required methods. Will add support for more loss functions and optimization methods later. If you want to train the network in parallel (or keep different sets of weights for a target network, or different architecture, etc.) then create a new instance of the class. Each new instance has its own weights and gradients and is thread safe (if you use *_thread(.) functions). The static class is the master net and retains its own weights.

This is a singleton static class. This class helps with network analysis, such as the expected error, and finite difference backprop checking.

===============================

For an example of creating and using a network, see main.cpp in the examples folder.

There is also an example with the MNIST Database in the examples folder. The provided .nn file has ~1% error.