Quick explanation

Performing binary classification with the network from NeuralNetwork.py file where an arbitrary architecture (depth and width) can be selected. The data consists of one 2D training set of s_t = 10000 data points of x_i∈ R² for i=1,...,s with corresponding target labels t = ± 1 evaluated on a validation set of s_v = 5000 and one 3D training set of s_t = 12000 data points with corresponding validation set s_v=6000 with same target labels.

To run and test the network on the pre-defined configuration and data, just type

python run.py

The classification error is defined as

where is the output of the network and s the size of the dataset. Furthermore are the tanh function used as activation functions with a local field such that the output of node i in layer l for input μ is defined as

where M_l is the number of nodes in layer l.

The network is trained by stochastic gradient descent sequential learning implying that the parameters are updated as

where η is the learning rate and is the cost vector for each layer evaluated by the chain rule as

with

being the cost value for the output layer.

Moreover, the weights are initiated with a modified glorot uniform initialization as

where is the univariate normal (gaussian) distribution with mean μ and variance σ and M_l, M_l+1 is the number of nodes in layers l and l+1 respectively. The thresholds are initialized to zero.

Results

Initilization of network

Initializing the network for two hidden layers with n₁ = n₂ = 5 hidden neurons each and training for 300 epochs with an initial learning rate of . The weights are initiated with the modified glorot initializer.

2D data

The results with the above initialization and on the 2D data, which looks like

is the following

And like that one can construct a custom machine learning classifier :)