ML_ELM_class_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ML_ELM class example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** ML_ELM class implements a multi-layer ELM **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an example we will use the popular MNIST dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "import os\n",
    "import tensorflow as tf\n",
    "import keras\n",
    "from keras.datasets import mnist;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, test = mnist.load_data(os.getcwd() + \"/elm_tf_test\" + \"mnist.txt\");\n",
    "x_train, y_train = train\n",
    "x_test, y_test = test\n",
    "del train, test\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes=10)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes=10)\n",
    "\n",
    "# the input has to be flattened in order to be feeded to the network\n",
    "\n",
    "x_train = x_train.reshape(-1, 28* 28)\n",
    "x_test = x_test.reshape(-1, 28 * 28)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_size = 28*28 \n",
    "output_size = 10 # mnist has 10 output classes "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating ML_ELM classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tfelm.ml_elm import ML_ELM\n",
    "\n",
    "mlelm1 = ML_ELM(input_size=input_size, output_size=output_size, name='mlelm1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This creates an ML_ELM network with 784 input neurons and 10 output neurons. \n",
    "For now the hidden layers have not been specified. \n",
    "An hidden layer is added to the network through the add_layer method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlelm1.add_layer(n_neurons=1024, l2norm=1e2); \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This adds an hidden layer comprised of 1024 hidden layer neurons. \n",
    "\n",
    "By default the activation is set to tf.sigmoid and the initialization of weights and biases of the hidden layer is a modified He initialization: \n",
    "\n",
    "The weights are initialized by sampling from a random normal distribution with variance of 2/n_in, where n_in is the size of the previous layer, in this case the input layer. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlelm1.add_layer(n_neurons=1024, activation=tf.tanh, l2norm=1e2); "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This adds another hidden layer comprised of 1024 hidden layer neurons with a tanh activation function. \n",
    "Note that the normalization l2 parameter now can be diffrent for each layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the Network: fit method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train the network there are two main methods: train and fit. \n",
    "Fit is the most  basic and simple method but is suitable only for small datasets. \n",
    "\n",
    "It needs in input two numpy arrays for the training instances and labels and an optional batch_size argument. \n",
    "Internally a TensorFlow Iterator and a TensorFlow Dataset objects are created from the numpy arrays for the training as this it is the most efficient way to train a model according to TensorLow documentation. \n",
    "\n",
    "It should be noted that, unlike conventional Neural Networks, the batch_size doesn't change the outcome of the training but it only affect training time and memory required to train the network. The smaller the less the memory required but the more the training time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-07-26 18:07:49.796033 ML-ELM Start training...\n",
      "AE Network parameters have been initialized\n",
      "Training AE 1/1\n",
      "2018-07-26 18:07:49.933323 Start training...\n",
      "25/120 ETA:0:00\n",
      "50/120 ETA:0:00\n",
      "75/120 ETA:0:00\n",
      "100/120 ETA:0:00\n",
      "Training of AE mlelm1 ended in 0:0:0.735682\n",
      "Evaluating AE performance...\n",
      "MSE: 569.7012939\n",
      "####################################################################################################\n",
      "Initializing last layer ELM...\n",
      "Training last layer ELM...\n",
      "25/120 ETA:0:00\n",
      "50/120 ETA:0:00\n",
      "75/120 ETA:0:00\n",
      "100/120 ETA:0:00\n",
      "Training of ELM mlelm1 ended in 0:0:0.467148\n",
      "####################################################################################################\n",
      "####################################################################################################\n",
      "TOTAL Training of ML_ELM mlelm1 ended in 0:0:1.994121\n",
      "####################################################################################################\n",
      "Evaluating network performance\n",
      "Accuracy: 0.9344500\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.93445003"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mlelm1.fit(x_train, y_train, batch_size=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the network has been trained, we can evaluate the performance on the test set via evaluate method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating network performance\n",
      "Accuracy: 0.9357001\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.93570006"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mlelm1.evaluate(x_test, y_test, batch_size=500) # it accepts batch size as also the evaluation is done by batching to allow bigger datasets to be evaluated as we will see.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To return a numpy array with actual predictions it exist a prediction method:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicting...\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "pred = mlelm1.predict(x_test, batch_size=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is pretty much the most basic functionalities offered by the API and are suitable for small/medium datasets as it is required the dataset is fitted into memory as an array. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training and evaluating bigger Datasets which cannot be fitted into memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the same MNIST dataset for example purpose only. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of calling the fit method we should call the train method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The train method requires a TensorFlow Iterator object. the TensorFlow Iterator object thus must be created esternally from the dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are various ways to create a TF iterator object from a dataset and this strongly depends on what is your input pipeline and in what format your dataset is.\n",
    "\n",
    "Tutorials, documentation and examples on Dataset and Iterator is available at: https://www.tensorflow.org/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** As an example suppose we have an input pipeline in which we want to do some pre-process and data augmentation on the original MNIST dataset: **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.preprocessing.image import ImageDataGenerator # we will use keras imagedatagen for simplicity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 2500\n",
    "n_epochs = 10 # as the dataset is augmented the training now will be done on more \"epochs\", the resulting dataset will be 10 times the original.\n",
    "              # It could be argued that calling these epochs is not strictly correct as each \"epoch\" is different from the previous: \n",
    "              # the dataset is augmented via random trsformations\n",
    "        \n",
    "# keras ImageDataGen requires a 4-D tensor in input:\n",
    "x_train = x_train.reshape(-1, 28, 28, 1)\n",
    "\n",
    "datagen = ImageDataGenerator(\n",
    "        \n",
    "    width_shift_range=0.05,\n",
    "    height_shift_range=0.05\n",
    ")\n",
    "\n",
    "# random height and weight shifting \n",
    "\n",
    "\n",
    "datagen.fit(x_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "batches_per_epochs = len(x_train) // batch_size\n",
    "\n",
    "def gen():\n",
    "    n_it = 0\n",
    "    \n",
    "    for x, y in datagen.flow(x_train, y_train, batch_size=batch_size):\n",
    "        x = x.reshape(batch_size, 28 * 28) # the network requires a flatten array in input we flatten here\n",
    "        if n_it % 100 == 0:\n",
    "            print(\"generator iteration: %d\" % n_it)\n",
    "        yield x, y\n",
    "        n_it += 1\n",
    "        if n_it >= batches_per_epochs * n_epochs:\n",
    "            break\n",
    "\n",
    "data = tf.data.Dataset.from_generator(generator=gen,\n",
    "                                      output_shapes=((batch_size, 28 * 28,), (batch_size, 10,)),\n",
    "                                      output_types=(tf.float32, tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we have defined a python generator from the keras ImageDataGenerator and we have used this generator to create a TensorFlow dataset. This because it isn't possible to create a Dataset directly from the keraas generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike the standard ELM, because here a multi-layer ELM has to be trained, the train method requires as an additional argument a tensorflow iterator initializer. \n",
    "In other words it is not possible to use a one shot iterator. \n",
    "instead an initizalizable iterator is required: \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-07-26 18:07:52.645734 ML-ELM Start training...\n",
      "AE Network parameters have been initialized\n",
      "Training AE 1/1\n",
      "2018-07-26 18:07:52.739873 Start training...\n",
      "generator iteration: 0\n",
      "25/240 ETA:0:49\n",
      "50/240 ETA:0:39\n",
      "75/240 ETA:0:34\n",
      "100/240 ETA:0:29\n",
      "generator iteration: 100\n",
      "125/240 ETA:0:23\n",
      "150/240 ETA:0:18\n",
      "175/240 ETA:0:13\n",
      "200/240 ETA:0:08\n",
      "generator iteration: 200\n",
      "225/240 ETA:0:03\n",
      "Training of AE mlelm1 ended in 0:0:51.433999\n",
      "Evaluating AE performance...\n",
      "generator iteration: 0\n",
      "generator iteration: 100\n",
      "generator iteration: 200\n",
      "MSE: 627.4333496\n",
      "####################################################################################################\n",
      "Initializing last layer ELM...\n",
      "Training last layer ELM...\n",
      "generator iteration: 0\n",
      "25/240 ETA:0:44\n",
      "50/240 ETA:0:39\n",
      "75/240 ETA:0:35\n",
      "100/240 ETA:0:28\n",
      "generator iteration: 100\n",
      "125/240 ETA:0:25\n",
      "150/240 ETA:0:21\n",
      "175/240 ETA:0:16\n",
      "200/240 ETA:0:08\n",
      "generator iteration: 200\n",
      "225/240 ETA:0:03\n",
      "Training of ELM mlelm1 ended in 0:0:51.908342\n",
      "####################################################################################################\n",
      "####################################################################################################\n",
      "TOTAL Training of ML_ELM mlelm1 ended in 0:2:34.236620\n",
      "####################################################################################################\n"
     ]
    }
   ],
   "source": [
    "iterator = data.make_initializable_iterator() # a TF iterator is created from the Dataset\n",
    "iterator_init_op = lambda : mlelm1.sess.run(iterator.initializer)\n",
    "mlelm1.train(iterator,iterator_init_op,  n_batches=batches_per_epochs*n_epochs)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are other ways to generate initializable iterators but this is by far the simplest. \n",
    "In the training process the network is trained layer by layer starting from the one near the input. For the training of each layer the iterator is re-initializated via its init op"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The train method has the optinal n_batches argument which serves only the purpose of extimating the ETA. \n",
    "Note that the train method does not return the network performance. \n",
    "This should be done via evaluate. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating network performance\n",
      "Accuracy: 0.9276228\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.9276228"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mlelm1.evaluate(x_test, y_test,  batch_size=1024)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find the performance on the training set, due to the fact that ELM are not trained with gradient descent as conventional Neural Networks, one should call the evaluate function passing an iterator on the training set. \n",
    "\n",
    "Note that in fact, the actual training set is different now when evaluating, due to the random data augmentation. Unfortunately this is the only way to asses training performance in such scenario without loading and saving the augmented dataset or resorting to gradient descent to train the ELM thus giving up fast training.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the iterator was made initializable it should be re-initialized: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating network performance\n",
      "generator iteration: 0\n",
      "generator iteration: 100\n",
      "generator iteration: 200\n",
      "Accuracy: 0.8924633\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.8924633"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iterator_init_op()\n",
    "mlelm1.evaluate(tf_iterator = iterator,  batch_size=1024)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Activation and Weights and Bias initialization "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softsign(x):\n",
    "    y = x / (1+ tf.abs(x))\n",
    "    return y\n",
    "\n",
    "# this is a simple function which implements a softsign activation function "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlelm1.add_layer(1024, activation=softsign) # third layer with softsign is added to the network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This softsign function can be passed to the add_layer method, in the same way any TensorFlow pre-defined tf.nn.relu, tf.nn.elu  function etc can be passed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The add_layer method supports also custom Weights and Bias Initialization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example if we wish to initialize the ELM with an orthogonal weight matrix and the Bias as a unit norm vector: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "ortho_w = tf.orthogonal_initializer()\n",
    "uni_b= tf.uniform_unit_scaling_initializer()\n",
    "\n",
    "init_w = tf.get_variable(name='init_w',shape=[input_size, 1024], initializer=ortho_w)\n",
    "init_b = tf.get_variable(name='init_b', shape=[1024], initializer=uni_b)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlelm1.add_layer(1024, activation= softsign, w_init= init_w, b_init = init_b) # fourth layer is added to the network "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have used pre-made TensoFlow initialization functions but note that numpy or every other function can be used.\n",
    "\n",
    "** The important thing is that to w_init and b_init are passed TensorFlow variables with desired values. **\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with numpy \n",
    "import numpy as np \n",
    "\n",
    "init_w = tf.Variable(name='init_w', initial_value=np.random.uniform(low=-1, high=1, size=[input_size, 1024]))\n",
    "\n",
    "mlelm1.add_layer(1024, activation= softsign, w_init=init_w, b_init = None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Note that when using custom initialization both b_init and w_init should be specified, setting b_init to None creates a network without bias **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:machine-learning]",
   "language": "python",
   "name": "conda-env-machine-learning-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}