ELM_class_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ELM class example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** ELM class implements a single hidden layer ELM **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an example we will use the popular MNIST dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import tensorflow as tf\n",
    "import keras\n",
    "from keras.datasets import mnist;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, test = mnist.load_data(os.getcwd() + \"/elm_tf_test\" + \"mnist.txt\");\n",
    "x_train, y_train = train\n",
    "x_test, y_test = test\n",
    "del train, test\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes=10)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes=10)\n",
    "\n",
    "# the input has to be flattened in order to be feeded to the network\n",
    "\n",
    "x_train = x_train.reshape(-1, 28* 28)\n",
    "x_test = x_test.reshape(-1, 28 * 28)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_size = 28*28 \n",
    "output_size = 10 # mnist has 10 output classes "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating ELM classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tfelm.elm import ELM\n",
    "\n",
    "elm1 = ELM(input_size=input_size, output_size=output_size, l2norm=10e1, name='elm1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This creates an ELM network with 784 input neurons and 10 output neurons. \n",
    "The l2norm is a regularization parameter used in training.\n",
    "For now the hidden layer size hasn't been specified. \n",
    "The hidden layer is added to the network through the add_layer method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "elm1.add_layer(n_neurons=1024); \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This adds an hidden layer comprised of 1024 hidden layer neurons. \n",
    "\n",
    "By default the activation is set to tf.sigmoid and the initialization of weights and biases of the hidden layer is a modified He initialization: \n",
    "\n",
    "The weights are initialized by sampling from a random normal distribution with variance of 2/n_in, where n_in is the size of the previous layer, in this case the input layer. \n",
    "\n",
    "Actual network initialization is done via the compile method:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "elm1 has been initialized\n"
     ]
    }
   ],
   "source": [
    "elm1.compile()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the Network: fit method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train the network there are two main methods: train and fit. \n",
    "Fit is the most  basic and simple method but is suitable only for small datasets. \n",
    "\n",
    "It needs in input two numpy arrays for the training instances and labels and an optional batch_size argument. \n",
    "Internally a TensorFlow Iterator and a TensorFlow Dataset objects are created from the numpy arrays for the training as this it is the most efficient way to train a model according to TensorLow documentation. \n",
    "\n",
    "It should be noted that, unlike conventional Neural Networks, the batch_size doesn't change the outcome of the training but it only affect training time and memory required to train the network. The smaller the less the memory required but the more the training time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating Dataset and Iterator from tensors\n",
      "25/120 ETA:0:00\n",
      "50/120 ETA:0:00\n",
      "75/120 ETA:0:00\n",
      "100/120 ETA:0:00\n",
      "Training of ELM elm1 ended in 0:0:0.672470\n",
      "####################################################################################################\n",
      "Evaluating network performance\n",
      "Accuracy: 0.9210833\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.92108333"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "elm1.fit(x_train, y_train, batch_size=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the network has been trained, we can evaluate the performance on the test set via evaluate method. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating network performance\n",
      "Accuracy: 0.9181999\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.9181999"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "elm1.evaluate(x_test, y_test, batch_size=500) # it accepts batch size as also the evaluation is done by batching to allow bigger datasets to be evaluated as we will see.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To return a numpy array with actual predictions it exist a prediction method:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicting...\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "pred = elm1.predict(x_test, batch_size=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is pretty much the most basic functionalities offered by the API and are suitable for small/medium datasets as it is required the dataset is fitted into memory as an array. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training and evaluating bigger Datasets which cannot be fitted into memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the same MNIST dataset for example purpose only. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of calling the fit method we should call the train method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The train method requires a TensorFlow Iterator object. the TensorFlow Iterator object thus must be created esternally from the dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are various ways to create a TF iterator object from a dataset and this strongly depends on what is your input pipeline and in what format your dataset is.\n",
    "\n",
    "Tutorials, documentation and examples on Dataset and Iterator is available at: https://www.tensorflow.org/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** As an example suppose we have an input pipeline in which we want to do some pre-process and data augmentation on the original MNIST dataset: **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.preprocessing.image import ImageDataGenerator # we will use keras imagedatagen for simplicity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 2500\n",
    "n_epochs = 10 # as the dataset is augmented the training now will be done on more \"epochs\", the resulting dataset will be 10 times the original.\n",
    "              # It could be argued that calling these epochs is not strictly correct as each \"epoch\" is different from the previous: \n",
    "              # the dataset is augmented via random trsformations\n",
    "        \n",
    "# keras ImageDataGen requires a 4-D tensor in input:\n",
    "x_train = x_train.reshape(-1, 28, 28, 1)\n",
    "\n",
    "datagen = ImageDataGenerator(\n",
    "        \n",
    "    width_shift_range=0.05,\n",
    "    height_shift_range=0.05\n",
    ")\n",
    "\n",
    "# random height and weight shifting \n",
    "\n",
    "\n",
    "datagen.fit(x_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "batches_per_epochs = len(x_train) // batch_size\n",
    "\n",
    "def gen():\n",
    "    n_it = 0\n",
    "    \n",
    "    for x, y in datagen.flow(x_train, y_train, batch_size=batch_size):\n",
    "        x = x.reshape(batch_size, 28 * 28) # the network requires a flatten array in input we flatten here\n",
    "        if n_it % 100 == 0:\n",
    "            print(\"generator iteration: %d\" % n_it)\n",
    "        yield x, y\n",
    "        n_it += 1\n",
    "        if n_it >= batches_per_epochs * n_epochs:\n",
    "            break\n",
    "\n",
    "data = tf.data.Dataset.from_generator(generator=gen,\n",
    "                                      output_shapes=((batch_size, 28 * 28,), (batch_size, 10,)),\n",
    "                                      output_types=(tf.float32, tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we have defined a python generator from the keras ImageDataGenerator and we have used this generator to create a TensorFlow dataset. This because it isn't possible to create a Dataset directly from the keraas generator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generator iteration: 0\n",
      "25/240 ETA:0:47\n",
      "50/240 ETA:0:39\n",
      "75/240 ETA:0:34\n",
      "100/240 ETA:0:29\n",
      "generator iteration: 100\n",
      "125/240 ETA:0:23\n",
      "150/240 ETA:0:18\n",
      "175/240 ETA:0:13\n",
      "200/240 ETA:0:08\n",
      "generator iteration: 200\n",
      "225/240 ETA:0:03\n",
      "Training of ELM elm1 ended in 0:0:50.281698\n",
      "####################################################################################################\n"
     ]
    }
   ],
   "source": [
    "iterator = data.make_one_shot_iterator() # a TF iterator is created from the Dataset\n",
    "\n",
    "elm1.train(iterator, n_batches=batches_per_epochs*n_epochs)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The train method has the optinal n_batches argument which serves only the purpose of extimating the ETA. \n",
    "Note that the train method does not return the network performance. \n",
    "This should be done via evaluate. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating network performance\n",
      "Accuracy: 0.9127730\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.912773"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "elm1.evaluate(x_test, y_test,  batch_size=1024)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find the performance on the training set, due to the fact that ELM are not trained with gradient descent as conventional Neural Networks, one should call the evaluate function passing an iterator on the training set. \n",
    "\n",
    "Note that in fact, the actual training set is different now when evaluating, due to the random data augmentation. Unfortunately this is the only way to asses training performance in such scenario without loading and saving the augmented dataset or resorting to gradient descent to train the ELM thus giving up fast training.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the iterator was made as one shot only it should be re-created: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating network performance\n",
      "generator iteration: 0\n",
      "generator iteration: 100\n",
      "generator iteration: 200\n",
      "Accuracy: 0.8673667\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.8673667"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iterator = data.make_one_shot_iterator()\n",
    "elm1.evaluate(tf_iterator = iterator,  batch_size=1024)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Instead of creating two times the iterator a better way is to create an initializable iterator in first place, before training :**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "iterator = data.make_initializable_iterator()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The iterator should be initialized: \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.Session() as sess: \n",
    "    elm1.sess.run(iterator.initializer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have accessed the TF session attribute in ELM object, which has its own session and initialized the iterator inside the ELM session.\n",
    "Now taht the iterator has been initialized the iterator can be used for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generator iteration: 0\n",
      "25/240 ETA:0:47\n",
      "50/240 ETA:0:39\n",
      "75/240 ETA:0:34\n",
      "100/240 ETA:0:28\n",
      "generator iteration: 100\n",
      "125/240 ETA:0:23\n",
      "150/240 ETA:0:18\n",
      "175/240 ETA:0:13\n",
      "200/240 ETA:0:08\n",
      "generator iteration: 200\n",
      "225/240 ETA:0:03\n",
      "Training of ELM elm1 ended in 0:0:49.798366\n",
      "####################################################################################################\n"
     ]
    }
   ],
   "source": [
    "elm1.train(iterator, n_batches=batches_per_epochs*n_epochs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating network performance\n",
      "generator iteration: 0\n",
      "generator iteration: 100\n",
      "generator iteration: 200\n",
      "Accuracy: 0.8676217\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.86762166"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this re-initialize the iterator before calling the evaluate on the training set\n",
    "\n",
    "with tf.Session() as sess: \n",
    "    elm1.sess.run(iterator.initializer)\n",
    "\n",
    "\n",
    "elm1.evaluate(tf_iterator = iterator,  batch_size=1024)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how the training accuracy is slightly different due to the random trasformation due to data augmentation. \n",
    "\n",
    "This concludes this brief tutorial for ELM class training "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Activation and Weights and Bias initialization "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softsign(x):\n",
    "    y = x / (1+ tf.abs(x))\n",
    "    return y\n",
    "\n",
    "# this is a simple function which implements a softsign activation function "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "elm1.add_layer(1024, activation=softsign)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This softsign function can be passed to the add_layer method, in the same way any TensorFlow pre-defined tf.nn.relu, tf.nn.elu  function etc can be passed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The add_layer method supports also custom Weights and Bias Initialization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example if we wish to initialize the ELM with an orthogonal weight matrix and the Bias as a unit norm vector: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From <ipython-input-22-1d0c2e78e8fc>:2: UniformUnitScaling.__init__ (from tensorflow.python.ops.init_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.initializers.variance_scaling instead with distribution=uniform to get equivalent behavior.\n"
     ]
    }
   ],
   "source": [
    "ortho_w = tf.orthogonal_initializer()\n",
    "uni_b= tf.uniform_unit_scaling_initializer()\n",
    "\n",
    "init_w = tf.get_variable(name='init_w',shape=[input_size, 1024], initializer=ortho_w)\n",
    "init_b = tf.get_variable(name='init_b', shape=[1024], initializer=uni_b)\n",
    "\n",
    "elm1.add_layer(1024, activation= softsign, w_init= init_w, b_init = init_b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have used pre-made TensoFlow initialization functions but note that numpy or every other function can be used.\n",
    "\n",
    "** The important hing is that to w_init and b_init are passed TensorFlow variables with desired values. **\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with numpy \n",
    "import numpy as np \n",
    "\n",
    "init_w = tf.Variable(name='init_w', initial_value=np.random.uniform(low=-1, high=1, size=[input_size, 1024]))\n",
    "\n",
    "elm1.add_layer(1024, activation= softsign, w_init=init_w, b_init = None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Note that when using custom initialization both b_init and w_init should be specified, setting b_init to None creates a network without bias **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:machine-learning]",
   "language": "python",
   "name": "conda-env-machine-learning-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}