08_artifical_neural_networks_ex7ex8.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![book](https://raw.githubusercontent.com/ageron/tensorflow-safari-course/master/images/intro_to_tf_course.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Try not to peek at the solutions when you go through the exercises. ;-)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's make sure this notebook works well in both Python 2 and Python 3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import absolute_import, division, print_function, unicode_literals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Artificial Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Exercise](https://c1.staticflickr.com/9/8101/8553474140_c50cf08708_b.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visit the [TensorFlow Playground](http://playground.tensorflow.org).\n",
    "* Try training the default neural network by clicking the \"Run\" button (top left). Notice how it quickly finds a good solution for the classification task. Notice that the neurons in the first hidden layer have learned simple patterns, while the neurons in the second hidden layer have learned to combine the simple patterns of the first hidden layer into more complex patterns). In general, the more layers, the more complex the patterns can be.\n",
    "* Try replacing the Tanh activation function with the ReLU activation function, and train the network again. Notice that it finds a solution even faster, but this time the boundaries are linear. This is due to the shape of the ReLU function.\n",
    "* Modify the network architecture to have just one hidden layer with three neurons. Train it multiple times (to reset the network weights, just add and remove a neuron). Notice that the training time varies a lot, and sometimes it even gets stuck in a local minimum.\n",
    "* Now remove one neuron to keep just 2. Notice that the neural network is now incapable of finding a good solution, even if you try multiple times. The model has too few parameters and it systematically underfits the training set.\n",
    "* Next, set the number of neurons to 8 and train the network several times. Notice that it is now consistently fast and never gets stuck. This highlights an important finding in neural network theory: large neural networks almost never get stuck in local minima, and even when they do these local optima are almost as good as the global optimum. However, they can still get stuck on long plateaus for a long time.\n",
    "* Now change the dataset to be the spiral (bottom right dataset under \"DATA\"). Change the network architecture to have 4 hidden layers with 8 neurons each. Notice that training takes much longer, and often gets stuck on plateaus for long periods of time. Also notice that the neurons in the highest layers (i.e. on the right) tend to evolve faster than the neurons in the lowest layers (i.e. on the left). This problem, called the \"vanishing gradients\" problem, can be alleviated using better weight initialization and other techniques, better optimizers (such as AdaGrad or Adam), or using Batch Normalization.\n",
    "* Go ahead and play with the other parameters to get a feel of what they do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the MNIST dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"tmp/data/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 3\n",
    "X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "X_batch.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for image_data in X_batch:\n",
    "    plt.imshow(image_data.reshape([28, 28]), cmap=\"binary\", interpolation=\"nearest\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Exercise](https://c1.staticflickr.com/9/8101/8553474140_c50cf08708_b.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8.1) Take a close look at the following neural network model and make sure you understand every line. Next, add an extra hidden layer composed of 100 neurons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 100\n",
    "n_outputs = 10\n",
    "\n",
    "graph = tf.Graph()\n",
    "with graph.as_default():\n",
    "    with tf.name_scope(\"inputs\"):\n",
    "        X = tf.placeholder(tf.float32, shape=[None, n_inputs], name=\"X\")\n",
    "        y = tf.placeholder(tf.int32, shape=[None], name=\"y\")\n",
    "\n",
    "    with tf.name_scope(\"hidden1\"):\n",
    "        b1 = tf.Variable(tf.zeros([n_hidden1]), name=\"b1\")\n",
    "        W1 = tf.Variable(tf.random_uniform([n_inputs, n_hidden1], -1.0, 1.0), name=\"W1\")\n",
    "        hidden1 = tf.nn.relu(tf.matmul(X, W1) + b1)\n",
    "    \n",
    "    with tf.name_scope(\"output\"):\n",
    "        b2 = tf.Variable(tf.zeros([n_outputs]), name=\"b2\")\n",
    "        W2 = tf.Variable(tf.random_uniform([n_hidden1, n_outputs], -1.0, 1.0), name=\"W2\")\n",
    "        logits = tf.matmul(hidden1, W2) + b2\n",
    "        Y_proba = tf.nn.softmax(logits, name=\"Y_proba\")\n",
    "    \n",
    "    with tf.name_scope(\"train\"):\n",
    "        xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=y)\n",
    "        loss = tf.reduce_mean(xentropy)\n",
    "        optimizer = tf.train.AdamOptimizer()\n",
    "        training_op = optimizer.minimize(loss)\n",
    "\n",
    "    with tf.name_scope(\"eval\"):\n",
    "        correct = tf.nn.in_top_k(logits, y, 1)\n",
    "        accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))\n",
    "\n",
    "    with tf.name_scope(\"init_and_save\"):\n",
    "        init = tf.global_variables_initializer()\n",
    "        saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8.2) Write the training code, and train the model for about 20 epochs (i.e. enough training iterations to go through the training set 20 times). Evaluate it on the test set: you should get over 95% accuracy.\n",
    "\n",
    "Hint: you should open a session, initialize the variables, then write the main training loop. Inside it you should use `minst.train.next_batch(batch_size)` to get the next training batch (say with `batch_size=50`), then run the `training_op`, feeding it the training batch (don't forget to feed both `X` and `y`). Every few hundred iterations, evaluate the model's accuracy on the validation set (`mnist.validation.images` and `mnist.validation.labels`), and print the result. At the end of training, save the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8.3) Bonus question: load the model you just trained and saved, and use it to make predictions on the first 200 images of the test set (`mnist.test`). Display the images that the model got wrong, and show the class probabilities that it guessed. Notice that some of the images it gets wrong are pretty poorly written, but some are obvious to us humans. We will see that Convolutional Neural Networks can do a much better job and reach human performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try not to peek at the solution below before you have done the exercise! :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![thinking](https://upload.wikimedia.org/wikipedia/commons/0/06/Filos_segundo_logo_%28flipped%29.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 8 - Solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 100\n",
    "n_hidden2 = 100\n",
    "n_outputs = 10\n",
    "\n",
    "graph = tf.Graph()\n",
    "with graph.as_default():\n",
    "    with tf.name_scope(\"inputs\"):\n",
    "        X = tf.placeholder(tf.float32, shape=[None, n_inputs], name=\"X\")\n",
    "        y = tf.placeholder(tf.int32, shape=[None], name=\"y\")\n",
    "\n",
    "    with tf.name_scope(\"hidden1\"):\n",
    "        b1 = tf.Variable(tf.zeros([n_hidden1]), name=\"b1\")\n",
    "        W1 = tf.Variable(tf.random_uniform([n_inputs, n_hidden1], -1.0, 1.0), name=\"W1\")\n",
    "        hidden1 = tf.nn.relu(tf.matmul(X, W1) + b1)\n",
    "\n",
    "    with tf.name_scope(\"hidden2\"):\n",
    "        b2 = tf.Variable(tf.zeros([n_hidden2]), name=\"b2\")\n",
    "        W2 = tf.Variable(tf.random_uniform([n_hidden1, n_hidden2], -1.0, 1.0), name=\"W2\")\n",
    "        hidden2 = tf.nn.relu(tf.matmul(hidden1, W2) + b2)\n",
    "\n",
    "    with tf.name_scope(\"output\"):\n",
    "        b3 = tf.Variable(tf.zeros([n_outputs]), name=\"b3\")\n",
    "        W3 = tf.Variable(tf.random_uniform([n_hidden2, n_outputs], -1.0, 1.0), name=\"W3\")\n",
    "        logits = tf.matmul(hidden2, W3) + b3\n",
    "        Y_proba = tf.nn.softmax(logits, name=\"Y_proba\")\n",
    "    \n",
    "    with tf.name_scope(\"train\"):\n",
    "        xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=y)\n",
    "        loss = tf.reduce_mean(xentropy)\n",
    "        optimizer = tf.train.AdamOptimizer()\n",
    "        training_op = optimizer.minimize(loss)\n",
    "\n",
    "    with tf.name_scope(\"eval\"):\n",
    "        correct = tf.nn.in_top_k(logits, y, 1)\n",
    "        accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))\n",
    "\n",
    "    with tf.name_scope(\"init_and_save\"):\n",
    "        init = tf.global_variables_initializer()\n",
    "        saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_epochs = 20\n",
    "batch_size = 50\n",
    "\n",
    "with tf.Session(graph=graph) as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        for iteration in range(mnist.train.num_examples // batch_size):\n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})\n",
    "        acc_train = accuracy.eval(feed_dict={X: X_batch, y: y_batch})\n",
    "        acc_val = accuracy.eval(feed_dict={X: mnist.validation.images, y: mnist.validation.labels})\n",
    "        print(epoch, \"Train accuracy:\", acc_train, \"Validation accuracy:\", acc_val)\n",
    "\n",
    "    save_path = saver.save(sess, \"./my_mnist_model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "graph = tf.Graph()\n",
    "with tf.Session(graph=graph) as sess:\n",
    "    saver = tf.train.import_meta_graph(\"./my_mnist_model.meta\")\n",
    "    saver.restore(sess, \"./my_mnist_model\")\n",
    "    X = graph.get_tensor_by_name(\"inputs/X:0\")\n",
    "    Y_proba = graph.get_tensor_by_name(\"output/Y_proba:0\")\n",
    "    Y_proba_val = Y_proba.eval(feed_dict={X: mnist.test.images})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for example_index in range(200):\n",
    "    y_proba = Y_proba_val[example_index]\n",
    "    y_pred = np.argmax(y_proba)\n",
    "    y_label = mnist.test.labels[example_index]\n",
    "    if y_pred != y_label:\n",
    "        print(\"Actual class:{}, Predicted class: {}, Main probabilities: {}\".format(\n",
    "                  y_label,\n",
    "                  y_pred,\n",
    "                  \", \".join([\"{}:{:.1f}%\".format(n, 100*p)\n",
    "                             for n, p in enumerate(y_proba) if p > 0.01])))\n",
    "        plt.imshow(mnist.test.images[example_index].reshape([28, 28]), cmap=\"binary\", interpolation=\"nearest\")\n",
    "        plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}