16_Introduction_to_Recurrent_Neural_Nets_20170425.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Driven Modeling\n",
    "### (Theme of this semester: CODING AS LITERACY)\n",
    "<hr style=\"height:3px;border:none;color:stillblue;background-color:black;\" />\n",
    "### PhD seminar series at Chair for Computer Aided Architectural Design (CAAD), ETH Zurich\n",
    "[Vahid Moosavi](https://vahidmoosavi.com/)\n",
    "<hr style=\"height:1px;border:none;color:stillblue;background-color:black;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a></a>\n",
    "\n",
    "# 16th Session \n",
    "<hr style=\"height:1px;border:none;color:stillblue;background-color:steelblue;\" />\n",
    "<span style=\"color:steelblue;font-size: 1em;\">25 April 2017</span>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Recurrent Neural Nets\n",
    "### To be discussed\n",
    "* ** Key Concepts**\n",
    "* **Simple Experiments**\n",
    "* **Interesting applications**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "import datetime\n",
    "import pandas as pd\n",
    "# import pandas.io.data\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "from matplotlib import pyplot as plt\n",
    "import sys\n",
    "import sompylib.sompy as SOM# from pandas import Series, DataFrame\n",
    "\n",
    "from ipywidgets import interact, HTML, FloatSlider\n",
    "\n",
    "%matplotlib inline\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Feed Forward Networks\n",
    "- **Assumption of independent data** \n",
    "\n",
    "![](Images/feedforward_rumelhart.png)\n",
    "\n",
    "\n",
    "<hr style=\"height:3px;border:none;color:stillblue;background-color:black;\" />\n",
    "\n",
    "# Recurrent Neural Networks\n",
    "- **When our decision depends on our current state**\n",
    "- ** Similar to (Hidden) Markov Chains, but not exactly the smae attitude: Here we don't build a global state model**\n",
    "\n",
    "![](Images/srn_elman.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<hr style=\"height:3px;border:none;color:stillblue;background-color:black;\" />\n",
    "# Unrolled RNN\n",
    "- **To linearize in time: Multiple copies of the same cell**\n",
    "\n",
    "![](Images/RNN-unrolled.png)\n",
    "source: http://colah.github.io/posts/2015-08-Understa/nding-LSTMs/\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:3px;border:none;color:stillblue;background-color:black;\" />\n",
    "\n",
    "## Basic self referential element of RNN cell\n",
    "# $$ h_t = \\tanh ( W_{hh} h_{t-1} + W_{xh} x_t )$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# http://karpathy.github.io/2015/05/21/rnn-effectiveness/\n",
    "class RNN:\n",
    "    def setup(self,hidden_size,input_size):\n",
    "        Wxh = np.random.randn(hidden_size, input_size)*0.01 # input to hidden\n",
    "        Whh = np.random.randn(hidden_size, hidden_size)*0.01 # hidden to hidden\n",
    "        Why = np.random.randn(input_size, hidden_size)*0.01 # hidden to output\n",
    "        self.W_hh = Whh\n",
    "        self.h = np.zeros((hidden_size))\n",
    "        self.W_xh = Wxh\n",
    "        self.W_hy = Why\n",
    "    def step(self, x):\n",
    "        # update the hidden state\n",
    "        self.h = np.tanh(np.dot(self.h,self.W_hh) + np.dot(self.W_xh, x)).T\n",
    "#         print self.h.shape\n",
    "        # compute the output vector\n",
    "        y = np.dot(self.W_hy, self.h)\n",
    "#         print self.W_hy.shape,self.h.shape , y.shape\n",
    "        return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "rnn1= RNN()\n",
    "rnn2 = RNN()\n",
    "hidden_size = 10\n",
    "input_size = 4\n",
    "rnn1.setup(hidden_size,input_size)\n",
    "rnn2.setup(hidden_size,input_size)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 0 0 1] [ -3.83812807e-07   1.26533933e-07  -3.69391726e-08   5.78367948e-07]\n",
      "[ -3.83812807e-07   1.26533933e-07  -3.69391726e-08   5.78367948e-07] [ -2.89510828e-08   9.30318564e-09   1.33466938e-08   2.30315094e-09]\n"
     ]
    }
   ],
   "source": [
    "# In every step hidden states get updated\n",
    "\n",
    "#two time steps\n",
    "\n",
    "x = np.asarray([1,0,0,1])\n",
    "\n",
    "y1 = rnn1.step(x)\n",
    "y2 = rnn2.step(y1)\n",
    "print x, y2\n",
    "\n",
    "x = y2\n",
    "y1 = rnn1.step(x)\n",
    "y2 = rnn2.step(y1)\n",
    "print x, y2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Then, we need to update the matrices via gradient descent\n",
    "### Simple examples at the end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<hr style=\"height:3px;border:none;color:stillblue;background-color:black;\" />\n",
    "\n",
    "# However, for a long time nobody (few) believed in RNNs\n",
    "- ** Memory problem**\n",
    "- **Vanishing and Exploding Gradients**\n",
    "\n",
    "\n",
    "\n",
    "# Long Short-Term Memory (LSTM)\n",
    "- **Learn what and where to remember and forget!**\n",
    "- **[A nice lecture on LSTM](https://www.youtube.com/watch?v=56TYLaQN4N8)**\n",
    "- **[A nice blog post](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)**\n",
    "![](Images/lstm_memorycell.png)\n",
    "image from: http://deeplearning.net/tutorial/lstm.html\n",
    "<hr style=\"height:3px;border:none;color:stillblue;background-color:black;\" />\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Deep RNN\n",
    "- ** Ecah cell unfolds horizontally**\n",
    "- ** We stack cells vertically**\n",
    "\n",
    "![](Images/deep_RNN.png)\n",
    "\n",
    "## Different Architectures\n",
    "![](Images/RNN_Archs.jpeg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<hr style=\"height:3px;border:none;color:stillblue;background-color:steelblue;\" />\n",
    "\n",
    "# Important concepts: In princple RNNs can learn any \"program\" via data\n",
    "\n",
    "# Some examples using TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## number of 1s in a binary number\n",
    "- **Architecture**: We are just interested in the last output (Many to one)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Source code with the blog post at http://monik.in/a-noobs-guide-to-implementing-rnn-lstm-using-tensorflow/\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "import numpy as np\n",
    "import random\n",
    "from random import shuffle\n",
    "import tensorflow as tf\n",
    "tf.reset_default_graph()\n",
    "pow2= 16\n",
    "\n",
    "train_input = ['{0:016b}'.format(i) for i in range(2**pow2)]\n",
    "shuffle(train_input)\n",
    "train_input = [map(int,i) for i in train_input]\n",
    "ti  = []\n",
    "for i in train_input:\n",
    "    temp_list = []\n",
    "    for j in i:\n",
    "            temp_list.append([j])\n",
    "    ti.append(np.array(temp_list))\n",
    "train_input = ti\n",
    "\n",
    "train_output = []\n",
    "for i in train_input:\n",
    "    count = 0\n",
    "    for j in i:\n",
    "        if j[0] == 1:\n",
    "            count+=1\n",
    "    temp_list = ([0]*(pow2+1))\n",
    "    temp_list[count]=1\n",
    "    train_output.append(temp_list)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1]\n",
      " [1]\n",
      " [1]\n",
      " [0]\n",
      " [1]\n",
      " [1]\n",
      " [0]\n",
      " [0]\n",
      " [1]\n",
      " [0]\n",
      " [1]\n",
      " [0]\n",
      " [1]\n",
      " [0]\n",
      " [1]\n",
      " [1]]\n",
      "[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0]\n"
     ]
    }
   ],
   "source": [
    "print train_input[0]\n",
    "print train_output[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test and training data loaded\n",
      "Epoch 0 error: 77.4999976158\n",
      "Epoch 100 error: 0.0\n",
      "Epoch 200 error 0.1%\n"
     ]
    }
   ],
   "source": [
    "NUM_EXAMPLES = 10000\n",
    "test_input = train_input[NUM_EXAMPLES:]\n",
    "test_output = train_output[NUM_EXAMPLES:]\n",
    "train_input = train_input[:NUM_EXAMPLES]\n",
    "train_output = train_output[:NUM_EXAMPLES]\n",
    "\n",
    "tf.reset_default_graph()\n",
    "print \"test and training data loaded\"\n",
    "\n",
    "\n",
    "data = tf.placeholder(tf.float32, [None, pow2,1]) #Number of examples, number of input, dimension of each input\n",
    "target = tf.placeholder(tf.float32, [None, (pow2+1)])\n",
    "num_hidden = 24\n",
    "num_layers=2\n",
    "cell = tf.nn.rnn_cell.LSTMCell(num_hidden,state_is_tuple=True)\n",
    "cell = tf.nn.rnn_cell.MultiRNNCell([cell] * num_layers, state_is_tuple=True)\n",
    "val, _ = tf.nn.dynamic_rnn(cell, data, dtype=tf.float32)\n",
    "val = tf.transpose(val, [1, 0, 2])\n",
    "last = tf.gather(val, int(val.get_shape()[0]) - 1)\n",
    "\n",
    "weight = tf.Variable(tf.truncated_normal([num_hidden, int(target.get_shape()[1])]))\n",
    "bias = tf.Variable(tf.constant(0.1, shape=[target.get_shape()[1]]))\n",
    "\n",
    "prediction = tf.nn.softmax(tf.matmul(last, weight) + bias)\n",
    "\n",
    "cross_entropy = -tf.reduce_sum(target * tf.log(tf.clip_by_value(prediction,1e-10,1.0)))\n",
    "optimizer = tf.train.AdamOptimizer()\n",
    "minimize = optimizer.minimize(cross_entropy)\n",
    "\n",
    "mistakes = tf.not_equal(tf.argmax(target, 1), tf.argmax(prediction, 1))\n",
    "error = tf.reduce_mean(tf.cast(mistakes, tf.float32))\n",
    "\n",
    "# init_op = tf.initialize_all_variables()\n",
    "init_op = tf.global_variables_initializer()\n",
    "sess = tf.Session()\n",
    "sess.run(init_op)\n",
    "\n",
    "batch_size = 200\n",
    "no_of_batches = int(len(train_input)) / batch_size\n",
    "epoch = 200\n",
    "for i in range(epoch):\n",
    "    ptr = 0\n",
    "    for j in range(no_of_batches):\n",
    "        inp, out = train_input[ptr:ptr+batch_size], train_output[ptr:ptr+batch_size]\n",
    "        ptr+=batch_size\n",
    "        sess.run(minimize,{data: inp, target: out})\n",
    "    if i%100 ==0:\n",
    "        incorrect = sess.run(error,{data: inp, target: out})\n",
    "        print \"Epoch {} error: {}\".format(i,incorrect*100)\n",
    "incorrect = sess.run(error,{data: test_input, target: test_output})\n",
    "\n",
    "\n",
    "\n",
    "print('Epoch {:2d} error {:3.1f}%'.format(i + 1, 100 * incorrect))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 10\n"
     ]
    }
   ],
   "source": [
    "tt = np.random.randint(0,2,size=pow2)[np.newaxis,:,np.newaxis] \n",
    "p = sess.run(tf.argmax(prediction, 1),{data:tt})\n",
    "print np.sum(tt),p[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sess.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summing Two numbers\n",
    "- **Architecture**: We are interested in all the outputs in time (many to many)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0, train error: 0.67, valid accuracy: 61.1 %\n",
      "Epoch 10, train error: 0.00, valid accuracy: 100.0 %\n",
      "Epoch 20, train error: 0.00, valid accuracy: 100.0 %\n",
      "Epoch 30, train error: 0.00, valid accuracy: 100.0 %\n",
      "Epoch 40, train error: 0.00, valid accuracy: 100.0 %\n",
      "Epoch 50, train error: 0.00, valid accuracy: 100.0 %\n",
      "Epoch 60, train error: 0.00, valid accuracy: 100.0 %\n",
      "Epoch 70, train error: 0.00, valid accuracy: 100.0 %\n",
      "Epoch 80, train error: 0.00, valid accuracy: 100.0 %\n",
      "Epoch 90, train error: -0.00, valid accuracy: 100.0 %\n",
      "Epoch 100, train error: -0.00, valid accuracy: 100.0 %\n",
      "Epoch 110, train error: -0.00, valid accuracy: 100.0 %\n",
      "Epoch 120, train error: -0.00, valid accuracy: 100.0 %\n",
      "Epoch 130, train error: -0.00, valid accuracy: 100.0 %\n",
      "Epoch 140, train error: -0.00, valid accuracy: 100.0 %\n",
      "Epoch 150, train error: -0.00, valid accuracy: 100.0 %\n",
      "Epoch 160, train error: -0.00, valid accuracy: 100.0 %\n",
      "Epoch 170, train error: -0.00, valid accuracy: 100.0 %\n",
      "Epoch 180, train error: -0.00, valid accuracy: 100.0 %\n",
      "Epoch 190, train error: -0.00, valid accuracy: 100.0 %\n"
     ]
    }
   ],
   "source": [
    "# https://gist.github.com/nivwusquorum/b18ce332bde37e156034e5d3f60f8a23\n",
    "\n",
    "import numpy as np\n",
    "import random\n",
    "import tensorflow as tf\n",
    "import tensorflow.contrib.layers as layers\n",
    "tf.reset_default_graph()\n",
    "# map_fn = tf.python.functional_ops.map_fn\n",
    "map_fn = tf.map_fn\n",
    "\n",
    "################################################################################\n",
    "##                           DATASET GENERATION                               ##\n",
    "##                                                                            ##\n",
    "##  The problem we are trying to solve is adding two binary numbers. The      ##\n",
    "##  numbers are reversed, so that the state of RNN can add the numbers        ##\n",
    "##  perfectly provided it can learn to store carry in the state. Timestep t   ##\n",
    "##  corresponds to bit len(number) - t.                                       ##\n",
    "################################################################################\n",
    "\n",
    "def as_bytes(num, final_size):\n",
    "    res = []\n",
    "    for _ in range(final_size):\n",
    "        res.append(num % 2)\n",
    "        num //= 2\n",
    "    return res\n",
    "\n",
    "def generate_example(num_bits):\n",
    "    a = random.randint(0, 2**(num_bits - 1) - 1)\n",
    "    b = random.randint(0, 2**(num_bits - 1) - 1)\n",
    "    res = a + b\n",
    "    return (as_bytes(a,  num_bits),\n",
    "            as_bytes(b,  num_bits),\n",
    "            as_bytes(res,num_bits))\n",
    "\n",
    "def generate_batch(num_bits, batch_size):\n",
    "    \"\"\"Generates instance of a problem.\n",
    "    Returns\n",
    "    -------\n",
    "    x: np.array\n",
    "        two numbers to be added represented by bits.\n",
    "        shape: b, i, n\n",
    "        where:\n",
    "            b is bit index from the end\n",
    "            i is example idx in batch\n",
    "            n is one of [0,1] depending for first and\n",
    "                second summand respectively\n",
    "    y: np.array\n",
    "        the result of the addition\n",
    "        shape: b, i, n\n",
    "        where:\n",
    "            b is bit index from the end\n",
    "            i is example idx in batch\n",
    "            n is always 0\n",
    "    \"\"\"\n",
    "    x = np.empty((num_bits, batch_size, 2))\n",
    "    y = np.empty((num_bits, batch_size, 1))\n",
    "\n",
    "    for i in range(batch_size):\n",
    "        a, b, r = generate_example(num_bits)\n",
    "        x[:, i, 0] = a\n",
    "        x[:, i, 1] = b\n",
    "        y[:, i, 0] = r\n",
    "    return x, y\n",
    "\n",
    "\n",
    "################################################################################\n",
    "##                           GRAPH DEFINITION                                 ##\n",
    "################################################################################\n",
    "\n",
    "INPUT_SIZE    = 2       # 2 bits per timestep\n",
    "RNN_HIDDEN    = 20\n",
    "OUTPUT_SIZE   = 1       # 1 bit per timestep\n",
    "TINY          = 1e-6    # to avoid NaNs in logs\n",
    "LEARNING_RATE = 0.01\n",
    "\n",
    "USE_LSTM = True\n",
    "\n",
    "inputs  = tf.placeholder(tf.float32, (None, None, INPUT_SIZE))  # (time, batch, in)\n",
    "outputs = tf.placeholder(tf.float32, (None, None, OUTPUT_SIZE)) # (time, batch, out)\n",
    "\n",
    "\n",
    "if USE_LSTM:\n",
    "    num_layers=2\n",
    "    cell = tf.nn.rnn_cell.BasicLSTMCell(RNN_HIDDEN, state_is_tuple=True)\n",
    "    cell = tf.nn.rnn_cell.MultiRNNCell([cell] * num_layers, state_is_tuple=True)\n",
    "else:\n",
    "    cell = tf.nn.rnn_cell.BasicRNNCell(RNN_HIDDEN)\n",
    "\n",
    "# Create initial state. Here it is just a constant tensor filled with zeros,\n",
    "# but in principle it could be a learnable parameter. This is a bit tricky\n",
    "# to do for LSTM's tuple state, but can be achieved by creating two vector\n",
    "# Variables, which are then tiled along batch dimension and grouped into tuple.\n",
    "batch_size    = tf.shape(inputs)[1]\n",
    "initial_state = cell.zero_state(batch_size, tf.float32)\n",
    "\n",
    "# Given inputs (time, batch, input_size) outputs a tuple\n",
    "#  - outputs: (time, batch, output_size)  [do not mistake with OUTPUT_SIZE]\n",
    "#  - states:  (time, batch, hidden_size)\n",
    "rnn_outputs, rnn_states = tf.nn.dynamic_rnn(cell, inputs, initial_state=initial_state, time_major=True)\n",
    "\n",
    "\n",
    "# project output from rnn output size to OUTPUT_SIZE. Sometimes it is worth adding\n",
    "# an extra layer here.\n",
    "final_projection = lambda x: layers.linear(x, num_outputs=OUTPUT_SIZE, activation_fn=tf.nn.sigmoid)\n",
    "\n",
    "# apply projection to every timestep.\n",
    "predicted_outputs = map_fn(final_projection, rnn_outputs)\n",
    "\n",
    "# compute elementwise cross entropy.\n",
    "error = -(outputs * tf.log(predicted_outputs + TINY) + (1.0 - outputs) * tf.log(1.0 - predicted_outputs + TINY))\n",
    "error = tf.reduce_mean(error)\n",
    "\n",
    "# optimize\n",
    "train_fn = tf.train.AdamOptimizer(learning_rate=LEARNING_RATE).minimize(error)\n",
    "\n",
    "# assuming that absolute difference between output and correct answer is 0.5\n",
    "# or less we can round it to the correct output.\n",
    "accuracy = tf.reduce_mean(tf.cast(tf.abs(outputs - predicted_outputs) < 0.5, tf.float32))\n",
    "\n",
    "\n",
    "################################################################################\n",
    "##                           TRAINING LOOP                                    ##\n",
    "################################################################################\n",
    "\n",
    "NUM_BITS = 10\n",
    "ITERATIONS_PER_EPOCH = 100\n",
    "BATCH_SIZE = 16\n",
    "\n",
    "valid_x, valid_y = generate_batch(num_bits=NUM_BITS, batch_size=100)\n",
    "\n",
    "session = tf.Session()\n",
    "# For some reason it is our job to do this:\n",
    "session.run(tf.global_variables_initializer())\n",
    "\n",
    "for epoch in range(200):\n",
    "    epoch_error = 0\n",
    "    for _ in range(ITERATIONS_PER_EPOCH):\n",
    "        # here train_fn is what triggers backprop. error and accuracy on their\n",
    "        # own do not trigger the backprop.\n",
    "        x, y = generate_batch(num_bits=NUM_BITS, batch_size=BATCH_SIZE)\n",
    "        epoch_error += session.run([error, train_fn], {\n",
    "            inputs: x,\n",
    "            outputs: y,\n",
    "        })[0]\n",
    "    epoch_error /= ITERATIONS_PER_EPOCH\n",
    "    valid_accuracy = session.run(accuracy, {\n",
    "        inputs:  valid_x,\n",
    "        outputs: valid_y,\n",
    "    })\n",
    "    if epoch%10==0:\n",
    "        print (\"Epoch %d, train error: %.2f, valid accuracy: %.1f %%\" % (epoch, epoch_error, valid_accuracy * 100.0))\n",
    "# "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "preds_valid = session.run(predicted_outputs, {\n",
    "        inputs:  valid_x,\n",
    "        outputs: valid_y,\n",
    "    })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1.  1.  0.  0.  1.  1.  1.  1.  1.  0.]\n",
      "[ 1.  1.  0.  0.  1.  1.  1.  1.  1.  0.]\n"
     ]
    }
   ],
   "source": [
    "i = 20\n",
    "print (np.around(preds_valid)[:,i,0])\n",
    "print (valid_y[:,i,0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "session.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MNIST RNN (Image Classification)\n",
    "\n",
    "- **Architecture**: \n",
    "    - We have sequences of rows (columns) of images. (many to one)\n",
    "    - We are interested to predict a class only at the end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "# From: https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/recurrent_network.ipynb\n",
    "\n",
    "'''\n",
    "A Recurrent Neural Network (LSTM) implementation example using TensorFlow library.\n",
    "This example is using the MNIST database of handwritten digits (http://yann.lecun.com/exdb/mnist/)\n",
    "Long Short Term Memory paper: http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf\n",
    "\n",
    "Author: Aymeric Damien\n",
    "Project: https://github.com/aymericdamien/TensorFlow-Examples/\n",
    "'''\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "import tensorflow as tf\n",
    "from tensorflow.python.ops import rnn, rnn_cell\n",
    "\n",
    "# Import MNIST data\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"/tmp/data/\", one_hot=True)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "''' To classify images using a reccurent neural network, we consider every image row as a sequence of pixels. Because MNIST image shape is 28*28px, we will then handle 28 sequences of 28 steps for every sample. '''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 1280, Minibatch Loss= 1.355216, Training Accuracy= 0.54688\n",
      "Iter 2560, Minibatch Loss= 1.186185, Training Accuracy= 0.64062\n",
      "Iter 3840, Minibatch Loss= 0.959371, Training Accuracy= 0.69531\n",
      "Iter 5120, Minibatch Loss= 0.756388, Training Accuracy= 0.76562\n",
      "Iter 6400, Minibatch Loss= 0.597719, Training Accuracy= 0.78125\n",
      "Iter 7680, Minibatch Loss= 0.966340, Training Accuracy= 0.64062\n",
      "Iter 8960, Minibatch Loss= 0.563785, Training Accuracy= 0.83594\n",
      "Iter 10240, Minibatch Loss= 0.472388, Training Accuracy= 0.82812\n",
      "Iter 11520, Minibatch Loss= 0.273660, Training Accuracy= 0.94531\n",
      "Iter 12800, Minibatch Loss= 0.521885, Training Accuracy= 0.82031\n",
      "Iter 14080, Minibatch Loss= 0.402212, Training Accuracy= 0.87500\n",
      "Iter 15360, Minibatch Loss= 0.248117, Training Accuracy= 0.94531\n",
      "Iter 16640, Minibatch Loss= 0.305744, Training Accuracy= 0.93750\n",
      "Iter 17920, Minibatch Loss= 0.227243, Training Accuracy= 0.92188\n",
      "Iter 19200, Minibatch Loss= 0.252387, Training Accuracy= 0.93750\n",
      "Iter 20480, Minibatch Loss= 0.116889, Training Accuracy= 0.98438\n",
      "Iter 21760, Minibatch Loss= 0.330990, Training Accuracy= 0.89062\n",
      "Iter 23040, Minibatch Loss= 0.106201, Training Accuracy= 0.96094\n",
      "Iter 24320, Minibatch Loss= 0.324091, Training Accuracy= 0.89844\n",
      "Iter 25600, Minibatch Loss= 0.340873, Training Accuracy= 0.89062\n",
      "Iter 26880, Minibatch Loss= 0.170801, Training Accuracy= 0.95312\n",
      "Iter 28160, Minibatch Loss= 0.202922, Training Accuracy= 0.95312\n",
      "Iter 29440, Minibatch Loss= 0.261573, Training Accuracy= 0.92188\n",
      "Iter 30720, Minibatch Loss= 0.219512, Training Accuracy= 0.92969\n",
      "Iter 32000, Minibatch Loss= 0.152274, Training Accuracy= 0.93750\n",
      "Iter 33280, Minibatch Loss= 0.172803, Training Accuracy= 0.93750\n",
      "Iter 34560, Minibatch Loss= 0.178191, Training Accuracy= 0.96094\n",
      "Iter 35840, Minibatch Loss= 0.190618, Training Accuracy= 0.96094\n",
      "Iter 37120, Minibatch Loss= 0.230897, Training Accuracy= 0.91406\n",
      "Iter 38400, Minibatch Loss= 0.111684, Training Accuracy= 0.96875\n",
      "Iter 39680, Minibatch Loss= 0.116336, Training Accuracy= 0.96875\n",
      "Iter 40960, Minibatch Loss= 0.208338, Training Accuracy= 0.92188\n",
      "Iter 42240, Minibatch Loss= 0.099843, Training Accuracy= 0.96094\n",
      "Iter 43520, Minibatch Loss= 0.175103, Training Accuracy= 0.93750\n",
      "Iter 44800, Minibatch Loss= 0.188875, Training Accuracy= 0.93750\n",
      "Iter 46080, Minibatch Loss= 0.076799, Training Accuracy= 0.96094\n",
      "Iter 47360, Minibatch Loss= 0.203019, Training Accuracy= 0.94531\n",
      "Iter 48640, Minibatch Loss= 0.190256, Training Accuracy= 0.92188\n",
      "Iter 49920, Minibatch Loss= 0.138491, Training Accuracy= 0.95312\n",
      "Iter 51200, Minibatch Loss= 0.084339, Training Accuracy= 0.96875\n",
      "Iter 52480, Minibatch Loss= 0.144099, Training Accuracy= 0.96094\n",
      "Iter 53760, Minibatch Loss= 0.046987, Training Accuracy= 0.97656\n",
      "Iter 55040, Minibatch Loss= 0.242961, Training Accuracy= 0.91406\n",
      "Iter 56320, Minibatch Loss= 0.127308, Training Accuracy= 0.95312\n",
      "Iter 57600, Minibatch Loss= 0.075414, Training Accuracy= 0.96875\n",
      "Iter 58880, Minibatch Loss= 0.185896, Training Accuracy= 0.94531\n",
      "Iter 60160, Minibatch Loss= 0.029974, Training Accuracy= 1.00000\n",
      "Iter 61440, Minibatch Loss= 0.117170, Training Accuracy= 0.95312\n",
      "Iter 62720, Minibatch Loss= 0.118068, Training Accuracy= 0.95312\n",
      "Iter 64000, Minibatch Loss= 0.145590, Training Accuracy= 0.94531\n",
      "Iter 65280, Minibatch Loss= 0.131703, Training Accuracy= 0.96094\n",
      "Iter 66560, Minibatch Loss= 0.101745, Training Accuracy= 0.98438\n",
      "Iter 67840, Minibatch Loss= 0.137928, Training Accuracy= 0.95312\n",
      "Iter 69120, Minibatch Loss= 0.102363, Training Accuracy= 0.96875\n",
      "Iter 70400, Minibatch Loss= 0.035047, Training Accuracy= 1.00000\n",
      "Iter 71680, Minibatch Loss= 0.211946, Training Accuracy= 0.95312\n",
      "Iter 72960, Minibatch Loss= 0.115560, Training Accuracy= 0.95312\n",
      "Iter 74240, Minibatch Loss= 0.072575, Training Accuracy= 0.96875\n",
      "Iter 75520, Minibatch Loss= 0.145280, Training Accuracy= 0.96094\n",
      "Iter 76800, Minibatch Loss= 0.154121, Training Accuracy= 0.96094\n",
      "Iter 78080, Minibatch Loss= 0.167381, Training Accuracy= 0.96094\n",
      "Iter 79360, Minibatch Loss= 0.094849, Training Accuracy= 0.95312\n",
      "Iter 80640, Minibatch Loss= 0.021832, Training Accuracy= 1.00000\n",
      "Iter 81920, Minibatch Loss= 0.072856, Training Accuracy= 0.97656\n",
      "Iter 83200, Minibatch Loss= 0.039065, Training Accuracy= 0.99219\n",
      "Iter 84480, Minibatch Loss= 0.072395, Training Accuracy= 0.96875\n",
      "Iter 85760, Minibatch Loss= 0.090926, Training Accuracy= 0.96094\n",
      "Iter 87040, Minibatch Loss= 0.026965, Training Accuracy= 1.00000\n",
      "Iter 88320, Minibatch Loss= 0.051769, Training Accuracy= 0.96875\n",
      "Iter 89600, Minibatch Loss= 0.086891, Training Accuracy= 0.96875\n",
      "Iter 90880, Minibatch Loss= 0.081950, Training Accuracy= 0.96094\n",
      "Iter 92160, Minibatch Loss= 0.084406, Training Accuracy= 0.96094\n",
      "Iter 93440, Minibatch Loss= 0.017629, Training Accuracy= 1.00000\n",
      "Iter 94720, Minibatch Loss= 0.102386, Training Accuracy= 0.96875\n",
      "Iter 96000, Minibatch Loss= 0.037047, Training Accuracy= 0.99219\n",
      "Iter 97280, Minibatch Loss= 0.046284, Training Accuracy= 0.99219\n",
      "Iter 98560, Minibatch Loss= 0.167275, Training Accuracy= 0.95312\n",
      "Iter 99840, Minibatch Loss= 0.086690, Training Accuracy= 0.97656\n",
      "Optimization Finished!\n",
      "('Testing Accuracy:', 0.984375)\n"
     ]
    }
   ],
   "source": [
    "# Parameters\n",
    "learning_rate = 0.001\n",
    "training_iters = 100000\n",
    "batch_size = 128\n",
    "display_step = 10\n",
    "\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# Network Parameters\n",
    "n_input = 28 # MNIST data input (img shape: 28*28)\n",
    "n_steps = 28 # timesteps\n",
    "n_hidden = 128 # hidden layer num of features\n",
    "n_classes = 10 # MNIST total classes (0-9 digits)\n",
    "\n",
    "\n",
    "\n",
    "# tf Graph input\n",
    "x = tf.placeholder(tf.float32, [None, n_steps, n_input])\n",
    "y = tf.placeholder(tf.float32, [None, n_classes])\n",
    "\n",
    "# data = tf.placeholder(tf.float32, [None, pow2,1]) #Number of examples, number of input, dimension of each input\n",
    "\n",
    "\n",
    "# Define weights\n",
    "weights = {\n",
    "    'out': tf.Variable(tf.random_normal([n_hidden, n_classes]))\n",
    "}\n",
    "biases = {\n",
    "    'out': tf.Variable(tf.random_normal([n_classes]))\n",
    "}\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# Prepare data shape to match `rnn` function requirements\n",
    "\n",
    "num_layers = 2\n",
    "lstm_cell = tf.nn.rnn_cell.LSTMCell(n_hidden,state_is_tuple=True)\n",
    "lstm_cell = tf.nn.rnn_cell.MultiRNNCell([lstm_cell] * num_layers, state_is_tuple=True)\n",
    "val, _ = tf.nn.dynamic_rnn(lstm_cell, x, dtype=tf.float32)\n",
    "val = tf.transpose(val, [1, 0, 2])\n",
    "last = tf.gather(val, int(val.get_shape()[0]) - 1)\n",
    "    \n",
    "pred = tf.matmul(last, weights['out']) + biases['out']\n",
    "\n",
    "# Define loss and optimizer\n",
    "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(pred, y))\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)\n",
    "\n",
    "# Evaluate model\n",
    "correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# Launch the graph\n",
    "sess1 = tf.Session()\n",
    "# For some reason it is our job to do this:\n",
    "sess1.run(tf.global_variables_initializer())\n",
    "\n",
    "\n",
    "# with tf.Session() as sess:\n",
    "#     sess.run(init)\n",
    "step = 1\n",
    "    # Keep training until reach max iterations\n",
    "while step * batch_size < training_iters:\n",
    "    batch_x, batch_y = mnist.train.next_batch(batch_size)\n",
    "    # Reshape data to get 28 seq of 28 elements\n",
    "    batch_x = batch_x.reshape((batch_size, n_steps, n_input))\n",
    "    # Run optimization op (backprop)\n",
    "    sess1.run(optimizer, feed_dict={x: batch_x, y: batch_y})\n",
    "    if step % display_step == 0:\n",
    "        # Calculate batch accuracy\n",
    "        acc = sess1.run(accuracy, feed_dict={x: batch_x, y: batch_y})\n",
    "        # Calculate batch loss\n",
    "        loss = sess1.run(cost, feed_dict={x: batch_x, y: batch_y})\n",
    "        print(\"Iter \" + str(step*batch_size) + \", Minibatch Loss= \" + \\\n",
    "              \"{:.6f}\".format(loss) + \", Training Accuracy= \" + \\\n",
    "              \"{:.5f}\".format(acc))\n",
    "    step += 1\n",
    "print(\"Optimization Finished!\")\n",
    "\n",
    "# Calculate accuracy for 128 mnist test images\n",
    "test_len = 128\n",
    "test_data = mnist.test.images[:test_len].reshape((-1, n_steps, n_input))\n",
    "test_label = mnist.test.labels[:test_len]\n",
    "print(\"Testing Accuracy:\", \\\n",
    "      sess1.run(accuracy, feed_dict={x: test_data, y: test_label}))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps hopefully by the next week\n",
    "\n",
    "- **Try this! Polygon example: Learn to draw convex polygons as a sequence of two dimensional points**\n",
    "- **Sentiment analysis**\n",
    "- **Text and time series** \n",
    "- **Multi-dimensional time series prediction**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Other applications!\n",
    "- handwriting generation\n",
    "    - https://www.cs.toronto.edu/~graves/handwriting.cgi?text=Vahid+Moosavi&style=&bias=0.15&samples=3\n",
    "    - https://nbviewer.jupyter.org/github/greydanus/scribe/blob/master/sample.ipynb\n",
    "    - http://greydanus.github.io/2016/08/21/handwriting/\n",
    "- Image captioning\n",
    "    - http://cs.stanford.edu/people/karpathy/deepimagesent/\n",
    "- Seq2Seq models for natural language translation\n",
    "\n",
    "- Learning and Translation in general\n",
    "    - http://binds.cs.umass.edu/papers/1995_Siegelmann_Science.pdf\n",
    "\n",
    "- Extensions to LSTM: Neural Turing Machine and differentiable-neural-computers\n",
    "    - https://arxiv.org/abs/1410.5401\n",
    "    - https://deepmind.com/blog/differentiable-neural-computers/\n",
    "\n",
    "\n",
    "## Two useful sources on RNN\n",
    "- http://www.deeplearningbook.org/contents/rnn.html\n",
    "- Awsome RNN: https://github.com/kjw0612/awesome-rnn"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}