1-seq2seq.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple dynamic seq2seq with TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial covers building seq2seq using dynamic unrolling with TensorFlow. \n",
    "\n",
    "I wasn't able to find any existing implementation of dynamic seq2seq with TF (as of 01.01.2017), so I decided to learn how to write my own, and document what I learn in the process.\n",
    "\n",
    "I deliberately try to be as explicit as possible. As it currently stands, TF code is the best source of documentation on itself, and I have a feeling that many conventions and design decisions are not documented anywhere except in the brains of Google Brain engineers. \n",
    "\n",
    "I hope this will be useful to people whose brains are wired like mine.\n",
    "\n",
    "**UPDATE**: as of r1.0 @ 16.02.2017, there is new official implementation in `tf.contrib.seq2seq`. See [tutorial #3](3-seq2seq-native-new.ipynb). Official tutorial reportedly be up soon. Personally I still find wiring dynamic encoder-decoder by hand insightful in many ways."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we implement plain seq2seq — forward-only encoder + decoder without attention. I'll try to follow closely the original architecture described in [Sutskever, Vinyals and Le (2014)](https://arxiv.org/abs/1409.3215). If you notice any deviations, please let me know."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Architecture diagram from their paper:\n",
    "![seq2seq architecutre](pictures/1-seq2seq.png)\n",
    "Rectangles are encoder and decoder's recurrent layers. Encoder receives `[A, B, C]` sequence as inputs. We don't care about encoder outputs, only about the hidden state it accumulates while reading the sequence. After input sequence ends, encoder passes its final state to decoder, which receives `[<EOS>, W, X, Y, Z]` and is trained to output `[W, X, Y, Z, <EOS>]`. `<EOS>` token is a special word in vocabulary that signals to decoder the beginning of translation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation details\n",
    "\n",
    "TensorFlow has its own [implementation of seq2seq](https://www.tensorflow.org/tutorials/seq2seq/). Recently it was moved from core examples to [`tensorflow/models` repo](https://github.com/tensorflow/models/tree/master/tutorials/rnn/translate), and uses deprecated seq2seq implementation. Deprecation happened because it uses **static unrolling**.\n",
    "\n",
    "**Static unrolling** involves construction of computation graph with a fixed sequence of time step. Such a graph can only handle sequences of specific lengths. One solution for handling sequences of varying lengths is to create multiple graphs with different time lengths and separate the dataset into this buckets.\n",
    "\n",
    "**Dynamic unrolling** instead uses control flow ops to process sequence step by step. In TF this is supposed to more space efficient and just as fast. This is now a recommended way to implement RNNs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vocabulary\n",
    "\n",
    "Seq2seq maps sequence onto another sequence. Both sequences consist of integers from a fixed range. In language tasks, integers usually correspond to words: we first construct a vocabulary by assigning to every word in our corpus a serial integer. First few integers are reserved for special tokens. We'll call the upper bound on vocabulary a `vocabulary size`.\n",
    "\n",
    "Input data consists of sequences of integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = [[5, 7, 8], [6, 3], [3], [1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While manipulating such variable-length lists are convenient to humans, RNNs prefer a different layout:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import helpers\n",
    "xt, xlen = helpers.batch(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[5, 7, 8], [6, 3], [3], [1]]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[5, 6, 3, 1],\n",
       "       [7, 3, 0, 0],\n",
       "       [8, 0, 0, 0]], dtype=int32)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sequences form columns of a matrix of size `[max_time, batch_size]`. Sequences shorter then the longest one are padded with zeros towards the end. This layout is called `time-major`. It is slightly more efficient then `batch-major`. We will use it for the rest of the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[3, 2, 1, 1]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xlen"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For some forms of dynamic layout it is useful to have a pointer to terminals of every sequence in the batch in separate tensor (see following tutorials)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple seq2seq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Encoder starts with empty state and runs through the input sequence. We are not interested in encoder's outputs, only in its `final_state`.\n",
    "\n",
    "Decoder uses encoder's `final_state` as its `initial_state`. Its inputs are a batch-sized matrix with `<EOS>` token at the 1st time step and `<PAD>` at the following. This is a rather crude setup, useful only for tutorial purposes. In practice, we would like to feed previously generated tokens after `<EOS>`.\n",
    "\n",
    "Decoder's outputs are mapped onto the output space using `[hidden_units x output_vocab_size]` projection layer. This is necessary because we cannot make `hidden_units` of decoder arbitrarily large, while our target space would grow with the size of the dictionary.\n",
    "\n",
    "This kind of encoder-decoder is forced to learn fixed-length representation (specifically, `hidden_units` size) of the variable-length input sequence and restore output sequence only from this representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import helpers\n",
    "\n",
    "tf.reset_default_graph()\n",
    "sess = tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.3.0'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model inputs and outputs "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First critical thing to decide: vocabulary size.\n",
    "\n",
    "Dynamic RNN models can be adapted to different batch sizes and sequence lengths without retraining (e.g. by serializing model parameters and Graph definitions via `tf.train.Saver`), but changing vocabulary size requires retraining the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "PAD = 0\n",
    "EOS = 1\n",
    "\n",
    "vocab_size = 10\n",
    "input_embedding_size = 20\n",
    "\n",
    "encoder_hidden_units = 20\n",
    "decoder_hidden_units = encoder_hidden_units"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice way to understand complicated function is to study its signature - inputs and outputs. With pure functions, only inputs-output relation matters.\n",
    "\n",
    "- `encoder_inputs` int32 tensor is shaped `[encoder_max_time, batch_size]`\n",
    "- `decoder_targets` int32 tensor is shaped `[decoder_max_time, batch_size]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "encoder_inputs = tf.placeholder(shape=(None, None), dtype=tf.int32, name='encoder_inputs')\n",
    "decoder_targets = tf.placeholder(shape=(None, None), dtype=tf.int32, name='decoder_targets')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll add one additional placeholder tensor: \n",
    "- `decoder_inputs` int32 tensor is shaped `[decoder_max_time, batch_size]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "decoder_inputs = tf.placeholder(shape=(None, None), dtype=tf.int32, name='decoder_inputs')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We actually don't want to feed `decoder_inputs` manually — they are a function of either `decoder_targets` or previous decoder outputs during rollout. However, there are different ways to construct them. It might be illustrative to explicitly specify them for out first seq2seq implementation.\n",
    "\n",
    "During training, `decoder_inputs` will consist of `<EOS>` token concatenated with `decoder_targets` along time axis. In this way, we always pass target sequence as the history to the decoder, regrardless of what it actually outputs predicts. This can introduce distribution shift from training to prediction. \n",
    "In prediction mode, model will receive tokens it previously generated (via argmax over logits), not the ground truth, which would be unknowable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that all shapes are specified with `None`s (dynamic). We can use batches of any size with any number of timesteps. This is convenient and efficient, however but there are obvious constraints: \n",
    "- Feed values for all tensors should have same `batch_size`\n",
    "- Decoder inputs and ouputs (`decoder_inputs` and `decoder_targets`) should have same `decoder_max_time`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Embeddings\n",
    "\n",
    "`encoder_inputs` and `decoder_inputs` are int32 tensors of shape `[max_time, batch_size]`, while encoder and decoder RNNs expect dense vector representation of words, `[max_time, batch_size, input_embedding_size]`. We convert one to another by using *word embeddings*. Specifics of working with embeddings are nicely described in [official tutorial on embeddings](https://www.tensorflow.org/tutorials/word2vec/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we initialize embedding matrix. Initializations are random. We rely on our end-to-end training to learn vector representations for words jointly with encoder and decoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "embeddings = tf.Variable(tf.random_uniform([vocab_size, input_embedding_size], -1.0, 1.0), dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use `tf.nn.embedding_lookup` to *index embedding matrix*: given word `4`, we represent it as 4th column of embedding matrix. \n",
    "This operation is lightweight, compared with alternative approach of one-hot encoding word `4` as `[0,0,0,1,0,0,0,0,0,0]` (vocab size 10) and then multiplying it by embedding matrix.\n",
    "\n",
    "Additionally, we don't need to compute gradients for any columns except 4th.\n",
    "\n",
    "Encoder and decoder will share embeddings. It's all words, right? Well, digits in this case. In real NLP application embedding matrix can get very large, with 100k or even 1m columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoder_inputs_embedded = tf.nn.embedding_lookup(embeddings, encoder_inputs)\n",
    "decoder_inputs_embedded = tf.nn.embedding_lookup(embeddings, decoder_inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoder\n",
    "\n",
    "The centerpiece of all things RNN in TensorFlow is `RNNCell` class and its descendants (like `LSTMCell`). But they are outside of the scope of this post — nice [official tutorial](https://www.tensorflow.org/tutorials/recurrent/) is available. \n",
    "\n",
    "`@TODO: RNNCell as a factory`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "encoder_cell = tf.contrib.rnn.LSTMCell(encoder_hidden_units)\n",
    "\n",
    "encoder_outputs, encoder_final_state = tf.nn.dynamic_rnn(\n",
    "    encoder_cell, encoder_inputs_embedded,\n",
    "    dtype=tf.float32, time_major=True,\n",
    ")\n",
    "\n",
    "del encoder_outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We discard `encoder_outputs` because we are not interested in them within seq2seq framework. What we actually want is `encoder_final_state` — state of LSTM's hidden cells at the last moment of the Encoder rollout.\n",
    "\n",
    "`encoder_final_state` is also called \"thought vector\". We will use it as initial state for the Decoder. In seq2seq without attention this is the only point where Encoder passes information to Decoder. We hope that backpropagation through time (BPTT) algorithm will tune the model to pass enough information throught the thought vector for correct sequence output decoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LSTMStateTuple(c=<tf.Tensor 'rnn/while/Exit_2:0' shape=(?, 20) dtype=float32>, h=<tf.Tensor 'rnn/while/Exit_3:0' shape=(?, 20) dtype=float32>)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoder_final_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow LSTM implementation stores state as a tuple of tensors. \n",
    "- `encoder_final_state.h` is activations of hidden layer of LSTM cell\n",
    "- `encoder_final_state.c` is final output, which can potentially be transfromed with some wrapper `@TODO: check correctness`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "decoder_cell = tf.contrib.rnn.LSTMCell(decoder_hidden_units)\n",
    "\n",
    "decoder_outputs, decoder_final_state = tf.nn.dynamic_rnn(\n",
    "    decoder_cell, decoder_inputs_embedded,\n",
    "\n",
    "    initial_state=encoder_final_state,\n",
    "\n",
    "    dtype=tf.float32, time_major=True, scope=\"plain_decoder\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we pass `encoder_final_state` as `initial_state` to the decoder, they should be compatible. This means the same cell type (`LSTMCell` in our case), the same amount of `hidden_units` and the same amount of layers (single layer). I suppose this can be relaxed if we additonally pass `encoder_final_state` through a one-layer MLP."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With encoder, we were not interested in cells output. But decoder's outputs are what we actually after: we use them to get distribution over words of output sequence.\n",
    "\n",
    "At this point `decoder_cell` output is a `hidden_units` sized vector at every timestep. However, for training and prediction we need logits of size `vocab_size`. Reasonable thing would be to put linear layer (fully-connected layer without activation function) on top of LSTM output to get non-normalized logits. This layer is called projection layer by convention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "decoder_logits = tf.contrib.layers.linear(decoder_outputs, vocab_size)\n",
    "\n",
    "decoder_prediction = tf.argmax(decoder_logits, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'fully_connected/BiasAdd:0' shape=(?, ?, 10) dtype=float32>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoder_logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RNN outputs tensor of shape `[max_time, batch_size, hidden_units]` which projection layer maps onto `[max_time, batch_size, vocab_size]`. `vocab_size` part of the shape is static, while `max_time` and `batch_size` is dynamic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "stepwise_cross_entropy = tf.nn.softmax_cross_entropy_with_logits(\n",
    "    labels=tf.one_hot(decoder_targets, depth=vocab_size, dtype=tf.float32),\n",
    "    logits=decoder_logits,\n",
    ")\n",
    "\n",
    "loss = tf.reduce_mean(stepwise_cross_entropy)\n",
    "train_op = tf.train.AdamOptimizer().minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sess.run(tf.global_variables_initializer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test forward pass\n",
    "\n",
    "Did I say that deep learning is a game of shapes? When building a Graph, TF will throw errors when static shapes are not matching. However, mismatches between dynamic shapes are often only discovered when we try to run something through the graph.\n",
    "\n",
    "\n",
    "So let's try running something. For that we need to prepare values we will feed into placeholders."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "this is key part where everything comes together\n",
    "\n",
    "@TODO: describe\n",
    "- how encoder shape is fixed to max\n",
    "- how decoder shape is arbitraty and determined by inputs, but should probably be longer then encoder's\n",
    "- how decoder input values are also arbitraty, and how we use GO token, and what are those 0s, and what can be used instead (shifted gold sequence, beam search)\n",
    "@TODO: add references\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch_encoded:\n",
      "[[6 3 9]\n",
      " [0 4 8]\n",
      " [0 0 7]]\n",
      "decoder inputs:\n",
      "[[1 1 1]\n",
      " [0 0 0]\n",
      " [0 0 0]\n",
      " [0 0 0]]\n",
      "decoder predictions:\n",
      "[[2 4 6]\n",
      " [2 4 2]\n",
      " [1 4 5]\n",
      " [5 4 4]]\n"
     ]
    }
   ],
   "source": [
    "batch_ = [[6], [3, 4], [9, 8, 7]]\n",
    "\n",
    "batch_, batch_length_ = helpers.batch(batch_)\n",
    "print('batch_encoded:\\n' + str(batch_))\n",
    "\n",
    "din_, dlen_ = helpers.batch(np.ones(shape=(3, 1), dtype=np.int32),\n",
    "                            max_sequence_length=4)\n",
    "print('decoder inputs:\\n' + str(din_))\n",
    "\n",
    "pred_ = sess.run(decoder_prediction,\n",
    "    feed_dict={\n",
    "        encoder_inputs: batch_,\n",
    "        decoder_inputs: din_,\n",
    "    })\n",
    "print('decoder predictions:\\n' + str(pred_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Successful forward computation, everything is wired correctly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training on the toy task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will teach our model to memorize and reproduce input sequence. Sequences will be random, with varying length.\n",
    "\n",
    "Since random sequences do not contain any structure, model will not be able to exploit any patterns in data. It will simply encode sequence in a thought vector, then decode from it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "head of the batch:\n",
      "[4, 3, 6, 7, 8, 2, 7, 9]\n",
      "[4, 9, 3, 2, 3, 9, 5]\n",
      "[2, 7, 4, 7]\n",
      "[8, 4, 6, 6, 9, 2]\n",
      "[5, 8, 8, 8, 6, 2]\n",
      "[2, 7, 3, 2, 4]\n",
      "[4, 8, 6]\n",
      "[7, 8, 7, 3]\n",
      "[7, 2, 3, 3, 7, 7, 6, 2]\n",
      "[4, 5, 4, 7, 6, 5, 8]\n"
     ]
    }
   ],
   "source": [
    "batch_size = 100\n",
    "\n",
    "batches = helpers.random_sequences(length_from=3, length_to=8,\n",
    "                                   vocab_lower=2, vocab_upper=10,\n",
    "                                   batch_size=batch_size)\n",
    "\n",
    "print('head of the batch:')\n",
    "for seq in next(batches)[:10]:\n",
    "    print(seq)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def next_feed():\n",
    "    batch = next(batches)\n",
    "    encoder_inputs_, _ = helpers.batch(batch)\n",
    "    decoder_targets_, _ = helpers.batch(\n",
    "        [(sequence) + [EOS] for sequence in batch]\n",
    "    )\n",
    "    decoder_inputs_, _ = helpers.batch(\n",
    "        [[EOS] + (sequence) for sequence in batch]\n",
    "    )\n",
    "    return {\n",
    "        encoder_inputs: encoder_inputs_,\n",
    "        decoder_inputs: decoder_inputs_,\n",
    "        decoder_targets: decoder_targets_,\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given encoder_inputs `[5, 6, 7]`, decoder_targets would be `[5, 6, 7, 1]`, where 1 is for `EOS`, and decoder_inputs would be `[1, 5, 6, 7]` - decoder_inputs are lagged by 1 step, passing previous token as input at current step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss_track = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch 0\n",
      "  minibatch loss: 2.3455774784088135\n",
      "  sample 1:\n",
      "    input     > [4 3 2 0 0 0 0 0]\n",
      "    predicted > [2 2 7 2 5 5 5 5 5]\n",
      "  sample 2:\n",
      "    input     > [8 5 9 6 9 4 0 0]\n",
      "    predicted > [2 1 1 9 9 9 9 9 9]\n",
      "  sample 3:\n",
      "    input     > [3 3 3 6 8 2 6 0]\n",
      "    predicted > [2 3 3 3 3 3 5 5 5]\n",
      "\n",
      "batch 1000\n",
      "  minibatch loss: 0.3058355748653412\n",
      "  sample 1:\n",
      "    input     > [3 3 5 9 3 4 6 7]\n",
      "    predicted > [3 3 5 9 3 4 6 7 1]\n",
      "  sample 2:\n",
      "    input     > [8 4 9 8 3 8 0 0]\n",
      "    predicted > [8 4 8 8 3 8 1 0 0]\n",
      "  sample 3:\n",
      "    input     > [6 5 7 0 0 0 0 0]\n",
      "    predicted > [6 5 7 1 0 0 0 0 0]\n",
      "\n",
      "batch 2000\n",
      "  minibatch loss: 0.1154913678765297\n",
      "  sample 1:\n",
      "    input     > [5 4 8 5 0 0 0 0]\n",
      "    predicted > [5 4 8 5 1 0 0 0 0]\n",
      "  sample 2:\n",
      "    input     > [8 2 6 2 6 0 0 0]\n",
      "    predicted > [8 2 6 2 6 1 0 0 0]\n",
      "  sample 3:\n",
      "    input     > [6 2 9 5 0 0 0 0]\n",
      "    predicted > [6 2 9 5 1 0 0 0 0]\n",
      "\n",
      "batch 3000\n",
      "  minibatch loss: 0.09871210902929306\n",
      "  sample 1:\n",
      "    input     > [6 7 9 6 0 0 0 0]\n",
      "    predicted > [6 7 9 6 1 0 0 0 0]\n",
      "  sample 2:\n",
      "    input     > [7 5 3 5 4 9 9 0]\n",
      "    predicted > [7 5 3 5 4 9 9 1 0]\n",
      "  sample 3:\n",
      "    input     > [2 5 6 2 4 9 7 6]\n",
      "    predicted > [2 5 6 2 4 9 7 3 1]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "max_batches = 3001\n",
    "batches_in_epoch = 1000\n",
    "\n",
    "try:\n",
    "    for batch in range(max_batches):\n",
    "        fd = next_feed()\n",
    "        _, l = sess.run([train_op, loss], fd)\n",
    "        loss_track.append(l)\n",
    "\n",
    "        if batch == 0 or batch % batches_in_epoch == 0:\n",
    "            print('batch {}'.format(batch))\n",
    "            print('  minibatch loss: {}'.format(sess.run(loss, fd)))\n",
    "            predict_ = sess.run(decoder_prediction, fd)\n",
    "            for i, (inp, pred) in enumerate(zip(fd[encoder_inputs].T, predict_.T)):\n",
    "                print('  sample {}:'.format(i + 1))\n",
    "                print('    input     > {}'.format(inp))\n",
    "                print('    predicted > {}'.format(pred))\n",
    "                if i >= 2:\n",
    "                    break\n",
    "            print()\n",
    "except KeyboardInterrupt:\n",
    "    print('training interrupted')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.0982 after 300100 examples (batch_size=100)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8FPX9x/HXJzf3lSA34RIFRNAIKEe9RbRFWv1Vbaul\nKtraqq1tf9RapbXWag+rrdWftdTWelWrFgWtRVFRK8gV7vsMBBKuJJCEXN/fH7uJSciSa7OzO3k/\nH488mJ2Z3fkMA+/Mznzn+zXnHCIi4i9xXhcgIiLhp3AXEfEhhbuIiA8p3EVEfEjhLiLiQwp3EREf\nUriLiPiQwl1ExIcU7iIiPpTg1YZTU1Ndenq6V5sXEYlJS5cu3e+cS6tvPc/CPT09nSVLlni1eRGR\nmGRmOxqyni7LiIj4kMJdRMSHFO4iIj6kcBcR8SGFu4iIDyncRUR8SOEuIuJDMRfu+48c46evr6G0\nvMLrUkREolbMhfs76/bxl4+28/GWA16XIiIStWIu3CcMCTx1u+dwkceViIhEr5gL9+4dkomPM3Yf\nUriLiIQSc+GeGB9H785t2HGw0OtSRESiVsyFO0D/bm3ZceCo12WIiEStmAz39G7t2L5f4S4iEkpM\nhnu39knkF5epOaSISAgxGe6Zuw4DsDIrz+NKRESiU0yG+1UZfQEo05m7iEidYjLcu7VLAuDN1Xs9\nrkREJDrFZLifld4VgINHSzyuREQkOsVkuMfFGWMGdGVvfrHXpYiIRKWYDHeAtA7J7C845nUZIiJR\nKXbDvX0yuQp3EZE6xW64d0im4FgZRSXlXpciIhJ1YjbcN+0rAOAPCzZ5XImISPSJ2XCfOqo3ACVl\nausuIlJbzIb7505OIyk+jrg487oUEZGoE7PhHhdnlJRX8H/vb/W6FBGRqBOz4V6duiEQEakppsP9\nmjH9ACircB5XIiISXWI63AektgUU7iIitcV0uCfEBcrXZRkRkZpiOtwrB+s4cqzM40pERKJLTIf7\ni0t2ATDlkYUeVyIiEl1iOty/MrY/APnFOnMXEakupsP9qow+AEw5rYfHlYiIRJd6w93M+prZAjNb\na2ZrzOz2OtYxM3vUzDab2UozO6Nlyq2pY0oiAG9pRCYRkRoSGrBOGXCnc26ZmXUAlprZf5xza6ut\ncykwJPgzFng8+GdEqCWkiEhN9Z65O+eynXPLgtMFwDqgd63VpgJ/cwGfAJ3NrGfYqxURkQZp1DV3\nM0sHRgOLai3qDeyq9jqL438BtCjndPouIlKpweFuZu2BfwJ3OOfym7IxM5thZkvMbElubm5TPuI4\n00YHfodszjkSls8TEfGDBoW7mSUSCPZnnXOv1LHKbqBvtdd9gvNqcM496ZzLcM5lpKWlNaXe48xd\nmQ3Ay8uywvJ5IiJ+0JDWMgb8GVjnnPttiNXmANcFW82MA/Kcc9lhrDOkjm0C94Tzi9TWXUSkUkNa\ny4wHvgasMrMVwXl3Af0AnHNPAPOAKcBmoBCYHv5S69anS1v2HymhqEThLiJSqd5wd859CJxwuCMX\nuJt5a7iKaownvnom4x54h0Fp7b3YvIhIVIrpJ1QBenRKoW1SPHlFpV6XIiISNWI+3AE6t0nksMJd\nRKSKL8K9TVI8q3fneV2GiEjUaMgN1ai3JfcoAEUl5bRJive4GhER7/nizL1SiUZkEhEBfBbuBcW6\n7i4iAj4J9xG9OwLwm7c3elyJiEh08EW4V7ZxX77zkMeViIhEB1+Ee4eUwH3h7QcKPa5ERCQ6+CLc\nZ156qtcliIhEFV+Ee/vkBC47LTA2iPp1FxHxSbgDzF0V6IRy3iqNpyoi4ptwr3Trc8u8LkFExHO+\nC3cREfFRuM+7bWLVdKmeVBWRVs434T6sV8eq6eLScg8rERHxnm/CvbriUp25i0jr5stwP1amM3cR\nad18Ge6rd+d7XYKIiKd8Ge53vbrK6xJERDzly3AvU2sZEWnlfBXuw4MtZvKLy3j4P+r+V0RaL1+F\n+9++MaZq+pF3NnlYiYiIt3wV7kkJvtodEZEm81UaxseZ1yWIiEQFX4V7m8R4r0sQEYkKvgp3M525\ni4iAz8K9trdWq293EWmdfB3ut/x9qdcliIh4wnfh/sPJQ2u81rB7ItIa+S7cv3XuYGZMGlj1emVW\nnofViIh4w3fhDvDp9oNV0yXqikBEWiFfhvvynYerpnMLjnlYiYiIN3wZ7vdPG1E1/a1nl+m6u4i0\nOvWGu5nNNrMcM1sdYvm5ZpZnZiuCP/eEv8zG+dIZfWq8/uvH270pRETEIw05c38amFzPOgudc6OC\nPz9rflnNk1LrSdXH3tviUSUiIt6oN9ydcx8AB+tbL5rpsoyItDbhuuZ+jpmtNLM3zWx4mD4zbPYf\nKWH+2n1elyEiEjHhCPdlQD/n3Ejg98BroVY0sxlmtsTMluTm5oZh0w333OKdEd2eiIiXmh3uzrl8\n59yR4PQ8INHMUkOs+6RzLsM5l5GWltbcTTdKcWl5RLcnIuKlZoe7mfWwYHeMZjYm+JkHmvu54fbx\nlqgrSUSkxTSkKeTzwH+BoWaWZWY3mNktZnZLcJUrgdVmlgk8ClztouAO5u0XDPG6BBERzyTUt4Jz\n7pp6lv8B+EPYKgqTPl3aeF2CiIhnfPmEKkBdXx2i4AuFiEhE+DbcR/TqdNy8J97f6kElIiKR59tw\nH9arI9eO7Vdj3oNvrfeoGhGRyPJtuAP07JjidQkiIp7wdbjf/LlBXpcgIuIJX4d7UsLxu/fBxsg+\nGSsi4gVfh3tdHluw2esSRERaXKsL90XbYrqDSxGRBml14Q6QPnOu1yWIiLQo34f7iN4dvS5BRCTi\nfB/uXzi9l9cliIhEnO/Dffr4AfxsatSNHyIi0qJ8H+6J8XFcd3b6cfPTZ85l+/6jkS9IRCQCfB/u\nJzJ/3T7mrswmfeZcdh0s9LocEZGwaTXhvv6+yXzvopNrzCstd7yyLAuADXsLvChLRKRFtJpwT0mM\n57ZaA3g8+NZ6FmzIASCu1fxNiEhr0OojrSLYxXtwpEAREV9odeF+92Wn1jk/XuEuIj7S6sK9fXLd\nIwvGKdxFxEdaXbiHEqdsFxEfaXXhHmoU1U+3H4poHSIiLan1hXuIdH/8fXUFLCL+0erCPZTi0gqv\nSxARCZtWF+4dUuq+oQqQnVcUwUpERFpOqwv3y07ryS+mncZDV448btnZD7xLfnGpB1WJiIRX6NNY\nn4qLM64d2w+AbfuP8vh7W2osz8k/RseURC9KExEJm1Z35l4fNXcXET9o1eGuHBcRv2rV4V4XBb6I\n+IHCvZbzf/M+6TPncriwxOtSRESarFWHe1JC6N3/+yc7IliJiEh4tepwv3nSIG6eNLDOZR9vORDh\nakREwqdVh3ubpHh+NOVUzuzf5bhlCncRiWWtOtwr/ePms70uQUQkrFrdQ0x1iQ/R3++Db62nvMJx\nx4VDaJukvyoRiR31JpaZzQYuB3KccyPqWG7AI8AUoBD4unNuWbgL9ULl06spCXF896KTNRSfiMSM\nhlyWeRqYfILllwJDgj8zgMebX1bk3TBhQMhlj767mV+/vSGC1YiINE+94e6c+wA4eIJVpgJ/cwGf\nAJ3NrGe4CoyUn1w+jH/dOj7k8r/9V00jRSR2hOOGam9gV7XXWcF5xzGzGWa2xMyW5ObmhmHT4XV6\n384hlxUUl7FmT14EqxERabqItpZxzj3pnMtwzmWkpaVFctMNtuwnF4VcdtmjH7J+b34EqxERaZpw\nhPtuoG+1132C82JS13ZJjOzTKeRyjbUqIrEgHOE+B7jOAsYBec657DB8rmdenBG63Xt5uYbjE5Ho\n15CmkM8D5wKpZpYF3AskAjjnngDmEWgGuZlAU8jpLVVspLRJig+5rKwixAjbIiJRpN5wd85dU89y\nB9watoqixKST0/hg4/E3fXccKPSgGhGRxlH3A430zCc7KNfZu4hEOYV7CANT24VcNuiueazKUrNI\nEYleCvcQfjTlFJ6eflbI5bM/2hbBakREGkfhHkJyQjznDu0ecnlSvP7qRCR6KaGaqKi03OsSRERC\nUrjXo2u7pDrnz8ncQ6ChENz3xlrueGF5JMsSETkhhXs93vjOBO6fNoK2SfFcdWafGsvufCmTT7Ye\n4M8fbuO1FXs8qlBE5HgagaIevTq34Stj+/OVsf2Z/WHNm6ivLNvNK8titqcFEfExnbk3QkK8BusQ\nkdigcG+EhDj9dYlIbFBaNUJCiLFWRUSijcK9ES4efhKn9uwYcvntLyznWFk5RSVqJiki3rLK5nyR\nlpGR4ZYsWeLJtpsrfebcetdZOetiOqYkRqAaEWlNzGypcy6jvvV05t4EN55gMO1KI2e9TUFxaQSq\nERE5nsK9CWZMGtig9Q4cKWnhSkRE6qZwb0GlGrVJRDyicG+CyrsU3Tsks/n+S0OuN/uj7VSo73cR\n8YDCvQkqgjehzSAhPo5hIVrQPL94J798a30kSxMRARTuTVLZwMgItHufPj495LpPfrCV8379Hr95\newO7DmqIPhGJDIV7E6R1SGZE7448eOVIAK7K6HvC9bftP8rv393MLX9fGonyRETUcVhTJMbH8cZ3\nJjb6fWv25LdANSIix9OZe4S9tGSXnmAVkRancI+wH7y8kh/+cyVlaiYpIi1Il2XCZP73JpEUH8/a\n7Px6r62/nrmHeIMHrxxJckJ8hCoUkdZEZ+5hMrh7B/p1a8uEIakAfHVcv5BD9AG8tmIP189eTL66\nKBCRFqCOw1pATn4xXdslkVdUypef/ITNOUdOuP4fv3IGU07rGaHqRCSWqeMwD3XvmEJCfBzd2ifz\n7zsm0b1D8gnXX7hpf4QqE5HWQuHewuLjjMU/vpCFPzwv5Dql5RV8svWAbrKKSNjohmqE9O3aNuSy\nl5dm8fLSLAB++oXhZKR34aSOKaS2P/EZv4hIKAp3Dzz85dP57ouZdS67d84aAOIMtj5wWSTLEhEf\n0WWZCHrt1vG8/u0JTBvdp951KxzkFhyrMa+guJTNOQUtVZ6I+IjO3CNoVN/OjVp/4kPvMqR7B26Y\nMIArRvdm3C/e4WhJOdt/qTN6ETkxhXsUKy6tYNXuPO54cQXr9xZwVN0WiEgD6bKMRxbfdQH3XD6s\nwes/8f6WFqxGRPymQeFuZpPNbIOZbTazmXUsP9fM8sxsRfDnnvCX6i/dO6bwjQYMtF2XNXvyqKhw\nePUAmohEv3ovy5hZPPAYcBGQBXxqZnOcc2trrbrQOXd5C9QotVz26IcAnH9Kd2Z//SwApv3xI6aN\n7s11Z6d7WJmIRIuGnLmPATY757Y650qAF4CpLVuWNMS763OqppfvPMw9/1rjYTUiEk0aEu69gV3V\nXmcF59V2jpmtNLM3zWx4XR9kZjPMbImZLcnNzW1CuVJbRYWjuPT4G61ZhwrZuE/NJkVaq3C1llkG\n9HPOHTGzKcBrwJDaKznnngSehEDHYWHadky774oR7M0r4rEFWzitdydeuuVs5mTu4fMje3HqPW/V\n+/6Bd807bt4f39vMQ29tAFCzSZFWqiHhvhuoPkhon+C8Ks65/GrT88zsj2aW6pxTj1j1+Nq4/jjn\nSO/Wjs+f3ouUxHj+Jzgm6/ZfXkb6zLmN+ryy8oqqYBeR1qshl2U+BYaY2QAzSwKuBuZUX8HMepiZ\nBafHBD/3QLiL9Ssz46qMvqQkHj9wx3fOH9yoz7rjxRU1XqfPnMvynYeaVZ+IxJ4G9ecevNTyOyAe\nmO2cu9/MbgFwzj1hZt8GvgmUAUXA95xzH5/oM/3cn3u4FZWU88nWA8xdlV3VwVhT/N/XzuSS4T3C\nWJmIRFpD+3PXYB0x5In3t/DLN9c36zPe+M4E3lmXw+0XBm6J/OClTNJT23HreY37hiAi3tBgHT40\ndkDXGq+b0iXw5b//kIfnb6xqYfPS0ix+9W9doxfxG/UtE0NG9+vCxp9fyt68YkrKy3l/437ue6P2\ns2QNc8cLK/h0+8Gq109/tI2vj2/aE7MiEn0U7jEmKSGOft0CA38MSmvPwk25vLeh8c8MvLVmb43X\ns15fy46Dhdxxwcl0apsYllpFxDu65u4DW3KPsHFvAR9syuX5xbvqf0M9Pp55Pn98bzM/uOQUzGDx\n1oOcd0p34uMsDNWKSHPohmortXp3Hpf//sOwf+7dl53KjRMHHje/uLScpPg44hT8IhGhG6qt1Ije\nnaqmv3hGXb1ENM3P565j1pyafddUVDhO+clb3DtnDQs35ZKTXxy27YlI8+iauw898dUzOFxYyiXD\ne/DKssDDxN3aJXHgaEmzPvfpj7ez48BRJp2cxmm9O7F+b6DvmucW7+SZT3bQp0sbPvzf85tdv4g0\nn8LdhyaP6Fk1/fT0sxjdtwud2iZy2aMLWbMnv8a6XdomcqiwtMGfvWBDLgtq3cAtrwhc2ss6VETW\noUI27TvCead0b8YeiEhz6bKMz507tHtV65e/fmMMt1XrzmDGpIEsv+diXpwxLmzbm/DgAqY//ekJ\n13lu0U5WZh0O2zZF5Hg6c29FUtsn872Lh3JVRl/e25DD14IDeyQlhP93/OrdeVXX//MKS3l1eRbX\nnZ2OGdz16ipAPVaKtCSFeyvUt2vbqmCHwE3YL47uzeo9eWzcd4T7rhjBpSN6kFtwjEsfWdikbVS2\n2Ln6rL5kZuWxLjufWa+v5aoz+4RjF0SkHmoKKVUKikt5ZP4mfjB5KMkJgR4q31yVzTefXdZi20xt\nn8S73z+XFTsPc++cNWzbf5RPf3whaR0a37WCSGugdu4SNpV9yn/+9F68nrmnxbfXLimeT+++kLZJ\ngS+WR46V0T75sy+ZpeUVxJnpoSppldTOXcKm8pr8I18eVWP+y7ec3SLbO1pSzrB7/k36zLk8/dE2\nRtz7bz7eEhj35ca/LmHIj99kUB0jUDXU0x9tY0oTLzeJxAqFu9Rryd0XsvwnFxEXZzVugo7s05ne\nndtUve7ZKSXs2571eqBjtGv/tIj0mXOZv25f1bLi0nLmrcrm+tmLKa9wTP7dB7y0ZBcVFY7DhSWE\n+lY66/W1rM3Or3OZiF/osow02rxV2Wzbf7SqD/iC4lKOlVVgQHZeMb9+e0ONzsz6d2vLjgOFYa/j\n9guG8Mg7m4C62+s/9KWR/M9Zn40Q+dv/bOT1zD1s238UgA0/n8yWnKN0bJPA0h2HmDoqfE/0irQU\nXXMXTy3dcYgvPf4xT3z1DCaP6MmmfQV8/S+fsvtwUcRquHxkT/5w7RkAHCsrZ+jddQ843jYpnsKS\ncp67aSznDEqtmu+c49dvb+CxBVvY9sAUgiNJinhK4S6eyyssrdF9cEWFY2DwWvnKWRfz6baD3PDX\n6Ps3cEqPDmQdKuLb5w+uGvlqWM+OvHrrOQA8898d9OiUwuThPXji/S1MHz+AdsEbvhUVLmQnasWl\n5Xz/pUzuuHAI723IZfr4AbopLI2mcJeoVNnypvLafXFpOZc9upAbJw7kmjH9qpbHgkeuHsXtLwQG\nJN/6iyms3J3HFY99xPM3jaNv1zaktk8mJTGenPxibnpmKZm7aj6V+9v/OZ0vnqF2/9I4DQ13PcQk\nETXvtol0SPnsn11KYjzv3Hlu1etFd13A0x9vZ8LgVJ5auJXObZN4dfluDyqt3/vV7iu8tmI33/tH\nJgDX/OkTAC4edhJPXpfB84t3HRfsAIUl5Y3a3uacI6R3a0tCvNpBSP105i5R763VeykqLaNz2yT6\nd23L+b95/7h1bvncIJ54f4sH1TXdz6YO57rgk8IfbMxlzICuPLdoJ9eO7UdKYjyLth4gtUMyg9La\ns2jrAb78ZOCXxsQhqTx1fQZPLdzGjRMHVD1w5pV9+cVM/8un/PnrGfTs1Kb+N0iz6LKM+Npv3t7A\noq0HeX7GOOIMzAznHNv2HyUxPo6JDy3wusQGe+6msVz7p0U15v3qypH84OWVACz7yUUsWJ/DnS9l\nVi3/7oUn8/D8jcQZPHV9BucN7c63n1/OZaf1ZMppgV5BDxeW0CYpnuSEeFbsOswry7K48+Kh/HTO\nGu79wnA6tWnecIr5xaW8tXove/OK+e1/NjJxSCrP3DC2WZ8p9VO4S6u2aV8Ba/bkc8eLgWvi1Z+u\nTW2fzP4jx5g8vAfXnd2fbu2TueR3H3hZbr3uu2IEP3ltdcjlf5l+FtP/EuiNc/zgbny0+UDVsvun\njeC+N9ZSXFrBN8YPYPZH27hxwgCy84qZ9YXhrM3OZ+n2g3zv4qGNqmnUz97mcGEp5wzqxsdbAttb\nOetiOqZoDN6WpHAXAf61Yjep7ZMZPziVbfuPUlZewZCTOhy33j+XZpGe2paTT+rAabPe9qBS7908\naSC9Orfh3jlruHZsP34x7TTKKxy3PruMb547iNP7dq6xfqib3w9dOZIfvrySdT+bTJuk5l8yKiop\nJ6egmP7d2jX7s/xA4S7SRO9tyGF0vy50TEngi49/zA0TBvDfLQc4vW9nzhnUjfXZBRSWlvPgm+sj\n2m7fC8N7dawa4GXp3Reyek8+iXHGgaMlfOf55Sd877M3jmX84NSQy99es5e9+cVcd3Y6xaXlPLto\nJxOHpJKdV8yEwam8unw300b35ut/WczCTftDPmtw9FgZCzftZ/KIHs3b2aCiknIqnKtq3hptFO4i\nEfDNvy/lzdV7eXHGOAaktmNtdj5d2yWx53ARWYeK+PncdQB8+L/nccnDH3C0pJz7p43gx6+GvsTi\nN9NG9+bV5bu5fGRP3liZzeh+nfnlF0dWXQpbeveFnPnz+TXeU3lpaUTvjqzeHfjl8odrR5PWPpmx\nA7sBUFJWwfYDR/n9u5t5PXMPb393Etv3H+Xd9TnkF5fy0JWnU1hSxv6CEob16ghQ1SXFroNFTPrV\nAl791jmM7tcFgDv/kcnGfQWs2p0HfNZc98VPdzKqbxeG9jj+G19dnlu0k0Fp7arq3HWwkJTEeNI6\nJPODlzKZOqo3E4aE/qVXH4W7SAQcLixhTuYevjauf51nlU8t3Mrg7u05d2h3Nucc4bbnl/P8TeMo\nLC3j7Afe5bmbxlJSVsGczD28smw3XdslcfAEY90+d9NYRvXtzH+3HIjKB8Ai4aoz+5B75FiNLi4A\nBndvz+acI3W+59yhaazPLmBvrUHcrxnTl1F9O3P2wFQm/armTfj0bm156voMLvxt4JfQtgem8NTC\nbZx3SndeWLyT8YNTq4aTLK9wHDxaQlqH5OOe5ah8ve2BKQz40bway5pC4S4Sw97fmEuFc2TuOswt\nnxvE3rxiUhLj6RGic7bKAJl320SmPFqzx8vPn96LopIy5q/LqZp3Usdk9uUfa7kd8IlObRLJKwr0\nWfTijHFVzVGru2ZMP7Lzio77ZTN1VC8euXp01bFZdNcFjP3FOwDMvW0Cw3t1alJNCneRVmTD3gJS\nEuNq3HTMKyoFR40uILbvP4oZ9O/Wjn+v2cvNzyzlmRvGsGnfEX72xlruvOhkBnVvz7eCA7S0TYrn\nX7eO56KH629NNKpvZ+6bOoKlOw5W9ebZ2o0Z0JXF2w4eN79NYjzr7pvcpM/UE6oirUhd14Prasee\nnvpZ+F8yvEfV5YGJQ9LISO/CiF6dMAu0nBkzoCsXnHoSABee2p2z0rvytbP7k5wQzwcbc0lKiOMr\nTy3ikatHcWb/LnRpm0S75ARO69OJw0Wl/G7+phPW/I3xA/j+JSdz8zNLGdazIzMvPYWCY2Wc9fP5\nHCurqFqveu+fsaauYAcoKm3c08lNoTN3EWmywpKyqhGzalu+8xCn9+nM91/OZNHWg3w08/wavXOe\n6LpzTvDaeFqH5Kp7GVtyj3BB8OnkL2f05Z31Odw0cQDbDxQyZkAXpo3uw/b9R/np62sY2adzVP9C\nqLxk0xS6LCMiUWlddj6J8XEM7t6+0e89cOQYZkbXdkn1rvvI/E2M7teZQ4Ul3P7CCi4edhKPfeUM\ncgqOsS+/mF6d2mAGr2fu4az0rjw8fyObc46QdSjQvPXp6Wdx7tDuTHpoATsPBsYjuHZsP+6/YgTX\nzV5MpzaJ3DBhAB3bJPLDl1dy4Mgxtjdg3II7LzqZmyYNJCWxac8AKNxFRMLEOce+/GMhb2hDoLvn\n5xbv5Moz+7A55wh9u7SlQ0oC76zP4R9LdvH7a0Y3OdCrU7iLiPhQWAfINrPJZrbBzDab2cw6lpuZ\nPRpcvtLMzmhK0SIiEh71hruZxQOPAZcCw4BrzGxYrdUuBYYEf2YAj4e5ThERaYSGnLmPATY757Y6\n50qAF4CptdaZCvzNBXwCdDaznmGuVUREGqgh4d4b2FXtdVZwXmPXwcxmmNkSM1uSm5tbe7GIiIRJ\nRMfrcs496ZzLcM5lpKWlRXLTIiKtSkPCfTfQt9rrPsF5jV1HREQipCHh/ikwxMwGmFkScDUwp9Y6\nc4Drgq1mxgF5zrnsMNcqIiINVG/fMs65MjP7NvBvIB6Y7ZxbY2a3BJc/AcwDpgCbgUJgesuVLCIi\n9fHsISYzywV2NPHtqcD+MJbjJe1LdPLLvvhlP0D7Uqm/c67em5aehXtzmNmShjyhFQu0L9HJL/vi\nl/0A7UtjRbS1jIiIRIbCXUTEh2I13J/0uoAw0r5EJ7/si1/2A7QvjRKT19xFROTEYvXMXURETiDm\nwr2+7oejjZltN7NVZrbCzJYE53U1s/+Y2abgn12qrf+j4L5tMLNLvKsczGy2meWY2epq8xpdu5md\nGfw72BzsGtqiZF9mmdnu4LFZYWZTon1fzKyvmS0ws7VmtsbMbg/Oj7njcoJ9icXjkmJmi80sM7gv\nPw3O9+64OOdi5ofAQ1RbgIFAEpAJDPO6rnpq3g6k1pr3EDAzOD0TeDA4PSy4T8nAgOC+xntY+yTg\nDGB1c2oHFgPjAAPeBC6Nkn2ZBXy/jnWjdl+AnsAZwekOwMZgvTF3XE6wL7F4XAxoH5xOBBYF6/Hs\nuMTamXtDuh+OBVOBvwan/wpcUW3+C865Y865bQSe+B3jQX0AOOc+AGoP396o2i3Q9XNH59wnLvAv\n92/V3hMxIfYllKjdF+dctnNuWXC6AFhHoAfWmDsuJ9iXUKJ5X5xz7kjwZWLwx+HhcYm1cG9Q18JR\nxgHzzWxgd9zgAAACEklEQVSpmc0IzjvJfdb3zl7gpOB0LOxfY2vvHZyuPT9afMcCo4fNrvaVOSb2\nxczSgdEEzhJj+rjU2heIweNiZvFmtgLIAf7jnPP0uMRauMeiCc65UQRGq7rVzCZVXxj87RyTTZZi\nufagxwlc4hsFZAO/8bachjOz9sA/gTucc/nVl8XacaljX2LyuDjnyoP/1/sQOAsfUWt5RI9LrIV7\nzHUt7JzbHfwzB3iVwGWWfcGvXwT/zAmuHgv719jadwena8/3nHNuX/A/ZAXwJz67BBbV+2JmiQTC\n8Fnn3CvB2TF5XOral1g9LpWcc4eBBcBkPDwusRbuDel+OGqYWTsz61A5DVwMrCZQ8/XB1a4H/hWc\nngNcbWbJZjaAwJi0iyNbdb0aVXvwK2m+mY0L3vW/rtp7PGU1h4KcRuDYQBTvS3C7fwbWOed+W21R\nzB2XUPsSo8clzcw6B6fbABcB6/HyuETyjnI4fgh0LbyRwN3lH3tdTz21DiRwRzwTWFNZL9ANeAfY\nBMwHulZ7z4+D+7YBD1qV1Kr/eQJfi0sJXPu7oSm1AxkE/oNuAf5A8OG5KNiXZ4BVwMrgf7ae0b4v\nwAQCX+1XAiuCP1Ni8bicYF9i8biMBJYHa14N3BOc79lx0ROqIiI+FGuXZUREpAEU7iIiPqRwFxHx\nIYW7iIgPKdxFRHxI4S4i4kMKdxERH1K4i4j40P8DLbGt5O66KzYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1228eefd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.plot(loss_track)\n",
    "print('loss {:.4f} after {} examples (batch_size={})'.format(loss_track[-1], len(loss_track)*batch_size, batch_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Something is definitely getting learned."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Limitations of the model\n",
    "\n",
    "We have no control over transitions of `tf.nn.dynamic_rnn`, it is unrolled in a single sweep. Some of the things that are not possible without such control:\n",
    "\n",
    "- We can't feed previously generated tokens without falling back to Python loops. This means *we cannot make efficient inference with dynamic_rnn decoder*!\n",
    "\n",
    "- We can't use attention, because attention conditions decoder inputs on its previous state\n",
    "\n",
    "Solution would be to use `tf.nn.raw_rnn` instead of `tf.nn.dynamic_rnn` for decoder, as we will do in tutorial #2. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fun things to try (aka Exercises)\n",
    "\n",
    "- In `copy_task` increasing `max_sequence_size` and `vocab_upper`. Observe slower learning and general performance degradation.\n",
    "\n",
    "- For `decoder_inputs`, instead of shifted target sequence `[<EOS> W X Y Z]`, try feeding `[<EOS> <PAD> <PAD> <PAD>]`, like we've done when we tested forward pass. Does it break things? Or slows learning?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}