bbc_classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# News articles classifier \n",
    "\n",
    "We use word vector representations to build a classifier of news. \n",
    "\n",
    "\n",
    "#### Outline\n",
    "1. We start with a baseline model (News_classifierV1) using word embeddings.\n",
    "2. We build a more sophisticated model (News_classifierV2) that incorporates an LSTM. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import sklearn\n",
    "from sklearn.utils import shuffle\n",
    "import re\n",
    "import matplotlib.pyplot as plt\n",
    "from utils import *\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 - Preparing the dataset\n",
    "\n",
    "### 1.1 - Dataset description\n",
    "\n",
    "We want to create a dataset (X, Y) where:\n",
    "- X is an array containing 2225 strings (the titles of the BBC articles).\n",
    "- Y is an array of shape (2225,) and contains an integer label between 0 and 4 corresponding to the categories:\n",
    "    * 0: Business\n",
    "    * 1: Entertainment\n",
    "    * 2: Politics\n",
    "    * 3: Sport\n",
    "    * 4: Tech\n",
    "\n",
    "\n",
    "Let's load the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "n_articles = 2225\n",
    "\n",
    "## We first declare X as a list, and then will turn it into an array\n",
    "X = []\n",
    "Y = np.zeros((n_articles,), dtype=int)\n",
    "\n",
    "directory = ['data/business', 'data/entertainment', 'data/politics', 'data/sport', 'data/tech']\n",
    "idx=0\n",
    "\n",
    "for i in range(5):\n",
    "    for filename in os.listdir(directory[i]):\n",
    "        f = open(directory[i] + '/' + filename, errors='replace')\n",
    "        first_line = f.readline()\n",
    "        X.append(first_line)\n",
    "        Y[idx] = i\n",
    "        idx = idx+1\n",
    "\n",
    "## We make the list X into an array X\n",
    "X = np.array(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the first few examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "UK economy facing 'major risks'\n",
      " [Business]\n",
      "Aids and climate top Davos agenda\n",
      " [Business]\n",
      "Asian quake hits European shares\n",
      " [Business]\n",
      "India power shares jump on debut\n",
      " [Business]\n",
      "Lacroix label bought by US firm\n",
      " [Business]\n"
     ]
    }
   ],
   "source": [
    "for idx in range(5):\n",
    "    print(X[idx], label_to_category(Y[idx]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 - Cleaning the data\n",
    "\n",
    "The main problem we may encounter is that some words appearing in the input strings are not present in our GloVe vocabulary. We decide to solve this isssue by simply removing these words from the input strings. Moreover, we remove any special characters and numbers and make all strings lowercase.\n",
    "\n",
    "First, let's load the vocabulary as a list `word_list` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Our vocabulary contains 400000 words.\n"
     ]
    }
   ],
   "source": [
    "## Get word list (vocabulary) from Glove file\n",
    "with open('data/glove.6B.50d.txt') as f:\n",
    "    text = f.readlines()\n",
    "\n",
    "word_list = [line.strip().split()[0] for line in text]\n",
    "\n",
    "print('Our vocabulary contains', len(word_list), 'words.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we normalize the text and print all words that are not present in the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13 phytopharm\n",
      "176 illva\n",
      "483 colludes\n",
      "700 mccririck\n",
      "823 holiff\n",
      "992 oaps\n",
      "1178 truanted\n",
      "1368 brizzel\n",
      "1910 blinx\n",
      "1912 gigapixel\n",
      "2010 dialler\n",
      "2123 robotiquette\n"
     ]
    }
   ],
   "source": [
    "## Remove numbers and special characters and set lowercase\n",
    "for i in range(n_articles):\n",
    "    X[i] = re.sub('[^A-Za-z ]+', ' ', X[i]).lower()\n",
    "    # remove whitespace from the beginning and end\n",
    "    X[i] = X[i].strip()\n",
    "        \n",
    "    # Print all words NOT in the vocabulary\n",
    "    for word in X[i].split() :\n",
    "        if not word in word_list :\n",
    "            print(i, word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we are dealing with only a few exceptions, we decide to simply remove  these words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(n_articles):\n",
    "    for word in X[i].split() :\n",
    "        if not word in word_list :\n",
    "            X[i] = X[i].replace(word, '')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take an overview of this dataset and the distribution of data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Titles (lenght of X):  2225 \n",
      "Number of Titles labelled Buiseness:  510 \n",
      "Number of Titles labelled Entertainment:  386 \n",
      "Number of Titles labelled Politics:  417 \n",
      "Number of Titles labelled Sport:  511 \n",
      "Number of Titles labelled Tech:  401 \n",
      "Total number of labelled Titles:  2225 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "Y0, Y1, Y2, Y3, Y4 = np.count_nonzero(Y == 0), np.count_nonzero(Y == 1), np.count_nonzero(Y == 2), np.count_nonzero(Y == 3), np.count_nonzero(Y == 4)\n",
    "\n",
    "print('Number of Titles (lenght of X): ', len(X), '\\n' \n",
    "'Number of Titles labelled Buiseness: ', Y0, '\\n'\n",
    "'Number of Titles labelled Entertainment: ', Y1, '\\n'     \n",
    "'Number of Titles labelled Politics: ', Y2, '\\n'     \n",
    "'Number of Titles labelled Sport: ', Y3, '\\n'     \n",
    "'Number of Titles labelled Tech: ', Y4, '\\n'\n",
    "'Total number of labelled Titles: ', Y0+Y1+Y2+Y3+Y4, '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 - Training and Test set\n",
    "\n",
    "We observe that the dataset is quite balanced with respect to the 5 categories.\n",
    "\n",
    "We shuffle the dataset and split it between training (2025 examples) and testing (200 examples)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = 2025\n",
    "\n",
    "X, Y = shuffle(X, Y)\n",
    "\n",
    "X_train, Y_train = X[:m], Y[:m]\n",
    "X_test, Y_test = X[m:], Y[m:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "maxLen = len(max(X_train, key=len).split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print some examples from the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fresh hope after argentine crisis [Business]\n",
      "umaga ready for lions [Sport]\n",
      "bond game fails to shake or stir [Tech]\n",
      "blair stresses prosperity goals [Politics]\n",
      "dawson joins england injury list [Sport]\n",
      "beckham rules out management move [Sport]\n",
      "mobile tv tipped as one to watch [Tech]\n",
      "blind student  hears in colour [Tech]\n",
      "ford gains from finance not cars [Business]\n",
      "us peer to peer pirates convicted [Tech]\n"
     ]
    }
   ],
   "source": [
    "for idx in range(10):\n",
    "    print(X_train[idx], label_to_category(Y_train[idx]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 - Building the baseline model\n",
    "\n",
    "\n",
    "#### Inputs and outputs\n",
    "* The input of the model is a string corresponding to a sentence.\n",
    "* The output will be a probability vector of shape (1,5), as there are 5 categories.\n",
    "* The (1,5) probability vector is passed to an argmax layer, which extracts the index of the category with the highest probability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### One-hot encoding\n",
    "To get our labels into a format suitable for training a softmax classifier, lets convert $Y$ from its current shape  $(m, 1)$ into a \"one-hot representation\" $(m, 5)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_oh_train = convert_to_one_hot(Y_train, C = 5)\n",
    "Y_oh_test = convert_to_one_hot(Y_test, C = 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 - Implementing News_classifierV1\n",
    "\n",
    "First step:\n",
    "* Convert each word in the input sentence into their word vector representations.\n",
    "* Then take an average of the word vectors. \n",
    "\n",
    "Let's load the `word_to_vec_map`, which contains all the vector representations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_to_index, index_to_word, word_to_vec_map = read_glove_vecs('data/glove.6B.50d.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Description:\n",
    "- `word_to_index`: dictionary mapping from words to their indices in the vocabulary \n",
    "- `index_to_word`: dictionary mapping from indices to their corresponding words in the vocabulary\n",
    "- `word_to_vec_map`: dictionary mapping words to their GloVe vector representation.\n",
    "\n",
    "As an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the index of hasard in the vocabulary is 174049\n",
      "the 259826th word in the vocabulary is neurotrophic\n"
     ]
    }
   ],
   "source": [
    "word = \"hasard\"\n",
    "idx = 259826\n",
    "print(\"the index of\", word, \"in the vocabulary is\", word_to_index[word])\n",
    "print(\"the\", str(idx) + \"th word in the vocabulary is\", index_to_word[idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now implement `sentence_to_avg()` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sentence_to_avg(sentence, word_to_vec_map):\n",
    "    \"\"\"\n",
    "    Converts a sentence (string) into a list of words (strings). Extracts the GloVe representation of each word\n",
    "    and averages its value into a single vector encoding the meaning of the sentence.\n",
    "    \n",
    "    Arguments:\n",
    "    sentence -- string, one training example from X\n",
    "    word_to_vec_map -- dictionary mapping every word in a vocabulary into its 50-dimensional vector representation\n",
    "    \n",
    "    Returns:\n",
    "    avg -- average vector encoding information about the sentence, numpy-array of shape (50,)\n",
    "    \"\"\"\n",
    "    \n",
    "    # Split sentence into list of lower case words\n",
    "    words = sentence.lower().split()\n",
    "\n",
    "    # Initialize the average word vector\n",
    "    avg = np.zeros(50,)\n",
    "    \n",
    "    # Average the word vectors\n",
    "    total = 0\n",
    "    for w in words:\n",
    "        total += word_to_vec_map[w]\n",
    "    avg = total / len(words)\n",
    "        \n",
    "    return avg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model\n",
    "\n",
    "We now implement the `model()` function. \n",
    "After using `sentence_to_avg()` we have 3 steps:\n",
    "* Pass the average through forward propagation\n",
    "* Compute the cost\n",
    "* Backpropagate to update the softmax parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def model(X, Y, word_to_vec_map, learning_rate = 0.01, num_iterations = 500):\n",
    "    \"\"\"\n",
    "    Model to train word vector representations in numpy.\n",
    "    \n",
    "    Arguments:\n",
    "    X -- input data, numpy array of sentences as strings, of shape (m, 1)\n",
    "    Y -- labels, numpy array of integers between 0 and 7, numpy-array of shape (m, 1)\n",
    "    word_to_vec_map -- dictionary mapping every word in a vocabulary into its 50-dimensional vector representation\n",
    "    learning_rate -- learning_rate for the stochastic gradient descent algorithm\n",
    "    num_iterations -- number of iterations\n",
    "    \n",
    "    Returns:\n",
    "    pred -- vector of predictions, numpy-array of shape (m, 1)\n",
    "    W -- weight matrix of the softmax layer, of shape (n_y, n_h)\n",
    "    b -- bias of the softmax layer, of shape (n_y,)\n",
    "    \"\"\"\n",
    "    \n",
    "    # Define number of training examples\n",
    "    m = Y.shape[0]      # number of training examples\n",
    "    n_y = 5             # number of classes  \n",
    "    n_h = 50            # dimensions of the GloVe vectors \n",
    "    \n",
    "    # Initialize parameters using Xavier initialization\n",
    "    W = np.random.randn(n_y, n_h) / np.sqrt(n_h)\n",
    "    b = np.zeros((n_y,))\n",
    "    \n",
    "    # Convert Y to Y_onehot with n_y classes\n",
    "    Y_oh = convert_to_one_hot(Y, C = n_y) \n",
    "    \n",
    "    # Optimization loop\n",
    "    for t in range(num_iterations): # Loop over the number of iterations\n",
    "        for i in range(m):          # Loop over the training examples\n",
    "            \n",
    "            # Average the word vectors of the words from the i'th training example\n",
    "            avg = sentence_to_avg(X[i], word_to_vec_map)\n",
    "\n",
    "            # Forward propagate the avg through the softmax layer\n",
    "            z = np.dot(W,avg) + b\n",
    "            a = softmax(z)\n",
    "\n",
    "            # Compute cost\n",
    "            cost = -1*np.dot(Y_oh, np.log(a))\n",
    "            \n",
    "            # Compute gradients \n",
    "            dz = a - Y_oh[i]\n",
    "            dW = np.dot(dz.reshape(n_y,1), avg.reshape(1, n_h))\n",
    "            db = dz\n",
    "\n",
    "            # Update parameters with Stochastic Gradient Descent\n",
    "            W = W - learning_rate * dW\n",
    "            b = b - learning_rate * db\n",
    "        \n",
    "        if t % 100 == 0:\n",
    "            print(\"Epoch: \" + str(t) + \" --- cost = \" + str(cost))\n",
    "            pred = predict(X, Y, W, b, word_to_vec_map) #predict is defined in utils.py\n",
    "\n",
    "    return pred, W, b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We train the model and learn the softmax parameters (W,b). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0 --- cost = [1.6217858  0.85731537 2.36928049 ... 0.85731537 0.85731537 1.66189324]\n",
      "Accuracy: 0.8069135802469136\n",
      "Epoch: 100 --- cost = [2.90939102 0.46946452 2.47849159 ... 0.46946452 0.46946452 1.75280549]\n",
      "Accuracy: 0.8790123456790123\n",
      "Epoch: 200 --- cost = [2.84056356 0.48392548 2.44812798 ... 0.48392548 0.48392548 1.67054283]\n",
      "Accuracy: 0.8790123456790123\n",
      "Epoch: 300 --- cost = [2.81939172 0.49315361 2.45239485 ... 0.49315361 0.49315361 1.63121982]\n",
      "Accuracy: 0.8809876543209877\n",
      "Epoch: 400 --- cost = [2.81114757 0.50087805 2.45912886 ... 0.50087805 0.50087805 1.60643082]\n",
      "Accuracy: 0.8819753086419753\n",
      "[[0.]\n",
      " [3.]\n",
      " [4.]\n",
      " ...\n",
      " [3.]\n",
      " [3.]\n",
      " [3.]]\n"
     ]
    }
   ],
   "source": [
    "pred, W, b = model(X_train, Y_train, word_to_vec_map)\n",
    "print(pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 - Examining test set performance "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set:\n",
      "Accuracy: 0.8819753086419753\n",
      "Test set:\n",
      "Accuracy: 0.855\n"
     ]
    }
   ],
   "source": [
    "print(\"Training set:\")\n",
    "pred_train = predict(X_train, Y_train, W, b, word_to_vec_map)\n",
    "print('Test set:')\n",
    "pred_test = predict(X_test, Y_test, W, b, word_to_vec_map)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Confusion matrix\n",
    "Let us try to understand which classes are more difficult for this model by plotting a confusion matrix. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(200,)\n",
      "Predicted  0.0  1.0  2.0  3.0  4.0  All\n",
      "Actual                                 \n",
      "0           41    0    4    1    1   47\n",
      "1            1   24    0    0    1   26\n",
      "2            1    0   29    1    1   32\n",
      "3            0    1    2   56    1   60\n",
      "4            4    5    4    1   21   35\n",
      "All         47   30   39   59   25  200\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARMAAAD3CAYAAAA+C7CYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAchUlEQVR4nO3debhcVZnv8e/vZIIIGCBhDiYqw2UMEGlbDI2gNIO3QexW0hcv3WAneKUd2n6E2M9tB5qrrQLthBiUBhRBbmOERkZp5qtAAsgUI2EQAmEIEIUIgYTf/WOt0uJwcs4+Vbuq9j7n/TxPPadqV9V+V53hPWvYay3ZJoQQ2tXX6wKEEEaGSCYhhFJEMgkhlCKSSQihFJFMQgiliGQSQihFJJMQRjhJUyVdK2mxpHslfTwf30TS1ZLuz183bnrPPElLJS2R9OeF4sR1JiGMbJK2BLa0fbukDYFFwOHA3wDP2v6SpBOBjW2fIGkn4Hxgb2Ar4GfA9rbXDhYnaiYhjHC2l9u+Pd9/HlgMbA0cBpyTX3YOKcGQj19ge7Xth4ClpMQyqLEll7tSJO0NjAPW2L6lR2Xos/1qF+L05LOOpriS5JpX5SVNA/YAbgE2t70cUsKRtFl+2dbAL5retiwfG9SIrZnkdt4lwKHA+ZKOl7RBF+IeKunzkr4oadMuJZJefdZRFRcYn+N35e9Gkodxu0fSwqbbnAHOtwFwEfAJ278bLPQAx4ZOorZH1C1/IyYAZwMfyMdmAFcD/wis38HYfwI8BPw1cAZwM/AOYNxI+qyjLW6Osx3wH8Cb8uO+TsVqilk4mQALhzjXOOBK4B+aji0h9aUAbAksyffnAfOaXncl8KdDlXfE1UycrCa1C3eTtIHtO4FPAIcAx3Qw/C7AVbZ/aPs40n+BTwN7Qvn/0Xr1WUdb3OwJ4DfAFyVNtf1qN2ookgrdhjiHgO8Bi22f2vTUJcDR+f7RwMVNx4+UNEHSdFIivXXIwnY6u/bqBhwMfBvYHRibj+0FPADs3qGYuwHnAjs2HftH4OfApJH0WUdLXGBXYEG+vyHwf4ALgan5WMdqKIDHjBlT6MYgNRPgnaRmyl3Anfl2CLApcA1wf/66SdN7/il/P5cABxcp74irmeQsjO3LgReAjwO75P9ii4ArGLhNWIYngDXAeyRNzuX4KnAPMLdDMXv1WbseV9KYHsR9mNTc+JHTSMgXSaMbp3S6hiKJvr6+QrfB2L7JtmzvZntGvl1m+xnbB9jeLn99tuk9J9t+i+0d8vd76PLmLFRrknYANgEWAq+6aTxc0pdJ/1FeAh4FPgXsY/vhkmKP6RdvD+BfSL/Y19m+O4/hv2r7yyXEeyswCbjH9kv9nuvYZ5W0MzCZVFV+qotx3wlMt/39/Hic7Ve6EHcL20/k+xOAfwcm2H5/vlZjHjAN+ExZv0v99fX1edy4cYVe+/LLLy+yPbMT5Siq9slE0hGkqudj+bYQONtNvdWS3kVqgmwPfMv2fSXE3d72r/P9MbbXNoYOc0KZS/qjN2mM/nDbd7cZ872kz/oMqRZ0su17+v2BdeKzHgz8K/AgqSNvju3HJI21vaYTcfN/+4mkIUwBX7d9Rn5uvUYi7dDn3RG4D/gacJ/tMyW9Afg3YIrtw3NCOQnYiPT9WNNu3P76+vo8fvz4Qq9dvXp1JJN2SBoH/ID0i3azpPcDbwdWA1+x/dt+rx9bxg89/1FfCPzE9l/nY42E0pervpOBjYG3AT93uvinnZjvAM4CZtu+Q9LpwHq2j8nPv+Z6lhI/637AfOAo27dKWkD6o/1Z/5hlxm0636eBtaT+kTtsn7aO15UWV9JU4AJSR+QBwHLgR6Tm6ieBbXMNZSNSbeXpMuL219fX5wkTJhR67UsvvdTzZDIS+kw2IvU2AywALiVdDzAbQNLbJR2anx/0cuAi8n+o40kjCC9L+gFATiRjm/641ti+32lkp61E0uRLtu/I9z8LbJKr4OQE9rac6KCEz5o9CczNiWQL0vD38ZK+A/xPgBy3tO9xP2uAqaQrNPeWdKqkL+a47+xEXNuPkkYv9iR1VF4BzCF1rn8XmCrp67Z/16lE0lDGaE631DqZ5Kr9qcARkmblP+SbSL3Vs/If2rZA41LitqthtleRhiB/SBqpWa8poTSq/LsDR0laT+X9pG8BfpzPP4Z0vcWbSMkUSdsAO5KaeaV81nyexbavzQ+PBU63fTjpCslD8n/x6ZT4Pe7nYuAJ29eQPttHgDfm57YoO27Tz+sEUhN1Mqlmsjtp1OOfSR2wp5cRb4iy1CqZ1LqZA6n9DHyY1G7+ge0b8vFrSf9Rf93h+JuSmgEv2j5K0m6kmtKN/TsqS4w5FlgPuNj2AZKOIl0i/bk84tAVki4nXQS1uIMxtgJOBv4f6Zqd75P6oC60fW6HYopUu/3fwJtJNZQTbf9E0nbACtvPdSJ2szFjxnj99dcv9NpVq1b1vJlT+7k5tl+SdB7pv8i83Hm2GtgM+O2gby4n/jOS5gJfkbSEVNvbt1OJJMdcA7wg6dFc5T8Q+NtOJpJG53LT4/eTvscd/aOy/bikR0l/2B+1/Z+503VpB2MaWC3p+8CNwDds/yQ/d3+n4g5kqGHfKql9MgGw/ZykM0k98HNJQ4VH2X6yS/FXSLqLdDHVe5wnT3VK/s85DpiVvx7Q6V/yRiLJTcejgH8APtgYPu2wM0m1sEX58fX9O347wfYSSScAb5I00fbvOx2zv6o0YYoYEckEwPbLwLWSbkgPO//L1qC0qMwhwIHtDv8Wkf+wX5Z0EnBbl/9bvkrqQzjC9pJuBMwdoo82akfd/NmSrl4+oovx/qBK/SFF1L7PpCqar33oYszaT4mvg17VSsaOHesNN9yw0GtXrlwZfSYjRbcTSY4ZiaQLepFIGupUM4lkEkKFRTIJIZQikkkIoW3Ks4broj4lbZEGWL5uJMaMuCMzbp2ugB3xyYQ0p2I0xIy4IzBunZJJNHNCqLCqJIoianGdyaRJk7zVVlu19N7nnnuOjTfeeOgXDmDixIktve/pp59mypQpLb0XoNWfyYoVK5g8eXJL723nl7bdz9uLuC+88ELLcVeuXMmkSZOG/b4nnniClStXFv5Gjx8/3kV/nsuXL4/rTIrYaqutOPfcjszpGtTMmb352bzyyitdjzl2bC1+FUpz8803dz3mhz/84WG/p041k9HQZxJCbZXVZyLpLElPSbqn6diPJN2Zbw9LujMfnybpxabnzihS1tH17yiEmilxaPhs4JukBZ4AsP3Bxn1Jp/DaWfYP2J4xnACRTEKoqDJHamzfoLQ16EBxBHwA2L+dGNHMCaHCujQ0PAt4st/s8+mS7pB0vaRZRU4SNZMQKmwYiWKypIVNj+fbnl/wvbOB85seLyctmv2MpL2An0ja2YPvTxzJJIQqG0YyWdHK0HBeAvQI0o6IADhtwbo6318k6QHSViILBzxJFskkhArrwtDwu4Ff2V7WFHMK8GzeceHNpDWNHxzqRNFnEkJFNSb6tbs9aD7X+aRV43aQtEzSsfmpI3ltEwdgX+AuSb8E/gM4zk1bh65L1ExCqLASR3Nmr+P43wxw7CLgouHG6EnNRNJBkpZIWqq0D28IYQB1mujX9WSitIHUt0grue8EzJa0U7fLEUIdRDIZ3N7AUtsP5hXlLwAO60E5Qqi0oolkNCeTrYFHmx4vy8dCCP3UKZn0ogN2oE/+ujn3SqtZzQHYYostOl2mECqpKomiiF7UTJaRdrVv2AZ4vP+LbM+3PdP2zFbXIwmh7soaGu6GXpTiNmA7SdMljSeNc1/Sg3KEUGl16zPpejPH9hpJxwNXAmOAs2zf2+1yhFAHVUkURfTkojXblwGX9SJ2CHUSySSEUIpIJiGEUkQyCSG0rUqdq0VEMgmhwqoy7FtEJJMQKixqJiGEUkQyCSG0LfpMQgiliWQSQihFJJOSTZw4kb322mvoF5bskUce6XpMgG233bYncXuh1U3a27X11t1f9WLcuHHDfk+dkkl9xp1CGGVKXlB6oL2GPyfpsaY9hQ9pem5eXlZ1iaQ/L1LeSCYhVFiJs4bPBg4a4Phptmfk22U55k6k2fw75/ecnpdbHVQkkxAqrKxkYvsGYMjtKrLDgAtsr7b9ELCUtNzqoCKZhFBhw0gmkyUtbLrNKRjieEl35WZQYxWylpZWrUUHbAij1TA6YFvZHvTbwEmkZVNPAk4BjqHg0qr9RTIJoaI6fdGa7SebYp0JXJofFlpatb9o5oRQYZ1ctlHSlk0P3wc0RnouAY6UNEHSdNJew7cOdb6omYRQYWXNGlbaa3g/Ut/KMuCzwH6SZpCaMA8DcwFs3yvpQuA+YA3wUdtrh4oRySSECiurmbOOvYa/N8jrTwZOHk6MSCYhVFRM9AshlKZOyaQnHbADXdobQni9Ou2b06vRnLMZ+NLeEEKTOiWTXu2bc4Okab2IHUJdNCb61UX0mYRQYVWpdRRR2WSS5xbMgdG1vkcIzeqUTCpbh7I93/ZM2zOnTJnS6+KE0BPRZxJCKEVVEkURvRoaPh/4ObCDpGWSju1FOUKosqK1kqoknF6N5gx0aW8IoZ+qJIoiopkTQoXF0HAIoW1VasIUEckkhAqLZBJCKEUkkxBCKSKZhBBKEckkhNC26IANIZSmTkPD9SlpCKNQWVfADrQgmaSvSPpV3oRrgaRJ+fg0SS/qj3sQn1GkrFEzGUSvZivff//9XY/51re+tesxe2n16tVdj2kPuY/V65TYzDkb+CZwbtOxq4F5ttdI+ldgHnBCfu4B2zOGEyBqJiFUVJlzcwbaa9j2VbbX5Ie/IG221bJIJiFU2DCSSat7DTccA1ze9Hi6pDskXS9pVpETRDMnhAobRjOnlb2GGzH+ibTZ1nn50HJgW9vPSNoL+ImknW3/brDzRDIJocI6PTQs6WjgvcABzp06tlcDq/P9RZIeALYHFg52rkgmIVRUpxeUlnQQqcP1z2z/vun4FOBZ22slvZm01/CDQ50vkkkIFVZWzUQD7zU8D5gAXJ3j/ML2ccC+wBckrQHWAsfZfnbAEzeJZBJChZWVTIaz17Dti4CLhhsjkkkIFRaX04cQShHJJITQtpjoF0IoTZ2SSdevgJU0VdK1khZLulfSx7tdhhDqoq+vr9CtCnpRM1kDfMr27ZI2BBZJutr2fT0oSwiVVqeaSdeTie3lpMt1sf28pMXA1kAkkxCaRJ/JMEiaBuwB3DLAc7FxeRj16pRMetbYkrQB6cKYTww0gSg2Lg8hNi4fkqRxpERynu0f96IMIdRBVRJFEetMJpK+AaxzaSjbH2sloNJ353vAYtuntnKOEEaDTk/0K9tgNZNBpxu3YR/gQ8Ddku7Mxz5j+7IOxQuhtkZEzcT2OZ0IaPsmoD7foRB6aEQkk4a8tsEJwE7Aeo3jtvfvYLlCCNQrmRRpkJ0HLAamA58HHgZu62CZQghZnUZziiSTTW1/D3jF9vW2jwHe3uFyhTDqlbk6fTcUGRp+JX9dLulQ4HHaXBI/hFBMVRJFEUWSyb9IeiPwKeAbwEbAJztaqhACUK/tQYdMJrYvzXd/C7yrs8UJITQbUTUTSf/OABev5b6TEEKHlNkfIuks0pYWT9neJR/bBPgRMI00sPIB28/l5+YBx5IWlP6Y7SuHilGkDnUp8NN8u4bUzHlhmJ8lhNCCEjtgzwYO6nfsROAa29uR/rZPzDF3Ao4Eds7vOV3SmKECFGnmvGaV6rxk/s8KFL5UvajutbLRdBne8pa3dD3mVVdd1fWYAAceeGBP4i5durTrMVvZLL3E1elvyLP0mx1G2v4C4BzgOtI1ZYcBF+TNuB6StBTYG/j5YDFamei3HRBrAoTQBcNIJpMlNU+BmW97/hDv2TyvL4Tt5ZI2y8e3Jm1k3rAsHxtUkT6T53ltn8kTpOwVQuiwbuw1PFDYAY4NWU0v0szZsKXihBDa0oVZw09K2jLXSrYEnsrHlwFTm163Den6skENWVJJ1xQ5FkIoX4evgL0EODrfPxq4uOn4kZImSJpO6tq4daiTDbaeyXrARFJbbGP+WPXZCNiqtbKHEIajw3sNfwm4UNKxwCPAXwHYvlfShaR1mdcAH7W9dqgYgzVz5gKfICWORfwxmfwO+FYLnyeEMEwd3msY4IB1vP5k4OThxBhsPZOvAV+T9Pe2vzGck4YQ2lelSXxFFOndeVXSpMYDSRtL+l+dK1IIoaFOs4aLJJO/s72y8SBfbvt3HStRCOEP6pRMily01idJzpeD5stqx3e2WCEEGGGzhoErST2+Z5AuXDkOuLyjpQohVKrWUUSRZHICaWe9j5BGdO4AtuxkoUIISZ2SyZB1KNuvkq7TfxCYSRpKWtxqQEnrSbpV0i8l3Svp862eK4SRbkT0mUjanjQNeTbwDGndA2y3u0DSamB/2y8o7ex3k6TLbf9iqDeGMNpUJVEUMVgz51fAjcB/t70UQFLbyzXmjtzGeijj8q03c/1DqLg6JZPBmjnvJ80QvlbSmZIOoKTNsySNUdrN7yngatu3DPCaOZIWSlr49NNPlxE2hFop2sSpSsJZZzKxvcD2B4EdSYumfBLYXNK3JbW1oo3ttbZnkGYj7i1plwFeM9/2TNszp0yZ0k64EGqrr6+v0K0KinTArrJ9nu33kv747yQv79aufDHcdbx+ObkQAvXqgB1WSrP9rO3vtLM1qKQpjcvzJa0PvJvUPxNC6KdOyaSVZRvbtSVwTr6Stg+4sGk7jRBCVqVEUUTXk4ntu4A9uh03hDqKZBJCKEUkkxBCKaoyUlNEJJMQKir6TEIIpYlkEkIoRYkLSu9Anl+XvRn4Z2ASabGzxmXmn7F9WSsxIpmEUGElLii9BJiRzzkGeAxYAPwtcJrtr7YbI5JJCBXWoWbOAcADtn9T5vnr01UcwigzzIl+kxsTY/NtziCnPhI4v+nx8ZLuknSW0h5ZLalFzcQ2r7zySk/i9kIvOt0OPLCtuZstW758eU/iTps2resxx48f/tLJwxgaLrTXsKTxwF8A8/KhbwMnkZYBOQk4BThm2AWlJskkhNGqA/9YDgZut/0kQONrjnUm0PLUlmjmhFBRHVrPZDZNTRylDcsb3gfc02p5o2YSQoWVWTORNBF4D2nr34YvS5pBauY83O+5YYlkEkKFlZlMbP8e2LTfsQ+Vdf5IJiFUWFwBG0IoRSSTEELbJMWs4RBCOaJmEkIoRSSTEEIpIpmEENpWt8WReta7k3f1u0NSrEwfwjrEVhfFfBxYDGzUwzKEUGlVSRRF9KRmImkb4FDgu72IH0Jd1Gl70F7VTP4N+DSwYY/ih1B5VWrCFNH1lCbpvcBTthcN8bo5jYVeVqxY0aXShVAtdeoz6UX9aB/gLyQ9DFwA7C/pB/1fZHu+7Zm2Z06ePLnbZQyhEiKZDML2PNvb2J5GWj7uv2wf1e1yhFAHdUomcZ1JCBVWlURRRE+Tie3rgOt6WYYQqqpKtY4iomYSQoVVZdi3iEgmIVRY1ExCCKWIZBJCaFvZfSb5cozngbXAGtszJW1C2oN4GmlB6Q/Yfq6V89enQRbCKNSBoeF32Z7RtGHXicA1trcDrsmPWxLJJIQK68J1JocB5+T75wCHt3qiaOaEUGHDGM2ZLGlh0+P5tuf3e42BqyQZ+E5+fnPbywFsL5e0WatljWQSQkUNs9ZRZK/hfWw/nhPG1ZJ+1V4JXyuaOSFUWJnNHNuP569PAQuAvYEnG1uE5q9PtVrWWtRMVq1axS233NL1uJtt1nKNry2rVq3qeszHHnus6zEBdtlll57E3XXXXXsSd7jKGs2R9Aagz/bz+f6BwBeAS4CjgS/lrxe3GqMWySSE0arEoeHNgQX5fGOBH9q+QtJtwIWSjgUeAf6q1QCRTEKosLKSie0Hgd0HOP4McEAZMSKZhFBRMdEvhFCamOgXQihF1ExCCKWIZBJCaFv0mYQQShPJJIRQikgmIYRSRDIJIbRNUq2GhjtaUknvk2RJO+bH0yTdk+/vJ+nSTsYPoe7qtG9Op9PebOAm0mZbIYRhimQCSNqAtBXosUQyCaEldUomnewzORy4wvavJT0raU/g2Q7GC2HEqUqiKKKTzZzZpI3JyV9nD+fNkuZIWihp4cqVK8suWwiVV7RWUpWE05GaiaRNgf2BXfJ6k2NI60+eXvQceX3K+QA77rijO1HOEKquKomiiE41c/4SONf23MYBSdcD23QoXggjUgwNpybNgn7HLgI+06F4IYxIo76ZY3u/AY59Hfh60+PrgOs6ET+EkaBKiaKI+tShQhiFyqiZSJoq6VpJiyXdK+nj+fjnJD0m6c58O6Sdssbl9CFUWEk1kzXAp2zfLmlDYJGkq/Nzp9n+ahlBIpmEUGFlJJO8Y19j177nJS0Gtm77xP1EMyeECiu7A1bSNGAPoLER1fGS7pJ0lqSN2ylrJJMQKqoxa7jIjbzXcNNtzgDn24A0qvoJ278Dvg28BZhBqrmc0k55o5kTQoUNo9Yx6F7DksaREsl5tn8MYPvJpufPBNqaxR81kxAqrKTRHAHfAxbbPrXp+JZNL3sfcE87ZY2aSQgVVtJozj7Ah4C7Jd2Zj30GmC1pBmmqy8PA3IHeXFQkkxAqqqyL1mzfBAx0osvaPnmTWiSTJUuWrJg1a9ZvWnz7ZGBFmeWpaMyIW/24bxruG+p0BWwtkontKa2+V9LCwTqmOqEXMSPuyIwbySSEUIo6zRqOZBJCRdVtot9oSCbzR0nMiDsC49YpmdSnDtWivGLbiIkpaW2e4XmPpP8raWKrcSWdLekv8/3vStppkNfuJ+kdAz03WFxJD0uaPJxyFdWLn22349ZpPZMRn0xGoBdtz7C9C/AycFzzk5LGtHJS2x+2fd8gL9kPGDCZhM6JZBK65UbgrbnWcK2kH5IuTBoj6SuSbsuTuOZCuhJS0jcl3Sfpp8BmjRNJuk7SzHz/IEm3S/qlpGvy5LDjgE/mWtEsSVMkXZRj3CZpn/zeTSVdJekOSd9h4OsbQkF1Siajoc9kRJI0FjgYuCIf2hvYxfZDeZLXb22/TdIE4GZJV5Fmi+4A7ApsDtwHnNXvvFOAM4F987k2sf2spDOAFxprX+TEdZrtmyRtC1wJ/Dfgs8BNtr8g6VDgdRPOQjGNiX51EcmkftZvuiT6RtKci3cAt9p+KB8/ENit0R8CvBHYDtgXON/2WuBxSf81wPnfDtzQOJftde119G5gp6b/ihspLbyzL3BEfu9PJT3X2scMUK8O2Egm9fOi7RnNB/Iv3KrmQ8Df276y3+sOIc3DGIwKvAZSE/lPbb84QFlia5KS1CmZ1KcOFYbjSuAjStPOkbS9pDcANwBH5j6VLYF3DfDenwN/Jml6fu8m+fjzwIZNr7sKOL7xIE8YI8f4H/nYwUBbC+6MZkX7S6qScCKZjEzfJfWH3C7pHuA7pFroAuB+4G7SwjjX93+j7adJ/Rw/lvRL4Ef5qf8E3tfogAU+BszMHbz38cdRpc8D+0q6ndTceqRDn3FUqFMykR010hCqaM899/SNN95Y6LUbbLDBol7MU2oWfSYhVFhVah1FRDIJoaJiaDiEUJqomYQQShHJJIRQijolk/o0yEIYhcoaGs7zrZZIWirpxE6UNZJJCBVV1kVrSjPJv0Way7UTaVX6dS430apIJiFUWEk1k72BpbYftP0ycAFwWNlljT6TECqspKHhrYFHmx4vA/6kjBM3i2QSQkUtWrToShVfpW49SQubHs9vWhFuoKpL6Ze+RzIJoaJsH1TSqZYBU5sebwM8XtK5/yD6TEIY+W4DtpM0XdJ44EjgkrKDRM0khBHO9hpJx5OWphgDnGX73rLjxKzhEEIpopkTQihFJJMQQikimYQQShHJJIRQikgmIYRSRDIJIZQikkkIoRSRTEIIpfj/SkCuUDnQFuYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 288x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(Y_test.shape)\n",
    "print(pd.crosstab(Y_test, pred_test.reshape(200,), rownames=['Actual'], colnames=['Predicted'], margins=True))\n",
    "plot_confusion_matrix(Y_test, pred_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like the main mistake is confusion between 'Business' (class 0) and 'Politics' (class 2) -- not so surprising even for human-level performance!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 - News_classifierV2 using LSTMs in Keras\n",
    "\n",
    "Let's build an LSTM model that takes word **sequences** as input.\n",
    "* This model will be able to account for the word ordering. \n",
    "* News_classifierV2 will continue to use pre-trained word embeddings to represent words.\n",
    "* We will feed word embeddings into an LSTM.\n",
    "* The LSTM will learn to predict the most appropriate category. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from keras.models import Model\n",
    "from keras.layers import Dense, Input, Dropout, LSTM, Activation\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.preprocessing import sequence\n",
    "from keras.initializers import glorot_uniform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 - The Embedding layer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inputs and outputs to the embedding layer\n",
    "\n",
    "* The `Embedding()` layer's input is an integer matrix of size **(batch size, max input length)**. \n",
    "    * This input corresponds to sentences converted into lists of indices (integers).\n",
    "    * The largest integer (the highest word index) in the input should be no larger than the vocabulary size.\n",
    "* The embedding layer outputs an array of shape (batch size, max input length, dimension of word vectors).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prepare the input sentences\n",
    "* We implement `sentences_to_indices`, which processes an array of sentences (X) and returns inputs to the embedding layer:\n",
    "    * Convert each training sentences into a list of indices (the indices correspond to each word in the sentence)\n",
    "    * Zero-pad all these lists so that their length is the length of the longest sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sentences_to_indices(X, word_to_index, max_len):\n",
    "    \"\"\"\n",
    "    Converts an array of sentences (strings) into an array of indices corresponding to words in the sentences.\n",
    "    The output shape should be such that it can be given to `Embedding()` (described in Figure 4). \n",
    "    \n",
    "    Arguments:\n",
    "    X -- array of sentences (strings), of shape (m, 1)\n",
    "    word_to_index -- a dictionary containing the each word mapped to its index\n",
    "    max_len -- maximum number of words in a sentence. You can assume every sentence in X is no longer than this. \n",
    "    \n",
    "    Returns:\n",
    "    X_indices -- array of indices corresponding to words in the sentences from X, of shape (m, max_len)\n",
    "    \"\"\"\n",
    "    \n",
    "    m = X.shape[0]  # number of training examples\n",
    "    \n",
    "    # Initialize X_indices as a numpy matrix of zeros and the correct shape (≈ 1 line)\n",
    "    X_indices = np.zeros((m, max_len))\n",
    "    \n",
    "    for i in range(m):  # loop over training examples\n",
    "        \n",
    "        # Convert ith training sentence in lower case and split it into words; get a list of words.\n",
    "        sentence_words = X[i].lower().split()\n",
    "        \n",
    "        # Initialize j to 0\n",
    "        j = 0\n",
    "        \n",
    "        # Loop over the words of sentence_words\n",
    "        for w in sentence_words:\n",
    "            # Set the (i,j)th entry of X_indices to the index of the correct word.\n",
    "            X_indices[i, j] = word_to_index[w]\n",
    "            j = j + 1\n",
    "                \n",
    "    return X_indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build embedding layer\n",
    "\n",
    "* Let's build the `Embedding()` layer in Keras, using pre-trained word vectors. \n",
    "* The embedding layer takes as input a list of word indices: `sentences_to_indices()` creates these word indices.\n",
    "* The embedding layer will return the word embeddings for a sentence. \n",
    "\n",
    "We implement `pretrained_embedding_layer()` with these steps:\n",
    "1. Initialize the embedding matrix as a numpy array of zeros.\n",
    "2. Fill in each row of the embedding matrix with the vector representation of a word\n",
    "3. Define the Keras embedding layer. Note:\n",
    "    * Setting `trainable = True`, will allow the optimization algorithm to modify the values of the word embeddings. In this case, we don't want the model to modify the word embeddings.\n",
    "4. Set the embedding weights to be equal to the embedding matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pretrained_embedding_layer(word_to_vec_map, word_to_index):\n",
    "    \"\"\"\n",
    "    Creates a Keras Embedding() layer and loads in pre-trained GloVe 50-dimensional vectors.\n",
    "    \n",
    "    Arguments:\n",
    "    word_to_vec_map -- dictionary mapping words to their GloVe vector representation.\n",
    "    word_to_index -- dictionary mapping from words to their indices in the vocabulary (400,001 words)\n",
    "\n",
    "    Returns:\n",
    "    embedding_layer -- pretrained layer Keras instance\n",
    "    \"\"\"\n",
    "    \n",
    "    vocab_len = len(word_to_index) + 1                  # adding 1 to fit Keras embedding (requirement)\n",
    "    emb_dim = word_to_vec_map[\"cucumber\"].shape[0]      # define dimensionality of your GloVe word vectors (= 50)\n",
    "    \n",
    "    # Initialize the embedding matrix as a numpy array of zeros.\n",
    "    # See instructions above to choose the correct shape.\n",
    "    emb_matrix = np.zeros((vocab_len, emb_dim))\n",
    "    \n",
    "    # Set each row \"idx\" of the embedding matrix to be \n",
    "    # the word vector representation of the idx'th word of the vocabulary\n",
    "    for word, idx in word_to_index.items():\n",
    "        emb_matrix[idx, :] = word_to_vec_map[word]\n",
    "\n",
    "    # Define Keras embedding layer with the correct input and output sizes\n",
    "    # Make it non-trainable.\n",
    "    embedding_layer = Embedding(vocab_len, emb_dim, trainable=False)\n",
    "\n",
    "    # Build the embedding layer, it is required before setting the weights of the embedding layer. \n",
    "    embedding_layer.build((None,))\n",
    "    \n",
    "    # Set the weights of the embedding layer to the embedding matrix.\n",
    "    embedding_layer.set_weights([emb_matrix])\n",
    "    \n",
    "    return embedding_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Building the Model\n",
    "\n",
    "Let's now build the News_classifierV2 model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def News_classifierV2(input_shape, word_to_vec_map, word_to_index):\n",
    "    \"\"\"\n",
    "    Function creating the model's graph.\n",
    "    \n",
    "    Arguments:\n",
    "    input_shape -- shape of the input, usually (max_len,)\n",
    "    word_to_vec_map -- dictionary mapping every word in a vocabulary into its 50-dimensional vector representation\n",
    "    word_to_index -- dictionary mapping from words to their indices in the vocabulary (400,001 words)\n",
    "\n",
    "    Returns:\n",
    "    model -- a model instance in Keras\n",
    "    \"\"\"\n",
    "    \n",
    "    # Define sentence_indices as the input of the graph.\n",
    "    # It should be of shape input_shape and dtype 'int32' (as it contains indices, which are integers).\n",
    "    sentence_indices = Input(input_shape, dtype='int32')\n",
    "    \n",
    "    # Create the embedding layer pretrained with GloVe Vectors\n",
    "    embedding_layer = pretrained_embedding_layer(word_to_vec_map, word_to_index)\n",
    "    \n",
    "    # Propagate sentence_indices through your embedding layer\n",
    "    embeddings = embedding_layer(sentence_indices)  \n",
    "    \n",
    "    # Propagate the embeddings through an LSTM layer with 128-dimensional hidden state\n",
    "    X = LSTM(128, return_sequences = True)(embeddings)\n",
    "    # Add dropout with a probability of 0.5\n",
    "    X = Dropout(0.5)(X)\n",
    "    # Propagate X trough another LSTM layer with 128-dimensional hidden state\n",
    "    # The returned output should be a single hidden state, not a batch of sequences.\n",
    "    X = LSTM(128, return_sequences=False)(X)\n",
    "    # Add dropout with a probability of 0.5\n",
    "    X = Dropout(0.5)(X)\n",
    "    # Propagate X through a Dense layer with 5 units\n",
    "    X = Dense(5)(X)\n",
    "    # Add a softmax activation\n",
    "    X = Activation('softmax')(X)\n",
    "    \n",
    "    # Create Model instance which converts sentence_indices into X.\n",
    "    model = Model(inputs=sentence_indices, outputs=X)\n",
    "    \n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Model Summary**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         [(None, 9)]               0         \n",
      "_________________________________________________________________\n",
      "embedding (Embedding)        (None, 9, 50)             20000050  \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, 9, 128)            91648     \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 9, 128)            0         \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 128)               131584    \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 5)                 645       \n",
      "_________________________________________________________________\n",
      "activation (Activation)      (None, 5)                 0         \n",
      "=================================================================\n",
      "Total params: 20,223,927\n",
      "Trainable params: 223,877\n",
      "Non-trainable params: 20,000,050\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = News_classifierV2((maxLen,), word_to_vec_map, word_to_index)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compile and train the model\n",
    "Compile model using `categorical_crossentropy` loss, `adam` optimizer and `['accuracy']` metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "News_classifierV2 `model` takes as input an array of shape (`m`, `max_len`) and outputs probability vectors of shape (`m`, `number of classes`). We thus have to convert X_train (array of sentences as strings) to X_train_indices (array of sentences as list of word indices), and Y_train (labels as indices) to Y_train_oh (labels as one-hot vectors)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_indices = sentences_to_indices(X_train, word_to_index, maxLen)\n",
    "Y_train_oh = convert_to_one_hot(Y_train, C = 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit the Keras model on `X_train_indices` and `Y_train_oh`. We will use `epochs = 50` and `batch_size = 32`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "64/64 [==============================] - 6s 34ms/step - loss: 1.1746 - accuracy: 0.5719\n",
      "Epoch 2/50\n",
      "64/64 [==============================] - 2s 31ms/step - loss: 0.4918 - accuracy: 0.8464 1s -\n",
      "Epoch 3/50\n",
      "64/64 [==============================] - 1s 16ms/step - loss: 0.4024 - accuracy: 0.8671\n",
      "Epoch 4/50\n",
      "64/64 [==============================] - 1s 15ms/step - loss: 0.3444 - accuracy: 0.8867\n",
      "Epoch 5/50\n",
      "64/64 [==============================] - 1s 15ms/step - loss: 0.3158 - accuracy: 0.9069\n",
      "Epoch 6/50\n",
      "64/64 [==============================] - 1s 15ms/step - loss: 0.2915 - accuracy: 0.9036\n",
      "Epoch 7/50\n",
      "64/64 [==============================] - 1s 15ms/step - loss: 0.2432 - accuracy: 0.9187\n",
      "Epoch 8/50\n",
      "64/64 [==============================] - 1s 15ms/step - loss: 0.2227 - accuracy: 0.9287\n",
      "Epoch 9/50\n",
      "64/64 [==============================] - 1s 15ms/step - loss: 0.1937 - accuracy: 0.9443\n",
      "Epoch 10/50\n",
      "64/64 [==============================] - 1s 15ms/step - loss: 0.1585 - accuracy: 0.9488\n",
      "Epoch 11/50\n",
      "64/64 [==============================] - 1s 15ms/step - loss: 0.1363 - accuracy: 0.9549\n",
      "Epoch 12/50\n",
      "64/64 [==============================] - 1s 15ms/step - loss: 0.1318 - accuracy: 0.9571\n",
      "Epoch 13/50\n",
      "64/64 [==============================] - 1s 15ms/step - loss: 0.1109 - accuracy: 0.9636\n",
      "Epoch 14/50\n",
      "64/64 [==============================] - 1s 15ms/step - loss: 0.0858 - accuracy: 0.9794\n",
      "Epoch 15/50\n",
      "64/64 [==============================] - 1s 15ms/step - loss: 0.0555 - accuracy: 0.9855\n",
      "Epoch 16/50\n",
      "64/64 [==============================] - 2s 24ms/step - loss: 0.1033 - accuracy: 0.9654\n",
      "Epoch 17/50\n",
      "64/64 [==============================] - 2s 26ms/step - loss: 0.0721 - accuracy: 0.9758\n",
      "Epoch 18/50\n",
      "64/64 [==============================] - 2s 24ms/step - loss: 0.0446 - accuracy: 0.9895\n",
      "Epoch 19/50\n",
      "64/64 [==============================] - 1s 16ms/step - loss: 0.0555 - accuracy: 0.9771\n",
      "Epoch 20/50\n",
      "64/64 [==============================] - 1s 15ms/step - loss: 0.0850 - accuracy: 0.9770\n",
      "Epoch 21/50\n",
      "64/64 [==============================] - 1s 15ms/step - loss: 0.0148 - accuracy: 0.9972\n",
      "Epoch 22/50\n",
      "64/64 [==============================] - 1s 15ms/step - loss: 0.0191 - accuracy: 0.9948\n",
      "Epoch 23/50\n",
      "64/64 [==============================] - 1s 15ms/step - loss: 0.0118 - accuracy: 0.9975\n",
      "Epoch 24/50\n",
      "64/64 [==============================] - 1s 20ms/step - loss: 0.0810 - accuracy: 0.9787\n",
      "Epoch 25/50\n",
      "64/64 [==============================] - 1s 19ms/step - loss: 0.0354 - accuracy: 0.9886\n",
      "Epoch 26/50\n",
      "64/64 [==============================] - 1s 16ms/step - loss: 0.0298 - accuracy: 0.9947\n",
      "Epoch 27/50\n",
      "64/64 [==============================] - 1s 16ms/step - loss: 0.0080 - accuracy: 0.9991\n",
      "Epoch 28/50\n",
      "64/64 [==============================] - 1s 14ms/step - loss: 0.0043 - accuracy: 0.9990\n",
      "Epoch 29/50\n",
      "64/64 [==============================] - 1s 16ms/step - loss: 0.0027 - accuracy: 1.0000\n",
      "Epoch 30/50\n",
      "64/64 [==============================] - 1s 17ms/step - loss: 0.0010 - accuracy: 1.0000\n",
      "Epoch 31/50\n",
      "64/64 [==============================] - 2s 23ms/step - loss: 0.0016 - accuracy: 1.0000\n",
      "Epoch 32/50\n",
      "64/64 [==============================] - 2s 31ms/step - loss: 9.5407e-04 - accuracy: 1.0000\n",
      "Epoch 33/50\n",
      "64/64 [==============================] - 2s 31ms/step - loss: 0.0016 - accuracy: 0.9999\n",
      "Epoch 34/50\n",
      "64/64 [==============================] - 1s 17ms/step - loss: 0.0164 - accuracy: 0.9954\n",
      "Epoch 35/50\n",
      "64/64 [==============================] - 1s 21ms/step - loss: 0.0145 - accuracy: 0.9962\n",
      "Epoch 36/50\n",
      "64/64 [==============================] - 1s 18ms/step - loss: 0.1625 - accuracy: 0.9578\n",
      "Epoch 37/50\n",
      "64/64 [==============================] - 1s 19ms/step - loss: 0.0471 - accuracy: 0.9877\n",
      "Epoch 38/50\n",
      "64/64 [==============================] - 2s 26ms/step - loss: 0.0439 - accuracy: 0.9864\n",
      "Epoch 39/50\n",
      "64/64 [==============================] - 1s 17ms/step - loss: 0.0141 - accuracy: 0.9952\n",
      "Epoch 40/50\n",
      "64/64 [==============================] - 1s 17ms/step - loss: 0.0035 - accuracy: 0.9996\n",
      "Epoch 41/50\n",
      "64/64 [==============================] - 1s 19ms/step - loss: 0.0023 - accuracy: 1.0000\n",
      "Epoch 42/50\n",
      "64/64 [==============================] - 2s 25ms/step - loss: 0.0014 - accuracy: 1.0000\n",
      "Epoch 43/50\n",
      "64/64 [==============================] - 2s 25ms/step - loss: 8.7017e-04 - accuracy: 1.0000\n",
      "Epoch 44/50\n",
      "64/64 [==============================] - 1s 20ms/step - loss: 7.7519e-04 - accuracy: 1.0000\n",
      "Epoch 45/50\n",
      "64/64 [==============================] - 1s 22ms/step - loss: 6.5019e-04 - accuracy: 1.0000\n",
      "Epoch 46/50\n",
      "64/64 [==============================] - 1s 18ms/step - loss: 7.7978e-04 - accuracy: 1.0000\n",
      "Epoch 47/50\n",
      "64/64 [==============================] - 1s 17ms/step - loss: 6.8541e-04 - accuracy: 1.0000\n",
      "Epoch 48/50\n",
      "64/64 [==============================] - 1s 17ms/step - loss: 4.9578e-04 - accuracy: 1.0000\n",
      "Epoch 49/50\n",
      "64/64 [==============================] - 1s 21ms/step - loss: 4.3916e-04 - accuracy: 1.0000\n",
      "Epoch 50/50\n",
      "64/64 [==============================] - 2s 23ms/step - loss: 3.0743e-04 - accuracy: 1.0000\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fabfa08e490>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_indices, Y_train_oh, epochs = 50, batch_size = 32, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 - Model evaluation\n",
    "\n",
    "\n",
    "#### Evaluate model on the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7/7 [==============================] - 1s 8ms/step - loss: 0.6690 - accuracy: 0.8950\n",
      "\n",
      "Test accuracy =  0.8949999809265137\n"
     ]
    }
   ],
   "source": [
    "X_test_indices = sentences_to_indices(X_test, word_to_index, max_len = maxLen)\n",
    "Y_test_oh = convert_to_one_hot(Y_test, C = 5)\n",
    "loss, acc = model.evaluate(X_test_indices, Y_test_oh)\n",
    "print()\n",
    "print(\"Test accuracy = \", acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us look at the mislabelled examples. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Expected category[Entertainment] prediction: wife swap makers sue us  copycat[Business]\n",
      "Expected category[Tech] prediction: cabs collect mountain of mobiles[Business]\n",
      "Expected category[Tech] prediction: ea to take on film and tv giants[Entertainment]\n",
      "Expected category[Entertainment] prediction: da vinci code is  lousy history[Tech]\n",
      "Expected category[Tech] prediction: warning over tsunami aid website[Entertainment]\n",
      "Expected category[Tech] prediction: gates opens biggest gadget fair[Entertainment]\n",
      "Expected category[Politics] prediction: ministers  naive  over phone taps[Tech]\n",
      "Expected category[Entertainment] prediction: new media battle for bafta awards[Tech]\n",
      "Expected category[Business] prediction: ban on forced retirement under[Politics]\n",
      "Expected category[Business] prediction: kraft cuts snack ads for children[Politics]\n",
      "Expected category[Tech] prediction: bush website blocked outside us[Politics]\n",
      "Expected category[Tech] prediction: halo fans  hope for sequel[Entertainment]\n",
      "Expected category[Sport] prediction: edu describes tunnel fracas[Tech]\n",
      "Expected category[Business] prediction: durex maker ssl awaits firm bid[Tech]\n",
      "Expected category[Tech] prediction: virgin radio offers  g broadcast[Entertainment]\n",
      "Expected category[Sport] prediction: legendary dutch boss michels dies[Business]\n",
      "Expected category[Business] prediction: giving financial gifts to children[Politics]\n",
      "Expected category[Sport] prediction: reaction from spanish press[Business]\n",
      "Expected category[Sport] prediction: tv calls after carroll error[Entertainment]\n",
      "Expected category[Tech] prediction: warning over tsunami aid website[Entertainment]\n",
      "Expected category[Entertainment] prediction: t in the park sells out in days[Business]\n"
     ]
    }
   ],
   "source": [
    "# Print the mislabelled examples\n",
    "C = 5\n",
    "y_test_oh = np.eye(C)[Y_test.reshape(-1)]\n",
    "X_test_indices = sentences_to_indices(X_test, word_to_index, maxLen)\n",
    "pred = model.predict(X_test_indices)\n",
    "for i in range(len(X_test)):\n",
    "    x = X_test_indices\n",
    "    num = np.argmax(pred[i])\n",
    "    if(num != Y_test[i]):\n",
    "        print('Expected category'+ label_to_category(Y_test[i]) + ' prediction: '+ X_test[i] + label_to_category(num).strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acknowledgments\n",
    "\n",
    "* This work is based on what I learned from Coursera's Machine Learning Specialization by Andrew Ng and in particular uses code taken by the assignments I completed for the course on Sequence Models. \n",
    "* I used a (Public Domain) pretrained GloVe word embedding representation, dowloadable <a href=\"https://www.kaggle.com/watts2/glove6b50dtxt\">here</a>.\n",
    "* BBC Dataset: D. Greene and P. Cunningham. \"Practical Solutions to the Problem of Diagonal Dominance in Kernel Document Clustering\", Proc. ICML 2006.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "nlp-sequence-models",
   "graded_item_id": "RNnEs",
   "launcher_item_id": "acNYU"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}