Ex4.Classifying-newswires.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://www.th-koeln.de/img/logo.svg\" style=\"float: right;\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4th exercise: <font color=\"#C70039\">Multi-Class classification of newswires</font>\n",
    "* Course: DIS21a.1\n",
    "* Lecturer: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Author of notebook modifications and adaptations: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Date: 10.07.2020\n",
    "\n",
    "<img src=\"https://miro.medium.com/max/700/1*HgXA9v1EsqlrRDaC_iORhQ.png\" style=\"float: center;\" width=\"400\">\n",
    "\n",
    "**GENERAL NOTE 1**: \n",
    "Please make sure you are reading the entire notebook, since it contains a lot of information on your tasks (e.g. regarding the set of certain paramaters or a specific computational trick), and the written mark downs as well as comments contain a lot of information on how things work together as a whole. \n",
    "\n",
    "**GENERAL NOTE 2**: \n",
    "* Please, when commenting source code, just use English language only. \n",
    "* When describing an observation (for instance, after you have run through your test plan) you may use German language.\n",
    "This applies to all exercises in DIS 21a.1.  \n",
    "\n",
    "--------------------\n",
    "\n",
    "### <font color=\"ce33ff\">DESCRIPTION</font>:\n",
    "The previous exercises were dealing with classification problems, that was to classify data into two mutually exclusive classes using a densely-connected neural network. This time you will deal with a problem where you have more than two classes. \n",
    "\n",
    "This notebook allows you for classifying Reuters newswires into 46 different mutually-exclusive topics (multi-class classification problem), and since each data point should be classified into only one category, the problem is more specifically spoken a \"single-label, multi-class classification\" problem (compare lecture slide p.163). \n",
    "If each data point could have belonged to multiple categories (in our case, topics) then we would be facing a \"multi-label, multi-class classification\" problem.\n",
    "\n",
    "The example works with the so-called _Reuters dataset_, a set of short newswires and their topics, published by Reuters in 1986. It's a very simple, widely used toy dataset for text classification. There are 46 different topics; some topics are more represented than others, but each topic has at least 10 examples in the training set. Like IMDB and MNIST, the Reuters dataset comes packaged as part of Keras.\n",
    "\n",
    "-----------------------\n",
    "\n",
    "### <font color=\"FFC300\">TASKS</font>:\n",
    "The tasks that you need to work on in this notebook are always indicated below as bullet points. \n",
    "If a task is more challenging and consists of several steps, this is indicated as well. \n",
    "Make sure you have worked down the task list and commented your doings. \n",
    "This should be done by using markdown.<br> \n",
    "<font color=red>Make sure you don't forget to specify your name and your matriculation number in the notebook before submitting it.</font>\n",
    "\n",
    "**YOUR TASKS in this exercise are as follows**:\n",
    "1. import the notebook to Google Colab.\n",
    "2. make sure you specified you name and your matriculation number in the header below my name and date.\n",
    "    * set the date too and remove mine.\n",
    "3. read the entire notebook carefully\n",
    "    * add comments whereever you feel it necessary for better understanding\n",
    "    * run the notebook for the first time and note the result in your markdown result table (your test plan). \n",
    "4. decode the newswire back to english words by what you have learned from the previous exercise.\n",
    "5. go into the section 'building the ANN' \n",
    "    * add the missing code that does create a network as shown in the image in the lecture slides on page 172 (File: 'DIS21a.1-7.HANDS_ON.First.DLNetwork.Architectures.for.Solving.Three.Interesting.Problems.pdf')\n",
    "    * set the activation function to ReLu\n",
    "    * set the correct activation function in the last layer/the output. What is correct when doing a single-label, multi-class classification?\n",
    "    * add the missing code for compiling the network by setting\n",
    "        * the loss function \n",
    "        * the optimizer\n",
    "        * an evaluation metric, that makes sense\n",
    "6. optimize the hyperparameters, build a new the model and evaluate it on the test data \n",
    "    * determine the minimum number of epochs and train a new model from scratch for this number of epochs.\n",
    "    * evaluate it on the test set from the data set you have loaded.\n",
    "7. make combinations of this according to your testplan. Make sure you combine with sense and intelligence and not just chaotically.\n",
    "    * Try using larger or smaller layers: 32 units, 128 units...\n",
    "    * We were using two hidden layers. Now try to use a single hidden layer, or three hidden layers.\n",
    "8. comment your observations\n",
    "    * when is the accuracy increasing / decreasing? Describe your findings!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## START OF THE NOTEBOOK CODE\n",
    "----------------------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow\n",
    "tensorflow.keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### loading the reuters newswire data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import reuters\n",
    "(train_data, train_labels), (test_data, test_labels) = reuters.load_data(num_words=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like with the IMDB data set, the argument `num_words=10000` restricts the data to the 10.000 most frequently occurring words found in the data.\n",
    "\n",
    "There are 8,982 training examples and 2,246 test examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with the IMDB reviews, each example is a list of integers (word indices)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data[10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <font color=\"#00ff00\">Task 4:</font> decode one entry back to words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoded_newswire"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The label associated with an example is an integer between 0 and 45, since there are 46 topics: a topic index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_labels[10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### data preparation\n",
    "\n",
    "The data is going to be vectorized with the exact same code as in the previous exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1.\n",
    "    return results\n",
    "\n",
    "# Our vectorized training data\n",
    "x_train = vectorize_sequences(train_data)\n",
    "# Our vectorized test data\n",
    "x_test = vectorize_sequences(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To vectorize the labels, one-hot-encoding is used, since it is a widely used format for categorical data, very often also referred to as \"categorical encoding\". \n",
    "\n",
    "Here, one-hot encoding of the labels consists in embedding each label as an all-zero vector with a 1 in the place of the label index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_one_hot(labels, dimension=46):\n",
    "    results = np.zeros((len(labels), dimension))\n",
    "    for i, label in enumerate(labels):\n",
    "        results[i, label] = 1.\n",
    "    return results\n",
    "\n",
    "# Our vectorized training labels\n",
    "one_hot_train_labels = to_one_hot(train_labels)\n",
    "# Our vectorized test labels\n",
    "one_hot_test_labels = to_one_hot(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=red>Note</font> that there is a more elegant and built-in way to do this in Keras, which you have already seen in action in the MNIST example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.utils import to_categorical\n",
    "\n",
    "one_hot_train_labels = to_categorical(train_labels)\n",
    "one_hot_test_labels = to_categorical(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### building the ANN\n",
    "\n",
    "This topic classification problem looks very similar to our previous movie review classification problem: in both cases, we are trying to classify short snippets of text. There is however a new constraint here: the number of output classes has gone from 2 to 46, i.e. the dimensionality of the output space is much larger. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <font color =\"#00ff00\">Task 5:</font> build and compile the ANN\n",
    "as described in the task list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import models\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "model = models.Sequential()\n",
    "\n",
    "'''ADD THE MISSING CODE HERE'''\n",
    "'''LOOK AT THE TEXT ABOVE TO SEE WHAT PARAMETERS THE NETWORK SHALL HAVE'''\n",
    "\n",
    "# your code\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validating the ANN model\n",
    "\n",
    "Let's set apart 1000 samples from the training data to use as a validation set. The testing data remains untouched."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_val = x_train[:1000]\n",
    "partial_x_train = x_train[1000:]\n",
    "\n",
    "y_val = one_hot_train_labels[:1000]\n",
    "partial_y_train = one_hot_train_labels[1000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train the ANN for 20 epochs with batch_size=512 and use the history object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(partial_x_train,\n",
    "                    partial_y_train,\n",
    "                    epochs=20,\n",
    "                    batch_size=512,\n",
    "                    validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history.history.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize the loss and accuracy by using pyplot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(loss) + 1)\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='validation loss')\n",
    "plt.title('training / validation loss')\n",
    "plt.xlabel('epochs')\n",
    "plt.ylabel('loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.clf()   # clear the old figure (if you forget it, this might cause problems sometimes.)\n",
    "\n",
    "acc = history.history['accuracy']\n",
    "val_acc = history.history['val_accuracy']\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='validation acc')\n",
    "plt.title('training / validation accuracy')\n",
    "plt.xlabel('eochs')\n",
    "plt.ylabel('loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see the ANN starts overfitting after some epochs. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <font color =\"#00ff00\">Task 6:</font> Optimize the hyperparameters, build a new the model and evaluate it on the test data \n",
    "1. determine the minimum number of epochs and train a new model from scratch for this number of epochs\n",
    "    * set all missing hyperparameters and other parameters that are needed\n",
    "2. evaluate it on the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = models.Sequential()\n",
    "\n",
    "#add your parameters here\n",
    "model.add(layers.Dense(xxx, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(xxx, activation='relu'))\n",
    "model.add(layers.Dense(xxx, activation='softmax'))\n",
    "\n",
    "#add your parameters here\n",
    "model.compile(ADD YOUR params HERE)\n",
    "\n",
    "#add your parameters here\n",
    "model.fit(partial_x_train,\n",
    "          partial_y_train,\n",
    "          epochs=xxxx,\n",
    "          batch_size=xxxx,\n",
    "          validation_data=(x_val, y_val))\n",
    "\n",
    "results = model.evaluate(x_test, one_hot_test_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The approach reaches an accuracy of ~78%. With a balanced binary classification problem, the accuracy reached by a purely random classifier would be 50%, so your results seem pretty good, at least when compared to a random baseline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import copy\n",
    "\n",
    "test_labels_copy = copy.copy(test_labels)\n",
    "np.random.shuffle(test_labels_copy)\n",
    "float(np.sum(np.array(test_labels) == np.array(test_labels_copy))) / len(test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <font color=\"#C70039\">Include your result table here and reflect a good test plan (see task list)</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### generating predictions on new data\n",
    "\n",
    "The `predict` method of your model returns a probability distribution over all 46 topics (which sums to 1). \n",
    "Now, generate topic predictions for all of the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each entry in predictions is a vector of length 46."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The coefficients in this vector sum to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(predictions[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The largest entry is the predicted class, i.e. the class with the highest probability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.argmax(predictions[0])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}