Ex11.Learning-and-Using-word-embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://www.th-koeln.de/img/logo.svg\" style=\"float: right;\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 11th exercise: <font color=\"#C70039\">Learning and using word embeddings</font>\n",
    "* Course: DIS21a.1\n",
    "* Lecturer: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Author of notebook modifications and adaptations: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Date:   09.09.2024\n",
    "\n",
    "<img src=\"https://media.springernature.com/lw685/springer-static/image/art%3A10.1007%2Fs12115-019-00362-9/MediaObjects/12115_2019_362_Fig2_HTML.png?as=webp\" style=\"float: center;\" width=\"600\">\n",
    "\n",
    "---------------------------------\n",
    "**GENERAL NOTE 1**: \n",
    "Please make sure you are reading the entire notebook, since it contains a lot of information about your tasks (e.g. regarding the set of certain paramaters or specific computational tricks, etc.), and the written mark downs as well as comments contain a lot of information on how things work together as a whole. \n",
    "\n",
    "**GENERAL NOTE 2**: \n",
    "* Please, when commenting source code, just use English language only. \n",
    "* When describing an observation (for instance, after you have run through your test plan) you may use German language.\n",
    "This applies to all exercises in DIS 21a.1.  \n",
    "\n",
    "---------------------\n",
    "\n",
    "### <font color=\"ce33ff\">DESCRIPTION</font>:\n",
    "\n",
    "This notebook allows for learning how to create and use word embeddings as another popular and powerful way for associating a word with a vector. While the vectors obtained through one-hot encoding are binary and sparse, which means mostly made of zeros as well as high-dimensional, which means show the same dimensionality as the number of words in the vocabulary, \"word embeddings\" are low-dimensional floating point vectors. In other words these are dense vectors, in contrast to sparse vectors. \n",
    "Unlike word vectors created by one-hot encoding, word embeddings are `learned` from data. It is common to see word embeddings that are 256-dimensional, 512-dimensional or 1024-dimensional when dealing with very large vocabularies. On the other hand, one-hot encoding words generally leads to vectors that are 20.000-dimensional or higher (capturing a vocabulary of 20.000 \n",
    "token in this case). So, word embeddings pack more information into far fewer dimensions. \n",
    "\n",
    "There are two ways to get word embeddings:\n",
    "\n",
    "1. Learn word embeddings jointly with the main task, for example document classification or sentiment prediction as already done in the previous exercises. \n",
    "    * start with random word vectors, then learn word vectors in the same way the weights of a neural network are learned.\n",
    "2. Load into the model word embeddings that were pre-computed using a different machine learning task than the one you are trying to solve. \n",
    "    * these are called \"pre-trained word embeddings\"\n",
    "\n",
    "Each of them is taught in a separate PART of this notebook.\n",
    "\n",
    "So let's go for it.\n",
    "\n",
    "-------------------------------------------------------------------------------------------------------------\n",
    "\n",
    "### <font color=\"FFC300\">TASKS</font>:\n",
    "Within this notebook, the tasks that you need to work on are always listed as bullet points below. \n",
    "If a task is more challenging and consists of several steps, this is indicated as well. \n",
    "Make sure you have worked down the task list and commented your doings. \n",
    "This should be done using markdown.<br> \n",
    "<font color=red>Make sure you don't forget to specify your name and your matriculation number in the notebook before submitting it.</font>\n",
    "\n",
    "**YOUR TASKS in this exercise are as follows**:\n",
    "1. import the notebook to Google Colab.\n",
    "2. make sure you specified you name and your matriculation number in the header below my name and date. \n",
    "    * set the date too and remove mine.\n",
    "3. read the entire notebook carefully. \n",
    "    * add comments whereever you feel it necessary for better understanding.\n",
    "    * run the notebook and try to follow and understand all steps and examples.\n",
    "4. in PART II, try to randomly pick different (n=200) training samples and try to find a model. Compare its performance.\n",
    "    * develop a testplan as you have done before, so you do not change parameters arbitrarily and use a table for the validation results. \n",
    "5. in PART II, increase the number of training samples for the task-specific embedding and compare it again with the pre-trained embedding. Does it become better? When does it beat the pre-trained embedding?\n",
    "    * develop a testplan as you have done before, so you do not change parameters arbitrarily and use a table for the validation results\n",
    "    \n",
    "-----------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"#C70039\">PART I</font>\n",
    "### Learning word embeddings with the `Embedding` layer\n",
    "\n",
    "The simplest way to associate a dense vector to a word would be to pick the vector at random. The problem with this approach is that the resulting embedding space would have no structure and that the words \"accurate\" and \"exact\" may end up with completely different embeddings even though they are interchangeable in many sentences. It would be very difficult for a deep neural network to make sense of such a noisy, unstructured embedding space. \n",
    "\n",
    "This is why within a good embedding space a geometric relationships between word vectors should reflect the semantic relationships between these words. Hence it would to be expected that the geometric distance (e.g. L2 distance) between any two word vectors do relate to the semantic distance of the associated words. I.e., words meaning very different things would be embedded to points far away from each other, while related words would be closer.\n",
    "\n",
    "However, pragmatically seen, what makes a good word embedding space depends heavily on the task at hand. Thus the perfect word embedding space for an English-language movie review sentiment analysis model may look very different from the perfect embedding space for an English-language legal document classification model, because the importance of certain semantic relationships varies from task to task. It is thus reasonable to learn a new embedding space with every new task. \n",
    "\n",
    "Thanks to backpropagation this learning is really easy and thanks to Keras it becomes even easier. \n",
    "Finally it is just about learning the weights of a layer, the so-called `Embedding` layer and this is what PART I is all about."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### START OF THE NOTEBOOK CODE\n",
    "----------------------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import tensorflow\n",
    "tensorflow.keras.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Embedding\n",
    "\n",
    "# The Embedding layer takes at least two arguments:\n",
    "# the number of possible tokens, here 1000 (1 + maximum word index),\n",
    "# and the dimensionality of the embeddings, here 64.\n",
    "embedding_layer = Embedding(1000, 64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Embedding` layer is best understood as a dictionary mapping integer indices (which stand for specific words) to dense vectors. It takes as input those integers, it looks them up in an internal dictionary and finally, it returns the associated vectors. Hence, it is effectively a dictionary lookup."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Embedding` layer takes as input a 2D tensor of integers, of shape `(samples, sequence_length)`, where each entry is a sequence of integers. It can embed sequences of variable lengths, so e.g. one could feed into the embedding layer above batches that could have shapes `(32, 10)` (batch of 32 sequences of length 10) or `(64, 15)` (batch of 64 sequences of length 15). Note that all sequences in a batch must have the same length since they need to be packed into a single tensor and so, sequences  shorter than others should be padded (filled-in) with zeros and sequences that are longer should be truncated (cut).\n",
    "\n",
    "The layer above returns a 3D floating point tensor, of shape `(samples, sequence_length, embedding_dimensionality)`. Such a 3D tensor can then be processed by a recurrent (RNN) or a convolution layer (CNN). For the RNN case see next exercise.\n",
    "\n",
    "When instantiating an `Embedding` layer, its weights (its internal dictionary of token vectors) are initially random, just as always like with any other layer. During training, these word vectors will be gradually adjusted via backpropagation, structuring the space into something that the downstream model can exploit. Once fully trained, the trained embedding space will show a lot of structure - a kind of structure specialized for the specific problem at hand.\n",
    "\n",
    "-----------\n",
    "\n",
    "Now, let us apply this idea to the IMDB movie review sentiment prediction task from the earlier exercise.\n",
    "Quickly prepare the data. The movie reviews will be restricted to the top 10.000 most common words (like the first time when working with this data set), and then the reviews are cut off after only 20 words. <br>\n",
    "The network will simply learn 8-dimensional embeddings for each of the 10.000 words, then turn the input integer sequences (2D integer tensor) into embedded sequences (3D float tensor), flatten the tensor to 2D and train a single `Dense` layer on top for classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import imdb\n",
    "from tensorflow.keras import preprocessing\n",
    "\n",
    "# Number of words to consider as features\n",
    "max_features = 10000\n",
    "# Cut texts after this number of words \n",
    "# (among top max_features most common words)\n",
    "maxlen = 20\n",
    "\n",
    "# Load the data as lists of integers.\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
    "\n",
    "# This turns the lists of integers into a 2D integer tensor of shape `(samples, maxlen)`\n",
    "x_train = preprocessing.sequence.pad_sequences(x_train, maxlen=maxlen)\n",
    "x_test = preprocessing.sequence.pad_sequences(x_test, maxlen=maxlen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "# Specify the maximum input length to the Embedding layer\n",
    "# Later flatten the embedded inputs\n",
    "model.add(Embedding(10000, 8, input_length=maxlen))\n",
    "# After the Embedding layer, the activations have shape `(samples, maxlen, 8)`.\n",
    "\n",
    "# Flatten the 3D tensor of embeddings into a 2D tensor of shape `(samples, maxlen * 8)`\n",
    "model.add(Flatten())\n",
    "\n",
    "# Add the classifier on top with a single Dense layer\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])\n",
    "model.summary()\n",
    "\n",
    "# use the history object again\n",
    "history = model.fit(x_train, y_train,\n",
    "                    epochs=10,\n",
    "                    batch_size=32,\n",
    "                    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The validation accuracy is about 72-76%, which is pretty good considering that only the first 20 words in every review were taken into account. \n",
    "\n",
    "But note that merely flattening the embedded sequences and training a single `Dense` layer on top leads to a model that treats each word in the input sequence separately, without considering inter-word relationships and structured sentences (for example it would treat both _\"this movie is shit\"_ and _\"this movie is the shit\"_ as being negative \"reviews\" which is obviously not true). \n",
    "\n",
    "It would be much better to add recurrent layers or 1D convolutional layers on top of the embedded sequences to learn features that take into account each sequence as a whole. However, that's the content of the next exercise.\n",
    "\n",
    "----------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"#C70039\">PART II</font>\n",
    "### Using `pre-trained` word embeddings\n",
    "\n",
    "Instead of learning word embeddings jointly with the problem, embedding vectors can be loaded from a pre-computed embedding space known to be highly structured and to exhibit useful properties that captures generic aspects of language structure. \n",
    "\n",
    "The reseaon behind using pre-trained word embeddings in natural language processing is very much the same as for using pre-trained CNNs in image classification. There is simply not enough data available to learn truly powerful features on its own but it is expected that the features needed are fairly generic, i.e. common visual features or semantic features. In this case it makes sense to re-use features learned on a different problem.\n",
    "\n",
    "Such word embeddings are generally computed using word occurrence statistics which are observations about what words co-occur in sentences or documents, using a variety of techniques, some involving neural networks, others not. \n",
    "The idea of a dense, low-dimensional embedding space for words computed in an unsupervised way, was initially explored by Bengio et al. in the early 2000s, but it only started really taking off in research and industry applications after the release of one of the most famous and successful word embedding scheme: the Word2Vec algorithm, developed by Mikolov at Google in 2013.<br>\n",
    "\n",
    "There are various pre-computed databases of word embeddings that can be downloaded and started with using a Keras `Embedding` layer. \"Word2Vec\" is one of them. Another popular one is called \"GloVe\", developed by Stanford researchers in 2014. \n",
    "It stands for \"Global Vectors for Word Representation\" and it is an embedding technique based on factorizing a matrix of word co-occurrence statistics. Its developers have made available pre-computed embeddings for millions of English tokens, obtained from Wikipedia data or from common crawl data.\n",
    "\n",
    "Let us take a look at how GloVe embeddings in a Keras model can be used. The same method will of course be valid for Word2Vec embeddings or any other word embedding database. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### START OF THE NOTEBOOK CODE\n",
    "----------------------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting it all together: from raw text to word embeddings\n",
    "\n",
    "Use a model similar to the one that was just used earlier -- embedding sentences in sequences of vectors, flattening them and training a `Dense` layer on top. But now, this will be done using pre-trained word embeddings and instead of using the pre-tokenized IMDB data packaged in Keras it will be started from scratch by downloading the original text data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the IMDB data as raw text\n",
    "\n",
    "First, head to `http://ai.stanford.edu/~amaas/data/sentiment/` and download the raw IMDB data set. Anyways, this data set is distributed and available all over the web and can be found anywhere. After downloading, uncompress it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, using some lines of code, collect the individual training reviews into a list of strings, one string per review, and also collect the review labels (positive / negative) into a `labels` list. Make sure you are uploading the data to your Google Drive if you are using Google Colab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "imdb_dir = '../data/acl_IMDB'\n",
    "train_dir = os.path.join(imdb_dir, 'train')\n",
    "\n",
    "labels = []\n",
    "texts = []\n",
    "\n",
    "for label_type in ['neg', 'pos']:\n",
    "    dir_name = os.path.join(train_dir, label_type)\n",
    "    for fname in os.listdir(dir_name):\n",
    "        if fname[-4:] == '.txt':\n",
    "            f = open(os.path.join(dir_name, fname), encoding=\"utf8\")\n",
    "            texts.append(f.read())\n",
    "            f.close()\n",
    "            if label_type == 'neg':\n",
    "                labels.append(0)\n",
    "            else:\n",
    "                labels.append(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization\n",
    "\n",
    "Vectorize the texts and prepare a training and validation split. Use the same concepts that were introduced earlier in the notebook of exercise #10.\n",
    "\n",
    "Because pre-trained word embeddings are meant to be particularly useful on problems where little training data is available, add the following restriction (otherwise, task-specific embeddings are always outperforming them): \n",
    "\n",
    "* restrict the training data to its first 200 samples only. Hence, try to let the model learn to classify movie reviews after looking at just 200 examples. That is almost nothing and we will see how far we can get with this. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "import numpy as np\n",
    "\n",
    "maxlen = 100  # cut reviews after 100 words\n",
    "training_samples = 200  # training on 200 samples only\n",
    "validation_samples = 10000  # validation on 10000 samples\n",
    "max_words = 10000  # consider the top 10000 words in the data set\n",
    "\n",
    "tokenizer = Tokenizer(num_words=max_words)\n",
    "tokenizer.fit_on_texts(texts)\n",
    "sequences = tokenizer.texts_to_sequences(texts)\n",
    "\n",
    "word_index = tokenizer.word_index\n",
    "print('Found %s unique tokens' % len(word_index))\n",
    "\n",
    "data = pad_sequences(sequences, maxlen=maxlen)\n",
    "\n",
    "labels = np.asarray(labels)\n",
    "print('Shape of data tensor:', data.shape)\n",
    "print('Shape of label tensor:', labels.shape)\n",
    "\n",
    "# Split the data into a training set and a validation set\n",
    "# But first, shuffle the data, since the data samples are \n",
    "# ordered (all negative first, then all positive)\n",
    "''' NOTE: Better always shuffle if you are not sure data is ordered'''\n",
    "indices = np.arange(data.shape[0])\n",
    "np.random.shuffle(indices)\n",
    "data = data[indices]\n",
    "labels = labels[indices]\n",
    "\n",
    "x_train = data[:training_samples]\n",
    "y_train = labels[:training_samples]\n",
    "x_val = data[training_samples: training_samples + validation_samples]\n",
    "y_val = labels[training_samples: training_samples + validation_samples]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Downloading the GloVe word embeddings\n",
    "\n",
    "Please visit `https://nlp.stanford.edu/projects/glove/` and download the pre-computed embeddings from 2014 English Wikipedia. \n",
    "By the way, this site is an excellent ressource for further reading where you can learn more about the GloVe algorithm. \n",
    "The pre-computed embeddings are packed together in a 822MB zip file named `glove.6B.zip`, containing 100-dimensional embedding vectors for 400.000 words. Unzip it and put it into the local disk data folder. Since its size is too large for Moodle, I recommend to download it yourself. If you are working on Google Colab please upload the necessary file \"glove.6B.100d.txt\" into your Google drive rather than the entire zip file content."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing the embeddings\n",
    "\n",
    "Parse the unzipped file to build an index, mapping words (as strings) to their vector representation (as number vectors)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "glove_dir = '../data/GloVE'\n",
    "\n",
    "embeddings_index = {}\n",
    "f = open(os.path.join(glove_dir, 'glove.6B.100d.txt'), encoding=\"utf8\")\n",
    "for line in f:\n",
    "    values = line.split()\n",
    "    word = values[0]\n",
    "    coefs = np.asarray(values[1:], dtype='float32')\n",
    "    embeddings_index[word] = coefs\n",
    "f.close()\n",
    "\n",
    "print('Found %s word vectors' % len(embeddings_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, build an embedding matrix that can be loaded into an `Embedding` layer. It must be a matrix of shape `(max_words,embedding_dim)`, where each entry `i` contains the `embedding_dim`-dimensional vector for the word of index `i` in our reference word index (built during tokenization). Note that the index `0` is not supposed to stand for any word or token. It is simply a placeholder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "embedding_dim = 100\n",
    "embedding_matrix = np.zeros((max_words, embedding_dim))\n",
    "\n",
    "for word, i in word_index.items():\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if i < max_words:\n",
    "        if embedding_vector is not None:\n",
    "            # Words not found in the embedding index will be all-zeros\n",
    "            embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model definition\n",
    "\n",
    "Use the same model architecture as before, using a Dense layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Embedding, Flatten, Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_words, embedding_dim, input_length=maxlen))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the GloVe embeddings in the model\n",
    "\n",
    "The `Embedding` layer has got a single weight matrix. It is a 2D float matrix where each entry `i` is the word vector meant to be associated with index `i`. Simple enough! Load the GloVe matrix into the `Embedding` layer, which is the first layer in the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model.layers[0].set_weights([embedding_matrix])\n",
    "model.layers[0].trainable = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, freeze the embedding layer by setting its `trainable` attribute to `False`, following the same idea that we are already familiar with in the context of pre.trained CNN features in the earlier exercise. The idea was that when parts of a model are pre-trained like this `Embedding` layer and parts are randomly initialized like this classifier, then the pre-trained parts should not be updated during training to avoid forgetting what they already know. \n",
    "The large gradient update triggered by the randomly initialized layers would be very disruptive to the already learned features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training and evaluating\n",
    "\n",
    "Compile the model and then train and validate it in one pass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['acc'])\n",
    "\n",
    "history = model.fit(x_train, y_train,\n",
    "                    epochs=10,\n",
    "                    batch_size=32,\n",
    "                    validation_data=(x_val, y_val))\n",
    "\n",
    "model.save_weights('Ex.11.pretrained_GloVe_model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot its performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model quickly starts overfitting which is not surprising due to the small number of training samples. Validation accuracy has high variance for the same reason but seems to reach high 50s.\n",
    "\n",
    "Note that the precision may vary since there are so few training samples, that performance heavily depends on which exact 200 samples were picked and in this example they were picked at random (see your task list). \n",
    "\n",
    "----------------\n",
    "### Another attempt\n",
    "We can also try to train the same model <u>without</u> loading the pre-trained word embeddings and <u>without</u> freezing the embedding layer. In that case, it will result in learning a task-specific embedding of the input tokens which is generally more powerful than pre-trained word embeddings when lots of data is available. However, in this case here, there are only 200 training samples available. So let's go for it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Embedding, Flatten, Dense\n",
    "\n",
    "#-----------------------------------------------------------------\n",
    "maxlen = 100  # cut reviews after 100 words\n",
    "training_samples = 200  # training on 200 samples only\n",
    "validation_samples = 10000  # validation on 10000 samples\n",
    "max_words = 10000  # consider the top 10,000 words in the data set\n",
    "# Number of words to consider as features\n",
    "embedding_dim=100\n",
    "#-----------------------------------------------------------------\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_words, embedding_dim, input_length=maxlen))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.summary()\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['acc'])\n",
    "\n",
    "history = model.fit(x_train, y_train,\n",
    "                    epochs=10,\n",
    "                    batch_size=32,\n",
    "                    validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the results again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Validation accuracy is somewhere in the low 50s. So in this case, pre-trained word embeddings do outperform jointly learned embeddings. \n",
    "\n",
    "Finally, evaluate the model on the test data. Do not forget to tokenize the test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "test_dir = os.path.join(imdb_dir, 'test')\n",
    "\n",
    "labels = []\n",
    "texts = []\n",
    "\n",
    "for label_type in ['neg', 'pos']:\n",
    "    dir_name = os.path.join(test_dir, label_type)\n",
    "    for fname in sorted(os.listdir(dir_name)):\n",
    "        if fname[-4:] == '.txt':\n",
    "            f = open(os.path.join(dir_name, fname),encoding=\"utf8\")\n",
    "            texts.append(f.read())\n",
    "            f.close()\n",
    "            if label_type == 'neg':\n",
    "                labels.append(0)\n",
    "            else:\n",
    "                labels.append(1)\n",
    "\n",
    "sequences = tokenizer.texts_to_sequences(texts)\n",
    "x_test = pad_sequences(sequences, maxlen=maxlen)\n",
    "y_test = np.asarray(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last but not least, load and evaluate the pre-trained model in the testing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model.load_weights('Ex.11.pretrained_GloVe_model.h5')\n",
    "model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an appalling test accuracy of 54-55%. Working with just a handful of training samples is hard and obviously crazy :-)  Take this is a warning within a lesson-learned!!!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}