week_4/data-pre-processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word Embeddings First Steps: Data Preparation\n",
    "\n",
    "In this series of ungraded notebooks, you'll try out all the individual techniques that you learned about in the lectures. Practicing on small examples will prepare you for the graded assignment, where you will combine the techniques in more advanced ways to create word embeddings from a real-life corpus. \n",
    "\n",
    "This notebook focuses on data preparation, which is the first step of any machine learning algorithm. It is a very important step because models are only as good as the data they are trained on and the models used require the data to have a particular structure to process it properly.\n",
    "\n",
    "To get started, import and initialize all the libraries you will need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import nltk\n",
    "import emoji\n",
    "import numpy as np\n",
    "from nltk.tokenize import word_tokenize\n",
    "from utils2 import get_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the data preparation phase, starting with a corpus of text, you will:\n",
    "\n",
    "- Clean and tokenize the corpus.\n",
    "\n",
    "- Extract the pairs of context words and center word that will make up the training data set for the CBOW model. The context words are the features that will be fed into the model, and the center words are the target values that the model will learn to predict.\n",
    "\n",
    "- Create simple vector representations of the context words (features) and center words (targets) that can be used by the neural network of the CBOW model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning and tokenization\n",
    "\n",
    "To demonstrate the cleaning and tokenization process, consider a corpus that contains emojis and various punctuation signs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a corpus\n",
    "corpus = 'Who ❤️ \"word embeddings\" in 2020? I do!!!'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, replace all interrupting punctuation signs — such as commas and exclamation marks — with periods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Corpus:  Who ❤️ \"word embeddings\" in 2020? I do!!!\n",
      "After cleaning punctuation:  Who ❤️ \"word embeddings\" in 2020. I do.\n"
     ]
    }
   ],
   "source": [
    "# Print original corpus\n",
    "print(f'Corpus:  {corpus}')\n",
    "\n",
    "# Do the substitution\n",
    "data = re.sub(r'[,!?;-]+', '.', corpus)\n",
    "\n",
    "# Print cleaned corpus\n",
    "print(f'After cleaning punctuation:  {data}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, use NLTK's tokenization engine to split the corpus into individual tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial string:  Who ❤️ \"word embeddings\" in 2020. I do.\n",
      "After tokenization:  ['Who', '❤️', '``', 'word', 'embeddings', \"''\", 'in', '2020', '.', 'I', 'do', '.']\n"
     ]
    }
   ],
   "source": [
    "# Print cleaned corpus\n",
    "print(f'Initial string:  {data}')\n",
    "\n",
    "# Tokenize the cleaned corpus\n",
    "data = nltk.word_tokenize(data)\n",
    "\n",
    "# Print the tokenized version of the corpus\n",
    "print(f'After tokenization:  {data}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, as you saw in the lecture, get rid of numbers and punctuation other than periods, and convert all the remaining tokens to lowercase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial list of tokens:  ['Who', '❤️', '``', 'word', 'embeddings', \"''\", 'in', '2020', '.', 'I', 'do', '.']\n",
      "After cleaning:  ['who', '❤️', 'word', 'embeddings', 'in', '.', 'i', 'do', '.']\n"
     ]
    }
   ],
   "source": [
    "# Print the tokenized version of the corpus\n",
    "print(f'Initial list of tokens:  {data}')\n",
    "\n",
    "# Filter tokenized corpus using list comprehension\n",
    "data = [ ch.lower() for ch in data\n",
    "         if ch.isalpha()\n",
    "         or ch == '.'\n",
    "         or emoji.get_emoji_regexp().search(ch)\n",
    "       ]\n",
    "\n",
    "# Print the tokenized and filtered version of the corpus\n",
    "print(f'After cleaning:  {data}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the heart emoji is considered as a token just like any normal word.\n",
    "\n",
    "Now let's streamline the cleaning and tokenization process by wrapping the previous steps in a function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the 'tokenize' function that will include the steps previously seen\n",
    "def tokenize(corpus):\n",
    "    data = re.sub(r'[,!?;-]+', '.', corpus)\n",
    "    data = nltk.word_tokenize(data)  # tokenize string to words\n",
    "    data = [ ch.lower() for ch in data\n",
    "             if ch.isalpha()\n",
    "             or ch == '.'\n",
    "             or emoji.get_emoji_regexp().search(ch)\n",
    "           ]\n",
    "    return data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply this function to the corpus that you'll be working on in the rest of this notebook: \"I am happy because I am learning\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Corpus:  I am happy because I am learning\n",
      "Words (tokens):  ['i', 'am', 'happy', 'because', 'i', 'am', 'learning']\n"
     ]
    }
   ],
   "source": [
    "# Define new corpus\n",
    "corpus = 'I am happy because I am learning'\n",
    "\n",
    "# Print new corpus\n",
    "print(f'Corpus:  {corpus}')\n",
    "\n",
    "# Save tokenized version of corpus into 'words' variable\n",
    "words = tokenize(corpus)\n",
    "\n",
    "# Print the tokenized version of the corpus\n",
    "print(f'Words (tokens):  {words}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Now try it out yourself with your own sentence.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['now',\n",
       " 'it',\n",
       " 'your',\n",
       " 'turn',\n",
       " 'try',\n",
       " 'with',\n",
       " 'your',\n",
       " 'own',\n",
       " 'sentence',\n",
       " '.',\n",
       " 'wow',\n",
       " 'this',\n",
       " 'is',\n",
       " 'pretty',\n",
       " 'amazing',\n",
       " '💜💃',\n",
       " '.']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Run this with any sentence\n",
    "tokenize(\"Now it's your turn: try with your own sentence! \\n wow this is pretty amazing 💜💃 !!!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sliding window of words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have transformed the corpus into a list of clean tokens, you can slide a window of words across this list. For each window you can extract a center word and the context words.\n",
    "\n",
    "The `get_windows` function in the next cell was introduced in the lecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the 'get_windows' function\n",
    "def get_windows(words, C):\n",
    "    i = C\n",
    "    while i < len(words) - C:\n",
    "        center_word = words[i]\n",
    "        context_words = words[(i - C):i] + words[(i+1):(i+C+1)]\n",
    "        yield context_words, center_word\n",
    "        i += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first argument of this function is a list of words (or tokens). The second argument, `C`, is the context half-size. Recall that for a given center word, the context words are made of `C` words to the left and `C` words to the right of the center word.\n",
    "\n",
    "Here is how you can use this function to extract context words and center words from a list of tokens. These context and center words will make up the training set that you will use to train the CBOW model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['i', 'am', 'because', 'i']\thappy\n",
      "['am', 'happy', 'i', 'am']\tbecause\n",
      "['happy', 'because', 'am', 'learning']\ti\n"
     ]
    }
   ],
   "source": [
    "# Print 'context_words' and 'center_word' for the new corpus with a 'context half-size' of 2\n",
    "for x, y in get_windows(['i', 'am', 'happy', 'because', 'i', 'am', 'learning'], 2):\n",
    "    print(f'{x}\\t{y}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first example of the training set is made of:\n",
    "\n",
    "- the context words \"i\", \"am\", \"because\", \"i\",\n",
    "\n",
    "- and the center word to be predicted: \"happy\".\n",
    "\n",
    "**Now try it out yourself. In the next cell, you can change both the sentence and the context half-size.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['neccisiy', 'is', 'mother', 'of']\tthe\n",
      "['is', 'the', 'of', 'invention']\tmother\n"
     ]
    }
   ],
   "source": [
    "# Print 'context_words' and 'center_word' for any sentence with a 'context half-size' of 1\n",
    "for x, y in get_windows(tokenize(\"neccisiy is the mother of invention\"), 2):\n",
    "    print(f'{x}\\t{y}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transforming words into vectors for the training set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To finish preparing the training set, you need to transform the context words and center words into vectors.\n",
    "\n",
    "### Mapping words to indices and indices to words\n",
    "\n",
    "The center words will be represented as one-hot vectors, and the vectors that represent context words are also based on one-hot vectors.\n",
    "\n",
    "To create one-hot word vectors, you can start by mapping each unique word to a unique integer (or index). We have provided a helper function, `get_dict`, that creates a Python dictionary that maps words to integers and back."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get 'word2Ind' and 'Ind2word' dictionaries for the tokenized corpus\n",
    "word2Ind, Ind2word = get_dict(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the dictionary that maps words to numeric indices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'am': 0, 'because': 1, 'happy': 2, 'i': 3, 'learning': 4}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print 'word2Ind' dictionary\n",
    "word2Ind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use this dictionary to get the index of a word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index of the word 'i': 3\n"
     ]
    }
   ],
   "source": [
    "# Print value for the key 'i' within word2Ind dictionary\n",
    "print(\"Index of the word 'i':\", word2Ind['i'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And conversely, here's the dictionary that maps indices to words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'am', 1: 'because', 2: 'happy', 3: 'i', 4: 'learning'}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print 'Ind2word' dictionary\n",
    "Ind2word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word which has index 2: happy\n"
     ]
    }
   ],
   "source": [
    "# Print value for the key '2' within Ind2word dictionary\n",
    "print(\"Word which has index 2:\", Ind2word[2] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, get the length of either of these dictionaries to get the size of the vocabulary of your corpus, in other words the number of different words making up the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of vocabulary: 5\n"
     ]
    }
   ],
   "source": [
    "# Save length of word2Ind dictionary into the 'V' variable\n",
    "V = len(word2Ind)\n",
    "\n",
    "# Print length of word2Ind dictionary\n",
    "print(\"Size of vocabulary:\", V)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting one-hot word vectors\n",
    "\n",
    "Recall from the lecture that you can easily convert an integer, $n$, into a one-hot vector.\n",
    "\n",
    "Consider the word \"happy\". First, retrieve its numeric index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Save index of word 'happy' into the 'n' variable\n",
    "n = word2Ind['happy']\n",
    "\n",
    "# Print index of word 'happy'\n",
    "n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create a vector with the size of the vocabulary, and fill it with zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0.])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create vector with the same length as the vocabulary, filled with zeros\n",
    "center_word_vector = np.zeros(V)\n",
    "\n",
    "# Print vector\n",
    "center_word_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can confirm that the vector has the right size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Assert that the length of the vector is the same as the size of the vocabulary\n",
    "len(center_word_vector) == V"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, replace the 0 of the $n$-th element with a 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace element number 'n' with a 1\n",
    "center_word_vector[n] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And you have your one-hot word vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 1., 0., 0.])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print vector\n",
    "center_word_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**You can now group all of these steps in a convenient function, which takes as parameters: a word to be encoded, a dictionary that maps words to indices, and the size of the vocabulary.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the 'word_to_one_hot_vector' function that will include the steps previously seen\n",
    "def word_to_one_hot_vector(word, word2Ind):\n",
    "    V = len(word2Ind)\n",
    "    one_hot_vector = np.zeros(V)\n",
    "    one_hot_vector[word2Ind[word]] = 1\n",
    "    return one_hot_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check that it works as intended."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 1., 0., 0.])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print output of 'word_to_one_hot_vector' function for word 'happy'\n",
    "word_to_one_hot_vector('happy', word2Ind)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What is the word vector for \"learning\"?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 1.])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print output of 'word_to_one_hot_vector' function for word 'learning'\n",
    "word_to_one_hot_vector('learning', word2Ind)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Expected output:\n",
    "\n",
    "    array([0., 0., 0., 0., 1.])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting context word vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create the vectors that represent context words, you will calculate the average of the one-hot vectors representing the individual words.\n",
    "\n",
    "Let's start with a list of context words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define list containing context words\n",
    "context_words = ['i', 'am', 'because', 'i']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using Python's list comprehension construct and the `word_to_one_hot_vector` function that you created in the previous section, you can create a list of one-hot vectors representing each of the context words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([0., 0., 0., 1., 0.]),\n",
       " array([1., 0., 0., 0., 0.]),\n",
       " array([0., 1., 0., 0., 0.]),\n",
       " array([0., 0., 0., 1., 0.])]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create one-hot vectors for each context word using list comprehension\n",
    "context_words_vectors = [word_to_one_hot_vector(w, word2Ind) for w in context_words]\n",
    "\n",
    "# Print one-hot vectors for each context word\n",
    "context_words_vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And you can now simply get the average of these vectors using numpy's `mean` function, to get the vector representation of the context words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.25, 0.25, 0.  , 0.5 , 0.  ])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute mean of the vectors using numpy\n",
    "np.mean(context_words_vectors, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the `axis=0` parameter that tells `mean` to calculate the average of the rows (if you had wanted the average of the columns, you would have used `axis=1`).\n",
    "\n",
    "**Now create the `context_words_to_vector` function that takes in a list of context words, a word-to-index dictionary, and a vocabulary size, and outputs the vector representation of the context words.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the 'context_words_to_vector' function that will include the steps previously seen\n",
    "def context_words_to_vector(context_words, word2Ind):\n",
    "    context_words_vectors = [word_to_one_hot_vector(w, word2Ind) for w in context_words]\n",
    "    context_words_vectors = np.mean(context_words_vectors, axis=0)\n",
    "    return context_words_vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And check that you obtain the same output as the manual approach above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.25, 0.25, 0.  , 0.5 , 0.  ])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print output of 'context_words_to_vector' function for context words: 'i', 'am', 'because', 'i'\n",
    "context_words_to_vector(['i', 'am', 'because', 'i'], word2Ind)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What is the vector representation of the context words \"am happy i am\"?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.5 , 0.  , 0.25, 0.25, 0.  ])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print output of 'context_words_to_vector' function for context words: 'am', 'happy', 'i', 'am'\n",
    "context_words_to_vector(['am', 'happy', 'i', 'am'], word2Ind)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Expected output:\n",
    "\n",
    "    array([0.5 , 0.  , 0.25, 0.25, 0.  ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Building the training set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now combine the functions that you created in the previous sections, to build a training set for the CBOW model, starting from the following tokenized corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['i', 'am', 'happy', 'because', 'i', 'am', 'learning']"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print corpus\n",
    "words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To do this you need to use the sliding window function (`get_windows`) to extract the context words and center words, and you then convert these sets of words into a basic vector representation using `word_to_one_hot_vector` and `context_words_to_vector`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context words:  ['i', 'am', 'because', 'i'] -> [0.25 0.25 0.   0.5  0.  ]\n",
      "Center word:  happy -> [0. 0. 1. 0. 0.]\n",
      "\n",
      "Context words:  ['am', 'happy', 'i', 'am'] -> [0.5  0.   0.25 0.25 0.  ]\n",
      "Center word:  because -> [0. 1. 0. 0. 0.]\n",
      "\n",
      "Context words:  ['happy', 'because', 'am', 'learning'] -> [0.25 0.25 0.25 0.   0.25]\n",
      "Center word:  i -> [0. 0. 0. 1. 0.]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Print vectors associated to center and context words for corpus\n",
    "for context_words, center_word in get_windows(words, 2):  # reminder: 2 is the context half-size\n",
    "    print(f'Context words:  {context_words} -> {context_words_to_vector(context_words, word2Ind)}')\n",
    "    print(f'Center word:  {center_word} -> {word_to_one_hot_vector(center_word, word2Ind)}')\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this practice notebook you'll be performing a single iteration of training using a single example, but in this week's assignment you'll train the CBOW model using several iterations and batches of example.\n",
    "Here is how you would use a Python generator function (remember the `yield` keyword from the lecture?) to make it easier to iterate over a set of examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the generator function 'get_training_example'\n",
    "def get_training_example(words, C, word2Ind):\n",
    "    for context_words, center_word in get_windows(words, C):\n",
    "        yield context_words_to_vector(context_words, word2Ind), word_to_one_hot_vector(center_word, word2Ind)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of this function can be iterated on to get successive context word vectors and center word vectors, as demonstrated in the next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context words vector:  [0.25 0.25 0.   0.5  0.  ]\n",
      "Center word vector:  [0. 0. 1. 0. 0.]\n",
      "\n",
      "Context words vector:  [0.5  0.   0.25 0.25 0.  ]\n",
      "Center word vector:  [0. 1. 0. 0. 0.]\n",
      "\n",
      "Context words vector:  [0.25 0.25 0.25 0.   0.25]\n",
      "Center word vector:  [0. 0. 0. 1. 0.]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Print vectors associated to center and context words for corpus using the generator function\n",
    "for context_words_vector, center_word_vector in get_training_example(words, 2, word2Ind):\n",
    "    print(f'Context words vector:  {context_words_vector}')\n",
    "    print(f'Center word vector:  {center_word_vector}')\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your training set is ready, you can now move on to the CBOW model itself which will be covered in the next lecture notebook.\n",
    "\n",
    "**Congratulations on finishing this lecture notebook!** Hopefully you now have a better understanding of how to prepare your data before feeding it to a continuous bag-of-words model. \n",
    "\n",
    "**Keep it up!**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}