week_2/creating-vocabulary-and-handling-unknown-words.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parts-of-Speech Tagging - First Steps: Working with text files, Creating a Vocabulary and Handling Unknown Words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lecture notebook you will create a vocabulary from a tagged dataset and learn how to deal with words that are not present in this vocabulary when working with other text sources. Aside from this you will also learn how to:\n",
    " \n",
    "- read text files\n",
    "- work with defaultdict\n",
    "- work with string data\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    "from collections import defaultdict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read Text Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A tagged dataset taken from the Wall Street Journal is provided in the file `WSJ_02-21.pos`. \n",
    "\n",
    "To read this file you can use Python's context manager by using the `with` keyword and specifying the name of the file you wish to read. To actually save the contents of the file into memory you will need to use the `readlines()` method and store its return value in a variable. \n",
    "\n",
    "Python's context managers are great because you don't need to explicitly close the connection to the file, this is done under the hood:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read lines from 'WSJ_02-21.pos' file and save them into the 'lines' variable\n",
    "with open(\"WSJ_02-21.pos\", 'r') as f:\n",
    "    lines = f.readlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To check the contents of the dataset you can print the first 5 lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t\tWord \tTag\n",
      "\n",
      "line number 1: In\tIN\n",
      "\n",
      "line number 2: an\tDT\n",
      "\n",
      "line number 3: Oct.\tNNP\n",
      "\n",
      "line number 4: 19\tCD\n",
      "\n",
      "line number 5: review\tNN\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Print columns for reference\n",
    "print(\"\\t\\tWord\", \"\\tTag\\n\")\n",
    "\n",
    "# Print first five lines of the dataset\n",
    "for i in range(5):\n",
    "    print(f'line number {i+1}: {lines[i]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each line within the dataset has a word followed by its corresponding tag. However since the printing was done using a formatted string it can be inferred that the **word** and the **tag** are separated by a tab (or some spaces) and there is a newline at the end of each line (notice that there is a space between each line). \n",
    "\n",
    "If you want to understand the meaning of these tags you can take a look [here](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html).\n",
    "\n",
    "To better understand how the information is structured in the dataset it is recommended to print an unformatted version of it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'In\\tIN\\n'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print first line (unformatted)\n",
    "lines[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed there is a tab between the word and the tag and a newline at the end of each line.\n",
    "\n",
    "### Creating a vocabulary\n",
    "\n",
    "Now that you understand how the dataset is structured, you will create a vocabulary out of it. A vocabulary is made up of every word that appeared at least 2 times in the dataset. \n",
    "For this, follow these steps:\n",
    "- Get only the words from the dataset\n",
    "- Use a defaultdict to count the number of times each word appears\n",
    "- Filter the dict to only include words that appeared at least 2 times\n",
    "- Create a list out of the filtered dict\n",
    "- Sort the list\n",
    "\n",
    "For step 1 you can use the fact that every word and tag are separated by a tab and that words always come first. Using list comprehension the words list can be created like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the words from each line in the dataset\n",
    "words = [line.split('\\t')[0] for line in lines]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Step 2 can be done easily by leveraging `defaultdict`. In case you aren't familiar with defaultdicts they are a special kind of dictionaries that **return the \"zero\" value of a type if you try to access a key that does not exist**. Since you want the frequencies of words, you should define the defaultdict with a type of `int`. \n",
    "\n",
    "Now you don't need to worry about the case when the word is not present within the dictionary because getting the value for that key will simply return a zero. Isn't that cool?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define defaultdict of type 'int'\n",
    "freq = defaultdict(int)\n",
    "\n",
    "# Count frequency of ocurrence for each word in the dataset\n",
    "for word in words:\n",
    "    freq[word] += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filtering the `freq` dictionary can be done using list comprehensions again (aren't they handy?). You should filter out words that appeared only once and also words that are just a newline character:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the vocabulary by filtering the 'freq' dictionary\n",
    "vocab = [k for k, v in freq.items() if (v > 1 and k != '\\n')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, the `sort` method will take care of the final step. Notice that it changes the list directly so you don't need to reassign the `vocab` variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Early\n",
      "Earnings\n",
      "Earth\n",
      "Earthquake\n",
      "East\n"
     ]
    }
   ],
   "source": [
    "# Sort the vocabulary\n",
    "vocab.sort()\n",
    "\n",
    "# Print some random values of the vocabulary\n",
    "for i in range(4000, 4005):\n",
    "    print(vocab[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have successfully created a vocabulary from the dataset. **Great job!** The vocabulary is quite extensive so it is not printed out but you can still do so by creating a cell and running something like `print(vocab)`. \n",
    "\n",
    "At this point you will usually write the vocabulary into a file for future use, but that is out of the scope of this notebook. If you are curious it is very similar to how you read the file at the beginning of this notebook.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processing new text sources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dealing with unknown words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have a vocabulary, you will use it when processing new text sources. **A new text will have words that do not appear in the current vocabulary**. To tackle this, you can simply classify each new word as an unknown one, but you can do better by creating a function that tries to classify the type of each unknown word and assign it a corresponding `unknown token`. \n",
    "\n",
    "This function will do the following checks and return an appropriate token:\n",
    "\n",
    "   - Check if the unknown word contains any character that is a digit \n",
    "       - return `--unk_digit--`\n",
    "   - Check if the unknown word contains any punctuation character \n",
    "       - return `--unk_punct--`\n",
    "   - Check if the unknown word contains any upper-case character \n",
    "       - return `--unk_upper--`\n",
    "   - Check if the unknown word ends with a suffix that could indicate it is a noun, verb, adjective or adverb \n",
    "        - return `--unk_noun--`, `--unk_verb--`, `--unk_adj--`, `--unk_adv--` respectively\n",
    "\n",
    "If a word fails to fall under any condition then its token will be a plain `--unk--`. The conditions will be evaluated in the same order as listed here. So if a word contains a punctuation character but does not contain digits, it will fall under the second condition. To achieve this behaviour some if/elif statements can be used along with early returns. \n",
    "\n",
    "This function is implemented next. Notice that the `any()` function is being heavily used. It returns `True` if at least one of the cases it evaluates is `True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def assign_unk(word):\n",
    "    \"\"\"\n",
    "    Assign tokens to unknown words\n",
    "    \"\"\"\n",
    "    \n",
    "    # Punctuation characters\n",
    "    # Try printing them out in a new cell!\n",
    "    punct = set(string.punctuation)\n",
    "    \n",
    "    # Suffixes\n",
    "    noun_suffix = [\"action\", \"age\", \"ance\", \"cy\", \"dom\", \"ee\", \"ence\", \"er\", \"hood\", \"ion\", \"ism\", \"ist\", \"ity\", \"ling\", \"ment\", \"ness\", \"or\", \"ry\", \"scape\", \"ship\", \"ty\"]\n",
    "    verb_suffix = [\"ate\", \"ify\", \"ise\", \"ize\"]\n",
    "    adj_suffix = [\"able\", \"ese\", \"ful\", \"i\", \"ian\", \"ible\", \"ic\", \"ish\", \"ive\", \"less\", \"ly\", \"ous\"]\n",
    "    adv_suffix = [\"ward\", \"wards\", \"wise\"]\n",
    "\n",
    "    # Loop the characters in the word, check if any is a digit\n",
    "    if any(char.isdigit() for char in word):\n",
    "        return \"--unk_digit--\"\n",
    "\n",
    "    # Loop the characters in the word, check if any is a punctuation character\n",
    "    elif any(char in punct for char in word):\n",
    "        return \"--unk_punct--\"\n",
    "\n",
    "    # Loop the characters in the word, check if any is an upper case character\n",
    "    elif any(char.isupper() for char in word):\n",
    "        return \"--unk_upper--\"\n",
    "\n",
    "    # Check if word ends with any noun suffix\n",
    "    elif any(word.endswith(suffix) for suffix in noun_suffix):\n",
    "        return \"--unk_noun--\"\n",
    "\n",
    "    # Check if word ends with any verb suffix\n",
    "    elif any(word.endswith(suffix) for suffix in verb_suffix):\n",
    "        return \"--unk_verb--\"\n",
    "\n",
    "    # Check if word ends with any adjective suffix\n",
    "    elif any(word.endswith(suffix) for suffix in adj_suffix):\n",
    "        return \"--unk_adj--\"\n",
    "\n",
    "    # Check if word ends with any adverb suffix\n",
    "    elif any(word.endswith(suffix) for suffix in adv_suffix):\n",
    "        return \"--unk_adv--\"\n",
    "    \n",
    "    # If none of the previous criteria is met, return plain unknown\n",
    "    return \"--unk--\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A POS tagger will always encounter words that are not within the vocabulary that is being used. By augmenting the dataset to include these `unknown word tokens` you are helping the tagger to have a better idea of the appropriate tag for these words. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting the correct tag for a word"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All that is left is to implement a function that will get the correct tag for a particular word taking special considerations for unknown words. Since the dataset provides each word and tag within the same line and a word being known depends on the vocabulary used, these two elements should be arguments to this function.\n",
    "\n",
    "This function should check if a line is empty and if so, it should return a placeholder word and tag, `--n--` and `--s--` respectively. \n",
    "\n",
    "If not, it should process the line to return the correct word and tag pair, considering if a word is unknown in which scenario the function `assign_unk()` should be used.\n",
    "\n",
    "The function is implemented next. Notice That the `split()` method can be used without specifying the delimiter, in which case it will default to any whitespace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_word_tag(line, vocab):\n",
    "    # If line is empty return placeholders for word and tag\n",
    "    if not line.split():\n",
    "        word = \"--n--\"\n",
    "        tag = \"--s--\"\n",
    "    else:\n",
    "        # Split line to separate word and tag\n",
    "        word, tag = line.split()\n",
    "        # Check if word is not in vocabulary\n",
    "        if word not in vocab: \n",
    "            # Handle unknown word\n",
    "            word = assign_unk(word)\n",
    "    return word, tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can try this function with some examples to test that it is working as intended:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('--n--', '--s--')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_word_tag('\\n', vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this line only includes a newline character it returns a placeholder word and tag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('In', 'IN')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_word_tag('In\\tIN\\n', vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This one is a valid line and the function does a fair job at returning the correct (word, tag) pair."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('--unk--', 'NN')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_word_tag('tardigrade\\tNN\\n', vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This line includes a noun that is not present in the vocabulary. \n",
    "\n",
    "The `assign_unk` function fails to detect that it is a noun so it returns an `unknown token`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('--unk_verb--', 'VB')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_word_tag('scrutinize\\tVB\\n', vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This line includes a verb that is not present in the vocabulary. \n",
    "\n",
    "In this case the `assign_unk` is able to detect that it is a verb so it returns an `unknown verb token`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulations on finishing this lecture notebook!** Now you should be more familiar with working with text data and have a better understanding of how a basic POS tagger works.\n",
    "\n",
    "**Keep it up!**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}