Preprocessing and classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 1: Preprocessing and Text Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Student Name: Vaibhav Singh Makhloga\n",
    "\n",
    "Student ID:1424591"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# General Info\n",
    "\n",
    "<b>Due date</b>: Wednesday, 20 March 2024 1pm\n",
    "\n",
    "<b>Submission method</b>: Canvas submission\n",
    "\n",
    "<b>Submission materials</b>: completed copy of this iPython notebook\n",
    "\n",
    "<b>Late submissions</b>: -20% per day (both week and weekend days counted)\n",
    "\n",
    "<b>Marks</b>: 9% of mark for class (with 8% on correctness + 1% on quality and efficiency of your code)\n",
    "\n",
    "<b>Materials</b>: See \"Using Jupyter Notebook and Python\" page on Canvas (under Modules>Resources) for information on the basic setup required for this class, including an iPython notebook viewer and the python packages NLTK, Numpy, Scipy, Matplotlib, Scikit-Learn, Gensim, Keras and Pytorch. We recommend installing all the data for NLTK, since you will need various parts of it to complete this assignment. You can also use any Python built-in packages, but do not use any other 3rd party packages (the packages listed above are all fine to use); if your iPython notebook doesn't run on the marker's machine, you will lose marks. <b> You should use Python 3.8</b>.  \n",
    "\n",
    "To familiarize yourself with NLTK, here is a free online book:  Steven Bird, Ewan Klein, and Edward Loper (2009). <a href=https://www.nltk.org/book/>Natural Language Processing with Python</a>. O'Reilly Media Inc. You may also consult the <a href=https://www.nltk.org/api/nltk.html>NLTK API</a>.\n",
    "\n",
    "<b>Evaluation</b>: Your iPython notebook should run end-to-end without any errors in a reasonable amount of time, and you must follow all instructions provided below, including specific implementation requirements and instructions for what needs to be printed (please avoid printing output we don't ask for). You should edit the sections below where requested, but leave the rest of the code as is. You should leave the output from running your code in the iPython notebook you submit, to assist with marking. The amount each question is worth is explicitly given. \n",
    "\n",
    "You will be marked not only on the correctness of your methods, but also the quality and efficency of your code: in particular, you should be careful to use Python built-in functions and operators when appropriate and pick descriptive variable names that adhere to <a href=\"https://www.python.org/dev/peps/pep-0008/\">Python style requirements</a>. If you think it might be unclear what you are doing, you should comment your code to help the marker make sense of it.\n",
    "\n",
    "<b>Updates</b>: Any major changes to the assignment will be announced via Canvas. Minor changes and clarifications will be announced on the discussion board; we recommend you check it regularly.\n",
    "\n",
    "<b>Academic misconduct</b>: For most people, collaboration will form a natural part of the undertaking of this homework, and we encourge you to discuss it in general terms with other students. However, this ultimately is still an individual task, and so reuse of code or other instances of clear influence will be considered cheating. We will be checking submissions for originality and will invoke the University’s <a href=\"http://academichonesty.unimelb.edu.au/policy.html\">Academic Misconduct policy</a> where inappropriate levels of collusion or plagiarism are deemed to have taken place. In regards to the use of artificial intelligence tools in the context of academic integrity, please see the university's statement <a href=\"https://academicintegrity.unimelb.edu.au/plagiarism-and-collusion/artificial-intelligence-tools-and-technologies\">here</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview\n",
    "\n",
    "In this homework, you'll be working with a collection of tweets. The task is to predict the geolocation (country) where the tweet comes from. This homework involves writing code to preprocess data and perform text classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing (4 marks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Instructions**: Download the data (as1-data.json) from Canvas and put it in the same directory as this iPython notebook. Run the code below to load the json data. This produces two objects, `x` and `y`, which contains a list of  tweets and corresponding country labels (it uses the standard [2 letter country code](https://www.iban.com/country-codes)) respectively. **No implementation is needed.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of tweets = 943\n",
      "Number of labels = 943\n",
      "\n",
      "Samples of data:\n",
      "Country = us \tTweet = @Addictd2Success thx u for following\n",
      "Country = us \tTweet = Let's just say, if I were to ever switch teams, Khalesi would be top of the list. #girlcrush\n",
      "Country = ph \tTweet = Taemin jonghyun!!! Your birits make me go~ http://t.co/le8z3dntlA\n",
      "Country = id \tTweet = depart.senior 👻 rapat perdana (with Nyayu, Anita, and 8 others at Ruang Aescullap FK Unsri Madang) — https://t.co/swRALlNkrQ\n",
      "Country = ph \tTweet = Done with internship with this pretty little lady!  (@ Metropolitan Medical Center w/ 3 others) [pic]: http://t.co/1qH61R1t5r\n",
      "Country = gb \tTweet = Wow just Boruc's clanger! Haha Sunday League stuff that, Giroud couldn't believe his luck! #clown\n",
      "Country = my \tTweet = I'm at Sushi Zanmai (Petaling Jaya, Selangor) w/ 5 others http://t.co/bcNobykZ\n",
      "Country = us \tTweet = Mega Fest!!!! Its going down🙏🙌  @BishopJakes\n",
      "Country = gb \tTweet = @EllexxxPharrell wow love the pic babe xx\n",
      "Country = us \tTweet = You have no clue how much you hurt me\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "x = []\n",
    "y = []\n",
    "data = json.load(open(\"as1-data.json\"))\n",
    "for k, v in data.items():\n",
    "    x.append(k)\n",
    "    y.append(v)\n",
    "    \n",
    "print(\"Number of tweets =\", len(x))\n",
    "print(\"Number of labels =\", len(y))\n",
    "print(\"\\nSamples of data:\")\n",
    "for i in range(10):\n",
    "    print(\"Country =\", y[i], \"\\tTweet =\", x[i])\n",
    "    \n",
    "assert(len(x) == 943)\n",
    "assert(len(y) == 943)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1 (1.0 mark)\n",
    "\n",
    "**Instructions**: Next we need to preprocess the collected tweets to create a bag-of-words representation (based on frequency). The preprocessing steps required here are: (1) tokenize each tweet into individual word tokens (using NLTK `TweetTokenizer`); (2) lowercase all words; (3) remove any word that does not contain any English letters in the alphabet (e.g. {_hello_, _#okay_, _abc123_} would be kept, but not {_123_, _!!_}) and (4) remove stopwords (based on NLTK `stopwords`). An empty tweet (after preprocessing) and its country label should be **excluded** from the output (`x_processed` and `y_processed`).\n",
    "\n",
    "**Task**: Complete the `preprocess_data(data, labels)` function. The function takes **a list of tweets** and **a corresponding list of country labels** as input, and returns **two lists**. For the first list, each element is a bag-of-words representation of a tweet (represented using a python dictionary). For the second list, each element is a corresponding country label. Note that while we do not need to preprocess the country labels (`y`), we need to have a new output list (`y_processed`) because some tweets maybe removed after the preprocessing (due to having an empty set of bag-of-words).\n",
    "\n",
    "**Check**: Use the assertion statements in <b>\"For your testing\"</b> below for the expected output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of preprocessed tweets = 943\n",
      "Number of preprocessed labels = 943\n",
      "\n",
      "Samples of preprocessed data:\n",
      "Country = us \tTweet = {'@addictd2success': 1, 'thx': 1, 'u': 1, 'following': 1}\n",
      "Country = us \tTweet = {\"let's\": 1, 'say': 1, 'ever': 1, 'switch': 1, 'teams': 1, 'khalesi': 1, 'would': 1, 'top': 1, 'list': 1, '#girlcrush': 1}\n",
      "Country = ph \tTweet = {'taemin': 1, 'jonghyun': 1, 'birits': 1, 'make': 1, 'go': 1, 'http://t.co/le8z3dntla': 1}\n",
      "Country = id \tTweet = {'depart.senior': 1, 'rapat': 1, 'perdana': 1, 'nyayu': 1, 'anita': 1, 'others': 1, 'ruang': 1, 'aescullap': 1, 'fk': 1, 'unsri': 1, 'madang': 1, 'https://t.co/swrallnkrq': 1}\n",
      "Country = ph \tTweet = {'done': 1, 'internship': 1, 'pretty': 1, 'little': 1, 'lady': 1, 'metropolitan': 1, 'medical': 1, 'center': 1, 'w': 1, 'others': 1, 'pic': 1, 'http://t.co/1qh61r1t5r': 1}\n",
      "Country = gb \tTweet = {'wow': 1, \"boruc's\": 1, 'clanger': 1, 'haha': 1, 'sunday': 1, 'league': 1, 'stuff': 1, 'giroud': 1, 'believe': 1, 'luck': 1, '#clown': 1}\n",
      "Country = my \tTweet = {\"i'm\": 1, 'sushi': 1, 'zanmai': 1, 'petaling': 1, 'jaya': 1, 'selangor': 1, 'w': 1, 'others': 1, 'http://t.co/bcnobykz': 1}\n",
      "Country = us \tTweet = {'mega': 1, 'fest': 1, 'going': 1, '@bishopjakes': 1}\n",
      "Country = gb \tTweet = {'@ellexxxpharrell': 1, 'wow': 1, 'love': 1, 'pic': 1, 'babe': 1, 'xx': 1}\n",
      "Country = us \tTweet = {'clue': 1, 'much': 1, 'hurt': 1}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to /Users/mac/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('stopwords')\n",
    "from nltk.tokenize import TweetTokenizer\n",
    "from nltk.corpus import stopwords\n",
    "import re\n",
    "\n",
    "tt = TweetTokenizer()\n",
    "stopwords = set(stopwords.words('english')) #note: stopwords are all in lowercase\n",
    "\n",
    "def preprocess_data(data, labels):\n",
    "    \n",
    "    ###\n",
    "    x_processed = []\n",
    "    y_processed = []\n",
    "    \n",
    "    for tweet, label in zip(data, labels):\n",
    "        # To tokenize the tweets we use TweetTokenizer\n",
    "        tokens = tt.tokenize(tweet)\n",
    "        \n",
    "        # this is done to filter words that do not contain english alphabets. Eg - ''#121'\n",
    "        tokens = [token for token in tokens if re.search('[a-zA-Z]', token)]\n",
    "        \n",
    "        # To remove stopwords from the dataset\n",
    "        tokens = [token.lower() for token in tokens if token.lower() not in stopwords]\n",
    "        \n",
    "        # create BOG representation\n",
    "        bag_of_words = {}\n",
    "        for token in tokens:\n",
    "            bag_of_words[token] = bag_of_words.get(token, 0) + 1\n",
    "        \n",
    "        # to not include empty tweets\n",
    "        if bag_of_words:\n",
    "            x_processed.append(bag_of_words)\n",
    "            y_processed.append(label)\n",
    "\n",
    "    return x_processed, y_processed\n",
    "  \n",
    "x_processed, y_processed = preprocess_data(x, y)\n",
    "\n",
    "print(\"Number of preprocessed tweets =\", len(x_processed))\n",
    "print(\"Number of preprocessed labels =\", len(y_processed))\n",
    "print(\"\\nSamples of preprocessed data:\")\n",
    "for i in range(10):\n",
    "    print(\"Country =\", y_processed[i], \"\\tTweet =\", x_processed[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**For your testing**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert(len(x_processed) == len(y_processed))\n",
    "assert(len(x_processed) > 800)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Instructions**: Hashtags (i.e. topic tags which start with #) pose an interesting tokenisation problem because they often include multiple words written without spaces or capitalization. Run the code below to collect all unique hashtags in the preprocessed data. **No implementation is needed.**\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of hashtags = 425\n",
      "['#100percentpay', '#1stsundayofoctober', '#1yearofalmostisneverenough', '#2011prdctn', '#2015eebritishfilmacademyawards', '#2k16', '#2littlebirds', '#365picture', '#5sosacousticatlanta', '#5sosfam', '#8thannualpubcrawl', '#affsuzukicup', '#aflpowertigers', '#ahimacon14', '#aim20', '#airasia', '#allcity', '#alliswell', '#allwedoiscurls', '#amazing', '#anferneehardaway', '#ariona', '#art', '#arte', '#artwork', '#ashes', '#asian', '#asiangirl', '#askcrawford', '#askherforfback', '#askolly', '#asksteven', '#at', '#australia', '#awesome', '#awesomepict', '#barcelona', '#bart', '#bayofislands', '#beautiful', '#bedimages', '#bell', '#beringmy', '#bettybooppose', '#bff', '#big', '#bigbertha', '#bigbreakfast', '#blackhat', '#blessedmorethanicanimagine', '#blessedsunday', '#blogtourambiente', '#bluemountains', '#bonekachika', '#boomtaob', '#booyaa', '#bored', '#boredom', '#bradersisterhood', '#breaktime', '#breedingground', '#bringithomemy', '#brooksengland', '#burgers', '#butitsalsokindofaphone', '#bye', '#camera', '#canadaelections', '#cbb', '#cbcolympics', '#cctv', '#cdnpoli', '#celebritycrush', '#chargers', '#chocolate', '#ciosummit', '#cleansidewalk', '#clown', '#coffeespoonart', '#colts', '#confused', '#cornell', '#country', '#craftbeer', '#creative', '#crepes', '#cumannanya', '#danny4mayor', '#data', '#date', '#datingsiteforyou', '#dearmind', '#deed', '#delightful', '#dennisrodman', '#design', '#devacurl', '#die', '#difd', '#diner', '#dinner', '#dragoncon', '#dus', '#dynamounlock', '#earrings', '#eeeeeehhh', '#election2015', '#electriccircus2014', '#endomondo', '#engine', '#english', '#europapark', '#excitables', '#fabulous', '#factorycampus', '#fall', '#familydinner', '#ff', '#fire', '#flambees', '#flashback', '#fly', '#focusateneo', '#followher', '#followme', '#foodporn', '#fotografías', '#fotorus', '#four', '#freaks', '#friday', '#fridaynight', '#fried', '#friends', '#friendshipflow', '#fries', '#frozenyoghurt', '#fun', '#future', '#galaxy', '#getfreetattooaviciipasses', '#girl', '#girlcrush', '#girls', '#givesmehope', '#goingout', '#google', '#graduated', '#grafunkthiremepls', '#grammyfans', '#grandmarnier', '#greenfood', '#grilled', '#gudnytall', '#gunner', '#gym', '#handbuiltbicycle', '#happybirthdaysandarapark', '#happyfriday', '#harimaumalaya', '#hb60', '#hens', '#hippy', '#holiday', '#hollywoodmusicawards', '#homemadegranola', '#hometomama', '#hot', '#hungergames', '#hungry', '#icu', '#ididntchoosethestudentlifeitchoseme', '#iloveyou', '#imsobored', '#imsosore', '#innoretail', '#insanity', '#insightmedia', '#insightmediasingapore', '#instaframeplus', '#instagood', '#instalook', '#isibaya', '#javaboy', '#jed', '#jewelry', '#jo', '#jordaan', '#jrsurfboards', '#justshare', '#kllive', '#ladygaga', '#latepost', '#laugh', '#laundryservice', '#lazysunday', '#learningcommunties', '#lebedeintennis', '#letmesleep', '#lfc', '#lgbt', '#life', '#lipstickfree', '#littlemonsters', '#loadsoffun', '#lol', '#longranger', '#lotsoflove', '#love', '#lovers', '#lovethisgirl', '#lovevibes', '#lte', '#magazinesandtvscreens', '#magic', '#makeupfree', '#makingemuklajawabnya', '#mamajeanneandme', '#mancrush', '#march', '#massacreconspiracy', '#mauce', '#mavic', '#me', '#meetup', '#melbourne', '#michaelkors', '#mindfulness', '#mkmedi', '#mobile', '#morning', '#mountains', '#movies', '#mtlnewtech', '#mtvhottest', '#mushroom', '#music', '#mustfollow', '#mwc14', '#mybabyemilia', '#myfriendsarebetterthanyours', '#nced', '#ncga', '#ncpol', '#nevergetsold', '#new', '#newlooks', '#newrecord', '#nextlevel', '#nfl', '#nickryrie', '#nochillzone', '#nofilter', '#notersnew2014', '#notreally', '#np', '#nye', '#of', '#offtochurch', '#ohyeah', '#oilandgas', '#olah', '#on', '#oops', '#openspace', '#oscars', '#oui', '#palacefansinthemorning', '#panther', '#panthers', '#partyhardpartyy', '#pats', '#pechanga', '#penny', '#peperoni', '#peppermoney', '#photo', '#photoby', '#photography', '#pic', '#pll', '#pmattheashes', '#pop-up', '#positivity', '#potd', '#procrastination', '#promise', '#purplefriday', '#rainorshine', '#reachingyougpeople', '#realgoodbikes', '#redbull', '#retail', '#revfcwh', '#rippaulwalker', '#ritenow', '#rollercoaster', '#rollersmusicawards', '#rollsroyce', '#rose', '#rosegold', '#rt', '#rundude', '#ryanpurrdler', '#sad', '#safm', '#saints', '#samsung', '#sandwich', '#sarahm', '#saulbass', '#scifigeek', '#security', '#seniorbabysenior', '#sexy', '#sfgiants', '#sggirls', '#shakes', '#shakethatbooty', '#shellvpower', '#shenanigans', '#shopaholic', '#siberuang', '#silver', '#singapore', '#singlefighter', '#sinvsmas', '#skeemsaam', '#skullsearcher', '#sl', '#socal', '#sorrynotsorry', '#southampton', '#spafy', '#spicy', '#spider', '#squad', '#stampede2014', '#startupfest', '#startuphub', '#starving', '#statoftheday', '#stayfreshsaturdaydbn', '#stkilda', '#stop', '#stopcomplaining', '#strictlyus', '#studio', '#summer', '#sun', '#sunrise', '#sunshine', '#supremelaundry', '#surfshop', '#swedumtl', '#sycip', '#taintedlove', '#takeabreak', '#tcschleissheim', '#tdwpliveinkl', '#teamcanada', '#tennis', '#thaiexpress', '#thaifood', '#thankful', '#thankyoulord', '#thankyoupatients', '#thecolorrun2014', '#thedisadvantagesofakindle', '#them', '#thenext5goldenyears', '#thevoiceau', '#thewalkingdead', '#thomassabo', '#throwback', '#thursday', '#ticklish', '#toronto', '#tpcoach', '#transit', '#truegrip', '#tuesday', '#tune', '#turbine', '#txlege', '#ujackbastards', '#umg', '#uniexamonasaturday', '#universal', '#uptonogood', '#urbangardening', '#uss', '#usydhereicome', '#usydoweek', '#utopia', '#vanilla', '#vca', '#vegan', '#veganfood', '#vegetables', '#vegetarian', '#video', '#vma', '#voteonedirection', '#vsco', '#vscocam', '#walking', '#watch', '#weare90s', '#wearesocial', '#white', '#wings', '#wok', '#wood', '#work', '#workmates', '#world', '#worldcup2014', '#yellow', '#yiamas', '#ynwa', '#youtube', '#yummy', '#yws13', '#zweihandvollfarm']\n"
     ]
    }
   ],
   "source": [
    "def get_all_hashtags(data):\n",
    "    hashtags = set([])\n",
    "    for d in data:\n",
    "        for word, frequency in d.items():\n",
    "            if word.startswith(\"#\") and len(word) > 1:\n",
    "                hashtags.add(word)\n",
    "    return hashtags\n",
    "\n",
    "hashtags = get_all_hashtags(x_processed)\n",
    "print(\"Number of hashtags =\", len(hashtags))\n",
    "print(sorted(hashtags))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2 (1.0 mark)\n",
    "\n",
    "**Instructions**: Our task here to tokenize the hashtags, by implementing the **MaxMatch algorithm** discussed in class.\n",
    "\n",
    "NLTK has a list of words that you can use for matching, see starter code below (`words`). Be careful about efficiency with respect to doing word lookups. One extra challenge you have to deal with is that the provided list of words (`words`) includes only lemmas: your MaxMatch algorithm should match inflected forms by converting them into lemmas using the NLTK lemmatizer before matching (provided by the function `lemmatize(word)`). Note that the list of words (`words`) is the only source that you'll use for matching (i.e. you do not need to find  other external word lists). If you are unable to make any longer match, your code should default to matching a single letter.\n",
    "\n",
    "For example, given \"#newrecord\", the algorithm should produce: \\[\"#\", \"new\", \"record\"\\].\n",
    "\n",
    "**Task**: Complete the `tokenize_hashtags(hashtags)` function by implementing the MaxMatch algorithm. The function takes as input **a set of hashtags**, and returns **a dictionary** where key=\"hashtag\" and value=\"a list of tokenised words\".\n",
    "\n",
    "**Check**: Use the assertion statements in <b>\"For your testing\"</b> below for the expected output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package words to /Users/mac/nltk_data...\n",
      "[nltk_data]   Package words is already up-to-date!\n",
      "[nltk_data] Downloading package wordnet to /Users/mac/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#vanilla ['#', 'vanilla']\n",
      "#vca ['#', 'v', 'ca']\n",
      "#vegan ['#', 'vega', 'n']\n",
      "#veganfood ['#', 'vega', 'n', 'food']\n",
      "#vegetables ['#', 'vegetable', 's']\n",
      "#vegetarian ['#', 'vegetarian']\n",
      "#video ['#', 'video']\n",
      "#vma ['#', 'v', 'ma']\n",
      "#voteonedirection ['#', 'vote', 'one', 'direction']\n",
      "#vsco ['#', 'v', 's', 'c', 'o']\n",
      "#vscocam ['#', 'v', 's', 'coca', 'm']\n",
      "#walking ['#', 'walking']\n",
      "#watch ['#', 'watch']\n",
      "#weare90s ['#', 'wear', 'e', '9', '0', 's']\n",
      "#wearesocial ['#', 'wear', 'es', 'o', 'c', 'i', 'al']\n",
      "#white ['#', 'white']\n",
      "#wings ['#', 'wing', 's']\n",
      "#wok ['#', 'wo', 'k']\n",
      "#wood ['#', 'wood']\n",
      "#work ['#', 'work']\n",
      "#workmates ['#', 'work', 'mate', 's']\n",
      "#world ['#', 'world']\n",
      "#worldcup2014 ['#', 'world', 'cup', '2', '0', '1', '4']\n",
      "#yellow ['#', 'yellow']\n",
      "#yiamas ['#', 'y', 'i', 'ama', 's']\n",
      "#ynwa ['#', 'yn', 'wa']\n",
      "#youtube ['#', 'you', 'tube']\n",
      "#yummy ['#', 'yummy']\n",
      "#yws13 ['#', 'y', 'w', 's', '1', '3']\n",
      "#zweihandvollfarm ['#', 'z', 'wei', 'hand', 'vol', 'l', 'farm']\n"
     ]
    }
   ],
   "source": [
    "from nltk.corpus import wordnet\n",
    "nltk.download('words')\n",
    "nltk.download('wordnet')\n",
    "\n",
    "lemmatizer = nltk.stem.wordnet.WordNetLemmatizer()\n",
    "words = set(nltk.corpus.words.words()) #a list of words provided by NLTK\n",
    "words = set([ word.lower() for word in words ]) #lowercase all the words for better matching\n",
    "\n",
    "\n",
    "\n",
    "def lemmatize(word):\n",
    "    lemma = lemmatizer.lemmatize(word,'v')\n",
    "    if lemma == word:\n",
    "        lemma = lemmatizer.lemmatize(word,'n')\n",
    "    return lemma\n",
    "\n",
    "\n",
    "def tokenize_hashtags(hashtags):\n",
    "    hashtags_t = {}\n",
    "    \n",
    "    for hashtag in hashtags:\n",
    "        tokens = []\n",
    "        rest = hashtag\n",
    "        \n",
    "        while rest:\n",
    "            \n",
    "            # this is for finding the longest match\n",
    "            found = False\n",
    "            for i in range(len(rest), 0, -1):\n",
    "                prefix = rest[:i]\n",
    "                lemma_prefix = lemmatize(prefix)\n",
    "                if lemma_prefix in words:\n",
    "                    tokens.append(prefix)\n",
    "                    rest = rest[i:]\n",
    "                    found = True\n",
    "                    break\n",
    "                    \n",
    "            # we take the single letter if no match is found\n",
    "            if not found:\n",
    "                tokens.append(rest[0])   \n",
    "                rest = rest[1:]\n",
    "        \n",
    "        hashtags_t[hashtag] = tokens\n",
    "    \n",
    "    return tokenized_hashtags\n",
    "\n",
    "#tokenise hashtags with MaxMatch\n",
    "tokenized_hashtags = tokenize_hashtags(hashtags)\n",
    "\n",
    "#print results\n",
    "for k, v in sorted(tokenized_hashtags.items())[-30:]:\n",
    "    print(k, v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**For your testing:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert(len(tokenized_hashtags) == len(hashtags))\n",
    "assert(tokenized_hashtags[\"#newrecord\"] == [\"#\", \"new\", \"record\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3 (1.0 mark)\n",
    "\n",
    "**Instructions**: Our next task is to tokenize the hashtags again, but this time using a **reversed version of the MaxMatch algorithm**, where matching begins at the end of the hashtag and progresses backwards (e.g. for <i>#helloworld</i>, we would process it right to left, starting from the last character <i>d</i>). Just like before, you should use the provided word list (`words`) for word matching.\n",
    "\n",
    "**Task**: Complete the `tokenize_hashtags_rev(hashtags)` function by the MaxMatch algorithm. The function takes as input **a set of hashtags**, and returns **a dictionary** where key=\"hashtag\" and value=\"a list of tokenised words\".\n",
    "\n",
    "**Check**: Use the assertion statements in <b>\"For your testing\"</b> below for the expected output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#vanilla ['#', 'vanilla']\n",
      "#vca ['#', 'v', 'ca']\n",
      "#vegan ['#', 'v', 'e', 'gan']\n",
      "#veganfood ['#', 'v', 'e', 'gan', 'food']\n",
      "#vegetables ['#', 'v', 'e', 'ge', 'tables']\n",
      "#vegetarian ['#', 'vegetarian']\n",
      "#video ['#', 'video']\n",
      "#vma ['#', 'v', 'ma']\n",
      "#voteonedirection ['#', 'vote', 'one', 'direction']\n",
      "#vsco ['#', 'v', 's', 'c', 'o']\n",
      "#vscocam ['#', 'v', 's', 'c', 'o', 'cam']\n",
      "#walking ['#', 'walking']\n",
      "#watch ['#', 'watch']\n",
      "#weare90s ['#', 'we', 'are', '9', '0', 's']\n",
      "#wearesocial ['#', 'we', 'are', 'social']\n",
      "#white ['#', 'white']\n",
      "#wings ['#', 'wing', 's']\n",
      "#wok ['#', 'w', 'ok']\n",
      "#wood ['#', 'wood']\n",
      "#work ['#', 'work']\n",
      "#workmates ['#', 'work', 'm', 'ates']\n",
      "#world ['#', 'world']\n",
      "#worldcup2014 ['#', 'world', 'cup', '2', '0', '1', '4']\n",
      "#yellow ['#', 'yellow']\n",
      "#yiamas ['#', 'y', 'i', 'a', 'mas']\n",
      "#ynwa ['#', 'yn', 'wa']\n",
      "#youtube ['#', 'you', 'tube']\n",
      "#yummy ['#', 'yummy']\n",
      "#yws13 ['#', 'y', 'w', 's', '1', '3']\n",
      "#zweihandvollfarm ['#', 'z', 'wei', 'hand', 'vol', 'l', 'farm']\n"
     ]
    }
   ],
   "source": [
    "def tokenize_hashtags_rev(hashtags):\n",
    "    ###\n",
    "    hashtags_tok_rev = {}\n",
    "    \n",
    "    for hashtag in hashtags:\n",
    "        tokens = []\n",
    "        rest = hashtag\n",
    "        \n",
    "        while rest:\n",
    "            # This is again to find the longest string match\n",
    "            found = False\n",
    "            for i in range(len(rest), 0, -1):\n",
    "                if rest[-i:] in words:\n",
    "                    tokens.insert(0, rest[-i:])\n",
    "                    rest = rest[:-i]\n",
    "                    found = True\n",
    "                    break\n",
    "            \n",
    "            # if we find no match then we have to select single letters\n",
    "            if not found:\n",
    "                tokens.insert(0, rest[-1])\n",
    "                rest = rest[:-1]\n",
    "        \n",
    "        hashtags_tok_rev[hashtag] = tokens\n",
    "    \n",
    "    return hashtags_tok_rev\n",
    "\n",
    "    \n",
    "#tokenise hashtags with the reversed version of MaxMatch\n",
    "tokenized_hashtags_rev = tokenize_hashtags_rev(hashtags)\n",
    "\n",
    "#print results\n",
    "for k, v in sorted(tokenized_hashtags_rev.items())[-30:]:\n",
    "    print(k, v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**For your testing:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert(len(tokenized_hashtags_rev) == len(hashtags))\n",
    "assert(tokenized_hashtags_rev[\"#newrecord\"] == [\"#\", \"new\", \"record\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4 (1.0 mark)\n",
    "\n",
    "**Instructions**: The two versions of MaxMatch will produce different results for some of the hashtags. For a hastag that has different results, our task here is to use a **unigram language model** (lecture 3) to score them to see which is better. Recall that in a unigram language model we compute P(<i>#</i>, <i>hello</i>, <i>world</i> = P(<i>#</i>)\\*P(<i>hello</i>)\\*P(<i>world</i>).\n",
    "\n",
    "You should: (1) use the NLTK's Brown corpus (`brown_words`) for collecting word frequencies (note: the words are already tokenised so no further tokenisation is needed); (2) lowercase all words in the corpus; (3) use add-one smoothing when computing the unigram probabilities; and (4) work in the log space to prevent numerical underflow.\n",
    "\n",
    "**Task**: Build a unigram language model with add-one smoothing using the word counts from the Brown corpus. Iterate through the hashtags, and for each hashtag where MaxMatch and reversed MaxMatch produce different results, print the following: (1) the hashtag; (2) the results produced by MaxMatch and reversed MaxMatch; and (3) the log probability of each result as given by the unigram language model. Note: you **do not** need to print the hashtags where MaxMatch and reversed MaxMatch produce the same results.\n",
    "\n",
    "An example output:\n",
    "```\n",
    "1. #abcd\n",
    "MaxMatch = [#, a, bc, d]; LogProb = -2.3\n",
    "Reversed MaxMatch = [#, a, b, cd]; LogProb = -3.5\n",
    "\n",
    "2. #efgh\n",
    "MaxMatch = [#, ef, g, h]; LogProb = -4.2\n",
    "Reversed MaxMatch = [#, e, fgh]; LogProb = -3.1\n",
    "\n",
    "```\n",
    "\n",
    "Have a look at the output, and see if the sequences with better language model scores (i.e. less negative) are generally more coherent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hashtag: #8thannualpubcrawl\n",
      "MaxMatch: ['#', '8', 'than', 'nu', 'alp', 'u', 'b', 'crawl']; LogProb: -90.1049\n",
      "Reversed MaxMatch: ['#', '8', 'th', 'annual', 'pub', 'crawl']; LogProb: -71.6476\n",
      "\n",
      "Hashtag: #longranger\n",
      "MaxMatch: ['#', 'long', 'ranger']; LogProb: -34.2982\n",
      "Reversed MaxMatch: ['#', 'l', 'on', 'granger']; LogProb: -44.3785\n",
      "\n",
      "Hashtag: #nickryrie\n",
      "MaxMatch: ['#', 'nick', 'r', 'yr', 'ie']; LogProb: -63.1132\n",
      "Reversed MaxMatch: ['#', 'nick', 'r', 'y', 'rie']; LogProb: -61.3214\n",
      "\n",
      "Hashtag: #thenext5goldenyears\n",
      "MaxMatch: ['#', 'then', 'ex', 't', '5', 'golden', 'year', 's']; LogProb: -81.0196\n",
      "Reversed MaxMatch: ['#', 'the', 'next', '5', 'golden', 'year', 's']; LogProb: -62.5637\n",
      "\n",
      "Hashtag: #thomassabo\n",
      "MaxMatch: ['#', 'thomas', 'sab', 'o']; LogProb: -48.1169\n",
      "Reversed MaxMatch: ['#', 'tho', 'mass', 'abo']; LogProb: -49.6410\n",
      "\n",
      "Hashtag: #panthers\n",
      "MaxMatch: ['#', 'panther', 's']; LogProb: -38.1922\n",
      "Reversed MaxMatch: ['#', 'pant', 'hers']; LogProb: -39.1877\n",
      "\n",
      "Hashtag: #cornell\n",
      "MaxMatch: ['#', 'cornel', 'l']; LogProb: -39.1877\n",
      "Reversed MaxMatch: ['#', 'cor', 'nell']; LogProb: -41.3277\n",
      "\n",
      "Hashtag: #weare90s\n",
      "MaxMatch: ['#', 'wear', 'e', '9', '0', 's']; LogProb: -66.3306\n",
      "Reversed MaxMatch: ['#', 'we', 'are', '9', '0', 's']; LogProb: -57.3560\n",
      "\n",
      "Hashtag: #hometomama\n",
      "MaxMatch: ['#', 'home', 'toma', 'ma']; LogProb: -46.7258\n",
      "Reversed MaxMatch: ['#', 'home', 'tom', 'ama']; LogProb: -44.8695\n",
      "\n",
      "Hashtag: #littlemonsters\n",
      "MaxMatch: ['#', 'little', 'monster', 's']; LogProb: -44.2226\n",
      "Reversed MaxMatch: ['#', 'lit', 't', 'lemon', 'st', 'ers']; LogProb: -73.8503\n",
      "\n",
      "Hashtag: #focusateneo\n",
      "MaxMatch: ['#', 'focus', 'aten', 'e', 'o']; LogProb: -59.3118\n",
      "Reversed MaxMatch: ['#', 'fo', 'c', 'u', 'sate', 'neo']; LogProb: -76.5987\n",
      "\n",
      "Hashtag: #rollersmusicawards\n",
      "MaxMatch: ['#', 'roller', 's', 'music', 'award', 's']; LogProb: -67.1544\n",
      "Reversed MaxMatch: ['#', 'roll', 'ers', 'music', 'award', 's']; LogProb: -68.1209\n",
      "\n",
      "Hashtag: #reachingyougpeople\n",
      "MaxMatch: ['#', 'reaching', 'you', 'g', 'people']; LogProb: -48.5768\n",
      "Reversed MaxMatch: ['#', 'reaching', 'yo', 'ug', 'people']; LogProb: -59.5077\n",
      "\n",
      "Hashtag: #strictlyus\n",
      "MaxMatch: ['#', 'strictly', 'us']; LogProb: -31.9783\n",
      "Reversed MaxMatch: ['#', 'strict', 'l', 'yus']; LogProb: -50.7097\n",
      "\n",
      "Hashtag: #bettybooppose\n",
      "MaxMatch: ['#', 'betty', 'boo', 'p', 'pose']; LogProb: -61.1267\n",
      "Reversed MaxMatch: ['#', 'betty', 'bo', 'oppose']; LogProb: -51.1758\n",
      "\n",
      "Hashtag: #endomondo\n",
      "MaxMatch: ['#', 'end', 'om', 'on', 'do']; LogProb: -47.9844\n",
      "Reversed MaxMatch: ['#', 'en', 'do', 'mon', 'do']; LogProb: -52.8259\n",
      "\n",
      "Hashtag: #fotografías\n",
      "MaxMatch: ['#', 'fot', 'og', 'ra', 'f', 'í', 'as']; LogProb: -84.8554\n",
      "Reversed MaxMatch: ['#', 'f', 'oto', 'gra', 'f', 'í', 'as']; LogProb: -81.9376\n",
      "\n",
      "Hashtag: #mamajeanneandme\n",
      "MaxMatch: ['#', 'mam', 'a', 'jeanne', 'and', 'me']; LogProb: -56.6451\n",
      "Reversed MaxMatch: ['#', 'm', 'ama', 'jeanne', 'and', 'me']; LogProb: -63.1704\n",
      "\n",
      "Hashtag: #usydhereicome\n",
      "MaxMatch: ['#', 'us', 'y', 'd', 'here', 'i', 'come']; LogProb: -64.1906\n",
      "Reversed MaxMatch: ['#', 'u', 'syd', 'here', 'i', 'come']; LogProb: -59.7843\n",
      "\n",
      "Hashtag: #ididntchoosethestudentlifeitchoseme\n",
      "MaxMatch: ['#', 'id', 'id', 'n', 'tch', 'o', 'ose', 'the', 'student', 'life', 'itch', 'ose', 'me']; LogProb: -143.6521\n",
      "Reversed MaxMatch: ['#', 'i', 'didnt', 'choose', 'the', 'student', 'life', 'it', 'cho', 'seme']; LogProb: -95.8977\n",
      "\n",
      "Hashtag: #arte\n",
      "MaxMatch: ['#', 'art', 'e']; LogProb: -32.9650\n",
      "Reversed MaxMatch: ['#', 'ar', 'te']; LogProb: -42.0209\n",
      "\n",
      "Hashtag: #scifigeek\n",
      "MaxMatch: ['#', 's', 'c', 'if', 'i', 'geek']; LogProb: -59.8568\n",
      "Reversed MaxMatch: ['#', 's', 'c', 'i', 'fi', 'geek']; LogProb: -67.5526\n",
      "\n",
      "Hashtag: #bedimages\n",
      "MaxMatch: ['#', 'bedim', 'age', 's']; LogProb: -47.4630\n",
      "Reversed MaxMatch: ['#', 'bed', 'ima', 'ges']; LogProb: -51.1758\n",
      "\n",
      "Hashtag: #toronto\n",
      "MaxMatch: ['#', 'toro', 'n', 'to']; LogProb: -42.1923\n",
      "Reversed MaxMatch: ['#', 'tor', 'onto']; LogProb: -37.9100\n",
      "\n",
      "Hashtag: #hungergames\n",
      "MaxMatch: ['#', 'hunger', 'game', 's']; LogProb: -45.1817\n",
      "Reversed MaxMatch: ['#', 'hunger', 'ga', 'mes']; LogProb: -52.4443\n",
      "\n",
      "Hashtag: #thankyoupatients\n",
      "MaxMatch: ['#', 'thank', 'youp', 'ati', 'en', 't', 's']; LogProb: -84.8662\n",
      "Reversed MaxMatch: ['#', 'thank', 'you', 'patient', 's']; LogProb: -50.7248\n",
      "\n",
      "Hashtag: #blogtourambiente\n",
      "MaxMatch: ['#', 'blo', 'g', 'tour', 'ambient', 'e']; LogProb: -73.7108\n",
      "Reversed MaxMatch: ['#', 'b', 'log', 'tou', 'ram', 'bien', 'te']; LogProb: -89.1086\n",
      "\n",
      "Hashtag: #fries\n",
      "MaxMatch: ['#', 'f', 'rie', 's']; LogProb: -49.2814\n",
      "Reversed MaxMatch: ['#', 'f', 'r', 'i', 'es']; LogProb: -54.2107\n",
      "\n",
      "Hashtag: #thewalkingdead\n",
      "MaxMatch: ['#', 'thew', 'alk', 'ing', 'dead']; LogProb: -64.8700\n",
      "Reversed MaxMatch: ['#', 'the', 'walking', 'dead']; LogProb: -35.6999\n",
      "\n",
      "Hashtag: #allwedoiscurls\n",
      "MaxMatch: ['#', 'all', 'wed', 'o', 'is', 'curl', 's']; LogProb: -72.1919\n",
      "Reversed MaxMatch: ['#', 'all', 'w', 'edo', 'is', 'curl', 's']; LogProb: -74.7946\n",
      "\n",
      "Hashtag: #innoretail\n",
      "MaxMatch: ['#', 'inn', 'ore', 'tail']; LogProb: -50.5064\n",
      "Reversed MaxMatch: ['#', 'in', 'no', 'retail']; LogProb: -35.3465\n",
      "\n",
      "Hashtag: #siberuang\n",
      "MaxMatch: ['#', 'sib', 'er', 'uang']; LogProb: -56.0279\n",
      "Reversed MaxMatch: ['#', 'si', 'ber', 'uang']; LogProb: -55.3347\n",
      "\n",
      "Hashtag: #wok\n",
      "MaxMatch: ['#', 'wo', 'k']; LogProb: -39.6230\n",
      "Reversed MaxMatch: ['#', 'w', 'ok']; LogProb: -38.4374\n",
      "\n",
      "Hashtag: #vegetables\n",
      "MaxMatch: ['#', 'vegetable', 's']; LogProb: -36.4875\n",
      "Reversed MaxMatch: ['#', 'v', 'e', 'ge', 'tables']; LogProb: -58.0838\n",
      "\n",
      "Hashtag: #hens\n",
      "MaxMatch: ['#', 'hen', 's']; LogProb: -35.7944\n",
      "Reversed MaxMatch: ['#', 'h', 'ens']; LogProb: -38.8428\n",
      "\n",
      "Hashtag: #singapore\n",
      "MaxMatch: ['#', 'sing', 'a', 'pore']; LogProb: -41.3222\n",
      "Reversed MaxMatch: ['#', 's', 'inga', 'pore']; LogProb: -51.7937\n",
      "\n",
      "Hashtag: #takeabreak\n",
      "MaxMatch: ['#', 'take', 'ab', 'reak']; LogProb: -48.2265\n",
      "Reversed MaxMatch: ['#', 'ta', 'kea', 'break']; LogProb: -51.5392\n",
      "\n",
      "Hashtag: #difd\n",
      "MaxMatch: ['#', 'di', 'f', 'd']; LogProb: -45.3496\n",
      "Reversed MaxMatch: ['#', 'd', 'if', 'd']; LogProb: -40.4684\n",
      "\n",
      "Hashtag: #palacefansinthemorning\n",
      "MaxMatch: ['#', 'palace', 'fan', 'sin', 'them', 'or', 'ning']; LogProb: -71.6178\n",
      "Reversed MaxMatch: ['#', 'palace', 'fan', 'sin', 'the', 'morning']; LogProb: -56.9324\n",
      "\n",
      "Hashtag: #swedumtl\n",
      "MaxMatch: ['#', 's', 'wed', 'um', 't', 'l']; LogProb: -71.0083\n",
      "Reversed MaxMatch: ['#', 's', 'we', 'dum', 't', 'l']; LogProb: -65.8329\n",
      "\n",
      "Hashtag: #grammyfans\n",
      "MaxMatch: ['#', 'gram', 'my', 'fan', 's']; LogProb: -54.3724\n",
      "Reversed MaxMatch: ['#', 'g', 'rammy', 'fan', 's']; LogProb: -61.1217\n",
      "\n",
      "Hashtag: #aflpowertigers\n",
      "MaxMatch: ['#', 'a', 'f', 'l', 'power', 'tiger', 's']; LogProb: -70.5002\n",
      "Reversed MaxMatch: ['#', 'a', 'f', 'l', 'power', 'ti', 'gers']; LogProb: -75.0220\n",
      "\n",
      "Hashtag: #startupfest\n",
      "MaxMatch: ['#', 'start', 'up', 'fest']; LogProb: -43.4396\n",
      "Reversed MaxMatch: ['#', 'star', 'tup', 'fest']; LogProb: -52.7698\n",
      "\n",
      "Hashtag: #southampton\n",
      "MaxMatch: ['#', 'south', 'am', 'p', 'ton']; LogProb: -52.0949\n",
      "Reversed MaxMatch: ['#', 's', 'out', 'ham', 'p', 'ton']; LogProb: -63.2789\n",
      "\n",
      "Hashtag: #vegan\n",
      "MaxMatch: ['#', 'vega', 'n']; LogProb: -38.3573\n",
      "Reversed MaxMatch: ['#', 'v', 'e', 'gan']; LogProb: -48.9821\n",
      "\n",
      "Hashtag: #magazinesandtvscreens\n",
      "MaxMatch: ['#', 'magazine', 'sand', 't', 'v', 'screen', 's']; LogProb: -76.2763\n",
      "Reversed MaxMatch: ['#', 'magazine', 'sand', 't', 'v', 's', 'c', 're', 'ens']; LogProb: -101.9918\n",
      "\n",
      "Hashtag: #melbourne\n",
      "MaxMatch: ['#', 'mel', 'bourn', 'e']; LogProb: -49.4239\n",
      "Reversed MaxMatch: ['#', 'm', 'elb', 'our', 'ne']; LogProb: -58.1224\n",
      "\n",
      "Hashtag: #instagood\n",
      "MaxMatch: ['#', 'in', 'stag', 'o', 'od']; LogProb: -54.5735\n",
      "Reversed MaxMatch: ['#', 'in', 's', 'ta', 'good']; LogProb: -50.2378\n",
      "\n",
      "Hashtag: #bradersisterhood\n",
      "MaxMatch: ['#', 'brad', 'ers', 'ist', 'er', 'hood']; LogProb: -80.5760\n",
      "Reversed MaxMatch: ['#', 'brad', 'er', 'sisterhood']; LogProb: -55.3347\n",
      "\n",
      "Hashtag: #foodporn\n",
      "MaxMatch: ['#', 'food', 'po', 'r', 'n']; LogProb: -57.0173\n",
      "Reversed MaxMatch: ['#', 'food', 'p', 'or', 'n']; LogProb: -48.6857\n",
      "\n",
      "Hashtag: #anferneehardaway\n",
      "MaxMatch: ['#', 'an', 'fern', 'e', 'eh', 'ar', 'daw', 'ay']; LogProb: -97.2247\n",
      "Reversed MaxMatch: ['#', 'an', 'f', 'er', 'nee', 'hard', 'away']; LogProb: -74.7728\n",
      "\n",
      "Hashtag: #potd\n",
      "MaxMatch: ['#', 'pot', 'd']; LogProb: -34.7218\n",
      "Reversed MaxMatch: ['#', 'po', 'td']; LogProb: -41.3277\n",
      "\n",
      "Hashtag: #uptonogood\n",
      "MaxMatch: ['#', 'up', 'ton', 'og', 'o', 'od']; LogProb: -70.5620\n",
      "Reversed MaxMatch: ['#', 'up', 'to', 'no', 'good']; LogProb: -37.9561\n",
      "\n",
      "Hashtag: #shopaholic\n",
      "MaxMatch: ['#', 'shop', 'aho', 'li', 'c']; LogProb: -61.0719\n",
      "Reversed MaxMatch: ['#', 'sho', 'paho', 'li', 'c']; LogProb: -65.2308\n",
      "\n",
      "Hashtag: #realgoodbikes\n",
      "MaxMatch: ['#', 'real', 'good', 'bike', 's']; LogProb: -54.6492\n",
      "Reversed MaxMatch: ['#', 'real', 'good', 'b', 'i', 'k', 'es']; LogProb: -70.1876\n",
      "\n",
      "Hashtag: #instalook\n",
      "MaxMatch: ['#', 'in', 'st', 'alo', 'ok']; LogProb: -58.2748\n",
      "Reversed MaxMatch: ['#', 'in', 's', 'ta', 'look']; LogProb: -50.9396\n",
      "\n",
      "Hashtag: #startuphub\n",
      "MaxMatch: ['#', 'start', 'up', 'hub']; LogProb: -40.9547\n",
      "Reversed MaxMatch: ['#', 'star', 'tup', 'hub']; LogProb: -50.2848\n",
      "\n",
      "Hashtag: #veganfood\n",
      "MaxMatch: ['#', 'vega', 'n', 'food']; LogProb: -47.3671\n",
      "Reversed MaxMatch: ['#', 'v', 'e', 'gan', 'food']; LogProb: -57.9918\n",
      "\n",
      "Hashtag: #statoftheday\n",
      "MaxMatch: ['#', 'st', 'at', 'oft', 'he', 'day']; LogProb: -59.0615\n",
      "Reversed MaxMatch: ['#', 's', 'tat', 'of', 'the', 'day']; LogProb: -52.0208\n",
      "\n",
      "Hashtag: #goingout\n",
      "MaxMatch: ['#', 'going', 'out']; LogProb: -28.3807\n",
      "Reversed MaxMatch: ['#', 'go', 'in', 'gout']; LogProb: -38.5200\n",
      "\n",
      "Hashtag: #imsobored\n",
      "MaxMatch: ['#', 'i', 'm', 'sob', 'ore', 'd']; LogProb: -68.7271\n",
      "Reversed MaxMatch: ['#', 'i', 'm', 'so', 'bo', 'red']; LogProb: -59.0836\n",
      "\n",
      "Hashtag: #movies\n",
      "MaxMatch: ['#', 'movie', 's']; LogProb: -35.4842\n",
      "Reversed MaxMatch: ['#', 'mo', 'v', 'i', 'es']; LogProb: -58.1529\n",
      "\n",
      "Hashtag: #txlege\n",
      "MaxMatch: ['#', 't', 'x', 'leg', 'e']; LogProb: -55.1144\n",
      "Reversed MaxMatch: ['#', 't', 'x', 'l', 'e', 'ge']; LogProb: -69.2671\n",
      "\n",
      "Hashtag: #photoby\n",
      "MaxMatch: ['#', 'photo', 'by']; LogProb: -31.6523\n",
      "Reversed MaxMatch: ['#', 'pho', 'toby']; LogProb: -42.0209\n",
      "\n",
      "Hashtag: #mtlnewtech\n",
      "MaxMatch: ['#', 'm', 't', 'l', 'newt', 'e', 'c', 'h']; LogProb: -88.1397\n",
      "Reversed MaxMatch: ['#', 'm', 't', 'l', 'new', 'tech']; LogProb: -64.4214\n",
      "\n",
      "Hashtag: #ariona\n",
      "MaxMatch: ['#', 'arion', 'a']; LogProb: -31.9692\n",
      "Reversed MaxMatch: ['#', 'ar', 'i', 'ona']; LogProb: -47.4782\n",
      "\n",
      "Hashtag: #askolly\n",
      "MaxMatch: ['#', 'ask', 'o', 'l', 'ly']; LogProb: -59.0460\n",
      "Reversed MaxMatch: ['#', 'as', 'kol', 'ly']; LogProb: -47.1385\n",
      "\n",
      "Hashtag: #butitsalsokindofaphone\n",
      "MaxMatch: ['#', 'but', 'its', 'also', 'kind', 'of', 'a', 'phone']; LogProb: -58.8561\n",
      "Reversed MaxMatch: ['#', 'but', 'its', 'also', 'kin', 'do', 'fa', 'phone']; LogProb: -76.8431\n",
      "\n",
      "Hashtag: #grandmarnier\n",
      "MaxMatch: ['#', 'grandma', 'r', 'ni', 'er']; LogProb: -63.7322\n",
      "Reversed MaxMatch: ['#', 'grand', 'm', 'arni', 'er']; LogProb: -63.3098\n",
      "\n",
      "Hashtag: #usydoweek\n",
      "MaxMatch: ['#', 'us', 'y', 'dow', 'e', 'e', 'k']; LogProb: -78.3063\n",
      "Reversed MaxMatch: ['#', 'us', 'y', 'do', 'week']; LogProb: -48.8883\n",
      "\n",
      "Hashtag: #getfreetattooaviciipasses\n",
      "MaxMatch: ['#', 'get', 'freet', 'at', 'too', 'a', 'vic', 'i', 'i', 'passe', 's']; LogProb: -100.2440\n",
      "Reversed MaxMatch: ['#', 'get', 'free', 'tattoo', 'a', 'vic', 'i', 'i', 'pass', 'es']; LogProb: -94.6286\n",
      "\n",
      "Hashtag: #jordaan\n",
      "MaxMatch: ['#', 'jo', 'r', 'da', 'an']; LogProb: -53.4256\n",
      "Reversed MaxMatch: ['#', 'j', 'or', 'da', 'an']; LogProb: -48.7447\n",
      "\n",
      "Hashtag: #oilandgas\n",
      "MaxMatch: ['#', 'oil', 'and', 'gas']; LogProb: -36.5883\n",
      "Reversed MaxMatch: ['#', 'o', 'i', 'land', 'gas']; LogProb: -48.1951\n",
      "\n",
      "Hashtag: #peperoni\n",
      "MaxMatch: ['#', 'pep', 'er', 'on', 'i']; LogProb: -52.6690\n",
      "Reversed MaxMatch: ['#', 'pep', 'e', 'ro', 'ni']; LogProb: -66.3212\n",
      "\n",
      "Hashtag: #1stsundayofoctober\n",
      "MaxMatch: ['#', '1', 'st', 'sunday', 'ofo', 'c', 'tobe', 'r']; LogProb: -92.6941\n",
      "Reversed MaxMatch: ['#', '1', 'st', 'sunday', 'of', 'october']; LogProb: -58.6938\n",
      "\n",
      "Hashtag: #askherforfback\n",
      "MaxMatch: ['#', 'ask', 'her', 'for', 'f', 'back']; LogProb: -51.5202\n",
      "Reversed MaxMatch: ['#', 'ask', 'her', 'f', 'orf', 'back']; LogProb: -60.6782\n",
      "\n",
      "Hashtag: #fotorus\n",
      "MaxMatch: ['#', 'fot', 'or', 'us']; LogProb: -41.1672\n",
      "Reversed MaxMatch: ['#', 'fo', 'torus']; LogProb: -42.0209\n",
      "\n",
      "Hashtag: #chargers\n",
      "MaxMatch: ['#', 'charger', 's']; LogProb: -38.8854\n",
      "Reversed MaxMatch: ['#', 'char', 'gers']; LogProb: -41.3277\n",
      "\n",
      "Hashtag: #ahimacon14\n",
      "MaxMatch: ['#', 'ah', 'ima', 'con', '1', '4']; LogProb: -67.2250\n",
      "Reversed MaxMatch: ['#', 'a', 'hi', 'macon', '1', '4']; LogProb: -58.8329\n",
      "\n",
      "Hashtag: #skeemsaam\n",
      "MaxMatch: ['#', 'skee', 'm', 'saa', 'm']; LogProb: -64.3684\n",
      "Reversed MaxMatch: ['#', 's', 'k', 'e', 'em', 's', 'aam']; LogProb: -84.0568\n",
      "\n",
      "Hashtag: #alliswell\n",
      "MaxMatch: ['#', 'all', 'is', 'well']; LogProb: -31.9994\n",
      "Reversed MaxMatch: ['#', 'al', 'li', 'swell']; LogProb: -50.6903\n",
      "\n",
      "Hashtag: #google\n",
      "MaxMatch: ['#', 'goo', 'g', 'l', 'e']; LogProb: -60.6548\n",
      "Reversed MaxMatch: ['#', 'go', 'ogle']; LogProb: -35.5799\n",
      "\n",
      "Hashtag: #thevoiceau\n",
      "MaxMatch: ['#', 'the', 'voice', 'a', 'u']; LogProb: -40.7632\n",
      "Reversed MaxMatch: ['#', 'the', 'v', 'o', 'i', 'c', 'ea', 'u']; LogProb: -78.2791\n",
      "\n",
      "Hashtag: #cdnpoli\n",
      "MaxMatch: ['#', 'c', 'd', 'n', 'pol', 'i']; LogProb: -63.0927\n",
      "Reversed MaxMatch: ['#', 'c', 'd', 'n', 'po', 'li']; LogProb: -70.9492\n",
      "\n",
      "Hashtag: #socal\n",
      "MaxMatch: ['#', 'soc', 'al']; LogProb: -38.7628\n",
      "Reversed MaxMatch: ['#', 'so', 'cal']; LogProb: -33.7339\n",
      "\n",
      "Hashtag: #shakes\n",
      "MaxMatch: ['#', 'shake', 's']; LogProb: -35.9950\n",
      "Reversed MaxMatch: ['#', 's', 'hak', 'es']; LogProb: -52.8924\n",
      "\n",
      "Hashtag: #instaframeplus\n",
      "MaxMatch: ['#', 'in', 'st', 'a', 'frame', 'plus']; LogProb: -55.4138\n",
      "Reversed MaxMatch: ['#', 'in', 's', 'ta', 'frame', 'plus']; LogProb: -62.3301\n",
      "\n",
      "Hashtag: #insightmediasingapore\n",
      "MaxMatch: ['#', 'insight', 'media', 'sing', 'a', 'pore']; LogProb: -63.5615\n",
      "Reversed MaxMatch: ['#', 'insight', 'me', 'di', 'as', 'inga', 'pore']; LogProb: -74.7149\n",
      "\n",
      "Hashtag: #cumannanya\n",
      "MaxMatch: ['#', 'cum', 'anna', 'n', 'ya']; LogProb: -62.6824\n",
      "Reversed MaxMatch: ['#', 'c', 'u', 'mannan', 'ya']; LogProb: -60.9823\n",
      "\n",
      "Hashtag: #crepes\n",
      "MaxMatch: ['#', 'crepe', 's']; LogProb: -38.1922\n",
      "Reversed MaxMatch: ['#', 'c', 're', 'pes']; LogProb: -49.8375\n",
      "\n",
      "Hashtag: #blessedmorethanicanimagine\n",
      "MaxMatch: ['#', 'blessed', 'more', 'than', 'i', 'can', 'imagine']; LogProb: -60.0585\n",
      "Reversed MaxMatch: ['#', 'blessed', 'more', 'th', 'ani', 'can', 'imagine']; LogProb: -75.0000\n",
      "\n",
      "Hashtag: #5sosacousticatlanta\n",
      "MaxMatch: ['#', '5', 'so', 'saco', 'ust', 'i', 'cat', 'lant', 'a']; LogProb: -91.7265\n",
      "Reversed MaxMatch: ['#', '5', 's', 'os', 'acoustic', 'atlanta']; LogProb: -70.9736\n",
      "\n",
      "Hashtag: #wearesocial\n",
      "MaxMatch: ['#', 'wear', 'es', 'o', 'c', 'i', 'al']; LogProb: -74.5576\n",
      "Reversed MaxMatch: ['#', 'we', 'are', 'social']; LogProb: -33.8134\n",
      "\n",
      "Hashtag: #harimaumalaya\n",
      "MaxMatch: ['#', 'ha', 'rima', 'um', 'ala', 'ya']; LogProb: -79.7243\n",
      "Reversed MaxMatch: ['#', 'h', 'ar', 'i', 'mau', 'mala', 'ya']; LogProb: -84.7116\n",
      "\n",
      "Hashtag: #lebedeintennis\n",
      "MaxMatch: ['#', 'l', 'e', 'bed', 'e', 'in', 'tennis']; LogProb: -70.1955\n",
      "Reversed MaxMatch: ['#', 'l', 'e', 'be', 'de', 'in', 'tennis']; LogProb: -65.1883\n",
      "\n",
      "Hashtag: #taintedlove\n",
      "MaxMatch: ['#', 'taint', 'ed', 'love']; LogProb: -47.2489\n",
      "Reversed MaxMatch: ['#', 'tain', 'ted', 'love']; LogProb: -48.5017\n",
      "\n",
      "Hashtag: #thedisadvantagesofakindle\n",
      "MaxMatch: ['#', 'the', 'disadvantage', 'sofa', 'kindle']; LogProb: -55.3236\n",
      "Reversed MaxMatch: ['#', 'the', 'di', 'sad', 'v', 'anta', 'ges', 'of', 'akindle']; LogProb: -92.5612\n",
      "\n",
      "Hashtag: #workmates\n",
      "MaxMatch: ['#', 'work', 'mate', 's']; LogProb: -43.1641\n",
      "Reversed MaxMatch: ['#', 'work', 'm', 'ates']; LogProb: -46.5574\n",
      "\n",
      "Hashtag: #meetup\n",
      "MaxMatch: ['#', 'meet', 'up']; LogProb: -29.4654\n",
      "Reversed MaxMatch: ['#', 'me', 'e', 'tup']; LogProb: -45.2393\n",
      "\n",
      "Hashtag: #excitables\n",
      "MaxMatch: ['#', 'excitable', 's']; LogProb: -38.8854\n",
      "Reversed MaxMatch: ['#', 'ex', 'c', 'i', 'tables']; LogProb: -51.7759\n",
      "\n",
      "Hashtag: #beringmy\n",
      "MaxMatch: ['#', 'beri', 'n', 'g', 'my']; LogProb: -56.3534\n",
      "Reversed MaxMatch: ['#', 'be', 'ring', 'my']; LogProb: -36.2114\n",
      "\n",
      "Hashtag: #sinvsmas\n",
      "MaxMatch: ['#', 'sin', 'v', 'sma', 's']; LogProb: -59.5781\n",
      "Reversed MaxMatch: ['#', 'sin', 'v', 's', 'mas']; LogProb: -59.5781\n",
      "\n",
      "Hashtag: #learningcommunties\n",
      "MaxMatch: ['#', 'learning', 'c', 'om', 'munt', 'ie', 's']; LogProb: -85.9984\n",
      "Reversed MaxMatch: ['#', 'learning', 'c', 'om', 'mun', 'ti', 'es']; LogProb: -88.4407\n",
      "\n",
      "Hashtag: #yiamas\n",
      "MaxMatch: ['#', 'y', 'i', 'ama', 's']; LogProb: -55.8648\n",
      "Reversed MaxMatch: ['#', 'y', 'i', 'a', 'mas']; LogProb: -49.6417\n",
      "\n",
      "Hashtag: #ladygaga\n",
      "MaxMatch: ['#', 'lady', 'gag', 'a']; LogProb: -39.9722\n",
      "Reversed MaxMatch: ['#', 'lady', 'g', 'aga']; LogProb: -48.8002\n",
      "\n",
      "Hashtag: #bringithomemy\n",
      "MaxMatch: ['#', 'bring', 'it', 'home', 'my']; LogProb: -42.3969\n",
      "Reversed MaxMatch: ['#', 'brin', 'git', 'home', 'my']; LogProb: -54.9345\n",
      "\n",
      "Hashtag: #stopcomplaining\n",
      "MaxMatch: ['#', 'stop', 'complain', 'ing']; LogProb: -48.7472\n",
      "Reversed MaxMatch: ['#', 'stop', 'c', 'om', 'p', 'lai', 'ning']; LogProb: -84.1051\n",
      "\n",
      "Hashtag: #givesmehope\n",
      "MaxMatch: ['#', 'give', 's', 'me', 'hope']; LogProb: -48.6764\n",
      "Reversed MaxMatch: ['#', 'gi', 'v', 'es', 'me', 'hope']; LogProb: -68.4528\n",
      "\n",
      "Hashtag: #nced\n",
      "MaxMatch: ['#', 'n', 'ce', 'd']; LogProb: -48.4325\n",
      "Reversed MaxMatch: ['#', 'n', 'c', 'ed']; LogProb: -44.9212\n",
      "\n",
      "Hashtag: #burgers\n",
      "MaxMatch: ['#', 'burg', 'ers']; LogProb: -42.0209\n",
      "Reversed MaxMatch: ['#', 'bur', 'gers']; LogProb: -42.0209\n",
      "\n",
      "Hashtag: #isibaya\n",
      "MaxMatch: ['#', 'is', 'iba', 'ya']; LogProb: -45.1971\n",
      "Reversed MaxMatch: ['#', 'i', 'si', 'baya']; LogProb: -46.7850\n",
      "\n",
      "Hashtag: #pechanga\n",
      "MaxMatch: ['#', 'pech', 'an', 'ga']; LogProb: -47.1076\n",
      "Reversed MaxMatch: ['#', 'p', 'e', 'changa']; LogProb: -47.9705\n",
      "\n",
      "Hashtag: #thankyoulord\n",
      "MaxMatch: ['#', 'thank', 'youl', 'or', 'd']; LogProb: -54.1476\n",
      "Reversed MaxMatch: ['#', 'thank', 'you', 'lord']; LogProb: -39.7759\n",
      "\n",
      "Hashtag: #booyaa\n",
      "MaxMatch: ['#', 'boo', 'ya', 'a']; LogProb: -43.6735\n",
      "Reversed MaxMatch: ['#', 'boo', 'y', 'aa']; LogProb: -52.8498\n",
      "\n",
      "Hashtag: #blackhat\n",
      "MaxMatch: ['#', 'black', 'hat']; LogProb: -32.6597\n",
      "Reversed MaxMatch: ['#', 'b', 'lac', 'khat']; LogProb: -51.3644\n",
      "\n",
      "Hashtag: #vscocam\n",
      "MaxMatch: ['#', 'v', 's', 'coca', 'm']; LogProb: -60.7339\n",
      "Reversed MaxMatch: ['#', 'v', 's', 'c', 'o', 'cam']; LogProb: -68.7811\n",
      "\n",
      "Hashtag: #olah\n",
      "MaxMatch: ['#', 'o', 'la', 'h']; LogProb: -45.5837\n",
      "Reversed MaxMatch: ['#', 'o', 'l', 'ah']; LogProb: -46.7633\n",
      "\n",
      "Hashtag: #makeupfree\n",
      "MaxMatch: ['#', 'make', 'up', 'free']; LogProb: -36.2440\n",
      "Reversed MaxMatch: ['#', 'ma', 'keup', 'free']; LogProb: -47.4714\n",
      "\n",
      "Hashtag: #letmesleep\n",
      "MaxMatch: ['#', 'let', 'mes', 'leep']; LogProb: -50.0746\n",
      "Reversed MaxMatch: ['#', 'let', 'me', 'sleep']; LogProb: -38.8100\n",
      "\n",
      "Hashtag: #pmattheashes\n",
      "MaxMatch: ['#', 'p', 'matt', 'he', 'ashes']; LogProb: -53.1946\n",
      "Reversed MaxMatch: ['#', 'p', 'mat', 'the', 'ashes']; LogProb: -50.7975\n",
      "\n",
      "Hashtag: #loadsoffun\n",
      "MaxMatch: ['#', 'load', 'so', 'f', 'fun']; LogProb: -51.1947\n",
      "Reversed MaxMatch: ['#', 'load', 's', 'of', 'fun']; LogProb: -48.7613\n",
      "\n",
      "Hashtag: #imsosore\n",
      "MaxMatch: ['#', 'i', 'm', 'soso', 're']; LogProb: -57.2656\n",
      "Reversed MaxMatch: ['#', 'i', 'm', 'so', 'sore']; LogProb: -48.6602\n",
      "\n",
      "Hashtag: #5sosfam\n",
      "MaxMatch: ['#', '5', 'so', 's', 'fam']; LogProb: -54.3426\n",
      "Reversed MaxMatch: ['#', '5', 's', 'os', 'fam']; LogProb: -61.2433\n",
      "\n",
      "Hashtag: #lovers\n",
      "MaxMatch: ['#', 'lover', 's']; LogProb: -35.8897\n",
      "Reversed MaxMatch: ['#', 'lo', 'v', 'ers']; LogProb: -49.5602\n",
      "\n",
      "Hashtag: #bored\n",
      "MaxMatch: ['#', 'bore', 'd']; LogProb: -34.8702\n",
      "Reversed MaxMatch: ['#', 'bo', 'red']; LogProb: -36.0395\n",
      "\n",
      "Hashtag: #canadaelections\n",
      "MaxMatch: ['#', 'canada', 'election', 's']; LogProb: -44.9803\n",
      "Reversed MaxMatch: ['#', 'canada', 'el', 'e', 'c', 'ti', 'ons']; LogProb: -82.2381\n",
      "\n",
      "Hashtag: #uniexamonasaturday\n",
      "MaxMatch: ['#', 'unie', 'x', 'am', 'ona', 'saturday']; LogProb: -71.5774\n",
      "Reversed MaxMatch: ['#', 'u', 'ni', 'ex', 'a', 'mona', 'saturday']; LogProb: -80.0398\n",
      "\n",
      "Hashtag: #rainorshine\n",
      "MaxMatch: ['#', 'rain', 'or', 'shine']; LogProb: -41.6289\n",
      "Reversed MaxMatch: ['#', 'r', 'ai', 'nor', 'shine']; LogProb: -58.6082\n",
      "\n",
      "Hashtag: #flambees\n",
      "MaxMatch: ['#', 'flamb', 'e', 'es']; LogProb: -52.3143\n",
      "Reversed MaxMatch: ['#', 'flam', 'bees']; LogProb: -39.2483\n",
      "\n",
      "Hashtag: #ritenow\n",
      "MaxMatch: ['#', 'rite', 'now']; LogProb: -32.6421\n",
      "Reversed MaxMatch: ['#', 'rit', 'enow']; LogProb: -42.0209\n",
      "\n",
      "Hashtag: #cbcolympics\n",
      "MaxMatch: ['#', 'c', 'b', 'coly', 'm', 'pic', 's']; LogProb: -82.6126\n",
      "Reversed MaxMatch: ['#', 'c', 'b', 'c', 'olympic', 's']; LogProb: -64.5554\n",
      "\n",
      "Hashtag: #tcschleissheim\n",
      "MaxMatch: ['#', 't', 'c', 's', 'c', 'h', 'lei', 's', 'she', 'i', 'm']; LogProb: -111.3210\n",
      "Reversed MaxMatch: ['#', 't', 'c', 's', 'c', 'h', 'l', 'e', 'is', 's', 'hei', 'm']; LogProb: -126.0685\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk.corpus import brown\n",
    "import math\n",
    "\n",
    "brown_words = brown.words()\n",
    "\n",
    "# We need to lowercase words from the brown corpus\n",
    "brown_words = [word.lower() for word in brown.words()]\n",
    "\n",
    "# NOw we calculate word frequencies for the unigram model \n",
    "word_freq = nltk.FreqDist(brown_words)\n",
    "total_words = len(brown_words)\n",
    "vocab_size = len(word_freq)\n",
    "\n",
    "# This is to do add-one smoothing \n",
    "word_freq_smooth = {word: (count + 1) / (total_words + vocab_size) for word, count in word_freq.items()}\n",
    "\n",
    "#Function for the unigram model\n",
    "def unigram_log_prob(words):\n",
    "    log_prob = 0.0\n",
    "    for word in words:\n",
    "        #add-one smoothing for unknown words\n",
    "        log_prob += math.log(word_freq_smooth.get(word, 1 / (total_words + vocab_size)))\n",
    "    return log_prob\n",
    "\n",
    "#Here we're gonna compare the maxmatch and maxmatch reverse tokens \n",
    "def compare_maxmatch(hashtags):\n",
    "    for hashtag in hashtags:\n",
    "        \n",
    "        maxmatch_tokens = tokenize_hashtags([hashtag])[hashtag]\n",
    "        rev_maxmatch_tokens = tokenize_hashtags_rev([hashtag])[hashtag]\n",
    "        \n",
    "        # This is to check if MaxMatch and reversed MaxMatch produce different results\n",
    "        if maxmatch_tokens != rev_maxmatch_tokens:\n",
    "            maxmatch_log_prob = unigram_log_prob(maxmatch_tokens)\n",
    "            rev_maxmatch_log_prob = unigram_log_prob(rev_maxmatch_tokens)\n",
    "            \n",
    "            print(f\"Hashtag: {hashtag}\")\n",
    "            print(f\"MaxMatch: {maxmatch_tokens}; LogProb: {maxmatch_log_prob:.4f}\")\n",
    "            print(f\"Reversed MaxMatch: {rev_maxmatch_tokens}; LogProb: {rev_maxmatch_log_prob:.4f}\\n\")\n",
    "\n",
    "\n",
    "#Checking the final results\n",
    "compare_maxmatch(hashtags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Classification (4 marks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5 (1.0 mark)\n",
    "\n",
    "**Instructions**: Here we are interested to do text classification, to predict the country of origin of a given tweet. The task here is to create training, development and test partitions from the preprocessed data (`x_processed`) and convert the bag-of-words representation into feature vectors.\n",
    "\n",
    "**Task**: Create training, development and test partitions with a 70%/15%/15% ratio. Remember to preserve the ratio of the classes for all your partitions. That is, say we have only 2 classes and 70% of instances are labelled class A and 30% of instances are labelled class B, then the instances in training, development and test partitions should also preserve this 7:3 ratio. You may use sklearn's builtin functions for doing data partitioning.\n",
    "\n",
    "Next, turn the bag-of-words dictionary of each tweet into a feature vector. You may also use sklearn's builtin functions for doing this (but if you don't want to use sklearn that's fine).\n",
    "\n",
    "You should produce 6 objects: `x_train`, `x_dev`, `x_test` which contain the input feature vectors, and `y_train`, `y_dev` and `y_test` which contain the labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set shape: (660, 4387)\n",
      "Development set shape: (141, 4387)\n",
      "Test set shape: (142, 4387)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction import DictVectorizer\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "x_train, x_dev, x_test = None, None, None\n",
    "y_train, y_dev, y_test = None, None, None\n",
    "\n",
    "###\n",
    "# Convert bag-of-words representation to feature vectors\n",
    "vectorizer = DictVectorizer()\n",
    "X = vectorizer.fit_transform(x_processed)\n",
    "\n",
    "# We're Splitting the data into training, development, and test sets 70%/15%/15% ratio. \n",
    "#Keeping the class ratio same as well. \n",
    "X_train, X_temp, y_train, y_temp = train_test_split(X, y_processed, test_size=0.3, random_state=42, stratify=y_processed)\n",
    "X_dev, X_test, y_dev, y_test = train_test_split(X_temp, y_temp, test_size=0.5, random_state=42, stratify=y_temp)\n",
    "\n",
    "# Converting back to arrays for consistency\n",
    "x_train = X_train.toarray()\n",
    "x_dev = X_dev.toarray()\n",
    "x_test = X_test.toarray()\n",
    "\n",
    "# Checking if it is done correctly\n",
    "print(\"Training set shape:\", x_train.shape)\n",
    "print(\"Development set shape:\", x_dev.shape)\n",
    "print(\"Test set shape:\", x_test.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 6 (1.0 mark)\n",
    "\n",
    "**Instructions**: Now, let's build some classifiers. Here, we'll be comparing Naive Bayes and Logistic Regression. For each, you need to first find a good value for their main regularisation hyper-parameters, which you should identify using the scikit-learn docs or other resources. Use the development set you created for this tuning process; do **not** use cross-validation in the training set, or involve the test set in any way. You don't need to show all your work, but you do need to print out the **accuracy** with enough different settings to strongly suggest you have found an optimal or near-optimal choice. We should not need to look at your code to interpret the output. Your code should not take more than 1 minute to run (or penalty applies).\n",
    "\n",
    "**Task**: Implement two text classifiers: Naive Bayes and Logistic Regression. Tune the hyper-parameters of these classifiers and print the task performance (accuracy) for different hyper-parameter settings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tuning hyperparameters for Naive Bayes:\n",
      "Hyperparameter = 0.01, Accuracy = 0.2411\n",
      "Hyperparameter = 0.1, Accuracy = 0.2553\n",
      "Hyperparameter = 0.5, Accuracy = 0.2482\n",
      "Hyperparameter = 1.0, Accuracy = 0.2340\n",
      "Hyperparameter = 5.0, Accuracy = 0.2199\n",
      "Hyperparameter = 10.0, Accuracy = 0.2340\n",
      "Best alpha for Naive Bayes: 0.1\n",
      "\n",
      "Tuning hyperparameters for Logistic Regression:\n",
      "Hyperparameter = 0.01, Accuracy = 0.2199\n",
      "Hyperparameter = 0.1, Accuracy = 0.2128\n",
      "Hyperparameter = 1.0, Accuracy = 0.2270\n",
      "Hyperparameter = 10.0, Accuracy = 0.2411\n",
      "Hyperparameter = 100.0, Accuracy = 0.2411\n",
      "Best C for Logistic Regression: 10.0\n"
     ]
    }
   ],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "\n",
    "# This function is to train and evaluate classifiers with different hyperparameter settings\n",
    "def tune_hyperparameters(classifier, hyperparameter_values, x_train, y_train, x_dev, y_dev):\n",
    "    best_accuracy = 0.0\n",
    "    best_hyperparameter = None\n",
    "    \n",
    "    for value in hyperparameter_values:\n",
    "        # Set the hyperparameter value\n",
    "        #for MultinomialNB we set the alpha value\n",
    "        if isinstance(classifier, MultinomialNB):\n",
    "            classifier.set_params(alpha=value)\n",
    "            #For logistic we set the C value\n",
    "        elif isinstance(classifier, LogisticRegression):\n",
    "            classifier.set_params(C=value)\n",
    "        \n",
    "        # Training the classifier \n",
    "        classifier.fit(x_train, y_train)\n",
    "        \n",
    "        # Evaluation is done on the development set only\n",
    "        y_pred = classifier.predict(x_dev)\n",
    "        accuracy = accuracy_score(y_dev, y_pred)\n",
    "        \n",
    "    \n",
    "        print(f\"Hyperparameter = {value}, Accuracy = {accuracy:.4f}\")\n",
    "        \n",
    "        # Finally, We Update the best hyperparameter if it gives us the best accuracy\n",
    "        if accuracy > best_accuracy:\n",
    "            best_accuracy = accuracy\n",
    "            best_hyperparameter = value\n",
    "    \n",
    "    return best_hyperparameter\n",
    "\n",
    "# A set of standard hyparameters value to try \n",
    "alpha_values = [0.01, 0.1, 0.5, 1.0, 5.0, 10.0]\n",
    "C_values = [0.01, 0.1, 1.0, 10.0, 100.0]\n",
    "\n",
    "# Tuning hyperparameters for Naive Bayes\n",
    "print(\"Tuning hyperparameters for Naive Bayes:\")\n",
    "nb_classifier = MultinomialNB()\n",
    "best_alpha = tune_hyperparameters(nb_classifier, alpha_values, x_train, y_train, x_dev, y_dev)\n",
    "print(\"Best alpha for Naive Bayes:\", best_alpha)\n",
    "\n",
    "# Tuning hyperparameters for Logistic Regression\n",
    "print(\"\\nTuning hyperparameters for Logistic Regression:\")\n",
    "lr_classifier = LogisticRegression(max_iter=1000)\n",
    "best_C = tune_hyperparameters(lr_classifier, C_values, x_train, y_train, x_dev, y_dev)\n",
    "print(\"Best C for Logistic Regression:\", best_C)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 7 (1.0 mark)\n",
    "\n",
    "**Instructions**: Using the best settings you have found, compare the two classifiers based on performance in the test set. Print out both **accuracy** and **macro-averaged F-score** for each classifier. Be sure to label your output. You may use sklearn's inbuilt functions.\n",
    "\n",
    "**Task**: Compute test performance in terms of accuracy and macro-averaged F-score for both Naive Bayes and Logistic Regression, using their optimal hyper-parameter settings based on their development performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performance on the test set:\n",
      "Naive Bayes Classifier:\n",
      "Accuracy: 0.2887\n",
      "Macro-averaged F-score: 0.2835\n",
      "\n",
      "Logistic Regression Classifier:\n",
      "Accuracy: 0.2394\n",
      "Macro-averaged F-score: 0.2246\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score, f1_score\n",
    "\n",
    "# Training Naive Bayes classifier with the best hyperparameter\n",
    "nb_classifier = MultinomialNB(alpha=best_alpha)\n",
    "nb_classifier.fit(x_train, y_train)\n",
    "\n",
    "# Training Logistic Regression classifier with the best hyperparameter\n",
    "lr_classifier = LogisticRegression(C=best_C, max_iter=1000)\n",
    "lr_classifier.fit(x_train, y_train)\n",
    "\n",
    "# We find predictions on the test set\n",
    "nb_predictions = nb_classifier.predict(x_test)\n",
    "lr_predictions = lr_classifier.predict(x_test)\n",
    "\n",
    "# Computing accuracy and macro-averaged F-score for Naive Bayes & Logistic regression.\n",
    "nb_accuracy = accuracy_score(y_test, nb_predictions)\n",
    "nb_f1_macro = f1_score(y_test, nb_predictions, average='macro')\n",
    "\n",
    "lr_accuracy = accuracy_score(y_test, lr_predictions)\n",
    "lr_f1_macro = f1_score(y_test, lr_predictions, average='macro')\n",
    "\n",
    "\n",
    "print(\"Performance on the test set:\")\n",
    "print(\"Naive Bayes Classifier:\")\n",
    "print(f\"Accuracy: {nb_accuracy:.4f}\")\n",
    "print(f\"Macro-averaged F-score: {nb_f1_macro:.4f}\\n\")\n",
    "\n",
    "print(\"Logistic Regression Classifier:\")\n",
    "print(f\"Accuracy: {lr_accuracy:.4f}\")\n",
    "print(f\"Macro-averaged F-score: {lr_f1_macro:.4f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "source": [
    "### Question 8 (1.0 mark)\n",
    "\n",
    "**Instructions**: Print the most important features and their weights for each class for the two classifiers.\n",
    "\n",
    "\n",
    "**Task**: For each of the classifiers (Logistic Regression and Naive Bayes) you've built in the previous question, print out the top-20 features (words) with the highest weight for each class (countries).\n",
    "\n",
    "An example output:\n",
    "```\n",
    "Classifier = Logistic Regression\n",
    "\n",
    "Country = au\n",
    "aaa (0.999) bbb (0.888) ccc (0.777) ...\n",
    "\n",
    "Country = ca\n",
    "aaa (0.999) bbb (0.888) ccc (0.777) ...\n",
    "\n",
    "Classifier = Naive Bayes\n",
    "\n",
    "Country = au\n",
    "aaa (-1.0) bbb (-2.0) ccc (-3.0) ...\n",
    "\n",
    "Country = ca\n",
    "aaa (-1.0) bbb (-2.0) ccc (-3.0) ...\n",
    "```\n",
    "\n",
    "Have a look at the output, and see if you notice any trend/pattern in the words for each country."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classifier = Logistic Regression\n",
      "\n",
      "Country = au\n",
      "australia (1.9312) #melbourne (1.8357) melbourne (1.7831) little (1.6712) @micksunnyg (1.6099) literally (1.5084) summerpoyi's (1.4988) https://t.co/7rcjjptvl7 (1.4988) @dasheryoung (1.4733) lachie (1.4733) @whennboys (1.3477) @jackgilinsky (1.3477) australian (1.3167) please (1.3035) http://t.co/blhj9cmxit (1.2570) @christorrano (1.2570) green (1.2226) ha (1.2147) beach (1.2073) @thomjrob (1.1496) \n",
      "\n",
      "Country = ca\n",
      "@aliclarke (1.9057) happen (1.7706) hate (1.6019) thing (1.5207) gonna (1.5171) bed (1.4644) really (1.4508) @cheetahbiatch (1.4187) xoxo (1.4187) learning (1.3198) healthy (1.3031) @samanthasharris (1.2818) @lola9793 (1.2814) #nochillzone (1.2718) joking (1.2320) right (1.2267) cutest (1.1999) awesome (1.1433) think (1.1357) carlyle (1.1306) \n",
      "\n",
      "Country = de\n",
      "roseninsel (1.7749) https://t.co/df7ficsci3 (1.7749) #utopia (1.4572) https://t.co/brkwmsvzrb (1.4572) jessica (1.4572) hyde (1.4572) gauting (1.4572) #truegrip (1.4572) done (1.3700) night (1.3153) adlib (1.3012) https://t.co/psveyka5g3 (1.3012) workout (1.2730) https://t.co/i7j5pmd3mx (1.2730) @kelsxmclaughlin (1.2364) cannot (1.2364) posted (1.2268) yay (1.2153) ja (1.1533) hahah (1.1533) \n",
      "\n",
      "Country = gb\n",
      "x (2.0774) stressed (1.9135) prom (1.6634) interesting (1.6634) http://t.co/w0tjrah9y7 (1.6634) this'll (1.6634) chelsea (1.5002) @jackclaudereadi (1.4682) http://t.co/4yniyezb0n (1.4438) well (1.4259) always (1.4243) favouriteeee (1.4060) http://t.co/wbale6zrsq (1.4060) tweet (1.3929) plz (1.3788) sure (1.3655) @ryan_hildyard (1.3249) modelling (1.3008) http://t.co/giq4dl9jcq (1.3008) seeing (1.2674) \n",
      "\n",
      "Country = id\n",
      "pic (2.2233) http://t.co/0uccot9gdn (2.0171) https://t.co/xxbmsbuf0r (1.9264) https://t.co/exshmyqmsl (1.8471) http://t.co/32szhx0tlk (1.5675) world (1.5449) @justinbieber (1.5337) @smandatas_261 (1.5223) pranciska (1.5223) http://t.co/wyldhnlap4 (1.5223) https://t.co/3sxyzl4crq (1.5223) selamat (1.4264) perfect (1.3377) @viccent22 (1.3044) change (1.2912) strong (1.2489) rt (1.2325) sore (1.1572) @brunomars (1.1433) https://t.co/vs4bfdcvdc (1.1433) \n",
      "\n",
      "Country = my\n",
      "thank (2.0116) @siti_nurbalqish (1.9795) @poemporns (1.7105) http://t.co/hf8harhgbu (1.7105) selangor (1.7073) johor (1.6576) kl (1.5143) @markthewise (1.5121) depressing (1.5121) @12_maiii (1.5119) hunny (1.4742) @aqmarnaimi_ (1.4742) back (1.4530) care (1.4288) http://t.co/6qowzr40be (1.4258) goodnight (1.3763) hahaha (1.3188) cave (1.2701) http://t.co/utnamcqqx5 (1.2626) #gunner (1.2626) \n",
      "\n",
      "Country = ph\n",
      "@home (1.9411) @fuccyoudis2o9 (1.6880) #sorrynotsorry (1.6666) @veronicavispo (1.6666) @qweeesha (1.6659) thankyou (1.6358) http://t.co/fznyhrrj (1.5382) lol (1.4879) evening (1.4339) ng (1.4255) space (1.3573) city (1.3514) fairlane (1.3312) http://t.co/tnbxjbaocu (1.3312) subdivision (1.3312) ko (1.3274) pastries (1.2511) perfection (1.2511) yet (1.2479) dis (1.1717) \n",
      "\n",
      "Country = sg\n",
      "singapore (3.4410) @tutiandonlyher (2.6487) https://t.co/potgoiy5rv (2.1278) @moonbowcloth_id (1.7008) https://t.co/bwenaysyrq (1.7008) yeap (1.5472) @stylomiloteen (1.5472) @megaekaputri_ (1.4766) caterpillar (1.4079) https://t.co/fcnojxrgld (1.4079) heart (1.3846) https://t.co/7yswk6lzsi (1.3757) https://t.co/rwqvgoxm26 (1.3120) @thebeatles (1.3120) songs (1.2788) hahahha (1.2788) keri (1.1786) watching (1.1650) addictive (1.1578) yesterday (1.0912) \n",
      "\n",
      "Country = us\n",
      "freakin (2.0552) much (1.7730) http://t.co/ykkwnmzsir (1.6830) @ansleytolbert (1.6439) @savannerdd (1.6439) get (1.6180) tonight (1.5210) craving (1.5097) chipotle (1.5097) ahhahaha (1.4210) @b_diddddy (1.4210) happy (1.3613) testimony (1.3183) @rcpolar (1.3183) beer (1.2756) pack (1.2378) past (1.2378) clue (1.2267) better (1.2129) ever (1.2123) \n",
      "\n",
      "Country = za\n",
      "man (2.3076) https://t.co/2fsyuigben (1.9637) good (1.6602) @zimtweets (1.6006) akubabuze (1.6006) god (1.4574) u (1.4526) https://t.co/upndf1513q (1.4478) @ferlo_mabkay (1.4478) goodnight (1.3502) feel (1.3456) kosciiiiiiielny (1.3379) @giftnana7 (1.3227) close (1.3109) @fbotha1 (1.2903) blessed (1.2903) sir (1.2732) @enyawmm (1.2732) difficult (1.2617) consultation (1.2526) \n",
      "\n",
      "Classifier = Naive Bayes\n",
      "\n",
      "Country = au\n",
      "melbourne (-5.0972) great (-5.2762) one (-5.2762) i'm (-5.2762) little (-5.4945) australia (-5.4945) love (-5.7741) even (-5.7741) keep (-5.7741) make (-5.7741) come (-5.7741) look (-6.1635) anything (-6.1635) victoria (-6.1635) tomorrow (-6.1635) lovely (-6.1635) australian (-6.1635) direction (-6.1635) want (-6.1635) @micksunnyg (-6.1635) \n",
      "\n",
      "Country = ca\n",
      "i'm (-5.2882) maybe (-5.2882) like (-5.5064) right (-5.5064) school (-5.5064) first (-5.5064) great (-5.5064) got (-5.7860) thanks (-5.7860) good (-5.7860) big (-5.7860) think (-5.7860) thing (-5.7860) u (-5.7860) learning (-5.7860) really (-6.1755) without (-6.1755) cool (-6.1755) #devacurl (-6.1755) come (-6.1755) \n",
      "\n",
      "Country = de\n",
      "i'm (-5.1454) night (-5.4250) posted (-5.4250) good (-5.8144) remstal (-5.8144) year (-5.8144) painting (-5.8144) could (-5.8144) bad (-5.8144) done (-5.8144) kernen (-5.8144) enough (-5.8144) photo (-5.8144) photodesign (-5.8144) miss (-5.8144) #rosegold (-5.8144) mkmedi (-5.8144) im (-5.8144) week (-5.8144) accelerating (-6.4610) \n",
      "\n",
      "Country = gb\n",
      "sure (-5.2322) got (-5.2322) x (-5.4504) i'm (-5.4504) get (-5.7300) home (-5.7300) know (-5.7300) need (-5.7300) always (-5.7300) plz (-5.7300) well (-5.7300) #ff (-6.1195) @skyhelpteam (-6.1195) oh (-6.1195) close (-6.1195) hope (-6.1195) nice (-6.1195) time (-6.1195) lol (-6.1195) make (-6.1195) \n",
      "\n",
      "Country = id\n",
      "pic (-4.5815) rt (-4.9815) others (-5.6584) i'm (-5.6584) photo (-5.6584) hiks (-5.6584) posted (-5.6584) cafe (-6.0478) sore (-6.0478) kawah (-6.0478) shaw (-6.0478) direction (-6.0478) love (-6.0478) senayan (-6.0478) way (-6.0478) one (-6.0478) overboard (-6.0478) donuts (-6.0478) room (-6.0478) selamat (-6.0478) \n",
      "\n",
      "Country = my\n",
      "i'm (-4.0603) selangor (-4.8791) jaya (-5.2099) w (-5.4282) take (-5.4282) thank (-5.4282) birthday (-5.4282) johor (-5.4282) petaling (-5.4282) happy (-5.4282) nk (-5.7078) hahaha (-5.7078) ni (-5.7078) care (-5.7078) haha (-5.7078) cendol (-5.7078) pls (-6.0972) know (-6.0972) best (-6.0972) house (-6.0972) \n",
      "\n",
      "Country = ph\n",
      "follow (-5.0479) city (-5.2269) others (-5.4452) sm (-5.4452) w (-5.4452) make (-5.4452) please (-5.4452) i'm (-5.4452) manila (-5.4452) dreams (-5.7247) lol (-5.7247) @iam3l (-5.7247) yesterday (-5.7247) come (-5.7247) evening (-5.7247) like (-5.7247) ka (-5.7247) ng (-5.7247) @justinbieber (-5.7247) mo (-5.7247) \n",
      "\n",
      "Country = sg\n",
      "singapore (-4.3783) i'm (-4.8252) w (-5.1560) others (-5.3743) work (-5.6538) make (-5.6538) shaw (-6.0433) keri (-6.0433) another (-6.0433) studios (-6.0433) irwan (-6.0433) time (-6.0433) starbucks (-6.0433) side (-6.0433) @zherenne (-6.0433) photo (-6.0433) come (-6.0433) east (-6.0433) pic (-6.0433) @f41bad80f87243a (-6.0433) \n",
      "\n",
      "Country = us\n",
      "get (-5.0125) like (-5.1915) i'm (-5.4098) tonight (-5.4098) happy (-5.4098) going (-5.4098) ever (-5.6894) love (-5.6894) much (-5.6894) u (-5.6894) birthday (-5.6894) day (-5.6894) beer (-5.6894) new (-6.0788) can't (-6.0788) every (-6.0788) dead (-6.0788) hopefully (-6.0788) home (-6.0788) hope (-6.0788) \n",
      "\n",
      "Country = za\n",
      "good (-5.0583) u (-5.2374) love (-5.4556) day (-5.4556) got (-5.4556) de (-5.4556) way (-5.4556) god (-5.4556) made (-5.7352) feel (-5.7352) man (-5.7352) guys (-5.7352) weekend (-5.7352) like (-5.7352) follow (-6.1247) blood (-6.1247) years (-6.1247) yebo (-6.1247) time (-6.1247) say (-6.1247) \n",
      "\n"
     ]
    }
   ],
   "source": [
    "#Finding top features for lOGISTIC REGRESSION\n",
    "print(\"Classifier = Logistic Regression\")\n",
    "print()\n",
    "for i, country in enumerate(lr_classifier.classes_):\n",
    "    print(\"Country =\", country)\n",
    "    top_features_idx = lr_classifier.coef_[i].argsort()[::-1][:20]\n",
    "    top_features = [vectorizer.feature_names_[idx] for idx in top_features_idx]\n",
    "    top_weights = lr_classifier.coef_[i][top_features_idx]\n",
    "    for feature, weight in zip(top_features, top_weights):\n",
    "        print(f\"{feature} ({weight:.4f})\", end=\" \")\n",
    "    print(\"\\n\")\n",
    "\n",
    "# fINDING TOP FEATURES FOR NAIVE BAYES\n",
    "print(\"Classifier = Naive Bayes\")\n",
    "print()\n",
    "for i, country in enumerate(nb_classifier.classes_):\n",
    "    print(\"Country =\", country)\n",
    "    top_features_idx = nb_classifier.feature_log_prob_[i].argsort()[::-1][:20]\n",
    "    top_features = [vectorizer.feature_names_[idx] for idx in top_features_idx]\n",
    "    top_log_probs = nb_classifier.feature_log_prob_[i][top_features_idx]\n",
    "    for feature, log_prob in zip(top_features, top_log_probs):\n",
    "        print(f\"{feature} ({log_prob:.4f})\", end=\" \")\n",
    "    print(\"\\n\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}