Notebooks/04_TFIDF_DocSearch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "424b8180",
   "metadata": {
    "papermill": {
     "duration": 0.006984,
     "end_time": "2023-02-20T20:30:55.884248",
     "exception": false,
     "start_time": "2023-02-20T20:30:55.877264",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<center>\n",
    "<img src='../assets/1/lang-pic.jpg' width=600>\n",
    "</center>\n",
    "    \n",
    "# 1. Introduction\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">1.1 NLP series</p>\n",
    "\n",
    "This is the **fourth in a series of notebooks** covering the **fundamentals of Natural Language Processing (NLP)**. I find that the best way to learn is by teaching others, hence why I am sharing my journey learning this field from scratch. I hope these notebooks can be helpful to you too. \n",
    "\n",
    "NLP series:\n",
    "\n",
    "1. [Tokenization](./01_Tokenization.ipynb)\n",
    "2. [Preprocessing](./02_Pre_Processing.ipynb)\n",
    "3. [Bag of Words and Similarity](./03_BOW_Similarity.ipynb)\n",
    "4. TF-IDF and Document Search\n",
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/JUSTSUJAY/NLP_One_Shot/blob/main/Notebooks/04_TFIDF_DocSearch.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>\n",
    "[![Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://kaggle.com/kernels/welcome?src=https://github.com/JUSTSUJAY/NLP_One_Shot/blob/main/Notebooks/04_TFIDF_DocSearch.ipynb)\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">1.2 Outline</p>\n",
    "\n",
    "Last time, we took our first step in converting text into numbers through a basic bag-of-words. This had a number of shortcomings however, so in this notebook we will look at an improvement to bag-of-words, namely **Term Frequency - Inverse Document Frequency (TF-IDF)**. This addresses the idea that **not all words are equal**. \n",
    "\n",
    "It is a technique which is still **highly popular** today. According to Wikipedia, \"A survey conducted in 2015 showed that 83% of text-based recommender systems in digital libraries use tf–idf\" ([reference](https://en.wikipedia.org/wiki/Tf%E2%80%93idf))."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01f1bc30",
   "metadata": {
    "papermill": {
     "duration": 0.005962,
     "end_time": "2023-02-20T20:30:55.896223",
     "exception": false,
     "start_time": "2023-02-20T20:30:55.890261",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 2. Encoding importance\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">2.1 Shortcomings of Bag-of-Words</p>\n",
    "\n",
    "<center>\n",
    "<img src='../assets/4/wordcloud.jpg' width=500>\n",
    "</center>\n",
    "<br>\n",
    "\n",
    "Last time we discussed that with a bag-of-words we **lose word order information**, although this can be partially remedied by using n-grams to encode context. \n",
    "\n",
    "When working with a **binary** bag-of-words there is another significant drawback. This is that **all words are treated as equally important**, although we know this is **not the case** in language. \n",
    "\n",
    "We could use a **frequency** bag-of-words but then some words like 'the' and 'it' **occur very frequently** and affect similarity calculations. We could remove stop words but this won't remove all of the frequent and redundant words. For example, in a corpus of food recipes, words like 'mix', 'bowl' and 'teaspoon' will appear in almost all documents and so won't be very informative."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d23c582e",
   "metadata": {
    "papermill": {
     "duration": 0.00551,
     "end_time": "2023-02-20T20:30:55.907812",
     "exception": false,
     "start_time": "2023-02-20T20:30:55.902302",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">2.2 Relative frequency</p>\n",
    "\n",
    "An improvement then would be to use the **relative frequency** of each word in the corpus. This is **calculated** as the number of times a word appears in a document divided by the total number of times it appears in the corpus. \n",
    "\n",
    "<br>\n",
    "\n",
    "$$\n",
    "\\large\n",
    "\\text{relative frequency} = \\frac{\\text{frequency in document}}{\\text{frequency in corpus}}\n",
    "$$\n",
    "\n",
    "<br>\n",
    "\n",
    "The idea is that words that appear **highly frequently** in some documents and rarely in the rest, are likely to be **meaningful to those documents** and will help distinguish between documents. On the other hand, words that appears roughly **uniformly** across all documents are **unlikely to be important**. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d779f774",
   "metadata": {
    "papermill": {
     "duration": 0.005536,
     "end_time": "2023-02-20T20:30:55.919134",
     "exception": false,
     "start_time": "2023-02-20T20:30:55.913598",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 3. TF-IDF\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">3.1 Term Frequency</p>\n",
    "\n",
    "<center>\n",
    "<img src='../assets/4/wren.jpg' width=600>\n",
    "</center>\n",
    "<br>\n",
    "\n",
    "**TF-IDF** stands for **Term Frequency - Inverse Document Frequency** and is made up of **two components**. The first is the **term frequency**. \n",
    "\n",
    "Whilst some people define the term frequency to be the relative frequency, it is more common to use the **raw frequency** of the token/term $t$ in document $d$.\n",
    "\n",
    "<br>\n",
    "\n",
    "$$\n",
    "\\Large\n",
    "\\text{tf}(t,d) = f_{t,d}\n",
    "$$\n",
    "\n",
    "<br>\n",
    "\n",
    "However, some documents may be much **longer** than others and so will naturally have higher frequencies across the board. For this reason, it is standard practice to apply the **log-transform** to reduce this bias. The term frequency then becomes\n",
    "\n",
    "<br>\n",
    "\n",
    "$$\n",
    "\\Large\n",
    "\\text{tf}(t,d) = \\log(1+f_{t,d})\n",
    "$$\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">3.2 Inverse Document Frequency</p>\n",
    "\n",
    "The second part of TF-IDF is the **inverse document frequency**. This is the part that will **emphasise the more important words** in each document. \n",
    "\n",
    "Given a token/term $t$ in a corpus $D$, we define\n",
    "\n",
    "<br>\n",
    "\n",
    "$$\n",
    "\\Large\n",
    "\\text{idf}(t,D) = \\log \\left(\\frac{N}{n_t} \\right)\n",
    "$$\n",
    "\n",
    "<br>\n",
    "\n",
    "where $N$ is the number of documents and $n_t$ is the number of documents $t$ appears in. Notice how as $n_t$ decreases the idf increases corresponding to a token that is more likely to be **important**.\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">3.3 TF-IDF score</p>\n",
    "\n",
    "To get the final tf-idf score, we simply **multiply** the tf with the idf. That is,\n",
    "\n",
    "<br>\n",
    "\n",
    "$$\n",
    "\\Large\n",
    "w_{t,d} = \\text{tf}(t,d) \\times \\text{idf}(t,D)\n",
    "$$\n",
    "\n",
    "<br>\n",
    "\n",
    "So the more frequently a word appears in a given document and the fewer times it appears in other documents the **higher** its TF-IDF score. \n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">3.4 Variations</p>\n",
    "\n",
    "There are **many variations** to the TF-IDF score. Like we discussed earlier, instead of raw frequency, we could use the **relative frequency** in the term frequency term. This is actually how Wikipedia presents the formula.\n",
    "\n",
    "The **sklearn implementation** of tf-idf doesn't apply the log-transform in the tf term. It also adds constants to the idf term to prevent division by zero and uses the **natural logarithm**. In particular, is uses the following formulas.\n",
    "\n",
    "<br>\n",
    "\n",
    "$$\n",
    "\\Large\n",
    "\\text{tf}(t,d) = f_{t,d}\n",
    "$$\n",
    "\n",
    "<br>\n",
    "\n",
    "$$\n",
    "\\Large\n",
    "\\text{idf}(t,D) = \\ln \\left(\\frac{N+1}{n_t+1} \\right) + 1\n",
    "$$\n",
    "\n",
    "<br>\n",
    "\n",
    "It also **normalizes** the output vector for each document so that each document has a vector of scores with **norm equal to 1**. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12e22e89",
   "metadata": {
    "papermill": {
     "duration": 0.005552,
     "end_time": "2023-02-20T20:30:55.930446",
     "exception": false,
     "start_time": "2023-02-20T20:30:55.924894",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 4. Application\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">4.1 TF-IDF with sklearn</p>\n",
    "\n",
    "<center>\n",
    "<img src='../assets/4/moon.jpg' width=450>\n",
    "</center>\n",
    "<br>\n",
    "\n",
    "Import the **libraries**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e5e828da",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:30:55.944494Z",
     "iopub.status.busy": "2023-02-20T20:30:55.943667Z",
     "iopub.status.idle": "2023-02-20T20:31:09.678555Z",
     "shell.execute_reply": "2023-02-20T20:31:09.677254Z"
    },
    "papermill": {
     "duration": 13.745557,
     "end_time": "2023-02-20T20:31:09.681842",
     "exception": false,
     "start_time": "2023-02-20T20:30:55.936285",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.metrics.pairwise import cosine_similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65bb6026",
   "metadata": {
    "papermill": {
     "duration": 0.005531,
     "end_time": "2023-02-20T20:31:09.693429",
     "exception": false,
     "start_time": "2023-02-20T20:31:09.687898",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "For this demo, we'll use use the **20 newsgroups dataset**, which is a collection of 18,000 newsgroup posts across 20 topics. We'll take the posts relating to the `sci.space` topic as that will be enough for our application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e308d927",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:31:09.707432Z",
     "iopub.status.busy": "2023-02-20T20:31:09.706355Z",
     "iopub.status.idle": "2023-02-20T20:31:20.948784Z",
     "shell.execute_reply": "2023-02-20T20:31:20.947179Z"
    },
    "papermill": {
     "duration": 11.2536,
     "end_time": "2023-02-20T20:31:20.952784",
     "exception": false,
     "start_time": "2023-02-20T20:31:09.699184",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "593\n",
      "\n",
      "Any lunar satellite needs fuel to do regular orbit corrections, and when\n",
      "its fuel runs out it will crash within months.  The orbits of the Apollo\n",
      "motherships changed noticeably during lunar missions lasting only a few\n",
      "days.  It is *possible* that there are stable orbits here and there --\n",
      "the Moon's gravitational field is poorly mapped -- but we know of none.\n",
      "\n",
      "Perturbations from Sun and Earth are relatively minor issues at low\n",
      "altitudes.  The big problem is that the Moon's own gravitational field\n",
      "is quite lumpy due to the irregular distribution of mass within the Moon.\n"
     ]
    }
   ],
   "source": [
    "# Load corpus\n",
    "corpus = fetch_20newsgroups(categories=['sci.space'], remove=('headers', 'footers', 'quotes'))\n",
    "\n",
    "# Preview data\n",
    "print(len(corpus.data))\n",
    "print(corpus.data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "941dcfff",
   "metadata": {
    "papermill": {
     "duration": 0.005564,
     "end_time": "2023-02-20T20:31:20.966389",
     "exception": false,
     "start_time": "2023-02-20T20:31:20.960825",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We need to **pre-process** the text first using a **tokenizer**. We'll do this using spacy, like we have seen in previous notebooks. We apply lemmatization, remove punctuation, spaces and non-alphabetic characters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5f99cf51",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:31:20.979875Z",
     "iopub.status.busy": "2023-02-20T20:31:20.979462Z",
     "iopub.status.idle": "2023-02-20T20:31:21.847144Z",
     "shell.execute_reply": "2023-02-20T20:31:21.845670Z"
    },
    "papermill": {
     "duration": 0.87766,
     "end_time": "2023-02-20T20:31:21.849927",
     "exception": false,
     "start_time": "2023-02-20T20:31:20.972267",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load english language model\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "\n",
    "# Disable named-entity recognition and parsing to save time\n",
    "unwanted_pipes = [\"ner\", \"parser\"]\n",
    "\n",
    "# Custom tokenizer using spacy\n",
    "def custom_tokenizer(doc):\n",
    "    with nlp.disable_pipes(*unwanted_pipes):\n",
    "        return [t.lemma_ for t in nlp(doc) if not t.is_punct and not t.is_space and t.is_alpha]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed0d8e78",
   "metadata": {
    "papermill": {
     "duration": 0.006253,
     "end_time": "2023-02-20T20:31:21.862417",
     "exception": false,
     "start_time": "2023-02-20T20:31:21.856164",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Similar to the bag-of-words countvectorizer, **sklearn** also provides a class to perform **TF-IDF**, namely`TfidfVectorizer`. To use our custom tokenizer, we pass it through as a parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f4da4c93",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:31:21.877215Z",
     "iopub.status.busy": "2023-02-20T20:31:21.876229Z",
     "iopub.status.idle": "2023-02-20T20:31:36.512419Z",
     "shell.execute_reply": "2023-02-20T20:31:36.511364Z"
    },
    "papermill": {
     "duration": 14.646355,
     "end_time": "2023-02-20T20:31:36.515197",
     "exception": false,
     "start_time": "2023-02-20T20:31:21.868842",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Initialise tf-idf tokenizer\n",
    "vectorizer = TfidfVectorizer(tokenizer=custom_tokenizer)\n",
    "\n",
    "# Fit vectorizer to corpus\n",
    "features = vectorizer.fit_transform(corpus.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70736ba0",
   "metadata": {
    "papermill": {
     "duration": 0.005667,
     "end_time": "2023-02-20T20:31:36.527125",
     "exception": false,
     "start_time": "2023-02-20T20:31:36.521458",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The output is a **sparse matrix** with dimensions (number of documents, size of vocabulary). The entries of the matrix are the **tf-idf scores** of each **document-token pair**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "39281c42",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:31:36.540929Z",
     "iopub.status.busy": "2023-02-20T20:31:36.540526Z",
     "iopub.status.idle": "2023-02-20T20:31:36.553270Z",
     "shell.execute_reply": "2023-02-20T20:31:36.552189Z"
    },
    "papermill": {
     "duration": 0.02271,
     "end_time": "2023-02-20T20:31:36.556008",
     "exception": false,
     "start_time": "2023-02-20T20:31:36.533298",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9469\n",
      "(593, 9469)\n"
     ]
    }
   ],
   "source": [
    "# Size of vocabulary\n",
    "print(len(vectorizer.get_feature_names_out()))\n",
    "\n",
    "# Dimensions of output matrix\n",
    "print(features.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "53a34358",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:31:36.570323Z",
     "iopub.status.busy": "2023-02-20T20:31:36.569288Z",
     "iopub.status.idle": "2023-02-20T20:31:36.576894Z",
     "shell.execute_reply": "2023-02-20T20:31:36.575246Z"
    },
    "papermill": {
     "duration": 0.017581,
     "end_time": "2023-02-20T20:31:36.579618",
     "exception": false,
     "start_time": "2023-02-20T20:31:36.562037",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 5076)\t0.10452465319204224\n",
      "  (0, 2355)\t0.12746673572641337\n",
      "  (0, 4351)\t0.15331277268825122\n",
      "  (0, 2464)\t0.10862134983649849\n",
      "  (0, 4927)\t0.17102243214182047\n",
      "  (0, 6721)\t0.09939758959196421\n",
      "  (0, 5997)\t0.10183273017124134\n",
      "  (0, 6531)\t0.08455248650428543\n",
      "  (0, 903)\t0.08929749232550656\n",
      "  (0, 321)\t0.11094564582599897\n",
      "  (0, 4907)\t0.0824741348739743\n",
      "  (0, 637)\t0.05104326044583604\n",
      "  (0, 4378)\t0.10269890253976902\n",
      "  (0, 5285)\t0.13259379932649137\n",
      "  (0, 6928)\t0.12524362655380208\n",
      "  (0, 2501)\t0.07376358403679217\n",
      "  (0, 8137)\t0.09512941822420008\n",
      "  (0, 3299)\t0.051873252060803496\n",
      "  (0, 6196)\t0.13901479196044814\n",
      "  (0, 5662)\t0.11219221685212125\n",
      "  (0, 4604)\t0.06321553828701997\n",
      "  (0, 9188)\t0.060883075560077576\n",
      "  (0, 1148)\t0.048917557559216875\n",
      "  (0, 5035)\t0.12319856435864923\n",
      "  (0, 6371)\t0.15331277268825122\n",
      "  :\t:\n",
      "  (592, 9343)\t0.031082224218866934\n",
      "  (592, 4009)\t0.06586873156378406\n",
      "  (592, 9426)\t0.20489473227563584\n",
      "  (592, 6144)\t0.03168401769186157\n",
      "  (592, 3198)\t0.07105031369088437\n",
      "  (592, 4088)\t0.04616367577056477\n",
      "  (592, 9316)\t0.044014023469863946\n",
      "  (592, 3456)\t0.02464244792646734\n",
      "  (592, 5841)\t0.03665680780759958\n",
      "  (592, 7412)\t0.029155503547787242\n",
      "  (592, 637)\t0.021755506546425055\n",
      "  (592, 4378)\t0.04377202057626157\n",
      "  (592, 3299)\t0.04421852620448071\n",
      "  (592, 4604)\t0.026943538579366717\n",
      "  (592, 5393)\t0.10206469598286\n",
      "  (592, 820)\t0.06127561580386092\n",
      "  (592, 1)\t0.06825011641886097\n",
      "  (592, 5802)\t0.169004890244449\n",
      "  (592, 8399)\t0.29814384400748034\n",
      "  (592, 9281)\t0.02551949122091724\n",
      "  (592, 4382)\t0.016312771353441658\n",
      "  (592, 377)\t0.10485993459585063\n",
      "  (592, 2377)\t0.055534284666880984\n",
      "  (592, 8522)\t0.19546129149308888\n",
      "  (592, 4929)\t0.1135203305081983\n"
     ]
    }
   ],
   "source": [
    "# What the matrix looks like\n",
    "print(features)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95983f60",
   "metadata": {
    "papermill": {
     "duration": 0.005837,
     "end_time": "2023-02-20T20:31:36.591659",
     "exception": false,
     "start_time": "2023-02-20T20:31:36.585822",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">4.2 Document Search</p>\n",
    "\n",
    "Now we have the tf-idf matrix, we can **measure similarity** exactly the same as before - by using **cosine similarity**. Given a document, we can find the other documents which are **most similar to the original one**. \n",
    "\n",
    "We will now go a step further and use it to build a basic **document search recommender system**. Given a **query** (i.e. a search term), we **transform** the query, **measure the similarity** with all the other documents and finally **return the most similar documents**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "143fc592",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:31:36.605359Z",
     "iopub.status.busy": "2023-02-20T20:31:36.604892Z",
     "iopub.status.idle": "2023-02-20T20:31:36.615288Z",
     "shell.execute_reply": "2023-02-20T20:31:36.614308Z"
    },
    "papermill": {
     "duration": 0.020203,
     "end_time": "2023-02-20T20:31:36.617820",
     "exception": false,
     "start_time": "2023-02-20T20:31:36.597617",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Transform the query\n",
    "query = [\"Mars\"]\n",
    "query_tfidf = vectorizer.transform(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3a9ff55a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:31:36.631712Z",
     "iopub.status.busy": "2023-02-20T20:31:36.631316Z",
     "iopub.status.idle": "2023-02-20T20:31:36.638288Z",
     "shell.execute_reply": "2023-02-20T20:31:36.637293Z"
    },
    "papermill": {
     "duration": 0.01664,
     "end_time": "2023-02-20T20:31:36.640575",
     "exception": false,
     "start_time": "2023-02-20T20:31:36.623935",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Calculate pairwise similarity with all documents in corpus\n",
    "cosine_similarities = cosine_similarity(features, query_tfidf).flatten()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "adb5cb9c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:31:36.654963Z",
     "iopub.status.busy": "2023-02-20T20:31:36.654031Z",
     "iopub.status.idle": "2023-02-20T20:31:36.661719Z",
     "shell.execute_reply": "2023-02-20T20:31:36.660238Z"
    },
    "papermill": {
     "duration": 0.017529,
     "end_time": "2023-02-20T20:31:36.664256",
     "exception": false,
     "start_time": "2023-02-20T20:31:36.646727",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[468 261 583 410 335]\n"
     ]
    }
   ],
   "source": [
    "# Return indices of top k matching documents\n",
    "def top_k(arr, k):\n",
    "    kth_largest = (k + 1) * -1\n",
    "    return np.argsort(arr)[:kth_largest:-1]\n",
    "\n",
    "# Return top 5 document indices\n",
    "top_related_indices = top_k(cosine_similarities, 5)\n",
    "print(top_related_indices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b725e787",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:31:36.678864Z",
     "iopub.status.busy": "2023-02-20T20:31:36.678460Z",
     "iopub.status.idle": "2023-02-20T20:31:36.684571Z",
     "shell.execute_reply": "2023-02-20T20:31:36.683221Z"
    },
    "papermill": {
     "duration": 0.015816,
     "end_time": "2023-02-20T20:31:36.686583",
     "exception": false,
     "start_time": "2023-02-20T20:31:36.670767",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.31678663 0.2700739  0.17534145 0.1489204  0.14629581]\n"
     ]
    }
   ],
   "source": [
    "# Corresponding cosine similarities\n",
    "print(cosine_similarities[top_related_indices])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ff0c6f73",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:31:36.700558Z",
     "iopub.status.busy": "2023-02-20T20:31:36.700128Z",
     "iopub.status.idle": "2023-02-20T20:31:36.705679Z",
     "shell.execute_reply": "2023-02-20T20:31:36.704597Z"
    },
    "papermill": {
     "duration": 0.015619,
     "end_time": "2023-02-20T20:31:36.708374",
     "exception": false,
     "start_time": "2023-02-20T20:31:36.692755",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What is the deal with life on Mars?  I save the \"face\" and heard \n",
      "associated theories. (which sound thin to me)\n",
      "\n",
      "Are we going back to Mars to look at this face agian?\n",
      "Does anyone buy all the life theories?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Top match\n",
    "print(corpus.data[top_related_indices[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4fb0a49a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T20:31:36.723733Z",
     "iopub.status.busy": "2023-02-20T20:31:36.723315Z",
     "iopub.status.idle": "2023-02-20T20:31:36.729769Z",
     "shell.execute_reply": "2023-02-20T20:31:36.728345Z"
    },
    "papermill": {
     "duration": 0.016933,
     "end_time": "2023-02-20T20:31:36.732227",
     "exception": false,
     "start_time": "2023-02-20T20:31:36.715294",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Not quite true.  One of the instruments on Mars Observer will be searching\n",
      "for potential fossil sites.   \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Second-best match\n",
    "print(corpus.data[top_related_indices[1]])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d3bdd4c",
   "metadata": {
    "papermill": {
     "duration": 0.005976,
     "end_time": "2023-02-20T20:31:36.744712",
     "exception": false,
     "start_time": "2023-02-20T20:31:36.738736",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 5. Conclusion\n",
    "\n",
    "**Very cool!** It is returning documents that **overlap** with our search term quite well, although they might not be entirely relevant. \n",
    "\n",
    "In practice it wouldn't be feasible to measure similarity against every single document in the corpus as it would **take too long**. You would need more sophisticated **matching algorithms** to be able to return results in a short amount of time. \n",
    "\n",
    "Keep in mind that **state-of-the-art search engines** like Google will also use many **additional features** like your search history, location, device, etc to provide the most accuracy results. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b4ee6ac",
   "metadata": {
    "papermill": {
     "duration": 0.006032,
     "end_time": "2023-02-20T20:31:36.757145",
     "exception": false,
     "start_time": "2023-02-20T20:31:36.751113",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**References:**\n",
    "* [NLP demystified](https://www.nlpdemystified.org/)\n",
    "\n",
    "### Coming UP\n",
    "#### [5. Naive Bayes Text Classification](./05_NaiveBayes_TextClf.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4b43816",
   "metadata": {
    "papermill": {
     "duration": 0.005936,
     "end_time": "2023-02-20T20:31:36.769366",
     "exception": false,
     "start_time": "2023-02-20T20:31:36.763430",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Thanks for reading!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 54.86958,
   "end_time": "2023-02-20T20:31:40.113540",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2023-02-20T20:30:45.243960",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}