Project4 - Retrieving information.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "96938a1f",
   "metadata": {},
   "source": [
    "Information retrieval is one of the highly used applications of NLP and it is\n",
    "quite tricky. The meaning of the words or sentences not only depends on\n",
    "the exact words used but also on the context and meaning. Two sentences\n",
    "may be of completely different words but can convey the same meaning.\n",
    "We should be able to capture that as well\n",
    "\n",
    "An Information retreival system allows user to efficiently seach documents and retrieve information based on the seach text\n",
    "\n",
    "There are multiple ways to do Information retrieval. But we will see how to\n",
    "do it using word embeddings, which is very effective since it takes context\n",
    "also into consideration."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a019a957",
   "metadata": {},
   "source": [
    "### 1. Import the library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ae4c318e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import gensim\n",
    "from gensim.models import Word2Vec\n",
    "import nltk\n",
    "import itertools\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import word_tokenize, sent_tokenize\n",
    "from nltk.tokenize.toktok import ToktokTokenizer\n",
    "import scipy\n",
    "import re\n",
    "\n",
    "stopword_list = stopwords.words('english')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1589270",
   "metadata": {},
   "source": [
    "### 2. Create/Import the document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3c7e26f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "Doc1 = [\"With the Union cabinet approving the amendments to the Motor Vehicles Act, 2016, those caught for drunken driving will have to have really deep pockets, as the fine payable in courthas been enhanced to Rs 10,000 for first-time offenders.\" ]\n",
    "Doc2 = [\"Natural language processing (NLP) is an area of computer science and artificial intelligence concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data.\"]\n",
    "Doc3 = [\"He points out that public transport is very good in Mumbai and New Delhi, where there is a good network of suburban and metro rail systems.\"]\n",
    "Doc4 = [\"But the man behind the wickets at the other end was watching just as keenly. With an affirmative nod from Dhoni,India captain Rohit Sharma promptly asked for a review. Sure enough, the ball would have clipped the top of middle and leg.\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9e11d868",
   "metadata": {},
   "outputs": [],
   "source": [
    "# put all doc in one list\n",
    "\n",
    "fin = Doc1 + Doc2 + Doc3 + Doc4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "79866178",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['With the Union cabinet approving the amendments to the Motor Vehicles Act, 2016, those caught for drunken driving will have to have really deep pockets, as the fine payable in courthas been enhanced to Rs 10,000 for first-time offenders.',\n",
       " 'Natural language processing (NLP) is an area of computer science and artificial intelligence concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data.',\n",
       " 'He points out that public transport is very good in Mumbai and New Delhi, where there is a good network of suburban and metro rail systems.',\n",
       " 'But the man behind the wickets at the other end was watching just as keenly. With an affirmative nod from Dhoni,India captain Rohit Sharma promptly asked for a review. Sure enough, the ball would have clipped the top of middle and leg.']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fin"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f13c6b8",
   "metadata": {},
   "source": [
    "### 3. Load the word2vec model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a07e620f",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = gensim.models.KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6f77210",
   "metadata": {},
   "source": [
    "### 4. Build the IR system"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d2200d8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preprocess the data\n",
    "\n",
    "def remove_stopwords(text, is_lower_case = False):\n",
    "    pattern = r'[^a-zA-z0-9\\s]'\n",
    "    text = re.sub(pattern,\"\", \"\".join(text))\n",
    "    tokens = word_tokenize(text)\n",
    "    tokens = [token.strip() for token in tokens]\n",
    "    if is_lower_case:\n",
    "        filtered_token = [token for token in tokens if token not in stopword_list]\n",
    "    else:\n",
    "        filtered_token = [token for token in tokens if token.lower() not in stopword_list]\n",
    "    filtered_text = \" \".join(filtered_token)\n",
    "    return filtered_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "52f05823",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to get the embedding vector for n dimension, we have used \"300\"\n",
    "\n",
    "def get_embedding(word):\n",
    "    if word in model.key_to_index:\n",
    "        return model[word]\n",
    "    else:\n",
    "        return np.zeros(300)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "a4f16196",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Getting the average vector for every document\n",
    "out_dict = {}\n",
    "for sen in fin:\n",
    "    average_vector = (np.mean(np.array([get_embedding(x) for x in nltk.word_tokenize(remove_stopwords(sen))]), axis=0))\n",
    "    dict = { sen : (average_vector) }\n",
    "    out_dict.update(dict)\n",
    "    \n",
    "# Function to calculate the similarity between query vector and the average vector in the document\n",
    "def get_sim(query_embedding, average_vector_doc):\n",
    "    sim = [(1 - scipy.spatial.distance.cosine(query_embedding, average_vector_doc))]\n",
    "    return sim\n",
    "\n",
    "# Rank all the documents based on the similarity to get relevant docs\n",
    "\n",
    "\n",
    "def ranked_doc(query):\n",
    "    query_words = (np.mean(np.array([get_embedding(x) for x in \n",
    "                                      nltk.word_tokenize(remove_stopwords(query.lower()))], dtype=float), axis=0))\n",
    "    rank = []\n",
    "    for k, v in out_dict.items():\n",
    "        rank.append((k, get_sim(query_words, v)))\n",
    "    rank = sorted(rank, key=lambda t:t[1], reverse = True)\n",
    "    print('Rank Document:')\n",
    "    return rank"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20a2aade",
   "metadata": {},
   "source": [
    "### 5. Result and application"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "0ac8c8c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rank Document:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('But the man behind the wickets at the other end was watching just as keenly. With an affirmative nod from Dhoni,India captain Rohit Sharma promptly asked for a review. Sure enough, the ball would have clipped the top of middle and leg.',\n",
       "  [0.35923325370062154]),\n",
       " ('He points out that public transport is very good in Mumbai and New Delhi, where there is a good network of suburban and metro rail systems.',\n",
       "  [0.23973446930269127]),\n",
       " ('Natural language processing (NLP) is an area of computer science and artificial intelligence concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data.',\n",
       "  [0.17995061678671642]),\n",
       " ('With the Union cabinet approving the amendments to the Motor Vehicles Act, 2016, those caught for drunken driving will have to have really deep pockets, as the fine payable in courthas been enhanced to Rs 10,000 for first-time offenders.',\n",
       "  [0.1794264798270916])]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ranked_doc('cricket')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "e771a808",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rank Document:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('With the Union cabinet approving the amendments to the Motor Vehicles Act, 2016, those caught for drunken driving will have to have really deep pockets, as the fine payable in courthas been enhanced to Rs 10,000 for first-time offenders.',\n",
       "  [0.3598919009839452]),\n",
       " ('But the man behind the wickets at the other end was watching just as keenly. With an affirmative nod from Dhoni,India captain Rohit Sharma promptly asked for a review. Sure enough, the ball would have clipped the top of middle and leg.',\n",
       "  [0.2037214882585675]),\n",
       " ('He points out that public transport is very good in Mumbai and New Delhi, where there is a good network of suburban and metro rail systems.',\n",
       "  [0.1706653724240128]),\n",
       " ('Natural language processing (NLP) is an area of computer science and artificial intelligence concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyze large amounts of natural language data.',\n",
       "  [0.08872308406410134])]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ranked_doc(\"driving\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "164457bb",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}