SBERT_CORD19_QA_InferSent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "T28QEDT-Uqj2"
   },
   "source": [
    "<p align=\"center\">\n",
    " <img src=\"http://www.di.uoa.gr/themes/corporate_lite/logo_el.png\" title=\"Department of Informatics and Telecommunications - University of Athens\"/> </p>\n",
    "\n",
    "---\n",
    "<h1 align=\"center\"> \n",
    "  Artificial Intelligence\n",
    "</h1>\n",
    "<h1 align=\"center\" > \n",
    "  Deep Learning for Natural Language Processing\n",
    "</h1>\n",
    "\n",
    "---\n",
    "<h2 align=\"center\"> \n",
    " <b>Konstantinos Nikoletos</b>\n",
    "</h2>\n",
    "\n",
    "<h3 align=\"center\"> \n",
    " <b>Winter 2020-2021</b>\n",
    "</h3>\n",
    "\n",
    "\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wk9SOEk6mCXZ"
   },
   "source": [
    "\n",
    "### __Task__ \n",
    "This exercise is about developing a document retrieval system to return titles of scientific\n",
    "papers containing the answer to a given user question. You will use the first version of\n",
    "the COVID-19 Open Research Dataset (CORD-19) in your work (articles in the folder\n",
    "comm use subset).\n",
    "\n",
    "\n",
    "For example, for the question “What are the coronaviruses?”, your system can return the\n",
    "paper title “Distinct Roles for Sialoside and Protein Receptors in Coronavirus Infection”\n",
    "since this paper contains the answer to the asked question.\n",
    "\n",
    "\n",
    "To achieve the goal of this exercise, you will need first to read the paper Sentence-BERT:\n",
    "Sentence Embeddings using Siamese BERT-Networks, in order to understand how you\n",
    "can create sentence embeddings. In the related work of this paper, you will also find other\n",
    "approaches for developing your model. For example, you can using Glove embeddings,\n",
    "etc. In this link, you can find the extended versions of this dataset to test your model, if\n",
    "you want. You are required to:\n",
    "\n",
    "\n",
    "<ol type=\"a\">\n",
    "  <li>Preprocess the provided dataset. You will decide which data of each paper is useful\n",
    "to your model in order to create the appropriate embeddings. You need to explain\n",
    "your decisions.</li>\n",
    "  <li>Implement at least 2 different sentence embedding approaches (see the related work\n",
    "of the Sentence-BERT paper), in order for your model to retrieve the titles of the\n",
    "papers related to a given question.</li>\n",
    "  <li>Compare your 2 models based on at least 2 different criteria of your choice. Explain\n",
    "why you selected these criteria, your implementation choices, and the results. Some\n",
    "questions you can pose are included here. You will need to provide the extra questions\n",
    "you posed to your model and the results of all the questions as well.</li>\n",
    "</ol>\n",
    "\n",
    "### __Notebook__\n",
    "\n",
    "In this notebook I tried to create embeddings using InferSent but I got stucked to some errors that I finally didn't have the time to fix. I added this notebook it to the zip I handed as I made a try (unsuccesful) for InferSent embeddings. \n",
    "\n",
    "\n",
    "\n",
    "## NOT WORKING!!!\n",
    "\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6OfAccyufasd"
   },
   "source": [
    "__Import__ of essential libraries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "dVSHtKTrfGiV",
    "outputId": "f1c615b1-1262-4aeb-a4dc-087f8d166d09"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
      "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/wordnet.zip.\n",
      "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     /root/nltk_data...\n",
      "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd \n",
    "import sys # only needed to determine Python version number\n",
    "import matplotlib # only needed to determine Matplotlib version \n",
    "import nltk\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "import pprint\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torchtext import data\n",
    "import logging\n",
    "nltk.download('punkt')\n",
    "nltk.download('wordnet')\n",
    "nltk.download('stopwords')\n",
    "nltk.download('averaged_perceptron_tagger')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vC9DWskzConr"
   },
   "source": [
    "Selecting device (GPU - CUDA if available)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NiRkpLhiCntl",
    "outputId": "2dade556-bef0-43c3-93f4-3dc77c382a69"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training on GPU.\n"
     ]
    }
   ],
   "source": [
    "# First checking if GPU is available\n",
    "train_on_gpu=torch.cuda.is_available()\n",
    "\n",
    "if(train_on_gpu):\n",
    "    print('Training on GPU.')\n",
    "else:\n",
    "    print('No GPU available, training on CPU.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cvK_TWa2YNg5"
   },
   "source": [
    "# Loading data\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "D6nnb0BFMZDx",
    "outputId": "30833dd2-0387-4b2f-e69e-1ff6a9114b8a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mounted at /content/drive\n"
     ]
    }
   ],
   "source": [
    "# Opening data file\n",
    "import io\n",
    "from google.colab import drive\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "import json\n",
    "\n",
    "drive.mount('/content/drive',force_remount=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wih8b1VKVeS2"
   },
   "source": [
    "Loading the dictionary if it has been created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "uqOeA0cwEjnZ"
   },
   "outputs": [],
   "source": [
    "#@title Select number of papers that will be feeded in the model { vertical-output: true, display-mode: \"both\" }\n",
    "number_of_papers = \"1000\" #@param [\"1000\",\"3000\",\"6000\"]\n",
    "import pickle\n",
    "\n",
    "CORD19_Dataframe = r\"/content/drive/My Drive/AI_4/CORD19_SentenceMap_\"+number_of_papers+\".pkl\"\n",
    "with open(CORD19_Dataframe, 'rb') as drivef:\n",
    "  CORD19Dictionary = pickle.load(drivef)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aJ16If_pdTXV"
   },
   "source": [
    "## Queries\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "f8QgA1OwHLSK"
   },
   "outputs": [],
   "source": [
    "query_list = [\n",
    "  'What are the coronoviruses?',\n",
    "  'What was discovered in Wuhuan in December 2019?',\n",
    "  'What is Coronovirus Disease 2019?',\n",
    "  'What is COVID-19?',\n",
    "  'What is caused by SARS-COV2?', 'How is COVID-19 spread?',\n",
    "  'Where was COVID-19 discovered?','How does coronavirus spread?'\n",
    "]\n",
    "\n",
    "proposed_answers = [\n",
    "  'Coronaviruses (CoVs) are common human and animal pathogens that can transmit zoonotically and cause severe respiratory disease syndromes. ',\n",
    "  'In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.',\n",
    "  'Coronavirus Disease 2019 (COVID-19) is an emerging disease with a rapid increase in cases and deaths since its first identification in Wuhan, China, in December 2019.',\n",
    "  'COVID-19 is a viral respiratory illness caused by a new coronavirus called SARS-CoV-2.',\n",
    "  'Coronavirus disease (COVID-19) is caused by SARS-COV2 and represents the causative agent of a potentially fatal disease that is of great global public health concern.', \n",
    "  'First, although COVID-19 is spread by the airborne route, air disinfection of cities and communities is not known to be effective for disease control and needs to be stopped.',\n",
    "  'In December 2019, a novel coronavirus, called COVID-19, was discovered in Wuhan, China, and has spread to different cities in China as well as to 24 other countries.',\n",
    "  'The new coronavirus was reported to spread via droplets, contact and natural aerosols from human-to-human.'\n",
    "]\n",
    "\n",
    "myquery_list = [\n",
    "  \"How long can the coronavirus survive on surfaces?\",\n",
    "  \"What means COVID-19?\",\n",
    "  \"Is COVID19 worse than flue?\",\n",
    "  \"When the vaccine will be ready?\",\n",
    "  \"Whats the proteins that consist COVID-19?\",\n",
    "  \"Whats the symptoms of COVID-19?\",\n",
    "  \"How can I prevent COVID-19?\",\n",
    "  \"What treatments are available for COVID-19?\",\n",
    "  \"Is hand sanitizer effective against COVID-19?\",\n",
    "  \"Am I at risk for serious complications from COVID-19 if I smoke cigarettes?\",\n",
    "  \"Are there any FDA-approved drugs (medicines) for COVID-19?\",\n",
    "  \"How are people tested?\",\n",
    "  \"Why is the disease being called coronavirus disease 2019, COVID-19?\",\n",
    "  \"Am I at risk for COVID-19 from mail, packages, or products?\",\n",
    "  \"What is community spread?\",\n",
    "  \"How can I protect myself?\",\n",
    "  \"What is a novel coronavirus?\",\n",
    "  \"Was Harry Potter a good magician?\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nwPB-YNeW6t2"
   },
   "source": [
    "# Results dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 297
    },
    "id": "_kBsV1FlW_4y",
    "outputId": "7daca108-7691-4aef-bf39-06777540bc73"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Query</th>\n",
       "      <th>Proposed_answer</th>\n",
       "      <th>Model_answer</th>\n",
       "      <th>Cosine_similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What are the coronoviruses?</td>\n",
       "      <td>Coronaviruses (CoVs) are common human and anim...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What was discovered in Wuhuan in December 2019?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is Coronovirus Disease 2019?</td>\n",
       "      <td>Coronavirus Disease 2019 (COVID-19) is an emer...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What is COVID-19?</td>\n",
       "      <td>COVID-19 is a viral respiratory illness caused...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What is caused by SARS-COV2?</td>\n",
       "      <td>Coronavirus disease (COVID-19) is caused by SA...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>How is COVID-19 spread?</td>\n",
       "      <td>First, although COVID-19 is spread by the airb...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Where was COVID-19 discovered?</td>\n",
       "      <td>In December 2019, a novel coronavirus, called ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>How does coronavirus spread?</td>\n",
       "      <td>The new coronavirus was reported to spread via...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             Query  ... Cosine_similarity\n",
       "0                      What are the coronoviruses?  ...               NaN\n",
       "1  What was discovered in Wuhuan in December 2019?  ...               NaN\n",
       "2                What is Coronovirus Disease 2019?  ...               NaN\n",
       "3                                What is COVID-19?  ...               NaN\n",
       "4                     What is caused by SARS-COV2?  ...               NaN\n",
       "5                          How is COVID-19 spread?  ...               NaN\n",
       "6                   Where was COVID-19 discovered?  ...               NaN\n",
       "7                     How does coronavirus spread?  ...               NaN\n",
       "\n",
       "[8 rows x 4 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resultsDf = pd.DataFrame(columns=['Number of papers','Embeddings creation time'])\n",
    "\n",
    "queriesDf = pd.DataFrame(columns=['Query','Proposed_answer','Model_answer','Cosine_similarity'])\n",
    "queriesDf['Query'] = query_list\n",
    "queriesDf['Proposed_answer'] = proposed_answers\n",
    "\n",
    "myQueriesDf = pd.DataFrame(columns=['Query','Model_anser','Cosine_similarity'])\n",
    "myQueriesDf['Query'] = myquery_list\n",
    "\n",
    "queriesDf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ODeyHCYINWRR"
   },
   "source": [
    "# InferSent\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "J7g1wtxuwlkx",
    "outputId": "7a189cd7-bf2d-4591-f40f-8ab9e23fe65d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting sentence-transformers\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/6a/e2/84d6acfcee2d83164149778a33b6bdd1a74e1bcb59b2b2cd1b861359b339/sentence-transformers-0.4.1.2.tar.gz (64kB)\n",
      "\u001b[K     |████████████████████████████████| 71kB 7.8MB/s \n",
      "\u001b[?25hCollecting transformers<5.0.0,>=3.1.0\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/98/87/ef312eef26f5cecd8b17ae9654cdd8d1fae1eb6dbd87257d6d73c128a4d0/transformers-4.3.2-py3-none-any.whl (1.8MB)\n",
      "\u001b[K     |████████████████████████████████| 1.8MB 23.9MB/s \n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: tqdm in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (4.41.1)\n",
      "Requirement already satisfied, skipping upgrade: torch>=1.6.0 in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (1.7.0+cu101)\n",
      "Requirement already satisfied, skipping upgrade: numpy in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (1.19.5)\n",
      "Requirement already satisfied, skipping upgrade: scikit-learn in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (0.22.2.post1)\n",
      "Requirement already satisfied, skipping upgrade: scipy in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (1.4.1)\n",
      "Requirement already satisfied, skipping upgrade: nltk in /usr/local/lib/python3.6/dist-packages (from sentence-transformers) (3.2.5)\n",
      "Collecting sentencepiece\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/14/67/e42bd1181472c95c8cda79305df848264f2a7f62740995a46945d9797b67/sentencepiece-0.1.95-cp36-cp36m-manylinux2014_x86_64.whl (1.2MB)\n",
      "\u001b[K     |████████████████████████████████| 1.2MB 45.2MB/s \n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: requests in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (2.23.0)\n",
      "Requirement already satisfied, skipping upgrade: packaging in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (20.9)\n",
      "Collecting tokenizers<0.11,>=0.10.1\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/fd/5b/44baae602e0a30bcc53fbdbc60bd940c15e143d252d658dfdefce736ece5/tokenizers-0.10.1-cp36-cp36m-manylinux2010_x86_64.whl (3.2MB)\n",
      "\u001b[K     |████████████████████████████████| 3.2MB 50.1MB/s \n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: regex!=2019.12.17 in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (2019.12.20)\n",
      "Requirement already satisfied, skipping upgrade: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (3.4.0)\n",
      "Collecting sacremoses\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/7d/34/09d19aff26edcc8eb2a01bed8e98f13a1537005d31e95233fd48216eed10/sacremoses-0.0.43.tar.gz (883kB)\n",
      "\u001b[K     |████████████████████████████████| 890kB 55.3MB/s \n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: filelock in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (3.0.12)\n",
      "Requirement already satisfied, skipping upgrade: dataclasses; python_version < \"3.7\" in /usr/local/lib/python3.6/dist-packages (from transformers<5.0.0,>=3.1.0->sentence-transformers) (0.8)\n",
      "Requirement already satisfied, skipping upgrade: future in /usr/local/lib/python3.6/dist-packages (from torch>=1.6.0->sentence-transformers) (0.16.0)\n",
      "Requirement already satisfied, skipping upgrade: typing-extensions in /usr/local/lib/python3.6/dist-packages (from torch>=1.6.0->sentence-transformers) (3.7.4.3)\n",
      "Requirement already satisfied, skipping upgrade: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->sentence-transformers) (1.0.0)\n",
      "Requirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.6/dist-packages (from nltk->sentence-transformers) (1.15.0)\n",
      "Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->transformers<5.0.0,>=3.1.0->sentence-transformers) (1.24.3)\n",
      "Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->transformers<5.0.0,>=3.1.0->sentence-transformers) (2.10)\n",
      "Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->transformers<5.0.0,>=3.1.0->sentence-transformers) (2020.12.5)\n",
      "Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->transformers<5.0.0,>=3.1.0->sentence-transformers) (3.0.4)\n",
      "Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /usr/local/lib/python3.6/dist-packages (from packaging->transformers<5.0.0,>=3.1.0->sentence-transformers) (2.4.7)\n",
      "Requirement already satisfied, skipping upgrade: zipp>=0.5 in /usr/local/lib/python3.6/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers<5.0.0,>=3.1.0->sentence-transformers) (3.4.0)\n",
      "Requirement already satisfied, skipping upgrade: click in /usr/local/lib/python3.6/dist-packages (from sacremoses->transformers<5.0.0,>=3.1.0->sentence-transformers) (7.1.2)\n",
      "Building wheels for collected packages: sentence-transformers, sacremoses\n",
      "  Building wheel for sentence-transformers (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for sentence-transformers: filename=sentence_transformers-0.4.1.2-cp36-none-any.whl size=103068 sha256=b96ac9a6f132028338c93ac3f7bb355560f2795e45bda0217cd5623410cb8f1f\n",
      "  Stored in directory: /root/.cache/pip/wheels/3d/33/d1/5703dd56199c09d4a1b41e0c07fb4e7765a84d787cbdc48ac3\n",
      "  Building wheel for sacremoses (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for sacremoses: filename=sacremoses-0.0.43-cp36-none-any.whl size=893261 sha256=3f5c4e00fa5e5c2c1aecac843b53f04301b35a9c26750e84bc6223fab6de17b8\n",
      "  Stored in directory: /root/.cache/pip/wheels/29/3c/fd/7ce5c3f0666dab31a50123635e6fb5e19ceb42ce38d4e58f45\n",
      "Successfully built sentence-transformers sacremoses\n",
      "Installing collected packages: tokenizers, sacremoses, transformers, sentencepiece, sentence-transformers\n",
      "Successfully installed sacremoses-0.0.43 sentence-transformers-0.4.1.2 sentencepiece-0.1.95 tokenizers-0.10.1 transformers-4.3.2\n"
     ]
    }
   ],
   "source": [
    "!pip install -U sentence-transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "llwHBS7CDofR"
   },
   "source": [
    "Linking with ``` models.py ``` in Drive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "a-cdSdLmq_eU"
   },
   "outputs": [],
   "source": [
    "!cp -r \"/content/drive/My Drive/AI_4/models.py\" '/content/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DIhYGAmJh_kL",
    "outputId": "30a30157-4e47-4ba6-e6dc-a682250e7828"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/content/drive/My Drive/AI_4\n",
      "/content/drive/My Drive/AI_4\n"
     ]
    }
   ],
   "source": [
    "%cd drive/My\\ Drive/AI_4\n",
    "!pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_YpgOtzsE8MB"
   },
   "source": [
    "# Initializing and tuning InferSent model\n",
    "---\n",
    "\n",
    "## __InterSent__\n",
    "\n",
    "Conneau et al. created a bi-directional LSTM with max-pooling over its outputs to generate sentence embeddings. This model was trained from scratch on both MG-NLI and SNLI datasets which highlight the first difference compared to SBERT. Since BERT is at the core of SBERT much of its language understanding comes from the language modeling pre-training task. SBERT used the MG-NLI and SNLI datasets for fine-tuning which should allow it to have a better understanding of language.\n",
    "The LSTM model was able to achieve a test score 84.5 on the SNLI dataset, outperforming at the time best performing competitor with 1.1 points.\n",
    "\n",
    "\n",
    "<p align=\"center\">\n",
    " <img src=\"https://miro.medium.com/max/527/1*H5POIKHhmD-L_nLylV6MSQ.png\" title=\"Department of Informatics and Telecommunications - University of Athens\"/>\n",
    " <p  align=\"center\">Hierarchical convolutional networks</p>\n",
    "</p>\n",
    "\n",
    "\n",
    "## __How Infersent works__\n",
    "\n",
    "The architecture consists of 2 parts:\n",
    "1. One is the sentence encoder that takes word vectors and encodes sentences into vectors\n",
    "2. Two, an NLI classifier that takes the encoded vectors in and outputs a class among entailment, contradiction and neutral.\n",
    "\n",
    "<p align=\"center\">\n",
    " <img src=\"https://miro.medium.com/max/700/1*wbuFlMRo_NTqg8w52M8THw.png\" title=\"Department of Informatics and Telecommunications - University of Athens\"/>\n",
    " <p  align=\"center\">Infersent Flow</p>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "LZfcVYWjGQ1h"
   },
   "outputs": [],
   "source": [
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "V5KbffTGfE9I"
   },
   "source": [
    "# Initializing corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "id": "l_L7TL9AfECp"
   },
   "outputs": [],
   "source": [
    "corpus = list(CORD19Dictionary.keys())[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "id": "xZ2Fe7Z0kwIP"
   },
   "outputs": [],
   "source": [
    "from models import InferSent\n",
    "import torch\n",
    "from sentence_transformers import SentenceTransformer, util\n",
    "import torch\n",
    "import time\n",
    "\n",
    "V = 2\n",
    "MODEL_PATH = 'encoder/infersent%s.pkl' % V\n",
    "params_model = {'bsize': 32, 'word_emb_dim': 300, 'enc_lstm_dim': 2048,\n",
    "                'pool_type': 'max', 'dpout_model': 0.0, 'version': V}\n",
    "\n",
    "model = InferSent(params_model).to(device)\n",
    "model.load_state_dict(torch.load(MODEL_PATH))\n",
    "\n",
    "W2V_PATH = 'Glove/glove.6B.300d.txt'\n",
    "model.set_w2v_path(W2V_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8QECyC0lEKxI"
   },
   "source": [
    "Creating a cosine similarity function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gduRcZ8k0Ftd"
   },
   "outputs": [],
   "source": [
    "def cosine(u, v):\n",
    "    return np.dot(u, v) / (np.linalg.norm(u) * np.linalg.norm(v))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tnwmq9RzERHy"
   },
   "source": [
    "Building vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SLIDr6cW0flV",
    "outputId": "50e19b87-9b6f-4c30-ff32-798fa14d2ef5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 155(/177) words with w2v vectors\n",
      "Vocab size : 155\n"
     ]
    }
   ],
   "source": [
    "model.build_vocab(corpus, tokenize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FbDuU-ZVbp_z"
   },
   "source": [
    "# Creating the embeddings\n",
    "Encoding the papers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1D70QFk_DXRP",
    "outputId": "041a4ada-52ec-4b40-ba1a-cb8ca6bf8814"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nb words kept : 294/342 (86.0%)\n",
      "Speed : 202.4 sentences/s (gpu mode, bsize=64)\n",
      "CPU times: user 29.7 ms, sys: 17.5 ms, total: 47.3 ms\n",
      "Wall time: 49.6 ms\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/content/drive/My Drive/AI_4/models.py:207: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
      "  sentences = np.array(sentences)[idx_sort]\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "corpus_embeddings = torch.Tensor(model.encode(corpus,tokenize=True,verbose=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0QhlJhuUmXVJ"
   },
   "source": [
    "## Saving corpus as tensors to drive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gLrP2DBmmWr4",
    "outputId": "ebe69d4f-264c-4c63-d65b-532f7ca90ac6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving corpus as tensors to drive\n"
     ]
    }
   ],
   "source": [
    "print(\"Saving corpus as tensors to drive\")\n",
    "corpus_embeddings_path = r\"/content/drive/My Drive/AI_4/corpus_embeddings_\"+number_of_papers+\"_InferSent.pt\"\n",
    "torch.save(corpus_embeddings,corpus_embeddings_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EPN5yBuqDXRR"
   },
   "source": [
    "# Loading embeddings if have been created and saved\n",
    "\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "eoG6aMmRDXRS"
   },
   "outputs": [],
   "source": [
    "corpus_embeddings_path = r\"/content/drive/My Drive/AI_4/corpus_embeddings_\"+number_of_papers+\".pt\"\n",
    "with open(corpus_embeddings_path, 'rb') as f:\n",
    "    corpus_embeddings = torch.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JzvlwPU1DXRT"
   },
   "source": [
    "# Evaluation\n",
    "---\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "id": "1ftz-IXTDXRU"
   },
   "outputs": [],
   "source": [
    "import re\n",
    "from  nltk import tokenize\n",
    "from termcolor import colored\n",
    "\n",
    "def paperTitle(answer,SentenceMap):\n",
    "  record = SentenceMap[answer]\n",
    "  print(\"Paper title:\",record[1])\n",
    "  print(\"Paper id:   \",record[0])  \n",
    "\n",
    "# Find the closest sentence of the corpus for each query sentence based on cosine similarity\n",
    "def evaluation(topk,query_list,corpus_embeddings,resultsDf):\n",
    "  query_answers = []\n",
    "  scores = []\n",
    "  for query in query_list:\n",
    "      query_embedding = embedder.encode(query,tokenize=True)\n",
    "\n",
    "      # We use cosine-similarity and torch.topk to find the highest scores\n",
    "      cos_scores = util.pytorch_cos_sim(query_embedding, corpus_embeddings)[0]\n",
    "      top_results = torch.topk(cos_scores, k=top_k)\n",
    "\n",
    "      print(\"\\n\\n======================\\n\\n\")\n",
    "      print(\"Query:\",colored(query,'green') )\n",
    "\n",
    "      for iter, score, idx in zip(range(0,top_k),top_results[0], top_results[1]):\n",
    "        answer = ' '.join([re.sub(r\"^\\[.*\\]\", \"\", x) for x in corpus[idx].split()])\n",
    "        if len(tokenize.word_tokenize(answer)) > 1:\n",
    "          print(\"Score: {:.4f}\".format(score))\n",
    "          paperTitle(corpus[idx],CORD19Dictionary)\n",
    "          print(\"Anser size: \",len(tokenize.word_tokenize(answer)))\n",
    "          print(\"Anser: \")\n",
    "          if iter == 0:\n",
    "            scores.append(score.item())\n",
    "            query_answers.append(answer)\n",
    "          print(colored(answer,'yellow'))\n",
    "        break\n",
    "\n",
    "  resultsDf['Model_answer'] = query_answers\n",
    "  resultsDf['Cosine_similarity'] = scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "X6932YyT5NYP",
    "outputId": "261f92d9-3633-4644-9cea-2ad8f71eb55f"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'W', '</p>']\" (idx=0) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'h', '</p>']\" (idx=1) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 't', '</p>']\" (idx=3) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', '</p>']\" (idx=4) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'r', '</p>']\" (idx=6) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'e', '</p>']\" (idx=7) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', '</p>']\" (idx=8) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 't', '</p>']\" (idx=9) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'h', '</p>']\" (idx=10) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'e', '</p>']\" (idx=11) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', '</p>']\" (idx=12) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'c', '</p>']\" (idx=13) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'o', '</p>']\" (idx=14) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'r', '</p>']\" (idx=15) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'o', '</p>']\" (idx=16) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'n', '</p>']\" (idx=17) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'o', '</p>']\" (idx=18) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'v', '</p>']\" (idx=19) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'i', '</p>']\" (idx=20) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'r', '</p>']\" (idx=21) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'u', '</p>']\" (idx=22) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 's', '</p>']\" (idx=23) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 'e', '</p>']\" (idx=24) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', 's', '</p>']\" (idx=25) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n",
      "/content/drive/My Drive/AI_4/models.py:195: UserWarning: No words in \"['<p>', '?', '</p>']\" (idx=26) have w2v vectors.                                Replacing by \"</s>\"..\n",
      "  Replacing by \"</s>\"..' % (sentences[i], i))\n"
     ]
    },
    {
     "ename": "KeyError",
     "evalue": "ignored",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-42-42be79681c81>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mtop_k\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0membedder\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mevaluation\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtop_k\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mquery_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mcorpus_embeddings\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mqueriesDf\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-34-6f6c289a1b85>\u001b[0m in \u001b[0;36mevaluation\u001b[0;34m(topk, query_list, corpus_embeddings, resultsDf)\u001b[0m\n\u001b[1;32m     13\u001b[0m   \u001b[0mscores\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m   \u001b[0;32mfor\u001b[0m \u001b[0mquery\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mquery_list\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m       \u001b[0mquery_embedding\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0membedder\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquery\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mtokenize\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     16\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m       \u001b[0;31m# We use cosine-similarity and torch.topk to find the highest scores\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/content/drive/My Drive/AI_4/models.py\u001b[0m in \u001b[0;36mencode\u001b[0;34m(self, sentences, bsize, tokenize, verbose)\u001b[0m\n\u001b[1;32m    216\u001b[0m         \u001b[0membeddings\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    217\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mstidx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msentences\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbsize\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 218\u001b[0;31m             \u001b[0mbatch\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_batch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msentences\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstidx\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mstidx\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mbsize\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    219\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_cuda\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    220\u001b[0m                 \u001b[0mbatch\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/content/drive/My Drive/AI_4/models.py\u001b[0m in \u001b[0;36mget_batch\u001b[0;34m(self, batch)\u001b[0m\n\u001b[1;32m    169\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    170\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mj\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 171\u001b[0;31m                 \u001b[0membed\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mword_vec\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mbatch\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mj\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    172\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    173\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mFloatTensor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0membed\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: '</p>'"
     ]
    }
   ],
   "source": [
    "top_k = 3\n",
    "embedder = model\n",
    "evaluation(top_k,query_list,corpus_embeddings,queriesDf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FifabnyIOYE0"
   },
   "outputs": [],
   "source": [
    "top_k = 3\n",
    "embedder = model\n",
    "evaluation(top_k,myquery_list,corpus_embeddings,myQueriesDf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P9ugX3IekOYd"
   },
   "source": [
    "# References\n",
    "\n",
    "[1] [Oficcial InferSent GitHub repository](https://github.com/facebookresearch/InferSent)\n",
    "\n",
    "[2] [Sentence embeddings examples](https://www.analyticsvidhya.com/blog/2020/08/top-4-sentence-embedding-techniques-using-python/)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "SBERT_CORD19_QA_InferSent.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}