02_building.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 2: Building a Simple Index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this assignment, we will build a simple search index, which we will use later for Boolean retrieval. The assignment tasks are again at the bottom of this document."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "Summaries_file = 'data/allergy_Summaries.pkl.bz2'\n",
    "Abstracts_file = 'data/allergy_Abstracts.pkl.bz2'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle, bz2\n",
    "from collections import namedtuple\n",
    "\n",
    "Summaries = pickle.load( bz2.BZ2File( Summaries_file, 'rb' ) )\n",
    "\n",
    "paper = namedtuple( 'paper', ['title', 'authors', 'year', 'doi'] )\n",
    "\n",
    "for (id, paper_info) in Summaries.items():\n",
    "    Summaries[id] = paper( *paper_info )\n",
    "    \n",
    "Abstracts = pickle.load( bz2.BZ2File( Abstracts_file, 'rb' ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at what the data looks like for an example of a paper:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "paper(title='Type 2 innate lymphoid cells control eosinophil homeostasis.', authors=['Nussbaum JC', 'Van Dyken SJ', 'von Moltke J', 'Cheng LE', 'Mohapatra A', 'Molofsky AB', 'Thornton EE', 'Krummel MF', 'Chawla A', 'Liang HE', 'Locksley RM'], year=2013, doi='10.1038/nature12526')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Summaries[24037376]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Eosinophils are specialized myeloid cells associated with allergy and helminth infections. Blood eosinophils demonstrate circadian cycling, as described over 80\\u2009years ago, and are abundant in the healthy gastrointestinal tract. Although a cytokine, interleukin (IL)-5, and chemokines such as eotaxins mediate eosinophil development and survival, and tissue recruitment, respectively, the processes underlying the basal regulation of these signals remain unknown. Here we show that serum IL-5 levels are maintained by long-lived type 2 innate lymphoid cells (ILC2) resident in peripheral tissues. ILC2 cells secrete IL-5 constitutively and are induced to co-express IL-13 during type 2 inflammation, resulting in localized eotaxin production and eosinophil accumulation. In the small intestine where eosinophils and eotaxin are constitutive, ILC2 cells co-express IL-5 and IL-13; this co-expression is enhanced after caloric intake. The circadian synchronizer vasoactive intestinal peptide also stimulates ILC2 cells through the VPAC2 receptor to release IL-5, linking eosinophil levels with metabolic cycling. Tissue ILC2 cells regulate basal eosinophilopoiesis and tissue eosinophil accumulation through constitutive and stimulated cytokine expression, and this dissociated regulation can be tuned by nutrient intake and central circadian rhythms. '"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Abstracts[24037376]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some Utility Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll define some utility functions that allow us to tokenize a string into terms, perform linguistic preprocessing on a list of terms, as well as a function to display information about a paper in a nice way. Note that these tokenization and preprocessing functions are rather naive. We will improve them in a later assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['lorem', 'ipsum', 'dolor', 'sit', 'amet']\n"
     ]
    }
   ],
   "source": [
    "def tokenize(text):\n",
    "    \"\"\"\n",
    "    Function that tokenizes a string in a rather naive way. Can be extended later.\n",
    "    \"\"\"\n",
    "    return text.split(' ')\n",
    "\n",
    "def preprocess(tokens):\n",
    "    \"\"\"\n",
    "    Perform linguistic preprocessing on a list of tokens. Can be extended later.\n",
    "    \"\"\"\n",
    "    result = []\n",
    "    for token in tokens:\n",
    "        result.append(token.lower())\n",
    "    return result\n",
    "\n",
    "print(preprocess(tokenize(\"Lorem ipsum dolor sit AMET\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1038/nature12526>Type 2 innate lymphoid cells control eosinophil homeostasis.</a></strong><br>2013. Nussbaum JC, Van Dyken SJ, von Moltke J, Cheng LE, Mohapatra A, Molofsky AB, Thornton EE, Krummel MF, Chawla A, Liang HE, Locksley RM<br>[ID: 24037376]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1038/nature12526>Type 2 innate lymphoid cells control eosinophil homeostasis.</a></strong><br>2013. Nussbaum JC, Van Dyken SJ, von Moltke J, Cheng LE, Mohapatra A, Molofsky AB, Thornton EE, Krummel MF, Chawla A, Liang HE, Locksley RM<br><small><strong>Abstract:</strong> <em>Eosinophils are specialized myeloid cells associated with allergy and helminth infections. Blood eosinophils demonstrate circadian cycling, as described over 80 years ago, and are abundant in the healthy gastrointestinal tract. Although a cytokine, interleukin (IL)-5, and chemokines such as eotaxins mediate eosinophil development and survival, and tissue recruitment, respectively, the processes underlying the basal regulation of these signals remain unknown. Here we show that serum IL-5 levels are maintained by long-lived type 2 innate lymphoid cells (ILC2) resident in peripheral tissues. ILC2 cells secrete IL-5 constitutively and are induced to co-express IL-13 during type 2 inflammation, resulting in localized eotaxin production and eosinophil accumulation. In the small intestine where eosinophils and eotaxin are constitutive, ILC2 cells co-express IL-5 and IL-13; this co-expression is enhanced after caloric intake. The circadian synchronizer vasoactive intestinal peptide also stimulates ILC2 cells through the VPAC2 receptor to release IL-5, linking eosinophil levels with metabolic cycling. Tissue ILC2 cells regulate basal eosinophilopoiesis and tissue eosinophil accumulation through constitutive and stimulated cytokine expression, and this dissociated regulation can be tuned by nutrient intake and central circadian rhythms. </em></small><br>[ID: 24037376]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display, HTML\n",
    "import re\n",
    "\n",
    "def display_summary( id, show_abstract=False, show_id=True, extra_text='' ):\n",
    "    \"\"\"\n",
    "    Function for printing a paper's summary through IPython's Rich Display System.\n",
    "    Trims long author lists, and adds a link to the paper's DOI (when available).\n",
    "    \"\"\"\n",
    "    s = Summaries[id]\n",
    "    lines = []\n",
    "    title = s.title\n",
    "    if s.doi != '':\n",
    "        title = '<a href=http://dx.doi.org/{:s}>{:s}</a>'.format(s.doi, title)\n",
    "    title = '<strong>' + title + '</strong>'\n",
    "    lines.append(title)\n",
    "    authors = ', '.join( s.authors[:20] ) + ('' if len(s.authors) <= 20 else ', ...')\n",
    "    lines.append(str(s.year) + '. ' + authors)\n",
    "    if (show_abstract):\n",
    "        lines.append('<small><strong>Abstract:</strong> <em>{:s}</em></small>'.format(Abstracts[id]))\n",
    "    if (show_id):\n",
    "        lines.append('[ID: {:d}]'.format(id))\n",
    "    if (extra_text != ''):\n",
    "         lines.append(extra_text)\n",
    "    display( HTML('<br>'.join(lines)) )\n",
    "\n",
    "display_summary(24037376)\n",
    "display_summary(24037376, show_abstract=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating our first index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now create an _inverted index_ based on the words in the abstracts of the papers in our dataset.\n",
    "\n",
    "We will implement our inverted index as a **Python dictionary with terms as keys and posting lists as values**. For the posting lists, instead of using Python lists and then implementing the different operations on them ourselves, we will use **Python sets** and use the predefined set operations to process these posting \"lists\". This will also ensure that each document is added at most once per term. The use of Python sets is not the most efficient solution but will work for our purposes. (As an optional additional exercise, you can try to implement the posting lists as Python lists for this and the following assignments.)\n",
    "\n",
    "Not every paper in our dataset has an abstract; we will only index papers for which an abstract is available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "inverted_index = defaultdict(set)\n",
    "\n",
    "# This may take a while:\n",
    "for (id, abstract) in Abstracts.items():\n",
    "    for term in preprocess(tokenize(abstract)):\n",
    "        inverted_index[term].add(id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what's in the index for the example term 'amsterdam':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{16513592, 18706961, 15504455}\n"
     ]
    }
   ],
   "source": [
    "print(inverted_index['amsterdam'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use this inverted index to answer simple one-word queries, for example to show all papers that contain the word 'rotterdam':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1046/j.1442-2042.10.s1.10.x>Systemic aspects of interstitial cystitis, immunology and linkage with autoimmune disorders.</a></strong><br>2003. van de Merwe JP, Yamada T, Sakamoto Y<br>[ID: 14641413]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1093/oxfordjournals.aje.a117332>Toxocara seroprevalence in 5-year-old elementary schoolchildren: relation with allergic asthma.</a></strong><br>1994. Buijs J, Borsboom G, van Gemund JJ, Hazebroek A, van Dongen PA, van Knapen F, Neijens HJ<br>[ID: 7977294]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "query_word = 'rotterdam'\n",
    "for i in inverted_index[query_word]:\n",
    "    display_summary(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your name:** Andrew Harrison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1\n",
    "\n",
    "Construct a function called `and_query` that takes as input a single string, consisting of one or more words, such that the function returns a list of matching documents. `and_query`, as its name suggests, should require that all query terms are present in the documents of the result list. Demonstrate the working of your function with an example (choose one that leads to fewer than 100 hits to not overblow this notebook file).\n",
    "\n",
    "(You can use the `tokenize` and `preprocess` functions we defined above to tokenize and preprocess your query. You can also exploit the fact that the posting lists are [sets](https://docs.python.org/3/library/stdtypes.html#set), which means you can easily perform set operations such as union, difference and intersect on them.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1080/08958370500515913>Relation between sources of particulate air pollution and biological effect parameters in samples from four European cities: an exploratory study.</a></strong><br>2006. Steerenberg PA, van Amelsvoort L, Lovik M, Hetland RB, Alberg T, Halatek T, Bloemen HJ, Rydzynski K, Swaen G, Schwarze P, Dybing E, Cassee FR<br>[ID: 16513592]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def and_query(query_words):\n",
    "    search_words = []\n",
    "    search_words = preprocess(tokenize(query_words))\n",
    "    display_results = set(inverted_index[search_words[0]])\n",
    "    for i in search_words:\n",
    "        temp_results = set()\n",
    "        for j in inverted_index[i]:\n",
    "            temp_results.add(j)\n",
    "        display_results = display_results.intersection(temp_results)\n",
    "    return list(display_results)\n",
    "\n",
    "for i in and_query(\"amsterdam lodz oslo rome\"):\n",
    "    display_summary(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2\n",
    "\n",
    "Construct a second function called `or_query` that works in the same way as `and_query` you just implemented, but returns as function value the documents that contain _at least one_ of the words in the query. Demonstrate the working of this second function also with an example (again, choose one that leads to fewer than 100 hits)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1016/j.taap.2004.04.011>Adjuvant activity of ambient particulate matter of different sites, sizes, and seasons in a respiratory allergy mouse model.</a></strong><br>2004. Steerenberg PA, Withagen CE, van Dalen WJ, Dormans JA, Cassee FR, Heisterkamp SH, van Loveren H<br>[ID: 15504455]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1016/j.tox.2004.02.007>Respiratory allergy adjuvant and inflammatory effects of urban ambient particles.</a></strong><br>2004. Dybing E, Løvdal T, Hetland RB, Løvik M, Schwarze PE<br>[ID: 15138057]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1542/peds.2004-2785>Early respiratory infections, asthma, and allergy: 10-year follow-up of the Oslo Birth Cohort.</a></strong><br>2005. Nafstad P, Brunekreef B, Skrondal A, Nystad W<br>[ID: 16061578]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1007/s00431-006-0410-y>Characterization of irritable bowel syndrome and constipation in children with allergic diseases.</a></strong><br>2007. Caffarelli C, Coscia A, Baldi F, Borghi A, Capra L, Cazzato S, Migliozzi L, Pecorari L, Valenti A, Cavagni G<br>[ID: 17345097]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1111/j.1399-3038.2009.00965.x>Diagnostic value of exhaled nitric oxide in childhood asthma and allergy.</a></strong><br>2010. Sachs-Olsen C, Lødrup Carlsen KC, Mowinckel P, Håland G, Devulapalli CS, Munthe-Kaas MC, Carlsen KH<br>[ID: 21083852]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>Behaviour on the nasal provocation test in patients affected by conjunctivitis and/or asthma of allergic origin.</strong><br>1981. Filiaci F, Lucarelli N<br>[ID: 7302475]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>Incidence of allergic rhinitis in children.</strong><br>1983. Filiaci F, Lucarelli N, Zardo F, Ralli G<br>[ID: 6857100]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1016/j.fct.2008.07.020>The serum bank of EuroPrevall - the prevalence, cost and basis of food allergy across Europe.</a></strong><br>2008. Vieths S, Reese G, Ballmer-Weber BK, Beyer K, Burney P, Fernandez-Rivas M, Summers C, Ree Rv, Mills C<br>[ID: 18706961]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>[The underdiagnosis and undertreatment of asthma in general population of the Lodz Province (Poland)].</strong><br>2010. Kupryś-Lipińska I, Elgalal A, Kuna P<br>[ID: 20162515]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>Skin disease in military personnel.</strong><br>2000. Selvaag E<br>[ID: 10741081]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1016/j.tox.2004.09.016>Ambient air particles from four European cities increase the primary cellular response to allergen in the draining lymph node.</a></strong><br>2005. Nygaard UC, Alberg T, Bleumink R, Aase A, Dybing E, Pieters R, Løvik M<br>[ID: 15596255]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>Month of birth and grass pollen or mite sensitization in children with respiratory allergy: a significant relationship.</strong><br>1988. Businco L, Cantani A, Farinella F, Businco E<br>[ID: 3396196]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>[Analysis of possible causes of recurrent respiratory tract infections in children from Lodz, Poland].</strong><br>2003. Stelmach I, Podsiadłowicz-Borzecka M, Jurałowicz D, Stelmach W<br>[ID: 14587430]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>Immunogenicity of hydrolysate formulas in children (Part 2): 41 case reports.</strong><br>2001. Cantani A, Micera M<br>[ID: 11436966]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.3109/1547691X.2012.755580>Pre-natal exposure to perfluoroalkyl substances may be associated with altered vaccine antibody levels and immune-related health outcomes in early childhood.</a></strong><br>2013. Granum B, Haug LS, Namork E, Stølevik SB, Thomsen C, Aaberge IS, van Loveren H, Løvik M, Nygaard UC<br>[ID: 23350954]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1111/j.1398-9995.2005.00773.x>Early effects of Asthma Prevention Program on asthma diagnosis and hospitalization in urban population of Poland.</a></strong><br>2005. Stelmach W, Majak P, Jerzynska J, Stelmach I<br>[ID: 15813804]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>The environment and childhood asthma (ECA) study in Oslo: ECA-1 and ECA-2.</strong><br>2002. Lødrup Carlsen KC<br>[ID: 12688621]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>[Influence of cow's milk protein allergy on the diagnosis of functional gastrointestinal diseases based on the Rome IV standard in infants and young children].</strong><br>2018. Feng BW, Fu SM, Zhang QS, Long XL, Xie XL, Ren W, Liang ZT, Yang ZL, Chen A<br>[ID: 29335084]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1097/MCG.0000000000000740>The Role of Duodenal Inflammation in Functional Dyspepsia.</a></strong><br>2017. Walker MM, Talley NJ<br>[ID: 27811629]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1155/2014/198461>A cross-sectional study of risk factors for irritable bowel syndrome in children 8-13 years of age in suzhou, china.</a></strong><br>2014. Zhu X, Chen W, Zhu X, Shen Y<br>[ID: 24899889]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>[Skin prick test with inhaled allergens in the general population of Lodz province].</strong><br>2009. Kupryś-Lipińska I, Elgalal A, Kuna P<br>[ID: 19591092]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong><a href=http://dx.doi.org/10.1080/08958370500515913>Relation between sources of particulate air pollution and biological effect parameters in samples from four European cities: an exploratory study.</a></strong><br>2006. Steerenberg PA, van Amelsvoort L, Lovik M, Hetland RB, Alberg T, Halatek T, Bloemen HJ, Rydzynski K, Swaen G, Schwarze P, Dybing E, Cassee FR<br>[ID: 16513592]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>Detection of latex allergens by immunoelectron microscopy in ambient air (PM10) in Oslo, Norway (1997-2003).</strong><br>2004. Namork E, Kurup VP, Aasvang GM, Johansen BV<br>[ID: 15552702]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<strong>House mite dust bronchial asthma: clinical and epidemiological observations.</strong><br>1986. Sirianni MC, Soddu S, Bonomo R<br>[ID: 3828095]"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def or_query(query_words):\n",
    "    search_words = []\n",
    "    search_words = preprocess(tokenize(query_words))\n",
    "    display_results = set(inverted_index[search_words[0]])\n",
    "    for i in search_words:\n",
    "        temp_results = set()\n",
    "        for j in inverted_index[i]:\n",
    "            temp_results.add(j)\n",
    "        display_results = display_results.union(temp_results)\n",
    "    return list(display_results)\n",
    "\n",
    "for i in or_query(\"amsterdam lodz oslo rome\"):\n",
    "    display_summary(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3\n",
    "\n",
    "Show how many hits the query \"to be or not to be\" returns for your two query functions (`and_query` and `or_query`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "and_query hits = 3536\n",
      "or_query hits = 44826\n"
     ]
    }
   ],
   "source": [
    "print(\"and_query hits =\", len(and_query(\"to be or not to be\")))\n",
    "print(\"or_query hits =\", len(or_query(\"to be or not to be\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 4\n",
    "\n",
    "Given the nature of our dataset, how many of the documents from task 3 do you think are actually about the [Shakespeare quote](https://en.wikipedia.org/wiki/To_be%2C_or_not_to_be)? What could you do to better handle such queries? (You don't have to implement this yet!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer:** [Given that the dataset is about academic publications in pubmed, it's likely that almost all of the documents have nothing to do with Shakespeare's quote. There may a very small number of author's taking creative liberty and using the well known quote in the context of their paper to make a point. And it's not ruled out, but highly unlikely that any of the authors are directly referring to Hamlet, Act 3, Scene 1 specifically, when Hamlet utters the words.\n",
    "\n",
    "To better handle the queries, we need to move beyond simple binary, single word queries. The words 'to, be, or, not' are again all stop words (listed as such in NLTK). And as we're doing single word binary, you're just checking whether these 4 stop words are all present in a specific abstract. Which is fairly useless. \n",
    "\n",
    "For a start, we could think to use binary word queries, but we have a multiple term phrase (e.g. 6 in this case), and it would blow up the size of your index going >2. So you'd want to move towards a better phrase query model, perhaps using positional indexes. Where you're recording in the postings where in each document the tokens appear for the term. You can then use an ajusted merge algorithm that uses the offset between the words to do phrase queries. This is still susceptible to the feast or famine issue in Boolean search however, and therefore you might want to head towards a vector space model instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 5\n",
    "\n",
    "Why does `and_query('eosinophil signal inflammation')` not return our example paper 24037376, even though it mentions eosinophil, signal, and inflammation in the abstract? (You do not have to implement anything to fix this yet!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Answer:** [Eosinophil is present as eosinophil, eosinophils, and eosinophilopoiesis. Inflammation is present, but as 'inflammation,' i.e. with a comma straight afterwards. As our tokenisation splits on spaces, it keeps it together as one token. Finally, signal is only present as the plural signals, thus the token is 'signals', and as we're not turning tokens into terms yet, again this is missed.\n",
    "If you do `and_query('eosinophil signals inflammation,')` then you do find the paper 24037376, however, this is obviously not a workable solution to know in advance possible mistokenisations!. So we need to head towards better tokenization, and towards using tokens, words, and terms.]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Submit the answers to the assignment via Canvas as a modified version of this Notebook file (file with `.ipynb` extension) that includes your code and your answers.\n",
    "\n",
    "Before submitting, restart the kernel and re-run the complete code (**Kernel > Restart & Run All**), and then check whether your assignment code still works as expected.\n",
    "\n",
    "Don't forget to add your name, and remember that the assignments have to be done individually and group submissions are **not allowed**."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}