tutorials/intro/Intro_Tutorial_1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro. to Snorkel: Extracting Spouse Relations from the News\n",
    "\n",
    "In this tutorial, we will walk through the process of using `Snorkel` to identify mentions of spouses in a corpus of news articles. The tutorial is broken up into 3 notebooks, each covering a step in the pipeline:\n",
    "1. Preprocessing\n",
    "2. Training\n",
    "3. Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part I: Preprocessing\n",
    "\n",
    "In this notebook, we preprocess several documents using `Snorkel` utilities, parsing them into a simple hierarchy of component parts of our input data, which we refer to as _contexts_. We'll also create _candidates_ out of these contexts, which are the objects we want to classify, in this case, possible mentions of spouses. Finally, we'll load some gold labels for evaluation.\n",
    "\n",
    "All of this preprocessed input data is saved to a database.  (Connection strings can be specified by setting the `SNORKELDB` environment variable.  In Snorkel, if no database is specified, then a SQLite database at `./snorkel.db` is created by default--so no setup is needed here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initializing a `SnorkelSession`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we initialize a `SnorkelSession`, which manages a connection to a database automatically for us, and will enable us to save intermediate results.  If we don't specify any particular database (see commented-out code below), then it will automatically create a SQLite database in the background for us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "\n",
    "# TO USE A DATABASE OTHER THAN SQLITE, USE THIS LINE\n",
    "# Note that this is necessary for parallel execution amongst other things...\n",
    "# os.environ['SNORKELDB'] = 'postgres:///snorkel-intro'\n",
    "\n",
    "from snorkel import SnorkelSession\n",
    "session = SnorkelSession()\n",
    "\n",
    "# Here, we just set how many documents we'll process for automatic testing- you can safely ignore this!\n",
    "n_docs = 500 if 'CI' in os.environ else 2591"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the Corpus\n",
    "\n",
    "Next, we load and pre-process the corpus of documents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configuring a `DocPreprocessor`\n",
    "\n",
    "We'll start by defining a `TSVDocPreprocessor` class to read in the documents, which are stored in a tab-seperated value format as pairs of document names and text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.parser import TSVDocPreprocessor\n",
    "\n",
    "doc_preprocessor = TSVDocPreprocessor('data/articles.tsv', max_docs=n_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running a `CorpusParser`\n",
    "\n",
    "We'll use [Spacy](https://spacy.io/), an NLP preprocessing tool, to split our documents into sentences and tokens, and provide named entity annotations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clearing existing...\n",
      "Running UDF...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2591/2591 [02:20<00:00, 27.89it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 36s, sys: 3.06 s, total: 2min 39s\n",
      "Wall time: 2min 42s\n"
     ]
    }
   ],
   "source": [
    "from snorkel.parser.spacy_parser import Spacy\n",
    "from snorkel.parser import CorpusParser\n",
    "\n",
    "corpus_parser = CorpusParser(parser=Spacy())\n",
    "%time corpus_parser.apply(doc_preprocessor, count=n_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then use simple database queries (written in the syntax of [SQLAlchemy](http://www.sqlalchemy.org/), which Snorkel uses) to check how many documents and sentences were parsed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Documents:', 2591)\n",
      "('Sentences:', 67778)\n"
     ]
    }
   ],
   "source": [
    "from snorkel.models import Document, Sentence\n",
    "\n",
    "print(\"Documents:\", session.query(Document).count())\n",
    "print(\"Sentences:\", session.query(Sentence).count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Candidates\n",
    "\n",
    "The next step is to extract _candidates_ from our corpus. A `Candidate` in Snorkel is an object for which we want to make a prediction. In this case, the candidates are pairs of people mentioned in sentences, and our task is to predict which pairs are described as married in the associated text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining a `Candidate` schema\n",
    "We now define the schema of the relation mention we want to extract (which is also the schema of the candidates).  This must be a subclass of `Candidate`, and we define it using a helper function. Here we'll define a binary _spouse relation mention_ which connects two `Span` objects of text.  Note that this function will create the table in the database backend if it does not exist:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.models import candidate_subclass\n",
    "\n",
    "Spouse = candidate_subclass('Spouse', ['person1', 'person2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing a basic `CandidateExtractor`\n",
    "\n",
    "Next, we'll write a basic function to extract **candidate spouse relation mentions** from the corpus.  The [Spacy](https://spacy.io/) parser we used performs _named entity recognition_ for us.\n",
    "\n",
    "We will extract `Candidate` objects of the `Spouse` type by identifying, for each `Sentence`, all pairs of n-grams (up to 7-grams) that were tagged as people. (An n-gram is a span of text made up of n tokens.) We do this with three objects:\n",
    "\n",
    "* A `ContextSpace` defines the \"space\" of all candidates we even potentially consider; in this case we use the `Ngrams` subclass, and look for all n-grams up to 7 words long\n",
    "\n",
    "* A `Matcher` heuristically filters the candidates we use.  In this case, we just use a pre-defined matcher which looks for all n-grams tagged by Spacy as \"PERSON\". The keyword argument `longest_match_only` means that we'll skip n-grams contained in other n-grams.\n",
    "\n",
    "* A `CandidateExtractor` combines this all together!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.candidates import Ngrams, CandidateExtractor\n",
    "from snorkel.matchers import PersonMatcher\n",
    "\n",
    "ngrams         = Ngrams(n_max=7)\n",
    "person_matcher = PersonMatcher(longest_match_only=True)\n",
    "cand_extractor = CandidateExtractor(Spouse, [ngrams, ngrams], [person_matcher, person_matcher])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll split up the documents into train, development, and test splits; and collect the associated sentences.\n",
    "\n",
    "Note that we'll filter out a few sentences that mention more than five people. These lists are unlikely to contain spouses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.models import Document\n",
    "from util import number_of_people\n",
    "\n",
    "docs = session.query(Document).order_by(Document.name).all()\n",
    "\n",
    "train_sents = set()\n",
    "dev_sents   = set()\n",
    "test_sents  = set()\n",
    "\n",
    "for i, doc in enumerate(docs):\n",
    "    for s in doc.sentences:\n",
    "        if number_of_people(s) <= 5:\n",
    "            if i % 10 == 8:\n",
    "                dev_sents.add(s)\n",
    "            elif i % 10 == 9:\n",
    "                test_sents.add(s)\n",
    "            else:\n",
    "                train_sents.add(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we'll apply the candidate extractor to the three sets of sentences. The results will be persisted in the database backend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clearing existing...\n",
      "Running UDF...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 53991/53991 [05:08<00:00, 174.77it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Number of candidates:', 22276)\n",
      "Clearing existing...\n",
      "Running UDF...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 6789/6789 [00:34<00:00, 199.34it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Number of candidates:', 2814)\n",
      "Clearing existing...\n",
      "Running UDF...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 6194/6194 [00:32<00:00, 187.88it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Number of candidates:', 2702)\n",
      "CPU times: user 6min 1s, sys: 6.11 s, total: 6min 7s\n",
      "Wall time: 6min 15s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "for i, sents in enumerate([train_sents, dev_sents, test_sents]):\n",
    "    cand_extractor.apply(sents, split=i)\n",
    "    print(\"Number of candidates:\", session.query(Spouse).filter(Spouse.split == i).count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Gold Labels\n",
    "\n",
    "Finally, we'll load gold labels for development and evaluation. Even though Snorkel is designed to create labels for data, we still use gold labels to evaluate the quality of our models. Fortunately, we need far less labeled data to _evaluate_ a model than to _train_ it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AnnotatorLabels created: 2698\n",
      "AnnotatorLabels created: 2608\n",
      "CPU times: user 1min 26s, sys: 746 ms, total: 1min 27s\n",
      "Wall time: 1min 28s\n"
     ]
    }
   ],
   "source": [
    "from util import load_external_labels\n",
    "\n",
    "%time missed = load_external_labels(session, Spouse, annotator_name='gold')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, in Part II, we will work towards building a model to predict these labels with high accuracy using data programming"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}