WeakSupervision.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1c5c3dbc-7e8d-47eb-8c78-591c4edc2347",
   "metadata": {},
   "source": [
    "# Development of Labelling Functions and Model Training\n",
    "\n",
    "In this notebook, we present the workflow of training a weakly supervised gene tagger for German medical text using the skweak framework. In particular, we:\n",
    "\n",
    "1. implement different labelling functions for detecting gene names\n",
    "2. aggregate their predictions using an Hidden Markov Model (Label Model)\n",
    "3. train a spaCy NER model on this data\n",
    "4. evaluate the results on a small set of gold-standard labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "bb5a326a-91b3-4057-af63-82389654e86c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from pathlib import Path\n",
    "from tqdm import tqdm\n",
    "\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "import spacy\n",
    "from spacy.tokens import Span, DocBin\n",
    "from spacy_transformers import Transformer\n",
    "from spacy_transformers.pipeline_component import DEFAULT_CONFIG\n",
    "\n",
    "from skweak import heuristics, gazetteers, generative, utils, base\n",
    "from skweak.base import SpanAnnotator\n",
    "from skweak.heuristics import SpanEditorAnnotator, VicinityAnnotator, SpanConstraintAnnotator\n",
    "from skweak.analysis import LFAnalysis\n",
    "\n",
    "import sklearn.metrics\n",
    "\n",
    "from evaluation import evaluate, get_results, compute_raw_numbers, _get_probs, show_errors\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9ca0dedc-eb7c-4e0d-96e0-e49f9ffee692",
   "metadata": {},
   "outputs": [],
   "source": [
    "stops = set(stopwords.words('german'))\n",
    "random_seed = 42"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1eeaf50d-2649-437d-83e5-6af3f8927bd0",
   "metadata": {},
   "source": [
    "# Load Data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d0d0caf",
   "metadata": {},
   "source": [
    "Here we customize the spaCy tokenizer without statistical model and loading the standard German spaCy model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8712ce01-00dd-4afd-af2d-eca3788c264e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<spacy.pipeline.sentencizer.Sentencizer at 0x7f8f3693c900>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nlp = spacy.load('de_core_news_md')\n",
    "infixes = nlp.Defaults.infixes + [r'([-])']\n",
    "nlp.tokenizer.infix_finditer = spacy.util.compile_infix_regex(infixes).finditer\n",
    "nlp.add_pipe('sentencizer')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "201a60cb",
   "metadata": {},
   "source": [
    "In the following, we construct a dataframe from all sentences in GGPONC and concatenate them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8f14c8ac-efb8-4fe5-8c50-a873f5714833",
   "metadata": {},
   "outputs": [],
   "source": [
    "GGPONC_PATH = 'data/ggponc/plain_text/sentences/all_files_sentences/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0df70ec8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10193"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences = list(Path(GGPONC_PATH).glob('*.txt'))\n",
    "len(sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3c9a6a59",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_df(file):\n",
    "    df = pd.read_csv(file, delimiter='\\t', names =['text'] )\n",
    "    df['file'] = file.stem\n",
    "    df['sentence_id'] = df.index\n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6710a0ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "dfs = [get_df(file) for file in sentences]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "197a0013",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "85996"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentence_df = pd.concat(dfs)\n",
    "len(sentence_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bae7c442-07f8-4727-a33a-cb84dca20aa4",
   "metadata": {},
   "source": [
    "# Labeling Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b8f5257-d74b-4e85-a0c1-f15879a2bcb3",
   "metadata": {},
   "source": [
    "## Gazetteer-based LFs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c52fdfc",
   "metadata": {},
   "source": [
    "The Clinical Interpretation of Variants in Cancer (CIViC) database is an opensource, open-access knowledgebase curated by experts on therapeutic, prognostic, diagnostic and predisposing relevance of inherited and somatic variants of every type. Both, genes and variants, are used in this project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "09a9d007-e413-4718-8e00-dbe9d1f6ee5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "civic_genes_df = pd.read_csv('data/molecular/nightly-GeneSummaries.tsv', sep='\\t')\n",
    "CIVIC_genes = civic_genes_df['name'].tolist()\n",
    "CIVIC_genes_lower = [c.lower() for c in CIVIC_genes]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "14ff4e2e-1545-4f11-96a5-e0a481e78017",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Skipping line 13: expected 29 fields, saw 33\n",
      "Skipping line 17: expected 29 fields, saw 30\n",
      "Skipping line 31: expected 29 fields, saw 30\n",
      "Skipping line 441: expected 29 fields, saw 30\n",
      "Skipping line 502: expected 29 fields, saw 30\n",
      "Skipping line 553: expected 29 fields, saw 31\n",
      "\n"
     ]
    }
   ],
   "source": [
    "civic_variant_df = pd.read_csv('data/molecular/nightly-VariantSummaries.tsv', sep='\\t', error_bad_lines=False )\n",
    "CIVIC_variants = civic_variant_df['variant'].tolist()\n",
    "CIVIC_variants_lower = [c.lower() for c in CIVIC_variants]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52b9bd79",
   "metadata": {},
   "source": [
    "\"cue_civic\" is based on the CIViC database. If a token contains a gene which is listed in the database, this and the next token are labelled a gene. This way, we do not restrict the function to a 100 percent match but leave it some leeway."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "912fc8c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def civic_fn(doc):\n",
    "    for tok in doc:\n",
    "        for cue in CIVIC_genes:\n",
    "            if tok.text.find(cue) == -1:\n",
    "                continue\n",
    "            else:\n",
    "                yield tok.i, tok.i+1, \"Gene or Protein\"\n",
    "lf_civic = heuristics.FunctionAnnotator(\"CIViC\", civic_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d750a40-b60a-47a1-8988-69a3d2ed81d2",
   "metadata": {},
   "source": [
    "Get all synonyms in Entrez for CIViC genes, remove short ones and German stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d9454936-02a2-4d19-900f-b06e0d183c6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "entrez_df = pd.read_csv('data/Homo_sapiens.gene_info', sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "96921b47-84ab-4f09-8864-4e33059bcde0",
   "metadata": {},
   "outputs": [],
   "source": [
    "symbols = set()\n",
    "for _, r in entrez_df.set_index('GeneID').loc[civic_variant_df.entrez_id].iterrows():\n",
    "    symbols.add(r.Symbol)\n",
    "    for s in r.Synonyms.split('|'):\n",
    "        if not s in ['R1', 'R2', 'eN', 'HNPCC'] and len(s) > 1 and not s.lower() in stops:\n",
    "            symbols.add(s.lower())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8cb9c469-a8b9-47b0-8f8d-ab7581a40044",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2028/2028 [00:00<00:00, 12862.40it/s]\n"
     ]
    }
   ],
   "source": [
    "from spacy.matcher import Matcher\n",
    "\n",
    "entrez_matcher = Matcher(nlp.vocab)\n",
    "pattern = []\n",
    "for s in nlp.pipe(tqdm(symbols), disable=[\"ner\", \"tok2vec\"]):\n",
    "    for pos in ['NOUN', 'PROPN', 'X']: # Consider only if first POS is one of these\n",
    "        p = [{'LOWER' : spl.text.lower() } for spl in s]\n",
    "        p[0]['POS'] = pos\n",
    "        pattern.append(p)\n",
    "        p2 = p + [{'LOWER' : '-'}, {'LOWER' : 'gen'}] #also consider if followed by -Gen\n",
    "        pattern.append(p2)\n",
    "entrez_matcher.add(\"entrez\", pattern)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ce0939e0-0515-4921-af01-3ede88f7a043",
   "metadata": {},
   "outputs": [],
   "source": [
    "def entrez_fn(doc):\n",
    "    matches = entrez_matcher(doc)\n",
    "    if matches:\n",
    "        # Keep longest matches only\n",
    "        spans = [doc[start:end] for _, start, end in matches]\n",
    "        spans = spacy.util.filter_spans(spans)\n",
    "        for s in spans:\n",
    "            yield s.start, s.end, 'Gene or Protein'\n",
    "lf_entrez = heuristics.FunctionAnnotator(\"Entrez\", entrez_fn)  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54562797-295f-4920-863e-8d141a49ea80",
   "metadata": {},
   "source": [
    "The Online Mendelian Inheritance in Man (OMIM) database is the encyclopedic collection of the human medical branch of genetics. \"omim\" is based on the OMIM database and checks whether tokens are present in its list of 16,767 approved gene symbols in lowercase as the diversity of genes often shows in volatile capitalization. To increase precision, genes with a length shorter than three characters are matched only correctly cased."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3b2354cf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16767\n"
     ]
    }
   ],
   "source": [
    "omim_list = pd.read_csv(\"data/molecular/mim2gene.csv\")\n",
    "omim_list = omim_list['name'].tolist()\n",
    "omim_list_lower = [o.lower() for o in omim_list]\n",
    "short_genes = []\n",
    "for u in omim_list:\n",
    "    if len(u)<3:\n",
    "        short_genes.append(u)\n",
    "less_short_genes = []\n",
    "for u in omim_list:\n",
    "    if len(u)<5 and len(u)>2:\n",
    "        less_short_genes.append(u)\n",
    "print(len(omim_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "dab88a11-60d2-4075-9ae0-4f22d1e5874d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def omim_fn(doc):\n",
    "    for tok in doc:\n",
    "        if tok.text.lower() in omim_list_lower and tok.text.lower() not in stops and len(tok.text.lower())>=3:\n",
    "            yield tok.i, tok.i+1, \"Gene or Protein\"\n",
    "lf_omim = heuristics.FunctionAnnotator(\"OMIM\", omim_fn) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02dd22df-8fc6-44eb-9915-4e80ee9ef7d9",
   "metadata": {},
   "source": [
    "The Catalogue of Somatic Mutations in Cancer (COSMIC) database harbors somatic cell mutations and additional information associated with cancer in humans."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "2890317a-11f0-4cd2-bc22-5f583ca95c7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "cosmic_census = pd.read_csv(\"data/molecular/cancer_gene_census.csv\")\n",
    "cosmic_census = cosmic_census['Gene Symbol'].tolist()\n",
    "cosmic_census_lower = [c.lower() for c in cosmic_census]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "11655485-9d45-443a-a736-3e9301e6be21",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cosmic_fn(doc):\n",
    "    for tok in doc:\n",
    "        for cue in cosmic_census:\n",
    "            if tok.text.find(cue) == -1:\n",
    "                continue\n",
    "            else:\n",
    "                yield tok.i, tok.i+1, \"Gene or Protein\"\n",
    "lf_cosmic = heuristics.FunctionAnnotator(\"COSMIC\", cosmic_fn)  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "309be9e6-7453-4e00-a897-ef021f7c8bce",
   "metadata": {},
   "source": [
    "Gazetteer based on common Protein names, sourced from Wikipedia and refined using the training part of the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ee7c77c8-907d-4078-aea1-b0194e20e8cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Proteins': [PD-L1]}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from skweak.gazetteers import Trie, GazetteerAnnotator\n",
    "\n",
    "terms = [t.strip() for t in open('proteins.txt', 'r').readlines()]\n",
    "\n",
    "trie = Trie()\n",
    "for term in terms:\n",
    "    trie.add([t.text for t in nlp(term)])\n",
    "\n",
    "lf_protein_gazetteer = GazetteerAnnotator('Proteins', tries = {'Gene or Protein' : trie })\n",
    "\n",
    "doc = nlp(\"PD-L1\")\n",
    "lf_protein_gazetteer(doc)\n",
    "doc.spans"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64987ef9-e82d-43c2-86b6-2d4fef3b05f0",
   "metadata": {},
   "source": [
    "## Rule-based LFs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68e52e0a",
   "metadata": {},
   "source": [
    "\"hgnc\" is based on the Human Genome Organization (HUGO) Gene Nomenclature Committee (HGNC) naming conventions for genes and leverages regular expressions to let the annotator abide by them. Those expressions comprise various combinations of letters and numbers and certain fixed terms for shorter terms to avoid underfitting. In addition, the CIViC database for variants has also been included for a better recall."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "653ed53d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def hgnc_fn(doc):\n",
    "    for tok in doc:\n",
    "        if re.search(r\"[a-zA-Z]{4}\\d{2}\", tok.text) or re.search(r\"[a-zA-Z]{5}\\d{1}\", tok.text)\\\n",
    "        or re.search(r\"[a-zA-Z]{4}\\d{1}\", tok.text) or re.search(r\"[A-Z]{5}\\d{1}\", tok.text)\\\n",
    "        or re.search(r\"[A-Z]{5}\\d{2}\", tok.text) or re.search(r\"[A-Z]{3}\\d{2}\", tok.text)\\\n",
    "        or re.search(r\"[a-zA-Z]{2}\\d{3}[a-zA-Z]{2}\", tok.text) or re.search(r\"[a-zA-Z]{1}\\d{3}[a-zA-Z]{1}\", tok.text)\\\n",
    "        or re.search(r\"[A-Z]{3}\\d{2}\", tok.text) or re.search(r\"[A-Z]{6}\\d{1}\", tok.text)\\\n",
    "        or re.search(r\"[A-Z]{3}\\d{3}\", tok.text) or re.search(r\"[p]\\d{2}\", tok.text)\\\n",
    "        or re.search(r\"CYP[a-zA-Z0-9]{3}\", tok.text) or re.search(r\"CYP[a-zA-Z0-9]{2}\", tok.text)\\\n",
    "        or re.search(r\"[A-Z]{3}\\d{1}\", tok.text) or re.search(r\"[A-Z]{2}\\d{2}\", tok.text)\\\n",
    "        or re.search(r\"^CK.\", tok.text) or re.search(r\"^PD-..\", tok.text) or re.search(r\"^PS[MA|A]\", tok.text) or tok.text.lower in CIVIC_variants_lower:\n",
    "            yield tok.i, tok.i+1, \"Gene or Protein\"\n",
    "lf_hgnc = heuristics.FunctionAnnotator(\"HGNC\", hgnc_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21803256-76bf-496e-87e5-7205333d3763",
   "metadata": {},
   "source": [
    "Rule-based matcher based on protein families"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "6a982767-0c6f-4ff7-839d-d52d72d1b5be",
   "metadata": {},
   "outputs": [],
   "source": [
    "protein_matcher = Matcher(nlp.vocab)\n",
    "patterns = []\n",
    "\n",
    "for suffix in ['[A-Z]*[Kk]inase[n]?$', '[A-Z]+[rR]ezeptor(en|s)?$', '^(RAS|ras)$']:\n",
    "    p = [{'TEXT' : { 'REGEX' : suffix}}]\n",
    "    patterns.append(p)\n",
    "    for _ in range(0, 3): # Consider also combinations like Rezepter-Tyrosinkinasen\n",
    "        p = [{'IS_ALPHA' : True}, {'lower' : '-'}] + p\n",
    "        patterns.append(p)\n",
    "protein_matcher.add('protein', patterns[-1::-1])\n",
    "\n",
    "def protein_families_fn(doc):\n",
    "    matches = protein_matcher(doc)\n",
    "    if matches:\n",
    "        # Keep longest matches only\n",
    "        spans = [doc[start:end] for _, start, end in matches]\n",
    "        spans = spacy.util.filter_spans(spans)\n",
    "        for s in spans:\n",
    "            yield s.start, s.end, 'Gene or Protein'\n",
    "\n",
    "lf_protein_families = heuristics.FunctionAnnotator(\"Protein Families\", protein_families_fn) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "59a961e6-ea23-416e-9549-c444a59a438e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 1, 'Gene or Protein'), (1, 4, 'Gene or Protein')]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(protein_families_fn(nlp(\"RAS k-RAS krass\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "c4aed24b-34e9-48b9-b4fa-927ceb9172a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "218\n",
      "16782\n"
     ]
    }
   ],
   "source": [
    "# Calculate data stats (number of genes, sentences, etc...) -> Table for Materials\n",
    "# Intersections of the gazetteers are calculated below.\n",
    "intersection_db = list(set(cosmic_census_lower) & set(CIVIC_genes_lower) & set(omim_list_lower))\n",
    "union_db = set(set(cosmic_census_lower).union(set(CIVIC_genes_lower)).union(set(omim_list_lower)))\n",
    "print(len(intersection_db))\n",
    "print(len(union_db))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa06edd0",
   "metadata": {},
   "source": [
    "We define a list of labelling functions which we want to deploy onto our textual input. Every row in our dataframe is subject to examination. Those listed here was the best performing allocation from ca. 30 labelling functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "d9bc3aae-f3c3-4d76-93f1-ea2b44d8e9f9",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 85996/85996 [20:11<00:00, 70.96it/s]\n"
     ]
    }
   ],
   "source": [
    "lfs = [lf_civic, lf_entrez, lf_omim, lf_cosmic, lf_protein_gazetteer, lf_hgnc, lf_protein_families]\n",
    "\n",
    "#For Quick Run with Random Sentences!\n",
    "#random_files = files_df.sample(n = 10000)\n",
    "all_docs = []\n",
    "\n",
    "for sentence_idx, doc in zip(tqdm(list(sentence_df.reset_index().iterrows())), nlp.pipe(sentence_df.text, disable=[\"ner\"])):\n",
    "    i, row = sentence_idx\n",
    "    for lf in lfs:\n",
    "        doc = lf(doc)\n",
    "    all_docs.append(doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6954ced-b4be-49cb-8416-bf95ce225033",
   "metadata": {},
   "source": [
    "Remove files which have been manually annotated from the training dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "34c1e291-67ce-449e-a2dc-4aaa37b16d24",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2000"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('data/molecular/annotated_sentences.txt') as fh:\n",
    "    annotated_sentences = [l.strip() for l in fh.readlines()]\n",
    "len(annotated_sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "9022b1d4-4b52-4a80-8c26-37a9fc6af7cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(83624, 83624)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs, s_index = zip(*[(d, si) for d, si in zip(all_docs, sentence_df.reset_index().index) if not d.text in annotated_sentences])\n",
    "docs = list(docs)\n",
    "filtered_sentence_df = sentence_df.reset_index().loc[list(s_index)]\n",
    "len(docs), len(filtered_sentence_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9567f81-8883-404e-8156-63f625b99612",
   "metadata": {},
   "source": [
    "## Training Set Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "6dbc1515-9a22-4c9a-8437-7c85bda222dd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CIViC</th>\n",
       "      <th>Entrez</th>\n",
       "      <th>OMIM</th>\n",
       "      <th>COSMIC</th>\n",
       "      <th>Proteins</th>\n",
       "      <th>HGNC</th>\n",
       "      <th>Protein Families</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Coverage</th>\n",
       "      <td>0.209893</td>\n",
       "      <td>0.405695</td>\n",
       "      <td>0.344048</td>\n",
       "      <td>0.223103</td>\n",
       "      <td>0.136651</td>\n",
       "      <td>0.365478</td>\n",
       "      <td>0.017907</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Overlaps</th>\n",
       "      <td>0.980420</td>\n",
       "      <td>0.685962</td>\n",
       "      <td>0.499573</td>\n",
       "      <td>0.929605</td>\n",
       "      <td>0.699248</td>\n",
       "      <td>0.381526</td>\n",
       "      <td>0.368852</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             CIViC    Entrez      OMIM    COSMIC  Proteins      HGNC  \\\n",
       "Coverage  0.209893  0.405695  0.344048  0.223103  0.136651  0.365478   \n",
       "Overlaps  0.980420  0.685962  0.499573  0.929605  0.699248  0.381526   \n",
       "\n",
       "          Protein Families  \n",
       "Coverage          0.017907  \n",
       "Overlaps          0.368852  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from skweak.analysis import LFAnalysis\n",
    "\n",
    "lfa = LFAnalysis(docs, ['Gene or Protein'])\n",
    "cov = lfa.lf_coverages().rename(index={'Gene or Protein' : 'Coverage'})\n",
    "overlap = lfa.lf_overlaps().rename(index={'Gene or Protein' : 'Overlaps'})\n",
    "pd.concat([cov, overlap])[[lf.name for lf in lfs]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9eeb61a-975b-43bf-9a30-4d7d3bc40b65",
   "metadata": {},
   "source": [
    "# Label Aggregation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "cf2dd2ea-50b3-4a17-8905-30545b57a78b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting iteration 1\n",
      "Number of processed documents: 1000\n",
      "Number of processed documents: 2000\n",
      "Number of processed documents: 3000\n",
      "Number of processed documents: 4000\n",
      "Finished E-step with 4624 documents\n",
      "Starting iteration 2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "         1      -33073.4892             +nan\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of processed documents: 1000\n",
      "Number of processed documents: 2000\n",
      "Number of processed documents: 3000\n",
      "Number of processed documents: 4000\n",
      "Finished E-step with 4624 documents\n",
      "Starting iteration 3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "         2      -32400.1132        +673.3760\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of processed documents: 1000\n",
      "Number of processed documents: 2000\n",
      "Number of processed documents: 3000\n",
      "Number of processed documents: 4000\n",
      "Finished E-step with 4624 documents\n",
      "Starting iteration 4\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "         3      -32386.9774         +13.1358\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of processed documents: 1000\n",
      "Number of processed documents: 2000\n",
      "Number of processed documents: 3000\n",
      "Number of processed documents: 4000\n",
      "Finished E-step with 4624 documents\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "         4      -32383.9291          +3.0482\n"
     ]
    }
   ],
   "source": [
    "# Training of HMM and Majority Voter\n",
    "#voter = skweak.voting.SequentialMajorityVoter(\"maj_voter\", labels=[\"Gen\"])\n",
    "#voter.fit(docs)\n",
    "hmm = generative.HMM(\"hmm\", [\"Gene or Protein\"])\n",
    "hmm.fit(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "00c04eb3-5190-4b49-80a1-457e98e1fa70",
   "metadata": {},
   "outputs": [],
   "source": [
    "for d in docs:\n",
    "    d = hmm(d)\n",
    "    d.ents = d.spans[\"hmm\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3066aa9-db13-490e-ac24-432bffdd6577",
   "metadata": {},
   "source": [
    "Consider subset of files where at least on LF has matched"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "d07ab611-b36a-4eb4-beb4-0002fda81327",
   "metadata": {},
   "outputs": [],
   "source": [
    "lf_match = []\n",
    "for d in docs:\n",
    "    lf_match.append(any([len(v) > 0 for s, v in d.spans.items() if s != 'hmm']))\n",
    "\n",
    "filtered_sentence_df['lf_match'] = lf_match\n",
    "by_file = filtered_sentence_df.groupby('file')['lf_match'].max()\n",
    "match_files = by_file[by_file.values].index\n",
    "\n",
    "filtered_docs = [d for d, match in zip(docs, filtered_sentence_df.file.isin(match_files)) if match]\n",
    "gene_docs = [d for d, match in zip(docs, lf_match) if match]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "19236a84-2a05-4d61-aefd-789e804bd7da",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(83624, 35501, 4624)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(docs), len(filtered_docs), len(gene_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e00ac4b-8333-44bf-9274-ab6734810bbb",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Labeling Function Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "595183ba-63b3-4ca7-8870-2647745676d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "gold_docs_dev = list(DocBin().from_disk('data/molecular/gold_dev.spacy').get_docs(nlp.vocab))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8764f34a",
   "metadata": {},
   "source": [
    "Our labeling functions must also be deployed onto the gold standard data to evaluate strong supervision against weak supervision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "97cce16c-4d79-4dc8-83dd-c54f60795f67",
   "metadata": {},
   "outputs": [],
   "source": [
    "def apply_hmm(gold_docs):\n",
    "    for g in tqdm(gold_docs):\n",
    "        if 'Gene or Protein' in g.spans:\n",
    "            del g.spans['Gene or Protein']\n",
    "        for lf in lfs:\n",
    "            g = lf(g)\n",
    "        g = hmm(g)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "6c5ac964-a203-4c0e-a291-9ab837c7a18f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:15<00:00, 64.38it/s]\n"
     ]
    }
   ],
   "source": [
    "apply_hmm(gold_docs_dev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "f79a91a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>tok_precision</th>\n",
       "      <th>tok_recall</th>\n",
       "      <th>tok_f1</th>\n",
       "      <th>tok_cee</th>\n",
       "      <th>tok_acc</th>\n",
       "      <th>coverage</th>\n",
       "      <th>ent_precision</th>\n",
       "      <th>ent_recall</th>\n",
       "      <th>ent_f1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>label</th>\n",
       "      <th>proportion</th>\n",
       "      <th>model</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"8\" valign=\"top\">Gene or Protein</th>\n",
       "      <th rowspan=\"8\" valign=\"top\">100.0 %</th>\n",
       "      <th>CIViC</th>\n",
       "      <td>0.979</td>\n",
       "      <td>0.366</td>\n",
       "      <td>0.532</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.944</td>\n",
       "      <td>0.465</td>\n",
       "      <td>0.624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>COSMIC</th>\n",
       "      <td>0.960</td>\n",
       "      <td>0.342</td>\n",
       "      <td>0.504</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.928</td>\n",
       "      <td>0.436</td>\n",
       "      <td>0.594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Entrez</th>\n",
       "      <td>0.937</td>\n",
       "      <td>0.450</td>\n",
       "      <td>0.608</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.902</td>\n",
       "      <td>0.503</td>\n",
       "      <td>0.646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HGNC</th>\n",
       "      <td>0.879</td>\n",
       "      <td>0.244</td>\n",
       "      <td>0.382</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.833</td>\n",
       "      <td>0.305</td>\n",
       "      <td>0.446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OMIM</th>\n",
       "      <td>0.955</td>\n",
       "      <td>0.411</td>\n",
       "      <td>0.574</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.926</td>\n",
       "      <td>0.524</td>\n",
       "      <td>0.670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Protein Families</th>\n",
       "      <td>1.000</td>\n",
       "      <td>0.067</td>\n",
       "      <td>0.126</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.076</td>\n",
       "      <td>0.142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Proteins</th>\n",
       "      <td>1.000</td>\n",
       "      <td>0.117</td>\n",
       "      <td>0.210</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.934</td>\n",
       "      <td>0.120</td>\n",
       "      <td>0.212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hmm</th>\n",
       "      <td>0.899</td>\n",
       "      <td>0.596</td>\n",
       "      <td>0.716</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.841</td>\n",
       "      <td>0.680</td>\n",
       "      <td>0.752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"8\" valign=\"top\">macro</th>\n",
       "      <th rowspan=\"8\" valign=\"top\"></th>\n",
       "      <th>CIViC</th>\n",
       "      <td>0.979</td>\n",
       "      <td>0.366</td>\n",
       "      <td>0.532</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.944</td>\n",
       "      <td>0.465</td>\n",
       "      <td>0.624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>COSMIC</th>\n",
       "      <td>0.960</td>\n",
       "      <td>0.342</td>\n",
       "      <td>0.504</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.928</td>\n",
       "      <td>0.436</td>\n",
       "      <td>0.594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Entrez</th>\n",
       "      <td>0.937</td>\n",
       "      <td>0.450</td>\n",
       "      <td>0.608</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.902</td>\n",
       "      <td>0.503</td>\n",
       "      <td>0.646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HGNC</th>\n",
       "      <td>0.879</td>\n",
       "      <td>0.244</td>\n",
       "      <td>0.382</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.833</td>\n",
       "      <td>0.305</td>\n",
       "      <td>0.446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OMIM</th>\n",
       "      <td>0.955</td>\n",
       "      <td>0.411</td>\n",
       "      <td>0.574</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.926</td>\n",
       "      <td>0.524</td>\n",
       "      <td>0.670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Protein Families</th>\n",
       "      <td>1.000</td>\n",
       "      <td>0.067</td>\n",
       "      <td>0.126</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.076</td>\n",
       "      <td>0.142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Proteins</th>\n",
       "      <td>1.000</td>\n",
       "      <td>0.117</td>\n",
       "      <td>0.210</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.934</td>\n",
       "      <td>0.120</td>\n",
       "      <td>0.212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hmm</th>\n",
       "      <td>0.899</td>\n",
       "      <td>0.596</td>\n",
       "      <td>0.716</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.841</td>\n",
       "      <td>0.680</td>\n",
       "      <td>0.752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"8\" valign=\"top\">micro</th>\n",
       "      <th rowspan=\"8\" valign=\"top\"></th>\n",
       "      <th>CIViC</th>\n",
       "      <td>0.979</td>\n",
       "      <td>0.366</td>\n",
       "      <td>0.532</td>\n",
       "      <td>0.819</td>\n",
       "      <td>0.976</td>\n",
       "      <td>0.374</td>\n",
       "      <td>0.944</td>\n",
       "      <td>0.465</td>\n",
       "      <td>0.624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>COSMIC</th>\n",
       "      <td>0.960</td>\n",
       "      <td>0.342</td>\n",
       "      <td>0.504</td>\n",
       "      <td>0.819</td>\n",
       "      <td>0.976</td>\n",
       "      <td>0.356</td>\n",
       "      <td>0.928</td>\n",
       "      <td>0.436</td>\n",
       "      <td>0.594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Entrez</th>\n",
       "      <td>0.937</td>\n",
       "      <td>0.450</td>\n",
       "      <td>0.608</td>\n",
       "      <td>0.819</td>\n",
       "      <td>0.976</td>\n",
       "      <td>0.481</td>\n",
       "      <td>0.902</td>\n",
       "      <td>0.503</td>\n",
       "      <td>0.646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HGNC</th>\n",
       "      <td>0.879</td>\n",
       "      <td>0.244</td>\n",
       "      <td>0.382</td>\n",
       "      <td>0.819</td>\n",
       "      <td>0.976</td>\n",
       "      <td>0.278</td>\n",
       "      <td>0.833</td>\n",
       "      <td>0.305</td>\n",
       "      <td>0.446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OMIM</th>\n",
       "      <td>0.955</td>\n",
       "      <td>0.411</td>\n",
       "      <td>0.574</td>\n",
       "      <td>0.819</td>\n",
       "      <td>0.976</td>\n",
       "      <td>0.43</td>\n",
       "      <td>0.926</td>\n",
       "      <td>0.524</td>\n",
       "      <td>0.670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Protein Families</th>\n",
       "      <td>1.000</td>\n",
       "      <td>0.067</td>\n",
       "      <td>0.126</td>\n",
       "      <td>0.819</td>\n",
       "      <td>0.976</td>\n",
       "      <td>0.067</td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.076</td>\n",
       "      <td>0.142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Proteins</th>\n",
       "      <td>1.000</td>\n",
       "      <td>0.117</td>\n",
       "      <td>0.210</td>\n",
       "      <td>0.819</td>\n",
       "      <td>0.976</td>\n",
       "      <td>0.117</td>\n",
       "      <td>0.934</td>\n",
       "      <td>0.120</td>\n",
       "      <td>0.212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hmm</th>\n",
       "      <td>0.899</td>\n",
       "      <td>0.596</td>\n",
       "      <td>0.716</td>\n",
       "      <td>0.326</td>\n",
       "      <td>0.989</td>\n",
       "      <td>0.663</td>\n",
       "      <td>0.841</td>\n",
       "      <td>0.680</td>\n",
       "      <td>0.752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"8\" valign=\"top\">weighted</th>\n",
       "      <th rowspan=\"8\" valign=\"top\"></th>\n",
       "      <th>CIViC</th>\n",
       "      <td>0.979</td>\n",
       "      <td>0.366</td>\n",
       "      <td>0.532</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.944</td>\n",
       "      <td>0.465</td>\n",
       "      <td>0.624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>COSMIC</th>\n",
       "      <td>0.960</td>\n",
       "      <td>0.342</td>\n",
       "      <td>0.504</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.928</td>\n",
       "      <td>0.436</td>\n",
       "      <td>0.594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Entrez</th>\n",
       "      <td>0.937</td>\n",
       "      <td>0.450</td>\n",
       "      <td>0.608</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.902</td>\n",
       "      <td>0.503</td>\n",
       "      <td>0.646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HGNC</th>\n",
       "      <td>0.879</td>\n",
       "      <td>0.244</td>\n",
       "      <td>0.382</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.833</td>\n",
       "      <td>0.305</td>\n",
       "      <td>0.446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OMIM</th>\n",
       "      <td>0.955</td>\n",
       "      <td>0.411</td>\n",
       "      <td>0.574</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.926</td>\n",
       "      <td>0.524</td>\n",
       "      <td>0.670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Protein Families</th>\n",
       "      <td>1.000</td>\n",
       "      <td>0.067</td>\n",
       "      <td>0.126</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.076</td>\n",
       "      <td>0.142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Proteins</th>\n",
       "      <td>1.000</td>\n",
       "      <td>0.117</td>\n",
       "      <td>0.210</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.934</td>\n",
       "      <td>0.120</td>\n",
       "      <td>0.212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hmm</th>\n",
       "      <td>0.899</td>\n",
       "      <td>0.596</td>\n",
       "      <td>0.716</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.841</td>\n",
       "      <td>0.680</td>\n",
       "      <td>0.752</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             tok_precision  tok_recall  \\\n",
       "label           proportion model                                         \n",
       "Gene or Protein 100.0 %    CIViC                     0.979       0.366   \n",
       "                           COSMIC                    0.960       0.342   \n",
       "                           Entrez                    0.937       0.450   \n",
       "                           HGNC                      0.879       0.244   \n",
       "                           OMIM                      0.955       0.411   \n",
       "                           Protein Families          1.000       0.067   \n",
       "                           Proteins                  1.000       0.117   \n",
       "                           hmm                       0.899       0.596   \n",
       "macro                      CIViC                     0.979       0.366   \n",
       "                           COSMIC                    0.960       0.342   \n",
       "                           Entrez                    0.937       0.450   \n",
       "                           HGNC                      0.879       0.244   \n",
       "                           OMIM                      0.955       0.411   \n",
       "                           Protein Families          1.000       0.067   \n",
       "                           Proteins                  1.000       0.117   \n",
       "                           hmm                       0.899       0.596   \n",
       "micro                      CIViC                     0.979       0.366   \n",
       "                           COSMIC                    0.960       0.342   \n",
       "                           Entrez                    0.937       0.450   \n",
       "                           HGNC                      0.879       0.244   \n",
       "                           OMIM                      0.955       0.411   \n",
       "                           Protein Families          1.000       0.067   \n",
       "                           Proteins                  1.000       0.117   \n",
       "                           hmm                       0.899       0.596   \n",
       "weighted                   CIViC                     0.979       0.366   \n",
       "                           COSMIC                    0.960       0.342   \n",
       "                           Entrez                    0.937       0.450   \n",
       "                           HGNC                      0.879       0.244   \n",
       "                           OMIM                      0.955       0.411   \n",
       "                           Protein Families          1.000       0.067   \n",
       "                           Proteins                  1.000       0.117   \n",
       "                           hmm                       0.899       0.596   \n",
       "\n",
       "                                             tok_f1 tok_cee tok_acc coverage  \\\n",
       "label           proportion model                                               \n",
       "Gene or Protein 100.0 %    CIViC              0.532                            \n",
       "                           COSMIC             0.504                            \n",
       "                           Entrez             0.608                            \n",
       "                           HGNC               0.382                            \n",
       "                           OMIM               0.574                            \n",
       "                           Protein Families   0.126                            \n",
       "                           Proteins           0.210                            \n",
       "                           hmm                0.716                            \n",
       "macro                      CIViC              0.532                            \n",
       "                           COSMIC             0.504                            \n",
       "                           Entrez             0.608                            \n",
       "                           HGNC               0.382                            \n",
       "                           OMIM               0.574                            \n",
       "                           Protein Families   0.126                            \n",
       "                           Proteins           0.210                            \n",
       "                           hmm                0.716                            \n",
       "micro                      CIViC              0.532   0.819   0.976    0.374   \n",
       "                           COSMIC             0.504   0.819   0.976    0.356   \n",
       "                           Entrez             0.608   0.819   0.976    0.481   \n",
       "                           HGNC               0.382   0.819   0.976    0.278   \n",
       "                           OMIM               0.574   0.819   0.976     0.43   \n",
       "                           Protein Families   0.126   0.819   0.976    0.067   \n",
       "                           Proteins           0.210   0.819   0.976    0.117   \n",
       "                           hmm                0.716   0.326   0.989    0.663   \n",
       "weighted                   CIViC              0.532                            \n",
       "                           COSMIC             0.504                            \n",
       "                           Entrez             0.608                            \n",
       "                           HGNC               0.382                            \n",
       "                           OMIM               0.574                            \n",
       "                           Protein Families   0.126                            \n",
       "                           Proteins           0.210                            \n",
       "                           hmm                0.716                            \n",
       "\n",
       "                                             ent_precision  ent_recall  ent_f1  \n",
       "label           proportion model                                                \n",
       "Gene or Protein 100.0 %    CIViC                     0.944       0.465   0.624  \n",
       "                           COSMIC                    0.928       0.436   0.594  \n",
       "                           Entrez                    0.902       0.503   0.646  \n",
       "                           HGNC                      0.833       0.305   0.446  \n",
       "                           OMIM                      0.926       0.524   0.670  \n",
       "                           Protein Families          1.000       0.076   0.142  \n",
       "                           Proteins                  0.934       0.120   0.212  \n",
       "                           hmm                       0.841       0.680   0.752  \n",
       "macro                      CIViC                     0.944       0.465   0.624  \n",
       "                           COSMIC                    0.928       0.436   0.594  \n",
       "                           Entrez                    0.902       0.503   0.646  \n",
       "                           HGNC                      0.833       0.305   0.446  \n",
       "                           OMIM                      0.926       0.524   0.670  \n",
       "                           Protein Families          1.000       0.076   0.142  \n",
       "                           Proteins                  0.934       0.120   0.212  \n",
       "                           hmm                       0.841       0.680   0.752  \n",
       "micro                      CIViC                     0.944       0.465   0.624  \n",
       "                           COSMIC                    0.928       0.436   0.594  \n",
       "                           Entrez                    0.902       0.503   0.646  \n",
       "                           HGNC                      0.833       0.305   0.446  \n",
       "                           OMIM                      0.926       0.524   0.670  \n",
       "                           Protein Families          1.000       0.076   0.142  \n",
       "                           Proteins                  0.934       0.120   0.212  \n",
       "                           hmm                       0.841       0.680   0.752  \n",
       "weighted                   CIViC                     0.944       0.465   0.624  \n",
       "                           COSMIC                    0.928       0.436   0.594  \n",
       "                           Entrez                    0.902       0.503   0.646  \n",
       "                           HGNC                      0.833       0.305   0.446  \n",
       "                           OMIM                      0.926       0.524   0.670  \n",
       "                           Protein Families          1.000       0.076   0.142  \n",
       "                           Proteins                  0.934       0.120   0.212  \n",
       "                           hmm                       0.841       0.680   0.752  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# HMM / LFs vs. Gold-Standard\n",
    "#evaluate(gold_docs, ['Gen'], ['lf15', 'hmm'])\n",
    "evaluate(gold_docs_dev, ['Gene or Protein'], [l.name for l in lfs] + ['hmm'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c793b3ee-500a-4d28-8086-27a5e456fb27",
   "metadata": {},
   "source": [
    "# Training of Transformer-based NER Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "60f62095-f8ed-4fd6-b7a5-672e130c5282",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to output/weak_training_lg.spacy...done\n"
     ]
    }
   ],
   "source": [
    "utils.docbin_writer(docs, f\"output/weak_training_lg.spacy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fc7b292-c94a-45f9-8e06-a862a3932019",
   "metadata": {},
   "source": [
    "Train test splits for strong supervision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "5f713abd-6637-481b-8fea-39972eb129f4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Write to output/strong_train.spacy...done\n",
      "Write to output/strong_dev.spacy...done\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "gold_docs_strong = list(DocBin().from_disk('data/molecular/gold_dev.spacy').get_docs(nlp.vocab))\n",
    "\n",
    "docs_strong_train, docs_strong_dev = train_test_split(gold_docs_strong, test_size=0.2, random_state=random_seed)\n",
    "utils.docbin_writer(docs_strong_train, 'output/strong_train.spacy')\n",
    "utils.docbin_writer(docs_strong_dev, 'output/strong_dev.spacy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe40f18b-2112-479f-a513-a7728b095a33",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train NER model on weak labels with spaCy\n",
    "!spacy train config.cfg --paths.train output/weak_training_lg.spacy  --paths.dev data/molecular/gold_dev.spacy --output output/weak_ner_lg --gpu-id 0 --code training.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4907d92c-3ead-411c-aa30-0d07215fb8c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Baseline: train NER model on strong labels with spaCy\n",
    "!spacy train config.cfg --paths.train output/strong_train.spacy --paths.dev output/strong_dev.spacy --output output/strong_ner --gpu-id 0 --code training.py\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42435ea6-98d4-4dbe-9c45-188ab8d4d690",
   "metadata": {},
   "source": [
    "# Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "8842e777-aa91-485e-8e61-a4fc70790679",
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_model_weak = spacy.load('output/weak_ner_lg/model-best/')\n",
    "ner_model_strong = spacy.load('output/strong_ner/model-best/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "ef897e37-9bae-49ee-b8c1-267f19ceffe5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display, Markdown\n",
    "\n",
    "def print_metrics(gold_docs, is_test : bool):\n",
    "    display(Markdown('__Labeling Function / HMM performance__'))\n",
    "    gold_ents = [d.ents for d in gold_docs]\n",
    "    \n",
    "    apply_hmm(gold_docs)\n",
    "    \n",
    "    display(evaluate(gold_docs, ['Gene or Protein'], [l.name for l in lfs] + ['hmm']).loc['Gene or Protein'])\n",
    "        \n",
    "    display(Markdown('__Weak NER Performance__'))\n",
    "    for d, gold_ent in zip(gold_docs, tqdm(gold_ents)):\n",
    "        d.set_ents([])\n",
    "        d = ner_model_weak(d)\n",
    "        d.spans['ner_model'] = d.ents\n",
    "        d.set_ents(gold_ent)\n",
    "    display(evaluate(gold_docs, ['Gene or Protein'], ['ner_model']).loc['Gene or Protein'])\n",
    "    \n",
    "    if is_test:\n",
    "        display(Markdown('__Strong NER Performance__'))\n",
    "        for d, gold_ent in zip(gold_docs, tqdm(gold_ents)):\n",
    "            d.set_ents([])\n",
    "            d = ner_model_strong(d)\n",
    "            d.spans['ner_model'] = d.ents\n",
    "            d.set_ents(gold_ent)\n",
    "        display(evaluate(gold_docs, ['Gene or Protein'], ['ner_model']).loc['Gene or Protein'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84741e61-358c-437b-99c8-37066162d9af",
   "metadata": {},
   "source": [
    "## Dev Set Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "5be94760-3fa5-40e9-bb8f-d798cd452f6b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "__Labeling Function / HMM performance__"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:15<00:00, 64.79it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>tok_precision</th>\n",
       "      <th>tok_recall</th>\n",
       "      <th>tok_f1</th>\n",
       "      <th>tok_cee</th>\n",
       "      <th>tok_acc</th>\n",
       "      <th>coverage</th>\n",
       "      <th>ent_precision</th>\n",
       "      <th>ent_recall</th>\n",
       "      <th>ent_f1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proportion</th>\n",
       "      <th>model</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"8\" valign=\"top\">100.0 %</th>\n",
       "      <th>CIViC</th>\n",
       "      <td>0.979</td>\n",
       "      <td>0.366</td>\n",
       "      <td>0.532</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.944</td>\n",
       "      <td>0.465</td>\n",
       "      <td>0.624</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>COSMIC</th>\n",
       "      <td>0.960</td>\n",
       "      <td>0.342</td>\n",
       "      <td>0.504</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.928</td>\n",
       "      <td>0.436</td>\n",
       "      <td>0.594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Entrez</th>\n",
       "      <td>0.937</td>\n",
       "      <td>0.450</td>\n",
       "      <td>0.608</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.902</td>\n",
       "      <td>0.503</td>\n",
       "      <td>0.646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HGNC</th>\n",
       "      <td>0.879</td>\n",
       "      <td>0.244</td>\n",
       "      <td>0.382</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.833</td>\n",
       "      <td>0.305</td>\n",
       "      <td>0.446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OMIM</th>\n",
       "      <td>0.955</td>\n",
       "      <td>0.411</td>\n",
       "      <td>0.574</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.926</td>\n",
       "      <td>0.524</td>\n",
       "      <td>0.670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Protein Families</th>\n",
       "      <td>1.000</td>\n",
       "      <td>0.067</td>\n",
       "      <td>0.126</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.076</td>\n",
       "      <td>0.142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Proteins</th>\n",
       "      <td>1.000</td>\n",
       "      <td>0.117</td>\n",
       "      <td>0.210</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.934</td>\n",
       "      <td>0.120</td>\n",
       "      <td>0.212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hmm</th>\n",
       "      <td>0.899</td>\n",
       "      <td>0.596</td>\n",
       "      <td>0.716</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.841</td>\n",
       "      <td>0.680</td>\n",
       "      <td>0.752</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             tok_precision  tok_recall  tok_f1 tok_cee  \\\n",
       "proportion model                                                         \n",
       "100.0 %    CIViC                     0.979       0.366   0.532           \n",
       "           COSMIC                    0.960       0.342   0.504           \n",
       "           Entrez                    0.937       0.450   0.608           \n",
       "           HGNC                      0.879       0.244   0.382           \n",
       "           OMIM                      0.955       0.411   0.574           \n",
       "           Protein Families          1.000       0.067   0.126           \n",
       "           Proteins                  1.000       0.117   0.210           \n",
       "           hmm                       0.899       0.596   0.716           \n",
       "\n",
       "                            tok_acc coverage  ent_precision  ent_recall  \\\n",
       "proportion model                                                          \n",
       "100.0 %    CIViC                                      0.944       0.465   \n",
       "           COSMIC                                     0.928       0.436   \n",
       "           Entrez                                     0.902       0.503   \n",
       "           HGNC                                       0.833       0.305   \n",
       "           OMIM                                       0.926       0.524   \n",
       "           Protein Families                           1.000       0.076   \n",
       "           Proteins                                   0.934       0.120   \n",
       "           hmm                                        0.841       0.680   \n",
       "\n",
       "                             ent_f1  \n",
       "proportion model                     \n",
       "100.0 %    CIViC              0.624  \n",
       "           COSMIC             0.594  \n",
       "           Entrez             0.646  \n",
       "           HGNC               0.446  \n",
       "           OMIM               0.670  \n",
       "           Protein Families   0.142  \n",
       "           Proteins           0.212  \n",
       "           hmm                0.752  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "__Weak NER Performance__"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊| 999/1000 [00:39<00:00, 24.99it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>tok_precision</th>\n",
       "      <th>tok_recall</th>\n",
       "      <th>tok_f1</th>\n",
       "      <th>tok_cee</th>\n",
       "      <th>tok_acc</th>\n",
       "      <th>coverage</th>\n",
       "      <th>ent_precision</th>\n",
       "      <th>ent_recall</th>\n",
       "      <th>ent_f1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proportion</th>\n",
       "      <th>model</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>100.0 %</th>\n",
       "      <th>ner_model</th>\n",
       "      <td>0.902</td>\n",
       "      <td>0.617</td>\n",
       "      <td>0.732</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.855</td>\n",
       "      <td>0.72</td>\n",
       "      <td>0.782</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      tok_precision  tok_recall  tok_f1 tok_cee tok_acc  \\\n",
       "proportion model                                                          \n",
       "100.0 %    ner_model          0.902       0.617   0.732                   \n",
       "\n",
       "                     coverage  ent_precision  ent_recall  ent_f1  \n",
       "proportion model                                                  \n",
       "100.0 %    ner_model                   0.855        0.72   0.782  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "gold_docs_dev_eval = list(DocBin().from_disk('data/molecular/gold_dev.spacy').get_docs(nlp.vocab))\n",
    "\n",
    "print_metrics(gold_docs_dev_eval, is_test=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c25322e-327f-4050-8d7c-ef4042170b6e",
   "metadata": {},
   "source": [
    "## Test Set Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "631cbb84-3c39-4281-9160-5029493a23a7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "__Labeling Function / HMM performance__"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:13<00:00, 73.50it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>tok_precision</th>\n",
       "      <th>tok_recall</th>\n",
       "      <th>tok_f1</th>\n",
       "      <th>tok_cee</th>\n",
       "      <th>tok_acc</th>\n",
       "      <th>coverage</th>\n",
       "      <th>ent_precision</th>\n",
       "      <th>ent_recall</th>\n",
       "      <th>ent_f1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proportion</th>\n",
       "      <th>model</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"8\" valign=\"top\">100.0 %</th>\n",
       "      <th>CIViC</th>\n",
       "      <td>0.933</td>\n",
       "      <td>0.350</td>\n",
       "      <td>0.510</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.841</td>\n",
       "      <td>0.473</td>\n",
       "      <td>0.606</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>COSMIC</th>\n",
       "      <td>0.927</td>\n",
       "      <td>0.342</td>\n",
       "      <td>0.500</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.854</td>\n",
       "      <td>0.473</td>\n",
       "      <td>0.608</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Entrez</th>\n",
       "      <td>0.951</td>\n",
       "      <td>0.525</td>\n",
       "      <td>0.676</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.890</td>\n",
       "      <td>0.608</td>\n",
       "      <td>0.722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>HGNC</th>\n",
       "      <td>0.853</td>\n",
       "      <td>0.190</td>\n",
       "      <td>0.310</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.836</td>\n",
       "      <td>0.280</td>\n",
       "      <td>0.420</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OMIM</th>\n",
       "      <td>0.904</td>\n",
       "      <td>0.363</td>\n",
       "      <td>0.518</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.818</td>\n",
       "      <td>0.493</td>\n",
       "      <td>0.616</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Protein Families</th>\n",
       "      <td>0.538</td>\n",
       "      <td>0.027</td>\n",
       "      <td>0.052</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.250</td>\n",
       "      <td>0.012</td>\n",
       "      <td>0.022</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Proteins</th>\n",
       "      <td>1.000</td>\n",
       "      <td>0.131</td>\n",
       "      <td>0.232</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.975</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hmm</th>\n",
       "      <td>0.864</td>\n",
       "      <td>0.596</td>\n",
       "      <td>0.706</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.789</td>\n",
       "      <td>0.689</td>\n",
       "      <td>0.736</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             tok_precision  tok_recall  tok_f1 tok_cee  \\\n",
       "proportion model                                                         \n",
       "100.0 %    CIViC                     0.933       0.350   0.510           \n",
       "           COSMIC                    0.927       0.342   0.500           \n",
       "           Entrez                    0.951       0.525   0.676           \n",
       "           HGNC                      0.853       0.190   0.310           \n",
       "           OMIM                      0.904       0.363   0.518           \n",
       "           Protein Families          0.538       0.027   0.052           \n",
       "           Proteins                  1.000       0.131   0.232           \n",
       "           hmm                       0.864       0.596   0.706           \n",
       "\n",
       "                            tok_acc coverage  ent_precision  ent_recall  \\\n",
       "proportion model                                                          \n",
       "100.0 %    CIViC                                      0.841       0.473   \n",
       "           COSMIC                                     0.854       0.473   \n",
       "           Entrez                                     0.890       0.608   \n",
       "           HGNC                                       0.836       0.280   \n",
       "           OMIM                                       0.818       0.493   \n",
       "           Protein Families                           0.250       0.012   \n",
       "           Proteins                                   0.975       0.112   \n",
       "           hmm                                        0.789       0.689   \n",
       "\n",
       "                             ent_f1  \n",
       "proportion model                     \n",
       "100.0 %    CIViC              0.606  \n",
       "           COSMIC             0.608  \n",
       "           Entrez             0.722  \n",
       "           HGNC               0.420  \n",
       "           OMIM               0.616  \n",
       "           Protein Families   0.022  \n",
       "           Proteins           0.200  \n",
       "           hmm                0.736  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "__Weak NER Performance__"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊| 999/1000 [00:40<00:00, 24.53it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>tok_precision</th>\n",
       "      <th>tok_recall</th>\n",
       "      <th>tok_f1</th>\n",
       "      <th>tok_cee</th>\n",
       "      <th>tok_acc</th>\n",
       "      <th>coverage</th>\n",
       "      <th>ent_precision</th>\n",
       "      <th>ent_recall</th>\n",
       "      <th>ent_f1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proportion</th>\n",
       "      <th>model</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>100.0 %</th>\n",
       "      <th>ner_model</th>\n",
       "      <td>0.901</td>\n",
       "      <td>0.613</td>\n",
       "      <td>0.73</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.819</td>\n",
       "      <td>0.718</td>\n",
       "      <td>0.766</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      tok_precision  tok_recall  tok_f1 tok_cee tok_acc  \\\n",
       "proportion model                                                          \n",
       "100.0 %    ner_model          0.901       0.613    0.73                   \n",
       "\n",
       "                     coverage  ent_precision  ent_recall  ent_f1  \n",
       "proportion model                                                  \n",
       "100.0 %    ner_model                   0.819       0.718   0.766  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "__Strong NER Performance__"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊| 999/1000 [00:38<00:00, 25.86it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>tok_precision</th>\n",
       "      <th>tok_recall</th>\n",
       "      <th>tok_f1</th>\n",
       "      <th>tok_cee</th>\n",
       "      <th>tok_acc</th>\n",
       "      <th>coverage</th>\n",
       "      <th>ent_precision</th>\n",
       "      <th>ent_recall</th>\n",
       "      <th>ent_f1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>proportion</th>\n",
       "      <th>model</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>100.0 %</th>\n",
       "      <th>ner_model</th>\n",
       "      <td>0.544</td>\n",
       "      <td>0.694</td>\n",
       "      <td>0.61</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>0.558</td>\n",
       "      <td>0.758</td>\n",
       "      <td>0.642</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      tok_precision  tok_recall  tok_f1 tok_cee tok_acc  \\\n",
       "proportion model                                                          \n",
       "100.0 %    ner_model          0.544       0.694    0.61                   \n",
       "\n",
       "                     coverage  ent_precision  ent_recall  ent_f1  \n",
       "proportion model                                                  \n",
       "100.0 %    ner_model                   0.558       0.758   0.642  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "gold_docs_test_eval = list(DocBin().from_disk('data/molecular/gold_test.spacy').get_docs(nlp.vocab))\n",
    "\n",
    "print_metrics(gold_docs_test_eval, is_test=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "589141b8",
   "metadata": {},
   "source": [
    "## Descriptive Statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "a313589f-d854-41c8-8df6-bccc2b0bf2d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_genes(doc_list):\n",
    "    return [e.text for d in doc_list for e in d.ents if e.label_ == 'Gene or Protein']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "a655dd6c-92c2-4582-84ea-fb8af767856b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(83624, 5617)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(docs), len(get_genes(docs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "1dff773c-af59-4896-87d2-0a65bff9a6cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000, 475)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(gold_docs_dev_eval), len(get_genes(gold_docs_dev_eval))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "7bad482c-d324-4052-b965-26d27b9e686e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000, 347)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(gold_docs_test_eval), len(get_genes(gold_docs_test_eval))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63e43db1-f326-4372-91bd-f2e413e089ae",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:ggponc]",
   "language": "python",
   "name": "conda-env-ggponc-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  },
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}