notebooks/Flair Training Sequence Labeling Models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Flair Training Sequence Labeling Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(C) 2019 by [Damir Cavar](http://damir.cavar.me/)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Version:** 0.2, September 2019"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Download:** This and various other Jupyter notebooks are available from my [GitHub repo](https://github.com/dcavar/python-tutorial-for-ipython)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the Flair tutorial 7 license and copyright restrictions, see the website below. For all the parts that I added, consider the license to be [Creative Commons Attribution-ShareAlike 4.0 International License](https://creativecommons.org/licenses/by-sa/4.0/) ([CA BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the [Flair Tutorial 7 Training a Model](https://github.com/zalandoresearch/flair/blob/master/resources/docs/TUTORIAL_7_TRAINING_A_MODEL.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial is using the CoNLL-03 Named Entity Recognition data set. See [this website](https://www.clips.uantwerpen.be/conll2003/ner/) for more details and to download an independent copy of the data set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sequence Labeling Model using CoNLL-03 Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will need the following modules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.data import TaggedCorpus\n",
    "from flair.data_fetcher import NLPTaskDataFetcher, NLPTask\n",
    "from flair.embeddings import TokenEmbeddings, WordEmbeddings, StackedEmbeddings\n",
    "from typing import List"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to use the CoNLL-03 corpus, you need to download it and unpack it in your Flair data and model folder. This folder should be in your home-directory and it is named *.flair*. Once you have downloaded the corpus, unpack it into a folder *.flair/datasets/conll_03*. If you do not want to use the CoNLL-03 corpus, but rather [the free W-NUT 17 corpus](https://noisy-text.github.io/2017/emerging-rare-entities.html), you can use the Flair command: *NLPTaskDataFetcher.load_corpus(NLPTask.WNUT_17)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you decide to download the CoNLL-03 corpus, use the following code. We load the corpus and down-sample it to 10% of its size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use the CoNLL-13 corpus\n",
    "# corpus: TaggedCorpus = NLPTaskDataFetcher.load_corpus(NLPTask.CONLL_03).downsample(0.1)\n",
    "corpus: TaggedCorpus = NLPTaskDataFetcher.load_corpus(NLPTask.WNUT_17)\n",
    "print(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Declare the tag type to be predicted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tag_type = 'ner'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the tag-dictionary for the tag-type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)\n",
    "print(tag_dictionary.idx2item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the embeddings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_types: List[TokenEmbeddings] = [\n",
    "\n",
    "    WordEmbeddings('glove'),\n",
    "\n",
    "    # comment in this line to use character embeddings\n",
    "    # CharacterEmbeddings(),\n",
    "\n",
    "    # comment in these lines to use flair embeddings\n",
    "    # FlairEmbeddings('news-forward'),\n",
    "    # FlairEmbeddings('news-backward'),\n",
    "]\n",
    "embeddings: StackedEmbeddings = StackedEmbeddings(embeddings=embedding_types)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load and initialize the sequence tagger:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.models import SequenceTagger\n",
    "\n",
    "tagger: SequenceTagger = SequenceTagger(hidden_size=256,\n",
    "                                        embeddings=embeddings,\n",
    "                                        tag_dictionary=tag_dictionary,\n",
    "                                        tag_type=tag_type,\n",
    "                                        use_crf=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load and initialize the trainer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.trainers import ModelTrainer\n",
    "\n",
    "trainer: ModelTrainer = ModelTrainer(tagger, corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have a GPU (otherwise maybe tweak the batch size, etc.), run the training with 150 epochs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer.train('resources/taggers/example-ner',\n",
    "              learning_rate=0.1,\n",
    "              mini_batch_size=32,\n",
    "              max_epochs=150)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the training curves and results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.visual.training_curves import Plotter\n",
    "plotter = Plotter()\n",
    "plotter.plot_training_curves('resources/taggers/example-ner/loss.tsv')\n",
    "plotter.plot_weights('resources/taggers/example-ner/weights.txt')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": false,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}