transformers_question_answering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TBjVwYpHJ7ra"
   },
   "source": [
    "# Transformer network: Question answering\n",
    "\n",
    "We explore an application of the transformer architecture, question answering. \n",
    "\n",
    "Question answering (QA) is a task of natural language processing that aims to automatically answer questions. The goal of *extractive* QA is to identify the portion of the text that contains the answer to a question. For example, when tasked with answering the question 'When will Jane go to Africa?' given the text data 'Jane visits Africa in September', the question answering model will highlight 'September'.\n",
    "\n",
    "* We will fine-tune a pre-trained transformer model to a custom dataset to answer questions about stories.\n",
    "* We will implement an extractive QA model in TensorFlow and in PyTorch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TBjVwYpHJ7ra"
   },
   "source": [
    "## 1 - Data preprocessing\n",
    "\n",
    "Run the following cell to load the [QA bAbI dataset](https://research.fb.com/downloads/babi/), which is one of the bAbI datasets generated by Facebook AI Research to advance natural language processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "XxU0G_PYLSXJ",
    "outputId": "44e7877f-5c33-45fc-ed83-3aa4920dcc40"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'story': {'answer': ['', '', 'office'], 'id': ['1', '2', '3'], 'supporting_ids': [[], [], ['1']], 'text': ['The office is north of the kitchen.', 'The garden is south of the kitchen.', 'What is north of the kitchen?'], 'type': [0, 0, 1]}}\n"
     ]
    }
   ],
   "source": [
    "from datasets import load_from_disk\n",
    "\n",
    "# Load a dataset and print the first example in the training set\n",
    "babi_dataset = load_from_disk('data/')\n",
    "print(babi_dataset['train'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XJwacC3bMhZM"
   },
   "source": [
    "Take a look at the format of the data. For a given story, there are two sentences which serve as the context, and one question. Each of these phrases has an ID. There is also a supporting fact ID which refers to a sentence in the story that helps answer the question. \n",
    "\n",
    "For example, for the question 'What is east of the hallway?', the supporting fact 'The bedroom is east of the hallway' has the ID '2'. There is also the answer, 'bedroom' for the question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "aizPXfGlLZ1D",
    "outputId": "0e1d47bc-9c1a-458a-983e-22f47f8184bd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'story': {'answer': ['', '', 'bedroom'],\n",
       "  'id': ['1', '2', '3'],\n",
       "  'supporting_ids': [[], [], ['2']],\n",
       "  'text': ['The bedroom is west of the office.',\n",
       "   'The bedroom is east of the hallway.',\n",
       "   'What is east of the hallway?'],\n",
       "  'type': [0, 0, 1]}}"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "babi_dataset['train'][102]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ewtXZUPjMm2l"
   },
   "source": [
    "Check and see if the entire dataset of stories has this format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "55BSWxwuM1hN"
   },
   "outputs": [],
   "source": [
    "type_set = set()\n",
    "for story in babi_dataset['train']:\n",
    "    if str(story['story']['type'] )not in type_set:\n",
    "        type_set.add(str(story['story']['type'] ))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bdJ8VMF1UT7S",
    "outputId": "2b959467-75e8-4e25-e7bb-481b657a2fce"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'[0, 0, 1]'}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type_set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JsHx1tcyMq_k"
   },
   "source": [
    "To make the data easier to work with, you will flatten the dataset to transform it from a dictionary structure to a table structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "YxixFI-pVOK9"
   },
   "outputs": [],
   "source": [
    "flattened_babi = babi_dataset.flatten()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "kXU43CqCdX98",
    "outputId": "e968ff5e-0db0-4e9d-e1e9-e93f965b2582"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['story.answer', 'story.id', 'story.supporting_ids', 'story.text', 'story.type'],\n",
       "        num_rows: 1000\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['story.answer', 'story.id', 'story.supporting_ids', 'story.text', 'story.type'],\n",
       "        num_rows: 1000\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flattened_babi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "OQw59MgT6Luh",
    "outputId": "ea5eac53-027e-42d3-d19f-98ed7863de2b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'story.answer': ['', '', 'office'],\n",
       " 'story.id': ['1', '2', '3'],\n",
       " 'story.supporting_ids': [[], [], ['1']],\n",
       " 'story.text': ['The office is north of the kitchen.',\n",
       "  'The garden is south of the kitchen.',\n",
       "  'What is north of the kitchen?'],\n",
       " 'story.type': [0, 0, 1]}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(iter(flattened_babi['train']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4vXfmhOPMvt1"
   },
   "source": [
    "Now it is much easier to access the information you need. You can now easily extract the answer, question, and facts from the story, and also join the facts into a single entry under 'sentences'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "O5NcABwkdbrf"
   },
   "outputs": [],
   "source": [
    "def get_question_and_facts(story):\n",
    "    dic = {}\n",
    "    dic['question'] = story['story.text'][2]\n",
    "    dic['sentences'] = ' '.join([story['story.text'][0], story['story.text'][1]])\n",
    "    dic['answer'] = story['story.answer'][2]\n",
    "    return dic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 115,
     "referenced_widgets": [
      "44b7bea3e09d4e5684921c66dd4c7514",
      "6af3ec5091d74bd1a95bf02a87dd240b",
      "7e1325e57bf9417e93d7ef180794ab3c",
      "3dab28395f3f475d8242e4d4d45ed059",
      "ca722dcd857c433c9058585e31a1673d",
      "7fb1118c0b4443b6b6dbb5803e9ec2e8",
      "58718e12f1b7459989ab5296846c4be6",
      "63b4ebafcead4c0784b5511219a6a198",
      "c42644a4e6184a1cbdb2b453b5dbb7d6",
      "364ba960eb474c9084cc71851594d345",
      "e8f1abd85f3e49f991d4c1312ffd416b",
      "929946fdfaa04cf59d3b31cf92fc08d1",
      "aa5c0d374889482697fc0f7ce9c81afe",
      "ff444b253e9a40e5bec755926d83740f",
      "89fdda6e6688476495ca297bfe010bf8",
      "cda72c45821a4eb89f1a3ab5510b26d3"
     ]
    },
    "id": "LHKNQ75afMoZ",
    "outputId": "6ceeae5c-392c-4553-c487-14a648eb9209"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a76cb43e3f8e4792978035d85accecfa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "841af5f668844317885833d986af4df7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "processed = flattened_babi.map(get_question_and_facts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KaTacKMufPba",
    "outputId": "2433d446-e985-45cd-a200-f9805b4056bd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'answer': 'bedroom',\n",
       " 'question': 'What is north of the garden?',\n",
       " 'sentences': 'The garden is north of the office. The bedroom is north of the garden.',\n",
       " 'story.answer': ['', '', 'bedroom'],\n",
       " 'story.id': ['1', '2', '3'],\n",
       " 'story.supporting_ids': [[], [], ['2']],\n",
       " 'story.text': ['The garden is north of the office.',\n",
       "  'The bedroom is north of the garden.',\n",
       "  'What is north of the garden?'],\n",
       " 'story.type': [0, 0, 1]}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "processed['train'][2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "IOrYr5LI0pbP",
    "outputId": "8142f23c-7dab-49b9-8027-fbe7364ae4e9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'answer': 'bathroom',\n",
       " 'question': 'What is north of the garden?',\n",
       " 'sentences': 'The bathroom is north of the garden. The hallway is north of the bathroom.',\n",
       " 'story.answer': ['', '', 'bathroom'],\n",
       " 'story.id': ['1', '2', '3'],\n",
       " 'story.supporting_ids': [[], [], ['1']],\n",
       " 'story.text': ['The bathroom is north of the garden.',\n",
       "  'The hallway is north of the bathroom.',\n",
       "  'What is north of the garden?'],\n",
       " 'story.type': [0, 0, 1]}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "processed['test'][2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oN7D3fszM2hy"
   },
   "source": [
    "The goal of extractive QA is to find the part of the text that contains the answer to the question. You will identify the position of the answer using the indexes of the string. For example, if the answer to some question was 'September', you would need to find the start and end string indices of the word 'September' in the context sentence 'Jane visits Africa in September.'\n",
    "\n",
    "\n",
    "Use this next function to get the start and end indices of the answer in each of the stories in your dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "J1JJx3PafSyR"
   },
   "outputs": [],
   "source": [
    "def get_start_end_idx(story):\n",
    "    str_idx = story['sentences'].find(story['answer'])\n",
    "    end_idx = str_idx + len(story['answer'])\n",
    "    return {'str_idx':str_idx,\n",
    "          'end_idx': end_idx}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 115,
     "referenced_widgets": [
      "8968319cdaca476fb15c11a388dce39a",
      "863c5ce96db84e3da162072c9a13c913",
      "a725734893004a45b61194f649f5f602",
      "c4a24656d67844e995d3b8e175c6c497",
      "4f5b06c3a5e44c6cade5bf83634d9f69",
      "afc33fa78b5d440192c435bfca6f7914",
      "f37bd346f8614fec92d6c5b5e9b66d2f",
      "b4c6a18610734036a16a14a43174c52e",
      "07aaa9b79a744856b19d723370d6e588",
      "afedd2328cf141f78775e4cfa7758267",
      "b39b85d8cb05418aa92e8476ad02f755",
      "0a8534ac52af4d48ad82b66463ad08c3",
      "3abb36da57c841838867c56e2a3a325b",
      "8b961844b5004905922531bd805a9d57",
      "31fc08a1e7e04f6b9b3ea400ccfaea75",
      "8cfbd3b14b23417993270f851a2d8ff9"
     ]
    },
    "id": "4e7BdgJJhwXi",
    "outputId": "d9c7a923-d2eb-4533-f37e-4f269f22eb89"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7159f02db42f4958bc9a1eeafffb5146",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c2626181e0de4654932125c368d17ef3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "processed = processed.map(get_start_end_idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "P8ytxyfvh0kB",
    "outputId": "c008b161-be24-40bb-a32d-47d92e624787"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'answer': 'garden', 'end_idx': 34, 'question': 'What is south of the bedroom?', 'sentences': 'The hallway is south of the garden. The garden is south of the bedroom.', 'story.answer': ['', '', 'garden'], 'story.id': ['1', '2', '3'], 'story.supporting_ids': [[], [], ['2']], 'story.text': ['The hallway is south of the garden.', 'The garden is south of the bedroom.', 'What is south of the bedroom?'], 'story.type': [0, 0, 1], 'str_idx': 28}\n",
      "answer: garden\n"
     ]
    }
   ],
   "source": [
    "num = 187\n",
    "print(processed['test'][num])\n",
    "start_idx = processed['test'][num]['str_idx']\n",
    "end_idx = processed['test'][num]['end_idx']\n",
    "print('answer:', processed['test'][num]['sentences'][start_idx:end_idx])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VVX3TA2xM-vJ"
   },
   "source": [
    "## 2 - Tokenizing and aligning with 🤗 library\n",
    "\n",
    "Now you have all the data you need to train a transformer model to perform question answering. We are ready for the same task as in named enntity Recognition - tokenizing and aligning the input. To feed text data to a transformer model, you will need to tokenize your input using a [🤗 Transformer tokenizer](https://huggingface.co/transformers/main_classes/tokenizer.html). It is crucial that the tokenizer you use must match the Transformer model type you are using. We will use the 🤗 [DistilBERT fast tokenizer](https://huggingface.co/transformers/model_doc/distilbert.html), which standardizes the length of your sequence to 512 and pads with zeros. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c892hk9NNF9O"
   },
   "source": [
    "Transformer models are often trained by tokenizers that split words into subwords. For instance, the word 'Africa' might get split into multiple subtokens. This can create some misalignment between the list of tags for the dataset and the list of labels generated by the tokenizer, since the tokenizer can split one word into several, or add special tokens. Before processing, it is important that you align the start and end indices with the tokens associated with the target answer word with a `tokenize_and_align()` function. In this case, since you are interested in the start and end indices of the answer, you will want to align the index of the sentence to match the index of the token for a word. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "UI-9P7VYitxv"
   },
   "outputs": [],
   "source": [
    "from transformers import DistilBertTokenizerFast\n",
    "tokenizer = DistilBertTokenizerFast.from_pretrained('tokenizer/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "Pex-YXJnnwb9"
   },
   "outputs": [],
   "source": [
    "def tokenize_align(example):\n",
    "    encoding = tokenizer(example['sentences'], example['question'], truncation=True, padding=True, max_length=tokenizer.model_max_length)\n",
    "    start_positions = encoding.char_to_token(example['str_idx'])\n",
    "    end_positions = encoding.char_to_token(example['end_idx']-1)\n",
    "    if start_positions is None:\n",
    "        start_positions = tokenizer.model_max_length\n",
    "    if end_positions is None:\n",
    "        end_positions = tokenizer.model_max_length\n",
    "    return {'input_ids': encoding['input_ids'],\n",
    "          'attention_mask': encoding['attention_mask'],\n",
    "          'start_positions': start_positions,\n",
    "          'end_positions': end_positions}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 115,
     "referenced_widgets": [
      "4d9152a30e824931983a425ee6d607a6",
      "1f2773e3e80c4dd8b6b26e171bf33bc7",
      "013f041c3e0b4e35bf2432fc345cb7bf",
      "ef4e12f29f1e458f811a400faf21bdcc",
      "f0e34f2bf626434fa73f0def26b3d1a5",
      "1e6c02317171453cbd3d4d665879b0d4",
      "5b6dbe662ca24834b7678638e101e1ff",
      "39029f730ae140c7902fca6dac5361ad",
      "723acefae33d448199fa5c1a9ec3f246",
      "32a5c82c7a9845c09c11bb4e30c2f1aa",
      "77273c2e4b4e4e4c8ee4b6b344749518",
      "f0ac3b9b8f664479940c6ee18fc2f13e",
      "393697738e724e9fad4d163de0a77840",
      "e592db98c0c34c5e800f5d7b6d3c099e",
      "568f11b4462f4b4e95f3ad5947bb275e",
      "7fefe9e1121a43558d773500aef8935c"
     ]
    },
    "id": "kKyLNWCvksOr",
    "outputId": "7af3d914-4546-430c-c2f0-206b732e5131"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0c290f954adf41a99be7dda555ab9162",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5845d2759c3b4e25a4b4126a010154e2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "qa_dataset = processed.map(tokenize_align)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "8v5odGZBmGw0"
   },
   "outputs": [],
   "source": [
    "qa_dataset = qa_dataset.remove_columns(['story.answer', 'story.id', 'story.supporting_ids', 'story.text', 'story.type'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "yBHzbjffmJa8",
    "outputId": "b0688636-fdec-4de0-c2d9-69372b1ddbac"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'answer': 'garden',\n",
       " 'attention_mask': [1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1,\n",
       "  1],\n",
       " 'end_idx': 10,\n",
       " 'end_positions': 2,\n",
       " 'input_ids': [101,\n",
       "  1996,\n",
       "  3871,\n",
       "  2003,\n",
       "  2167,\n",
       "  1997,\n",
       "  1996,\n",
       "  5723,\n",
       "  1012,\n",
       "  1996,\n",
       "  6797,\n",
       "  2003,\n",
       "  2148,\n",
       "  1997,\n",
       "  1996,\n",
       "  5723,\n",
       "  1012,\n",
       "  102,\n",
       "  2054,\n",
       "  2003,\n",
       "  2167,\n",
       "  1997,\n",
       "  1996,\n",
       "  5723,\n",
       "  1029,\n",
       "  102],\n",
       " 'question': 'What is north of the bathroom?',\n",
       " 'sentences': 'The garden is north of the bathroom. The hallway is south of the bathroom.',\n",
       " 'start_positions': 2,\n",
       " 'str_idx': 4}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qa_dataset['train'][200]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qw79BQfW4feu"
   },
   "source": [
    "### Points to remember\n",
    "- The goal of *extractive* QA is to identify the portion of the text that contains the answer to a question.\n",
    "- Transformer models are often trained by tokenizers that split words into subwords.\n",
    "  - Before processing, it is important that you align the start and end indices with the tokens associated with the target answer word.\n",
    "</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rFfJozZvNZWG"
   },
   "source": [
    "## 3 - Pre-trained transformer model\n",
    "Now that you have finished tokenizing and aligning your data, you can feed it into a pre-trained 🤗 transformer model! You will use a DistilBERT model, which matches the tokenizer you used to preprocess your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "8sdX5XY0Gwwc"
   },
   "outputs": [],
   "source": [
    "train_ds = qa_dataset['train']\n",
    "test_ds = qa_dataset['test']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before running the cell below combine the model parts by running the Linux/macOS command in the `model/pytorch` directory of your local computer: `cat tf_model.h5.* > tf_model.h5`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Be5k3ilHsJ6q",
    "outputId": "f2f7fea3-1394-4aaf-b159-994a38476994",
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "All model checkpoint layers were used when initializing TFDistilBertForQuestionAnswering.\n",
      "\n",
      "All the layers of TFDistilBertForQuestionAnswering were initialized from the model checkpoint at model/tensorflow.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertForQuestionAnswering for predictions without further training.\n"
     ]
    }
   ],
   "source": [
    "from transformers import TFDistilBertForQuestionAnswering\n",
    "model = TFDistilBertForQuestionAnswering.from_pretrained(\"model/tensorflow\", return_dict=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-aQVOG4ANcd2",
    "tags": []
   },
   "source": [
    "We will execute two implemenations for fine-tuning the pre-trained model with our dataset, one in TensorFlow and one in PyTorch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 - TensorFlow implementation\n",
    "\n",
    "### Train and test datasets\n",
    "\n",
    "* In the TensorFlow implementation, you will have to set the data format type to tensors, which may create ragged tensors (tensors of different lengths). \n",
    "* You will have to convert the ragged tensors to normal tensors using the `to_tensor()` method, which pads the tensors and sets the dimensions to `[None, tokenizer.model_max_length]` so you can feed different size tensors into your model based on the batch size.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "FbpplBxNtanH"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "columns_to_return = ['input_ids','attention_mask', 'start_positions', 'end_positions']\n",
    "\n",
    "train_ds.set_format(type='tf', columns=columns_to_return)\n",
    "\n",
    "train_features = {x: train_ds[x].to_tensor(default_value=0, shape=[None, tokenizer.model_max_length]) for x in ['input_ids', 'attention_mask']}\n",
    "train_labels = {\"start_positions\": tf.reshape(train_ds['start_positions'], shape=[-1,1]),\n",
    "                'end_positions': tf.reshape(train_ds['end_positions'], shape=[-1,1])}\n",
    "\n",
    "\n",
    "train_tfdataset = tf.data.Dataset.from_tensor_slices((train_features, train_labels)).batch(8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "0_Jj8Av6rEuN"
   },
   "source": [
    "### Training \n",
    "\n",
    "* Create a custom training function using [tf.GradientTape()](https://www.tensorflow.org/api_docs/python/tf/GradientTape)\n",
    "* Target two loss functions, one for the start index and one for the end index. \n",
    "* `tf.GradientTape()` records the operations performed during forward prop for automatic differentiation during backprop. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "PtZz249vQbLn",
    "outputId": "24cdf861-af63-4581-a0ae-2de29d1880ed"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting epoch: 0\n",
      "Training loss (for one batch) at step 0: 6.4653\n",
      "Training loss (for one batch) at step 20: 1.3121\n",
      "Training loss (for one batch) at step 40: 0.9882\n",
      "Training loss (for one batch) at step 60: 0.8768\n",
      "Training loss (for one batch) at step 80: 0.5850\n",
      "Training loss (for one batch) at step 100: 0.2006\n",
      "Training loss (for one batch) at step 120: 0.6227\n",
      "Starting epoch: 1\n",
      "Training loss (for one batch) at step 0: 0.6469\n",
      "Training loss (for one batch) at step 20: 0.1895\n",
      "Training loss (for one batch) at step 40: 0.5063\n",
      "Training loss (for one batch) at step 60: 0.3557\n",
      "Training loss (for one batch) at step 80: 0.4518\n",
      "Training loss (for one batch) at step 100: 0.0930\n",
      "Training loss (for one batch) at step 120: 0.6264\n",
      "Starting epoch: 2\n",
      "Training loss (for one batch) at step 0: 0.4142\n",
      "Training loss (for one batch) at step 20: 0.1267\n",
      "Training loss (for one batch) at step 40: 0.5097\n",
      "Training loss (for one batch) at step 60: 0.2753\n",
      "Training loss (for one batch) at step 80: 0.1139\n",
      "Training loss (for one batch) at step 100: 0.0811\n",
      "Training loss (for one batch) at step 120: 0.5382\n"
     ]
    }
   ],
   "source": [
    "EPOCHS = 3\n",
    "loss_fn1 = tf.keras.losses.SparseCategoricalCrossentropy( from_logits=True)\n",
    "loss_fn2 = tf.keras.losses.SparseCategoricalCrossentropy( from_logits=True)\n",
    "opt = tf.keras.optimizers.Adam(learning_rate=3e-5)\n",
    "\n",
    "losses = []\n",
    "for epoch in range(EPOCHS):\n",
    "    print(\"Starting epoch: %d\"% epoch )\n",
    "    for step, (x_batch_train, y_batch_train) in enumerate(train_tfdataset):\n",
    "        with tf.GradientTape() as tape:\n",
    "            answer_start_scores, answer_end_scores = model(x_batch_train)\n",
    "            loss_start = loss_fn1(y_batch_train['start_positions'], answer_start_scores)\n",
    "            loss_end = loss_fn2(y_batch_train['end_positions'], answer_end_scores)\n",
    "            loss = 0.5 * (loss_start + loss_end)\n",
    "        losses.append(loss)\n",
    "        grads = tape.gradient(loss, model.trainable_weights)\n",
    "        opt.apply_gradients(zip(grads, model.trainable_weights))\n",
    "\n",
    "        if step % 20 == 0:\n",
    "            print(\"Training loss (for one batch) at step %d: %.4f\"% (step, \n",
    "                                                                   float(loss_start)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q8ggB0JUWQuW"
   },
   "source": [
    "Take a look at your losses and try playing around with some of the hyperparameters for better results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 282
    },
    "id": "fK91EPvRYFcX",
    "outputId": "6b7099dd-f918-4905-e3a3-fcce2880e506"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fec2cf834a8>]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAznElEQVR4nO3dd3gc5bX48e+7fdWtbsu944oLxmBjsCkxJYGQcANJgFCub4AUkpsQSIV7b3JJQgJJLiV0CCR0fhAgdBsbbGPL4I57k6t6L6vy/v7YmdGstLLWRmXWez7Po8fS7mj3eCSdOXPed95RWmuEEEI4l6u/AxBCCHF0kqiFEMLhJFELIYTDSaIWQgiHk0QthBAO5+mNF83OztbDhw/vjZcWQogT0po1a0q11jnRnuuVRD18+HAKCwt746WFEOKEpJTa29Vz0voQQgiHk0QthBAOJ4laCCEcThK1EEI4nCRqIYRwOEnUQgjhcJKohRDC4RyTqFvbNP/3/nY+2FbS36EIIYSjOCZRu12Kvy7dxbubj/R3KEII4SiOSdQABRlBDlY29HcYQgjhKI5K1IMHBDkgiVoIISI4KlEXZAQ5UCGJWggh7JyVqAcEqWlqoaqhub9DEUIIx3BWos5IApCqWgghbByVqAdlBABkQFEIIWwclagLBgQBZEBRCCFsHJWos5P9+DwuSdRCCGHjqETtcimZ+SGEEB3ElKiVUhlKqReUUluUUp8ppU7rrYAKMoLsl4paCCEssVbUfwLe1FqPB6YCn/VWQFJRCyFEpG4TtVIqHZgHPAKgtQ5prSt7K6CCAUFKa5tobG7trbcQQoi4EktFPQIoAR5TSn2qlHpYKZXccSOl1CKlVKFSqrCk5PhXwCvICM/8OFTVeNyvIYQQJ5JYErUHmA7cr7WeBtQBt3bcSGv9oNZ6ptZ6Zk5OznEHNMhI1NL+EEKIsFgS9X5gv9b6Y+PrFwgn7l4x2JpLXd9bbyGEEHGl20SttT4MFCmlxhkPnQ1s7q2A8tMDuJRU1EIIYfLEuN13gaeVUj5gF3BNbwXkdbvISwtwoFJ61EIIATEmaq31WmBm74bSriAjKK0PIYQwOOrKRFOB3EBACCEsjkzUgzKCHKpsRGvd36EIIUS/c2Sizkzy0dKmqW1q6e9QhBCi3zkyUacneQGorJc7vQghhCMTdUYwnKjlllxCCOHURJ3kA6SiFkIIcGyiNlofDaF+jkQIIfqfMxN1UHrUQghhcmSiTpMetRBCWByZqANeN0Gvm8p6aX0IIYQjEzWE+9TS+hBCCAcn6vSgl0ppfQghhHMT9YAkH1VSUQshhHMTdUaSV6bnCSEETk/UUlELIYRzE3V60EdlQ7OsoCeESHiOTdQZSV5CLW00Nrf1dyhCCNGvnJuog3IZuRBCgJMTtSx1KoQQgIMTdXpQVtATQghwcKJur6il9SGESGzOT9RydaIQIsF5YtlIKbUHqAFagRat9czeDAogQ1ofQggBxJioDfO11qW9FkkHAa8Ln8clsz6EEAnPsa0PpRTpQa+s9yGESHixJmoNvK2UWqOUWhRtA6XUIqVUoVKqsKSkpEeCSw14qGlq6ZHXEkKIeBVrop6rtZ4OnA/cpJSa13EDrfWDWuuZWuuZOTk5PRJcqt9DnSRqIUSCiylRa60PGP8WAy8Ds3ozKFOy30NtoyRqIURi6zZRK6WSlVKp5ufAecDG3g4MIMXvoVYqaiFEgotl1kce8LJSytz+71rrN3s1KkNKQBK1EEJ0m6i11ruAqX0QSydSUQshhIOn54GRqBtbZE1qIURCc3SiTvZ7aGnTNLXImtRCiMTl6ESdGgh3ZqT9IYRIZI5O1Cn+cKKWudRCiETm6ESdbCTqGplLLYRIYI5O1Kl+aX0IIYSjE3VKQFofQgjh6ESdLBW1EEI4O1GnSo9aCCGcnail9SGEEA5P1EGvG5eS1ocQIrE5OlErpUj2e6T1IYRIaI5O1BC+6EVaH0KIRBYXiVpaH0KIROb8RC1rUgshEpzzE7VU1EKIBBcfiVoGE4UQCSwuErUMJgohEpnjE3Wy30ONJGohRAJzfKJODYQrarkdlxAiUTk+Uaf4PbRpaGhu7e9QhBCiXzg+UVsr6MmAohAiQcWcqJVSbqXUp0qp13ozoI7M+yZKn1oIkaiOpaL+PvBZbwXSlWSfrKAnhEhsMSVqpdRg4ELg4d4Np7P0JC8AlfXNff3WQgjhCLFW1PcAtwBtXW2glFqklCpUShWWlJT0RGwAZCX7ACira+qx1xRCiHjSbaJWSl0EFGut1xxtO631g1rrmVrrmTk5OT0WYFaKH4Cy2lCPvaYQQsSTWCrqOcCXlFJ7gGeABUqpp3o1Kpu0gAef20VJrVTUQojE1G2i1lrfprUerLUeDlwOvK+1/mavR2ZQSpGV4pOKWgiRsBw/jxogO8VPmVTUQogE5TmWjbXWS4AlvRLJUWSl+CiViloIkaCkohZCCIeLi0RtVtSyMJMQIhHFR6JO9hFqbZM7vQghElJcJOok4zJyWUFPCJGI4iJRB71uABpCkqiFEIknLhJ1ks9I1FJRCyESUFwk6oCRqOulohZCJKC4SNRm66NRErUQIgHFRaKW1ocQIpHFRaI2K2ppfQghElF8JGqpqIUQCSw+ErVMzxNCJLC4SNRywYsQIpHFRaL2e8JhSkUthEhEcZGoXS5FwOuSiloIkZDiIlFDuP0hFbUQIhHFTaIOet0yPU8IkZDiJ1H73DRK60MIkYDiJ1F73dSHZD1qIUTiiatELYOJQohEFD+J2uemobmtv8MQQog+Fz+J2uumQVofQogE1G2iVkoFlFKrlFLrlFKblFJ39EVgHSX53dQ2SqIWQiQeTwzbNAELtNa1Sikv8KFS6l9a65W9HFuE/LQAxTVNtLZp3C7Vl28thBD9qtuKWofVGl96jQ/dq1FFMSgjSEubprS2qa/fWggh+lVMPWqllFsptRYoBt7RWn8cZZtFSqlCpVRhSUlJD4cJgzICAByobOjx1xZCCCeLKVFrrVu11icDg4FZSqlJUbZ5UGs9U2s9Mycnp4fDDFfUAAclUQshEswxzfrQWlcCi4GFvRLNUZiJ+lBlY1+/tRBC9KtYZn3kKKUyjM+DwLnAll6Oq5O0gJcUv0daH0KIhBPLrI+BwBNKKTfhxP6c1vq13g2ri0DSAxyqkkQthEgs3SZqrfV6YFofxNKtnFQ/JTUy60MIkVji5spECCfq0tpQf4chhBB9Kq4SdXaKX+ZRCyESTtwl6vpQqyx3KoRIKHGWqH0AlNZI+0MIkTjiK1Gn+gEokfaHECKBxFWizkkxErXM/BBCJJD4StRGRS0DikKIRBJXiToz2ehRS6IWQiSQuErUXreLAUleSdRCiIQSV4kajLnUMutDCJFA4jNRS0UthEgg8ZeoU/0yPU8IkVDiLlHnpPgplel5QogEEneJOjvVR12olYZQa3+HIoQQfSL+EnWKzKUWQiSWuEvU1tWJkqiFEAki/hJ1qlxGLoRILHGXqHONRF1cLTe5FUIkhrhL1FkpftwuxZFqqaiFEIkh7hK126XISfFzRCpqIUSCiLtEDZCX5ueI9KiFEAkiThN1QHrUQoiE0W2iVkoNUUotVkptVkptUkp9vy8CO5q8tACHJVELIRJELBV1C/CfWusJwGzgJqXUhN4N6+jy0vxU1jfT2CxXJwohTnzdJmqt9SGt9SfG5zXAZ0BBbwd2NHlpAQAZUBRCJIRj6lErpYYD04CPozy3SClVqJQqLCkp6aHwohuSmQRAUXlDr76PEEI4QcyJWimVArwI3Ky1ru74vNb6Qa31TK31zJycnJ6MsZOhRqLeV17fq+8jhBBOEFOiVkp5CSfpp7XWL/VuSN3LSwvgc7skUQshEkIssz4U8Ajwmdb6j70fUvfcLsXgAUGKJFELIRJALBX1HOBKYIFSaq3xcUEvx9WtwZlJUlELIRKCp7sNtNYfAqoPYjkmI7OT+fvOMj7aUcqc0dn9HY4QQvSauLwyEeDG+aPIS/dzz7vb+jsUIYToVXGbqHNTA8wYOoBDVTKXWghxYovbRA3Gmh81TWit+zsUIYToNXGdqHNS/YRa2qhqaO7vUIQQotfEdaLONS4lL5YlT4UQJ7D4TtTGbblkzQ8hxIksrhO1uThTsdyWSwhxAovrRG3d6FZaH0KIE1hcJ+pkv4dUv4eDlbKKnhDixBXXiRpg/MBUNh/qtJifEEKcMOI+UU8clM7mg9W0tslcaiHEiSnuE/WkgnQamlvZXVob8/dorbl38Q4Oy1WNQog4EPeJenJBOgDLtpfG/D37Kxr4/Vtb+fpDK3srLCGE6DFxn6jH5qVw6ohM/vTe9ogrFMtqu54J0tQSvinurtK6Xo9PCCE+r7hP1EopfnDuWCrrm7nsgeX84v9t5I0Nh5jxP++yZm951O9pCLVZn8uMESGE03W7HnU8OHlIBh6XYtuRWrYdqWX1nnCCLipvYMawztvXh1qsz19bf5BhWcl8YWJ+X4UrhBDH5IRI1AGvG7dL0WLM/NhyuAaAhubWqNvbH//NG1sA2HPnhb0cpRBCHJ+4b32YRmQnd3qsq1X1GqMkcFkqVQjhVCdMon7oqpmcc1JexGOV9dETdbRKW5ZKFUI41QmTqIdkJnHj/FERj1U1hKJuWx/qnKhlvRAhhFOdMIka2hdpAhiVkxxRJVc3NvPYR7vRWtMQLVHLCnxCCIc6IQYTTTm2RD0gyRfR+rjtpQ28vv4QU4dkRO1RF9fIVYpCCGfqtqJWSj2qlCpWSm3si4A+D7/HbX2eHvRGJOrPjIWbGkKt1Ida8bgUXreynj/e1sfKXWXMufN9ahrju8f90Y5SPthW0t9hCCGiiKX18TiwsJfj6DG/+uIEHr/mFNKTvB2uVAz3q8vqQjQ0txL0ugl42xP7M6v2UdPYzB/f3sq/PbAi5vfbsL+KA5UNHIjxwpm2Ns1zhUVRq/r+9I2HP+bqR1f1dxhCiCi6bX1orZcqpYb3QSw94po5IwBYuq2UA5UNPPLhbsbnp1pJu6y2icbmVgI+Ny4FNY3hi1/2lNVz64sbeH3DIQD2V9QzeEBSt+9XXh8+AJTXRR+47Oi5wiJufWkDlfUhFs0b1f039LGKuhADkn39HYYQwqbHBhOVUouUUoVKqcKSkv4/hfZ5wv+1/35tM1fZKsXS2iYaQuGKOmhU1P9x5kjOGpdjJWmAFTvLYnqfSiNRV9TF1vow2wv2Nk1f21Fcy/Id0RexWru/sm+DEUJ0q8cStdb6Qa31TK31zJycnJ562eM2Ni8FgNNHZUWsVV1WG6I+1EqSz03QFz6hGJmdzF2XTeVLUwdx79enMyDJy49fWM+bGw9FfW07s5KuqI+toi7cWwHQr+tn37d4Bz96fl3EY6mB8L74dF/l5359rTXvbzkia4QL0UNOqOl5dl+eVsC6X57H7746BYBkn5vx+amU1oZ71AGvm6A3/N8P+jxkp/j58xXTuHDKQH547lgAnli+N+I1D1c1cul9H0WsY21W0hUxtD5CLW2UGIOWdU0t3Wzde6oamqns4gKfAxWff5GqjQequfbxQpZt7/8zKyFOBCdsolZKkZ7kZfCAJO68dDKvfGcu2Sl+yurCPeqg102SUVEneSPbEFeeNpxvnzmK1XvKqbbN5vj7qn18sq+SJ1bssR4zK2mzV72rpLbLy9HtVXddlLncfaWmqYX6UCvNre2rCJpzy8vrPv988jLjNUprYzvLEEIcXSzT8/4BrADGKaX2K6Wu6/2wetbls4YyOjeFrBRfROvDnPWR5O/cLz77pFxa2jRvrG9vf6QZ7QF79Wwm321HarjlhXUs+MMHPLkishI3n3/pkwPW1/1ZUdcaA6jmv6GWNmtBq7IYB0WPptp43coY20EQng3ztxV7HDcbRggn6DZRa62v0FoP1Fp7tdaDtdaP9EVgvSEvLcDh6kYOVDYQ8LlJ8hmJ2td58suMoQOYPjSD37+11ZoxYiagUuOmBDuKa6yq8aMdZbxoJOInVuyxqupP9lXwf+9v57y7l/LbN7dYr18Xij1RH6xsYPitr/NRFwOAx6rWOEiYZwv2KzXLeqAKNueUVx/D+imvbzjEL17ZxJ/f2/6531+IE80J2/qI5rIZg2lr01TWN+N3u6xZH2bCtnO5FHd8aRJldSGm3vE2d7+zzUrQO0vqaGpp5Zw/Lo34nu8tGMPvvjqFXSV1rN9fRXF1I5fet5y73t4WsZ1Sx1ZRm73eF9fsj2n7nSW17D7K3WusRN0Q/re+Ofxvit8T8zTDozFft6s+eDTmHXlq++FMo7I+xI+eX3dMZwBC9KWEStRj8lL56QUnATAyJ5mgkaCD3uhT5SYPTmfO6CwA/vTeduvqxr1ldWw8UNVp+7NPyuWsceEZLyt3lbG0i/s4DhmQxFubjjD5V291GevOklp+/v820NLaRrkxYBnr/Oaz//AB8+9a0uXzZsvDrKjNRaoGDwjS0NwadS2UY2FW1MeyImFjS7hf7vf0/a/k25uP8MKa/fzP65/1+XsLEYuEStQA184dwfZfn89N80dblXSyv+vrfv58+TRu/+IEIDx1LT8tAMDPXm6/on7e2HBynjgojdzUACNzksOJelsJ2Sn+TsmnICMIhAf1isrrufDPyzrN2/7hc+t4auU+thyusSr5aJX/sWpsbiVkDCLWdGh9mBf4lNU1caiqgR3Fsd/Z3c48AHS1zGw0Tc1mou77+eXmWMVSuYReOFTCJWoAr9sVnhUS9OJxqaMmwKwUP9+aM4Ipg8N3O581IpMLJg+07iKz9MfzefLaWez+3wtQKrx2yKkjsli9p4IPtpUwb0x2xKXqAGnB9gPDnW9uYdPBan79xuaIbVqMZFpZ30xReT3QfVtAa23duBegKkqitL+G1fqwVdQQnht+x6ubufHpNQC899kRLntgecyVtnm157FV1OHXdrtUN1v2PLMNVVzTJIOZDnP1o6u47IHl/R1Gv0vIRG26fNZQnr7+1E6JNJozxmQDEPC6OH1UtvX4oIxwhW0maYAvTR1EbVMLVQ3NXDKtwKqgTcm2wcvXjVkl247URvRIPUbCOljVwN6ycKJesbOM+5bs6DLGJ1fsZdzP37S+3l3WuU9ttj3A3voIPzYk06yoQ+wurWNvWT3bj9Rw3ROFrN5TEfO86ONJ1Oa2/ZEo6yIOXv23uNaByoa4X9yrp32wrYTVeyr6O4x+l9CJOj3o5dSRWTFtO3N4JgAlNU1MHZJuPe5xd96Fs0dmMmVwOgUZQeaMzuavV87gP42LaKDzdMArZw+jtU1z7eOrWbmrjPuX7GS/ceFJUXk9e8vDCXfL4Rp+9+bWLpPZP1bti/h6jzGg+OzqfVaCj6ioGyMr6qFGor728dVsPVJDU0sb5969lBSjNfTeZ8VAeHri0W5dZia7YxmcM6c8RrupQ2+rsR28anpgMHP7kRq2HK4+pu/RWjPnzvf55sMfR33+lbUH+MkL6z9XXOuKKrnjn5uO6bZza/aW88Nn19ImV5n2q4RO1Mdizqhs/m3mYH5y/njG5qUedVulFA9fNZNnFs3G7VIMyUziu2ePYXhWEmeNy8HjitztV8wayk8vOIlP9lVy+YMr+e2bW6z5zP/aeJjG5raI7e1XRto1tURut6sk3GN++dMDPLe6CIhMSmZCNZPj2LwUrps7go5/x3ddNoULJw9k6fYSth6uYeav3+WSez/qshVjr6hj+QPfWVJrDdT2R6K2V9T2M47jde7dS1l4z7KYtm0ItfJcYRGHjJ/puv2dB6kBvv/MWp4tLPpccV3z+Goe+2jPMc2V/8r9K3jp0wNyq7p+Jok6Rj6Pi999dSrj89Pwul1cN3cE/3XxxC63z00LWK0E05Ifz+fxa2ZZrQbTuPxUrps7gk9+cS7j8yMPAjuKa1EKTh6SYT12sKqBzQer2Xigit+88ZmVDDtW2hsPhpNfeV3I+uOMrKjNwcTwY0k+D7eeP77T/2XCwHQmFqRxqKqRRz/cTWubZt3+Kp5Yvsd6nXVFlfzs5Q20tWnrdds0HK5uPOpURK01Vz+6ij1Ge6fjvukL9jnt9v3z5sZD/OSF9bS16ahVaHldiNfWH+zyde0/l5rGZnYU1xDqcDD97ZtbuOWF9Ty4dBdAxBrp0Xye9VNcRnvueJYJ6I9pk32xVozWmm1HaqyxncNVjSy8Z6m1bHFzaxuPfbS7y5ZUVUNzn7TrTqg7vPSlX1w04bi/t66p/Qc7bWiGNYCWmezjje+dwfee+ZTXbFdETilIZ0hmEmuLKgH4r39uZsvhGrxuRXOrZsawAXxhYn6ninpdUSVaa8rrQtQ0tlDb1GL9wiX53J0q6iSfG2+UVs7gAUFG54QXuXq2sIiFE/NpbGnl/iU7GZDk46cvbyDgddHY3Mb1Z4ykprGFoZlJ7Cuv5/Q732fGsAG8eMPpUffFzpJaq81jj6X96xYq6pupa2rp9kymo9Y2zdl/WML3zh7DpdMHd7ldre3nYT/j+PZTn1j/52vnjOCXX4z8mS/4wxIq65s5fVQ2mVGmTh6ubmRQRpCF9yy1DkRzRmfx9PWzrW2OVIcr6ZW7wrN+8oxZRV2pC7WQFvAedZuupPjdlNaGe+FTbQf+WFQ1NDOkw2PLtpeQ7PcwfeiA44qnO/bkqLWOGAfqKct3lvGNhz8m1e9h/e3n8cKaIrYcruGplXv5ycLxPLRsF797cyutbZrrzxjZ6fun3vE2kwrSeO27Z/R4bHZSUfcDTbhSeOxbp/DyjXMinnO5FNkp4VuKjc1L4Ywx2fzm0snW6naANeOkuTX8Oo9+uBuIrKinD82grC7EvvJ6KozZH5N+9ZbVxx6amWRdVWkmR3M++Yjs5E4xjcpNsb4+f3I+/3PJJAoygvz05Q3Ge4cPEuuKKqltauHikwdZ26/Z2/Vg0AfbIueamzNLtNbc8sI6Jt/+Nhf+eRlfuX95zD3vfWX1PP7RbirrQ+wpq+eHz6076vZ1TS3W8gBdVY6PfrSbnSXt0xW3Hq6xph/aLxKyV97m9maShvAVrPaq2jxImz/TltbOVaT9jKS+6firN3Ma6v6K+m627Py+NR1aQk0trVz5yCouvW95zEsCm1rbdEzfY5/eGWqNLEK01ry96XDUqnt3aR3bjtR0enz5ztKI36FnVu2zxl1qmlqoC7XiMn4ea/dV8tr6g1bLsDnKz8WswjceOLbxiOMhibof/PKiiVw3dwRzx2RHfX7GsHCF8u0zR/G3605l4qB0axaI6ecXnsTMYQNYMD6X7cZ8Z3tFfdqo8CDpsu2lEb/Mq/dU4HUrxuSlWosn1YdaCHhd1i/pP/59Nv/39WksnJjPzeeMAdoHGgHmjs5m8IAkbjir840PzFP4KYMzePyaUwAYlN5eJW48UMV22x/RuqJKCjKC3HjWKHJT/dZVkjtLanmucD+tbZrqhmZqm1p4eNlu6/tao7Qj6kMtNLe2cdlfl3P7PzdzsDK2+2DWNraQb8RYa6vi0gKRJ5xn/+ED1hlnNfZEYP/jtyf6nV3MQy/cUw7Ar17ZyMpd5RHPRVsut8iWWG94eg3/2tD98rttbTpiP0N7KyHW1sdbmw5bn3c89d9Z3D6jaNPB6H31jj7eVWZcCFbCFQ+t5ONdR0/W9n3RcWro6xsOsehva/jaX1dw20vtg6xF5fXMv2sJX7k/ckpfbVMLX3+o/S5GbW2aW1/awKMftf9O1TQ2W+2hFbvK+PHz662DbEmUW/XtLYvtgNcTJFH3g/z0AL+4aELUNgPARVMG8sK3T+PL0wqsxzr2wb5+6lBeuOF0pgxOp6I+RGNza0RCHp+fBsCmg52P9sOykslL9VNaE/5D2FtWHzGFMD89wEVTBvHAlTO4+ZzwbBV7rFlGxb/gpNxOr735UDWpfg/zxmZz1rhcrp87gsPVjfzg2bVc/0QhF/3lQ7712Gpr+/K6ELlpfm5ZOJ5TRmRS19RKY3OrNbj22DWn8NbN85g0KJ31xtWgFXUh5v72fW59cQNHqhv549tbaWlt46K/fMif3t3OEeOO8uZsGej6kv0rH/mYVXvKrZaDmWi11hFV3EAjkS/eWsyKnWUs39l+JmCv/Oyfd3UZf3FNE3VNLTyxYq91MZOpqaWt08+6qLw9sX66r5Ibnv6Estqmo1alv31rC+fevZS9timaZmyx3DZuV0kt//n8OqviNyvqbUdq+N4/PqVwb/sBJtoqiW9sOMSf3g2v27K7tA6tNV97cCWXP7iSEuPn82EXa9e0tLZxz7vb2Ffengg7tsTMdlnh3gr+sarImmXz1Mq9EfHa/z/QPlhbGmWVyOqGlogDUoPt5xDt5tfHe0HY8ZBE7UBKKWYOz4zoyZmtBaXCFbe5kFR2ih+tYfuRyF+a/PQAPreLrVGmiY3MTiY71U9Dcyt1TS1sO1LDuPzu+7+v3DSHN29u78WlBbw8ee0svrtgNACTCsIHhy9PL7CuMByQ7KNNh2eevPvZEXweFwcqGyg2erPldSEyk8L93SSvm92ldXzhnqVWoh6ZncyYvFQKMoIcqKjnhTX7mf+HJRyqauTZwiKuemQVf35/B8u2l7KrpC6iurNXPPb2S1NLK5/sq0BrzTLjMv/MZB8+j8uanhceJGojJ9XPbeePZ8VtZzN1SAbLtpdyxUMr+ceq9hkY5pomH24vpcSWeEtqm6IeIMrrQlGT5ZDM8MGy4xWd9paL6aa/f8IVD63sNBvDPMt40lhL/aK/fMh1j6824gwn1OU7y/jNG591OkjYvbHhEFrDP78zF4AXP9nPmr0VvLr2IK+uO8gvX9mE163ITvFZ67RAeDpffaiFRz/czb2Ld7DpYBXz71rCW5uOWNvY44hm6fYS7nl3e8TNLTom6uYO4zFPrtjLF+5eyitrw4O7Po+LNXvLrYOefR/WNDZHnTlV3dhstQntXAqKqzvvK/OMSSm45rFVXc7G6gkymBgnclLDVeyT187ijDE5nR7ffCjy9DPZ5yEn1c/Ww517dSNzUsgyBr+ueGgle8rqucRWvXcl2gDUvLE5TB2SQWNzKz88dxwet4qovgcktQ+y/ePfZxPwuvjyfcuZ9Zv3uPtrU6moDzFhUDjBm1eI7i2rt6brmZVuwYAgb246zI+eX8es4Zl4PYqPdpSx1Ti9N2+jttF2BmGvJu95dxtnjMmmqaWNf3+ykGXbS/nNlye37y+/h1S/x5qeZ1Z7t39xIhdOGQjA3NFZ3Lt4p/U9eWl+jlQ38aPn1/HTlzcQammzLozyuBSltaGop8wV9aGo7YdJg9IpKm9gZ0kta/ZWWO+7Icq6MjuM1sO6okprCYPFW4u54ak1XDB5oFUN1jS28N6WYuvAc8WsIZTUhHhw6S5rXXYNnD0+N+Kagn9tPMyMYQMYY9wpafnOMpbfv5xF89oH1FL8HvLSApTWNrF4azH3Ld7B6j0VfHfBaDYcqCLU2sbLxoqSL33SvqCYeSBaW1TJ79/awojsFAYkeTlzbA4et4uGUDgJ2/vC9tZHZX3IWv/d9PePI68hCLW08ZX7V3DGmGz+dt2pEa2adUVVUVevrG5ojnoDkKlDMqJW1NuMRK01LN5awr2Ld/Dfl0zqtF1PkEQdJ378hXFMLkhn7ujIvrY58Lhka+RVg8l+N9kpvqiVm1lRA6w3TgXHHeOMCrv0oJefXRh9FkxmcvsMhbF5KaTaZizc/upmaptarBkTQdsVmyt2ljEgyWtdNTrI1pq59xvTeX/LET7a0V6RmVd42hOj2V/84blj+eM72zjz90tI8rnZeqSGnFS/NRBqSgl4ePrjfeyvaLDubZmf7reenzAwPWJ7c0C2tU1bA4TmYl1j8lIprW2KqLBN5XWhqAN6Ewel8a+Nh/mGcdHLnNHnkpHkY+OBKsbnp1oDjhA+SJTWNvHJvgorUd+/eCeNzW0R656bpt7xNgCTCzL4+qlDmXrH2zy1ci9tOnxQeX39IZbeMt9qdewqqeMbpw7F63bhcSlrvfKSmiYKMoL853ljyU0N8NCyXZTWhvjx8+usFsibGw9b4yWvrAtXuG9vbq+ozd5za5uOOPD94qIJFFc38s5n7dua6kMtVNSFWH+git+/teWoA3jJPrd1Y45l20u57vHVuF3KmplUVFEfMaDrUuGppOGKOjJRKxWedfX8mv3sK6unTWuGG4Pta4siB8l7c5qetD7iRMDr5pJpBZ2mKOUaCfedzUdID7YnwaDPbVXbdj6Pi2lDM8hJiXzOrGp7WoatojbbCw98cwbfmT+aqoZmWtu0VXXb5xBvOVxDfnp7crb30HNS/YzIbp+Fcs5JuRH9RJNZUV83dwTzxuZQVFGP1+3inq+dzI0dBkIPVzUSMNo1H9gWZ7LPhTfvw2mqa2ql49Ik5qnz6NwUyqJU1AUZQSNRdz6AThwUeSAormmiqqGZvWX1zO5wBa3Z8rjn3e3ct2QH8+9awqo9kQOT0QxICv+O5KcFaNPhAdM/XzGNA5UNfLCtmENVDby58RANza1kpoR/Li22sY+SmiZy0/xcOn0wc8dkk53iZ8OBKkprQ1x+yhBGZCdbg9vJPneXg3AFGUHrBtTmla///dpm/rp0F7tKOvf2//D2Nqb99ztc/egqNkcZd7E7eWhGxNfvbSlm5a4yZo/MQqnwz/qQrU1RYKxxU93Q0qntlJvqZ1BGkPpQK/N+v5izjFUpi6sbKSpvsMYuoH0FyN4giTrOmRV1S5tmmu0X1Gx9QHge9M8vPInHrzmFLf+1kDF5qWSltCfQ1783l2FZkVPyeop9frF5kFk4KZ9vzh5m2yacPDr+UeentR9MzAWjTPYphPZWUPt7wZHqJmvRrQevnMHK287mn9+dy8UnF3Sak+1SkbMrrpw9jI9uXUBuavsf4vAO0xaVij5tSykYlZNMVUMzB21nNG6XOmqiHtPhQLCzuJbbX90EwPzxkQO3+ysarPbV797cag1cTh0cmew7MpNjrrFvB2UEOXdCHjmpfp5auY/T/vd9a/54VpS54QerGiIO8tmp7dv84NyxnDoivNTCkMwgCycNjHxvoyW2t6ye3DQ/s4Znkp8WYP2vzuOtm+dZbSM780Iv+0Go44y8gDcyjdkPePd87WQgvFzC8KxkslP8HK5qtOavAwxKNxN1c6f12AemBztduHbT059w35LwmcBZ49p/Lr25Towk6jgXtK38N3NY+4UHQa/bSuInD8ng+jNGcta4XGsKXlZy+LnhWUmdKrmelJEU/eKMPFsSNivq6g5TwOwVtZmoz5uQB0B2io9zTsrlvm9M5/JZkZdiFGQErbOL9KAXpRQBrzviYpIxtnnht54/nl9/ebI1YPXSjafzi4smdFpMy957v3bOCO42kgDAM4tmc/Vp4YPP6JwUK8Evs61JnuL3kJnso7wuxOZD7VXhWeNyWPyjsyjICEbMl7/h6U94+dMD/OCcscyLksRumj+aCybnRzz2xanh+etZyT6+NjO8X9b98jxevOE0hmQGrXEGc18Mygjidbv42swhvL+lOOK1MpM7n5HtKqmLOFMzB4InDkojLy3A0KxwUptSkMHCSZGxmT33A5UNZAS93PmVyTx53SxcLsW4/FT+dt2p3HXZ1IjvOXVkZqf/W0cdD7rTbGMp9kHygowgA9ONuzzZDpSZyT4CXlf4ps/1zSycmM+3zxzFt04fzrkT8ljQ4SD5+oZDPL58D/lpAWbb4otlNs3xkh71CeTK04Zbd5NxuZQ1vWxkdudq2edx8cS1s5jYSy0Pk30w0c7ewjGr7tvOP4nMZB/Ds5JZt7+K6+YOt7bJSPLx8o2nW9MOlVI8fPUp1vPLb11AY3Mrd7+7nZvmj+KGpz6hsr6ZtGD0A4U92Xz7zHAbZFJBGhsPVB/1SrtvnDqUg5UNna5SnFSQbk3XuvmcsVYb54NtJcwakcmq3eWkBjwMSPaxfVN4uxvOGsX9S3Zy/qR86wzh79fP5sa/r7Gm5F07ZwTfN+aydzQwPWCdCQ1MD7D4R2ex2qg8CwYE+c2lk7nj4okEvG5mDMtk2S0LrO8111U3T92/fupQ/rn+IC2t2ko40a627LjvzJ62OZXUrLZnDBtgVchXzBqK1pqvzBjMy5+G++cZST5r/XM7e3vpnJNyueKUofz1g/Dc/J9dcBIrd5V1OvMak5vK+v1VDM9K4tFvnRIRn701MSgjSF5agD2ldRyqasTndhFqbSM14CEt4A33rlvbmD4sg0XzIltjP7/wpIgbS5w+KovfXzbVmvYH4Vvm9dYVlJKoTwCvfmcOKX4P6UEvPzhnLC9/Gh5hN9cNOWNs59YAwJldPN6TvG4XWck+rpkzvMttzDvXDMlM4n8umdzldtOOkkDNwca/XDENaK/kO160Yor2x/TMotO6XWvk11+OjC814KGmsYUUv4evnTKEqYMzmDw4PWI64MKJ+Uai9lo94qxkHzefM4ZFZ4yMOOuYPDidZbcsYPitrwNw0sCuB3kHZgSts5ChmUkEvG5rPxRkBHG7FG5X9CV882ytD/PfD348n40HqrjoLx9aMQLWIJzJ3g66fNZQfB4XX581FIBLpw/G73Vz4eSBuF2KHb8+31phssg2Lzq9iwPoaONMZ3x+Kg9ffUrEzyM31c/EQWl8dqiaeWNy2FFSy6f7Kq3knp3iZ2ROSsSFUOlBrxX/oIwAA9MDvGMMbM4fl8PirSWkBrykBb3WFMJoB+rrzxjJ6NwU6xqAh66aSbLfE3HQqA+1sqesvtOVvT1BEvUJYMrgDOvz758zxqrALjm5gJnDMjv12Praml+cG/Xx00dlsXxnmXX63JPMSv5oNyJ4dtHsiEHIFL/HGtiK1ds/mGfdENjrdjHZ6BHn2qq6L0zK5/dvbSUt4LH6q/9x5kj8Hne3d7QZE2U2zqvfmcOn+yqZUpBuzTQYZrQczH5rx7ZNR7lW6yNybRF7e8g8gH7w4/nsKqnjiodWAjDQ9j0pfg9XnTbc+trtUnzJ1qKwLwNsPyB1laiTfB4eu+YUJgwMnzkFbPvH5VL87IKTqKhvZtaITP7y3nYjUYf3kTnuYh6Ex+enopQiPy3AnrJ6q6I2zRmdbSRqj3VATw96IxZAs7PPPDIvx7e3qoJeN3e9tZV7vzE96vd/HpKoT2BKqX5P0kdz/zdn8Om+ipjvBXksLpoykPe3FFsVWjSxrkV+NAPTgwxM75wUh2Qm8ZcrpnHykAwKMoKkBT2kBrxcffow0oIerp0zIqbXjxb/lMEZ1sHZrN7Mgc6gz83vvjIlorcbzaSCdIZmJnHykMjq0T6AaCavvLQAOSl+zh6fy8icZOZFGbyNhf0g2NXYBcB82wCdq8OB1n7gyksLoBSMN846sm2DnIU/P8eal5+fHuBAZXgQ1F4tm0s1pAa81vjD2SflRl1j3nydjsxEHfS6+Y8zR7J4awkNodaIsaOeEFOiVkotBP4EuIGHtdZ39mgUIiGlB70Ro+Y96dLpg5k9Muuo98PsbfbBrzPG5DA+P5WB6UFuPGt0t99rb6mYrpw9rNMZQl5agAe+OSNiUOvfTum4zl1nBRlBlt4yv9Pj9sRobw+5XIpHvnVKp+2PhVKKL04dxIfbS455ANu86tXu4mmDGJOXwsD0IDecNSriAGJP2qNzU6isb8blUpw2Kos3bz6D+lArgzKCKBU++/n+OWNYV1TFN2cP7TKG1ChVtLmS4VWnDePGs0bzvQVjOh1ceoLq7m4PSik3sA04F9gPrAau0Fpv7up7Zs6cqQsLC3syTiESSnF1I9WNzYzOPf4LkY6X2R/fc+eFff7e0ZTWNpHs8xx3lVofaqGpuS3qmdumg1WMy0vtsoruaPWecgoyghFtkPpQCwGP+3MnaKXUGq31zGjPxVJuzAJ2aK13GS/2DHAx0GWiFkJ8PrlpAauP3NcevmqmdbNhJ8hO6TxN8Fgk+Tx0NQxyrJX9KcM7t5SSfL1/1hbLOxQA9nsA7QdO7biRUmoRsAhg6NCuTx+EEM52jjFXXThHj13worV+UGs9U2s9Myen96d9CSFEooglUR+AiLvwDDYeE0II0QdiSdSrgTFKqRFKKR9wOfBq74YlhBDC1G2PWmvdopT6DvAW4el5j2qtN/V6ZEIIIYAY51Frrd8A3ujlWIQQQkQhq+cJIYTDSaIWQgiHk0QthBAO1+0l5Mf1okqVAHuP89uzgej3kXcGp8cHEmNPcXqMTo8PJMZjMUxrHfUilF5J1J+HUqqwq+vdncDp8YHE2FOcHqPT4wOJsadI60MIIRxOErUQQjicExP1g/0dQDecHh9IjD3F6TE6PT6QGHuE43rUQgghIjmxohZCCGEjiVoIIRzOMYlaKbVQKbVVKbVDKXVrf8djUkrtUUptUEqtVUoVGo9lKqXeUUptN/7tfH/53o3pUaVUsVJqo+2xqDGpsD8b+3W9Uqrnb5EcW3y3K6UOGPtxrVLqAttztxnxbVVKfaG34zPec4hSarFSarNSapNS6vvG447Yj0eJzzH7USkVUEqtUkqtM2K8w3h8hFLqYyOWZ41VN1FK+Y2vdxjPD+/HGB9XSu227ceTjcf7/O8lJlrrfv8gvCrfTmAk4APWARP6Oy4jtj1AdofHfgfcanx+K/DbPo5pHjAd2NhdTMAFwL8ABcwGPu6n+G4HfhRl2wnGz9sPjDB+D9x9EONAYLrxeSrh+4JOcMp+PEp8jtmPxr5IMT73Ah8b++Y54HLj8QeAG4zPbwQeMD6/HHi2D37OXcX4OPDVKNv3+d9LLB9Oqait+zJqrUOAeV9Gp7oYeML4/Angkr58c631UqA8xpguBp7UYSuBDKXUwH6IrysXA89orZu01ruBHYR/H3qV1vqQ1voT4/Ma4DPCt51zxH48Snxd6fP9aOyLWuNLr/GhgQXAC8bjHfehuW9fAM5WSvX8Lbtji7Erff73EgunJOpo92U82i9lX9LA20qpNSp8X0iAPK31IePzw4ATbjLXVUxO2rffMU4nH7W1i/o9PuMUfBrhastx+7FDfOCg/aiUciul1gLFwDuEK/lKrXVLlDisGI3nq4Csvo5Ra23ux18b+/FupZR5B91+/32MximJ2snmaq2nA+cDNyml5tmf1OHzJUfNcXRiTMD9wCjgZOAQ8Id+jcaglEoBXgRu1lpX259zwn6MEp+j9qPWulVrfTLhW/TNAsb3ZzzRdIxRKTUJuI1wrKcAmcBP+i/C7jklUTv2voxa6wPGv8XAy4R/GY+Yp0PGv8X9F6Glq5gcsW+11keMP5g24CHaT8v7LT6llJdwEnxaa/2S8bBj9mO0+Jy4H424KoHFwGmE2wXmTUnscVgxGs+nA2X9EONCo7WktdZNwGM4ZD92xSmJ2pH3ZVRKJSulUs3PgfOAjYRju9rY7Grglf6JMEJXMb0KXGWMZs8Gqmyn9n2mQ5/vy4T3oxnf5caMgBHAGGBVH8SjgEeAz7TWf7Q95Yj92FV8TtqPSqkcpVSG8XkQOJdwL30x8FVjs4770Ny3XwXeN85a+jrGLbaDsSLcQ7fvx37/e+mkv0czzQ/Co63bCPe4ftbf8RgxjSQ8kr4O2GTGRbiv9h6wHXgXyOzjuP5B+LS3mXAP7bquYiI8en2vsV83ADP7Kb6/Ge+/nvAfw0Db9j8z4tsKnN9H+3Au4bbGemCt8XGBU/bjUeJzzH4EpgCfGrFsBH5pPD6S8EFiB/A84DceDxhf7zCeH9mPMb5v7MeNwFO0zwzp87+XWD7kEnIhhHA4p7Q+hBBCdEEStRBCOJwkaiGEcDhJ1EII4XCSqIUQwuEkUQshhMNJohZCCIf7/11+exo7SmZoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib.pyplot import plot\n",
    "\n",
    "plot(losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "64OtEmyUWUiM"
   },
   "source": [
    "You have successfully trained your model to help automatically answer questions. Try asking it a question about a story."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "eFniMzpp1bpz",
    "outputId": "0ce0e2a3-3d6a-4e6e-adff-d0c16b622c9a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What is south of the bedroom? Garden\n"
     ]
    }
   ],
   "source": [
    "question, text = 'What is south of the bedroom?','The hallway is south of the garden. The garden is south of the bedroom.'\n",
    "input_dict = tokenizer(text, question, return_tensors='tf')\n",
    "outputs = model(input_dict)\n",
    "start_logits = outputs[0]\n",
    "end_logits = outputs[1]\n",
    "\n",
    "all_tokens = tokenizer.convert_ids_to_tokens(input_dict[\"input_ids\"].numpy()[0])\n",
    "answer = ' '.join(all_tokens[tf.math.argmax(start_logits, 1)[0] : tf.math.argmax(end_logits, 1)[0]+1])\n",
    "print(question, answer.capitalize())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f07OtnCpuKFa"
   },
   "source": [
    "We just implemented our first QA model in TensorFlow. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9UaM5pY9u8EW"
   },
   "source": [
    "## 5 - PyTorch implementation\n",
    "\n",
    "[PyTorch](https://pytorch.org/) is an open-source machine learning framework developed by Facebook's AI Research lab that can be used for computer vision and natural language processing. Iit is quite compatible with the bAbI dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nD9akXoXxMjd"
   },
   "source": [
    "### Train and test dataset\n",
    "\n",
    "Go ahead and try creating a train and test dataset by importing PyTorch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "JxMYWSG173ch"
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "columns_to_return = ['input_ids','attention_mask', 'start_positions', 'end_positions']\n",
    "train_ds.set_format(type='pt', columns=columns_to_return)\n",
    "test_ds.set_format(type='pt', columns=columns_to_return)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OeuzZKlPHAAQ"
   },
   "source": [
    "For the accuracy metrics for the PyTorch implementation, you will change things up a bit and use the [F1 score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) for start and end indicies over the entire test dataset as the loss functions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "aD9tDpZfJsIB"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "\n",
    "def compute_metrics(pred):\n",
    "    start_labels = pred.label_ids[0]\n",
    "    start_preds = pred.predictions[0].argmax(-1)\n",
    "    end_labels = pred.label_ids[1]\n",
    "    end_preds = pred.predictions[1].argmax(-1)\n",
    "    \n",
    "    f1_start = f1_score(start_labels, start_preds, average='macro')\n",
    "    f1_end = f1_score(end_labels, end_preds, average='macro')\n",
    "    \n",
    "    return {\n",
    "        'f1_start': f1_start,\n",
    "        'f1_end': f1_end,\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "laX5cYQRHMXb"
   },
   "source": [
    "### Training\n",
    "\n",
    "Now it is time to load a pre-trained model. \n",
    "\n",
    "**Note:** You will be using the DistilBERT instead of TFDistilBERT for a PyTorch implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "del model # We delete the tensorflow model to avoid memory issues"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before running the cell below combine the model parts by running the Linux/macOS command in the `model/pytorch` directory of your local computer: `cat pytorch_model.bin.* > pytorch_model.bin`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YXFCsNcY79jx",
    "outputId": "09af112f-e1e9-4a47-c988-37ee2a068df2"
   },
   "outputs": [],
   "source": [
    "from transformers import DistilBertForQuestionAnswering\n",
    "\n",
    "pytorch_model = DistilBertForQuestionAnswering.from_pretrained(\"model/pytorch\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xCUdMmCxHP6_"
   },
   "source": [
    "Instead of a custom training loop, you will use the [🤗 Trainer](https://huggingface.co/transformers/main_classes/trainer.html), which contains a basic training loop and is fairly easy to implement in PyTorch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 329
    },
    "id": "1htmS3TV-2Bk",
    "outputId": "cc21bfbb-da09-47f9-ee16-7db0096d35e7"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='375' max='375' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [375/375 00:13, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>50</td>\n",
       "      <td>1.553700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>100</td>\n",
       "      <td>0.783700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>150</td>\n",
       "      <td>0.601800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>200</td>\n",
       "      <td>0.435200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>250</td>\n",
       "      <td>0.358600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>300</td>\n",
       "      <td>0.295800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>350</td>\n",
       "      <td>0.413400</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=375, training_loss=0.6163130976359049, metrics={'train_runtime': 13.9401, 'train_samples_per_second': 26.901, 'total_flos': 31058547624000, 'epoch': 3.0})"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import Trainer, TrainingArguments\n",
    "\n",
    "training_args = TrainingArguments(\n",
    "    output_dir='results',          # output directory\n",
    "    overwrite_output_dir=True,\n",
    "    num_train_epochs=3,              # total number of training epochs\n",
    "    per_device_train_batch_size=8,  # batch size per device during training\n",
    "    per_device_eval_batch_size=8,   # batch size for evaluation\n",
    "    warmup_steps=20,                # number of warmup steps for learning rate scheduler\n",
    "    weight_decay=0.01,               # strength of weight decay\n",
    "    logging_dir=None,            # directory for storing logs\n",
    "    logging_steps=50\n",
    ")\n",
    "\n",
    "trainer = Trainer(\n",
    "    model=pytorch_model,                 # the instantiated 🤗 Transformers model to be trained\n",
    "    args=training_args,                  # training arguments, defined above\n",
    "    train_dataset=train_ds,         # training dataset\n",
    "    eval_dataset=test_ds,\n",
    "    compute_metrics=compute_metrics             # evaluation dataset\n",
    ")\n",
    "\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 207
    },
    "id": "lDzbm7vzAiPJ",
    "outputId": "7cd62f51-a04b-4583-bc0e-e459813d3103"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <style>\n",
       "            /* Turns off some styling */\n",
       "            progress {\n",
       "                /* gets rid of default border in Firefox and Opera. */\n",
       "                border: none;\n",
       "                /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "                background-size: auto;\n",
       "            }\n",
       "        </style>\n",
       "      \n",
       "      <progress value='125' max='125' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [125/125 00:02]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "{'eval_loss': 0.32361623644828796,\n",
       " 'eval_f1_start': 0.7895575553055868,\n",
       " 'eval_f1_end': 0.7883126550868487,\n",
       " 'eval_runtime': 2.0277,\n",
       " 'eval_samples_per_second': 493.171,\n",
       " 'epoch': 3.0}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.evaluate(test_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QAgrcs2pHvVu"
   },
   "source": [
    "Now it is time to ask your PyTorch model a question.\n",
    "* Before testing your model with a question, you can tell PyTorch to send your model and inputs to the GPU if your machine has one, or the CPU if it does not. \n",
    "* You can then proceed to tokenize your input and create PyTorch tensors and send them to your device. \n",
    "* The rest of the pipeline is relatively similar to the one implemented in TensorFlow.   \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "yfBe9AFABqUr",
    "outputId": "b5ca6039-8ce2-4e75-9161-1c96a0f39425"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What is east of the hallway? Kitchen\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n",
    "\n",
    "pytorch_model.to(device)\n",
    "\n",
    "question, text = 'What is east of the hallway?','The kitchen is east of the hallway. The garden is south of the bedroom.'\n",
    "\n",
    "input_dict = tokenizer(text, question, return_tensors='pt')\n",
    "\n",
    "input_ids = input_dict['input_ids'].to(device)\n",
    "attention_mask = input_dict['attention_mask'].to(device)\n",
    "\n",
    "outputs = pytorch_model(input_ids, attention_mask=attention_mask)\n",
    "\n",
    "start_logits = outputs[0]\n",
    "end_logits = outputs[1]\n",
    "\n",
    "all_tokens = tokenizer.convert_ids_to_tokens(input_dict[\"input_ids\"].numpy()[0])\n",
    "answer = ' '.join(all_tokens[torch.argmax(start_logits, 1)[0] : torch.argmax(end_logits, 1)[0]+1])\n",
    "\n",
    "print(question, answer.capitalize())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eGzuHkMZ4q9I"
   },
   "source": [
    " \n",
    "We have accomplished the following:\n",
    "* Perform extractive question qnswering \n",
    "* Fine-tune a pre-trained transformer model to a custom dataset\n",
    "* Implement a QA model in TensorFlow and in PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G8tAV-584vKE"
   },
   "source": [
    "### Points to remember\n",
    "- Transformer models are often trained by tokenizers that split words into subwords.\n",
    "  - Before processing, it is important that you align the start and end indices with the tokens associated with the target answer word.\n",
    "- PyTorch is a relatively light and easy to implement framework that can make rapid prototyping easier, while TensorFlow has advantages in scaling and is more widely used in production.\n",
    "  - `tf.GradientTape` allows you to build custom training loops in TensorFlow.\n",
    "  - The `Trainer` API in PyTorch gives you a basic training loop that is compatible with 🤗 models and datasets."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "QA-dataset.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "013f041c3e0b4e35bf2432fc345cb7bf": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1e6c02317171453cbd3d4d665879b0d4",
      "max": 1000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_f0e34f2bf626434fa73f0def26b3d1a5",
      "value": 1000
     }
    },
    "07aaa9b79a744856b19d723370d6e588": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_b39b85d8cb05418aa92e8476ad02f755",
       "IPY_MODEL_0a8534ac52af4d48ad82b66463ad08c3"
      ],
      "layout": "IPY_MODEL_afedd2328cf141f78775e4cfa7758267"
     }
    },
    "0a8534ac52af4d48ad82b66463ad08c3": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8cfbd3b14b23417993270f851a2d8ff9",
      "placeholder": "​",
      "style": "IPY_MODEL_31fc08a1e7e04f6b9b3ea400ccfaea75",
      "value": " 1000/1000 [01:40&lt;00:00,  9.90ex/s]"
     }
    },
    "1e6c02317171453cbd3d4d665879b0d4": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1f2773e3e80c4dd8b6b26e171bf33bc7": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "31fc08a1e7e04f6b9b3ea400ccfaea75": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "32a5c82c7a9845c09c11bb4e30c2f1aa": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "364ba960eb474c9084cc71851594d345": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "39029f730ae140c7902fca6dac5361ad": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "393697738e724e9fad4d163de0a77840": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "3abb36da57c841838867c56e2a3a325b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "3dab28395f3f475d8242e4d4d45ed059": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_63b4ebafcead4c0784b5511219a6a198",
      "placeholder": "​",
      "style": "IPY_MODEL_58718e12f1b7459989ab5296846c4be6",
      "value": " 1000/1000 [00:10&lt;00:00, 97.35ex/s]"
     }
    },
    "44b7bea3e09d4e5684921c66dd4c7514": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_7e1325e57bf9417e93d7ef180794ab3c",
       "IPY_MODEL_3dab28395f3f475d8242e4d4d45ed059"
      ],
      "layout": "IPY_MODEL_6af3ec5091d74bd1a95bf02a87dd240b"
     }
    },
    "4d9152a30e824931983a425ee6d607a6": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_013f041c3e0b4e35bf2432fc345cb7bf",
       "IPY_MODEL_ef4e12f29f1e458f811a400faf21bdcc"
      ],
      "layout": "IPY_MODEL_1f2773e3e80c4dd8b6b26e171bf33bc7"
     }
    },
    "4f5b06c3a5e44c6cade5bf83634d9f69": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "568f11b4462f4b4e95f3ad5947bb275e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "58718e12f1b7459989ab5296846c4be6": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "5b6dbe662ca24834b7678638e101e1ff": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "63b4ebafcead4c0784b5511219a6a198": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6af3ec5091d74bd1a95bf02a87dd240b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "723acefae33d448199fa5c1a9ec3f246": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_77273c2e4b4e4e4c8ee4b6b344749518",
       "IPY_MODEL_f0ac3b9b8f664479940c6ee18fc2f13e"
      ],
      "layout": "IPY_MODEL_32a5c82c7a9845c09c11bb4e30c2f1aa"
     }
    },
    "77273c2e4b4e4e4c8ee4b6b344749518": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e592db98c0c34c5e800f5d7b6d3c099e",
      "max": 1000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_393697738e724e9fad4d163de0a77840",
      "value": 1000
     }
    },
    "7e1325e57bf9417e93d7ef180794ab3c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7fb1118c0b4443b6b6dbb5803e9ec2e8",
      "max": 1000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_ca722dcd857c433c9058585e31a1673d",
      "value": 1000
     }
    },
    "7fb1118c0b4443b6b6dbb5803e9ec2e8": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7fefe9e1121a43558d773500aef8935c": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "863c5ce96db84e3da162072c9a13c913": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8968319cdaca476fb15c11a388dce39a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_a725734893004a45b61194f649f5f602",
       "IPY_MODEL_c4a24656d67844e995d3b8e175c6c497"
      ],
      "layout": "IPY_MODEL_863c5ce96db84e3da162072c9a13c913"
     }
    },
    "89fdda6e6688476495ca297bfe010bf8": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8b961844b5004905922531bd805a9d57": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8cfbd3b14b23417993270f851a2d8ff9": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "929946fdfaa04cf59d3b31cf92fc08d1": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_cda72c45821a4eb89f1a3ab5510b26d3",
      "placeholder": "​",
      "style": "IPY_MODEL_89fdda6e6688476495ca297bfe010bf8",
      "value": " 1000/1000 [00:08&lt;00:00, 123.32ex/s]"
     }
    },
    "a725734893004a45b61194f649f5f602": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_afc33fa78b5d440192c435bfca6f7914",
      "max": 1000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_4f5b06c3a5e44c6cade5bf83634d9f69",
      "value": 1000
     }
    },
    "aa5c0d374889482697fc0f7ce9c81afe": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "afc33fa78b5d440192c435bfca6f7914": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "afedd2328cf141f78775e4cfa7758267": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b39b85d8cb05418aa92e8476ad02f755": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8b961844b5004905922531bd805a9d57",
      "max": 1000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_3abb36da57c841838867c56e2a3a325b",
      "value": 1000
     }
    },
    "b4c6a18610734036a16a14a43174c52e": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c42644a4e6184a1cbdb2b453b5dbb7d6": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_e8f1abd85f3e49f991d4c1312ffd416b",
       "IPY_MODEL_929946fdfaa04cf59d3b31cf92fc08d1"
      ],
      "layout": "IPY_MODEL_364ba960eb474c9084cc71851594d345"
     }
    },
    "c4a24656d67844e995d3b8e175c6c497": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b4c6a18610734036a16a14a43174c52e",
      "placeholder": "​",
      "style": "IPY_MODEL_f37bd346f8614fec92d6c5b5e9b66d2f",
      "value": " 1000/1000 [01:41&lt;00:00,  9.86ex/s]"
     }
    },
    "ca722dcd857c433c9058585e31a1673d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "cda72c45821a4eb89f1a3ab5510b26d3": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e592db98c0c34c5e800f5d7b6d3c099e": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e8f1abd85f3e49f991d4c1312ffd416b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ff444b253e9a40e5bec755926d83740f",
      "max": 1000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_aa5c0d374889482697fc0f7ce9c81afe",
      "value": 1000
     }
    },
    "ef4e12f29f1e458f811a400faf21bdcc": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_39029f730ae140c7902fca6dac5361ad",
      "placeholder": "​",
      "style": "IPY_MODEL_5b6dbe662ca24834b7678638e101e1ff",
      "value": " 1000/1000 [01:25&lt;00:00, 11.68ex/s]"
     }
    },
    "f0ac3b9b8f664479940c6ee18fc2f13e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7fefe9e1121a43558d773500aef8935c",
      "placeholder": "​",
      "style": "IPY_MODEL_568f11b4462f4b4e95f3ad5947bb275e",
      "value": " 1000/1000 [01:24&lt;00:00, 11.77ex/s]"
     }
    },
    "f0e34f2bf626434fa73f0def26b3d1a5": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "f37bd346f8614fec92d6c5b5e9b66d2f": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ff444b253e9a40e5bec755926d83740f": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}