ProjectCode/BERT_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XqGbCc8C4UAn"
   },
   "source": [
    "# DeepPavlov: Transfer Learning with BERT\n",
    "### Valerie Nayak\n",
    "### November 18, 2019\n",
    "\n",
    "Notebook from TowardsDataScience\n",
    "\n",
    "Today we will cover following tasks:\n",
    "* classification\n",
    "* tagging (Named Enitity Recognition)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "H2WFTCgH4UAp"
   },
   "source": [
    "## BERT input representation\n",
    "Text preprocessing for BERT relies on tokenizing text on subtokens (or WordPieces). Then BERT internally represents each subtoken as sum of three vectors:\n",
    "* subtoken embedding\n",
    "* segment embedding\n",
    "* position embedding\n",
    "\n",
    "<img src=\"https://github.com/deepmipt/dp_tutorials/blob/master/img/BERT_input.png?raw=1\" width=\"75%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2yMoFE9R4UAq"
   },
   "source": [
    "## BERT for text classification\n",
    "When we want to use BERT model for text classification task we can train only one dense layer on top of the output from the last BERT Transformer layer for special `[CLS]` token.\n",
    "\n",
    "<img src=\"https://github.com/deepmipt/dp_tutorials/blob/master/img/BERT_classification.png?raw=1\" width=\"75%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SUKEWk0BDLWA"
   },
   "source": [
    "Install DeepPavlov library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 973
    },
    "colab_type": "code",
    "id": "Mwzl_A0MDJ2v",
    "outputId": "1502c612-5fb7-44fb-9717-128f8d3c5462"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting deeppavlov\n",
      "  Downloading https://files.pythonhosted.org/packages/d7/9d/453d101981b293441be889ba91d6983abdeeb2b3abc070b47a4044f6a64b/deeppavlov-0.7.1-py3-none-any.whl (735kB)\n",
      "Collecting pytelegrambotapi==3.6.6 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/25/5e/9711642455c4e17b1202d4f6403ede0fef37fc145038aee7193f3b24445e/pyTelegramBotAPI-3.6.6.tar.gz (49kB)\n",
      "Collecting tqdm==4.32.2 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/9f/3d/7a6b68b631d2ab54975f3a4863f3c4e9b26445353264ef01f465dc9b0208/tqdm-4.32.2-py2.py3-none-any.whl (50kB)\n",
      "Collecting numpy==1.16.4 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/20/ed/e036d31a9b2c750f270cbb1cfc1c0f94ac78ae504eea7eec3267be4e294a/numpy-1.16.4-cp36-cp36m-win_amd64.whl (11.9MB)\n",
      "Collecting h5py==2.9.0 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/01/1e/115c4403544a91001d9c618748b2e8786db45544e36b8a6cf3c525e9b57f/h5py-2.9.0-cp36-cp36m-win_amd64.whl (2.4MB)\n",
      "Collecting pymorphy2-dicts-ru (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/7c/9b/358faaff410f65a4ad159275e897b5956dcb20576c5b8e764b971c1634d7/pymorphy2_dicts_ru-2.4.404381.4453942-py2.py3-none-any.whl (8.0MB)\n",
      "Collecting scipy==1.3.0 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/9e/fd/9a995b7fc18c6c17ce570b3cfdabffbd2718e4f1830e94777c4fd66e1179/scipy-1.3.0-cp36-cp36m-win_amd64.whl (30.5MB)\n",
      "Collecting Cython==0.29.12 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/2c/f1/57f1413b5f5a16d67e63401227894b9667505c2727631a14af708892128a/Cython-0.29.12-cp36-cp36m-win_amd64.whl (1.7MB)\n",
      "Collecting overrides==1.9 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/de/55/3100c6d14c1ed177492fcf8f07c4a7d2d6c996c0a7fc6a9a0a41308e7eec/overrides-1.9.tar.gz\n",
      "Collecting pandas==0.24.2 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/d0/4e/9db3468e504ac9aeadb37eb32bcf0a74d063d24ad1471104bd8a7ba20c97/pandas-0.24.2-cp36-cp36m-win_amd64.whl (8.8MB)\n",
      "Collecting fastapi==0.38.1 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/37/59/1a42dde38f1ae2a7a318947e54fabd1fc02ac423ecc91957c417065e7cc6/fastapi-0.38.1-py3-none-any.whl (160kB)\n",
      "Collecting pymorphy2==0.8 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/a3/33/fff9675c68b5f6c63ec8c6e6ff57827dda28a1fa5b2c2d727dffff92dd47/pymorphy2-0.8-py2.py3-none-any.whl (46kB)\n",
      "Collecting pyopenssl==19.0.0 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl (53kB)\n",
      "Collecting scikit-learn==0.21.2 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/a9/bc/18663f6d75838b73353ba49fabd631347e68470ec9e623d7b3f3ccd4f426/scikit_learn-0.21.2-cp36-cp36m-win_amd64.whl (5.9MB)\n",
      "Collecting keras==2.2.4 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/5e/10/aa32dad071ce52b5502266b5c659451cfd6ffcbf14e6c8c4f16c0ff5aaab/Keras-2.2.4-py2.py3-none-any.whl (312kB)\n",
      "Collecting uvicorn==0.9.0 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/5a/a9/2ab492155d9b76cf109c2370e201822ba3c7f4aed85f5a1b4d22907e7206/uvicorn-0.9.0.tar.gz\n",
      "Collecting rusenttokenize==0.0.5 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/25/4c/a2f00be5def774a3df2e5387145f1cb54e324607ec4a7e23f573645946e7/rusenttokenize-0.0.5-py3-none-any.whl\n",
      "Collecting nltk==3.2.5 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/cc/87/76e691bbf1759ad6af5831649aae6a8d2fa184a1bcc71018ca6300399e5f/nltk-3.2.5.tar.gz (1.2MB)\n",
      "Collecting fuzzywuzzy==0.17.0 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/d8/f1/5a267addb30ab7eaa1beab2b9323073815da4551076554ecc890a3595ec9/fuzzywuzzy-0.17.0-py2.py3-none-any.whl\n",
      "Collecting requests==2.22.0 (from deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)\n",
      "Requirement already satisfied: six in c:\\users\\valerie\\anaconda3\\lib\\site-packages (from pytelegrambotapi==3.6.6->deeppavlov) (1.11.0)\n",
      "Requirement already satisfied: python-dateutil>=2.5.0 in c:\\users\\valerie\\anaconda3\\lib\\site-packages (from pandas==0.24.2->deeppavlov) (2.7.3)\n",
      "Requirement already satisfied: pytz>=2011k in c:\\users\\valerie\\anaconda3\\lib\\site-packages (from pandas==0.24.2->deeppavlov) (2018.5)\n",
      "Collecting starlette<=0.12.8,>=0.11.1 (from fastapi==0.38.1->deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/97/4a/90a6a8685fcbdcf544a2293d0d0211ecba3cd7f309b831499d5c895383cb/starlette-0.12.8.tar.gz (45kB)\n",
      "Collecting pydantic<=0.32.2,>=0.32.2 (from fastapi==0.38.1->deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/02/cf/80e08943ac1af41e373f478418a76cac232d3b36ea827c2f89dbbef9a698/pydantic-0.32.2-py36.py37.py38-none-any.whl (63kB)\n",
      "Collecting dawg-python>=0.7 (from pymorphy2==0.8->deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/6a/84/ff1ce2071d4c650ec85745766c0047ccc3b5036f1d03559fd46bb38b5eeb/DAWG_Python-0.7.2-py2.py3-none-any.whl\n",
      "Collecting pymorphy2-dicts<3.0,>=2.4 (from pymorphy2==0.8->deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/02/51/2465fd4f72328ab50877b54777764d928da8cb15b74e2680fc1bd8cb3173/pymorphy2_dicts-2.4.393442.3710985-py2.py3-none-any.whl (7.1MB)\n",
      "Collecting docopt>=0.6 (from pymorphy2==0.8->deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\n",
      "Requirement already satisfied: cryptography>=2.3 in c:\\users\\valerie\\anaconda3\\lib\\site-packages (from pyopenssl==19.0.0->deeppavlov) (2.7)\n",
      "Collecting joblib>=0.11 (from scikit-learn==0.21.2->deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/8f/42/155696f85f344c066e17af287359c9786b436b1bf86029bb3411283274f3/joblib-0.14.0-py2.py3-none-any.whl (294kB)\n",
      "Collecting keras-preprocessing>=1.0.5 (from keras==2.2.4->deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/28/6a/8c1f62c37212d9fc441a7e26736df51ce6f0e38455816445471f10da4f0a/Keras_Preprocessing-1.1.0-py2.py3-none-any.whl (41kB)\n",
      "Requirement already satisfied: pyyaml in c:\\users\\valerie\\anaconda3\\lib\\site-packages (from keras==2.2.4->deeppavlov) (3.12)\n",
      "Collecting keras-applications>=1.0.6 (from keras==2.2.4->deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/71/e3/19762fdfc62877ae9102edf6342d71b28fbfd9dea3d2f96a882ce099b03f/Keras_Applications-1.0.8-py3-none-any.whl (50kB)\n",
      "Collecting click==7.* (from uvicorn==0.9.0->deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)\n",
      "Collecting h11==0.8.* (from uvicorn==0.9.0->deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/f9/f3/8e4cf5fa1a3d8bda942a0b1cf92f87815494216fd439f82eb99073141ba0/h11-0.8.1-py2.py3-none-any.whl (55kB)\n",
      "Collecting websockets==8.* (from uvicorn==0.9.0->deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/78/48/9e5bb47b0c26bf02ffcff691a0ef0b1b1ef7824e9b240bad3f494a886897/websockets-8.1-cp36-cp36m-win_amd64.whl (66kB)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\\users\\valerie\\anaconda3\\lib\\site-packages (from requests==2.22.0->deeppavlov) (3.0.4)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\\users\\valerie\\anaconda3\\lib\\site-packages (from requests==2.22.0->deeppavlov) (1.23)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\valerie\\anaconda3\\lib\\site-packages (from requests==2.22.0->deeppavlov) (2019.9.11)\n",
      "Requirement already satisfied: idna<2.9,>=2.5 in c:\\users\\valerie\\anaconda3\\lib\\site-packages (from requests==2.22.0->deeppavlov) (2.7)\n",
      "Collecting dataclasses>=0.6; python_version < \"3.7\" (from pydantic<=0.32.2,>=0.32.2->fastapi==0.38.1->deeppavlov)\n",
      "  Downloading https://files.pythonhosted.org/packages/e1/d2/6f02df2616fd4016075f60157c7a0452b38d8f7938ae94343911e0fb0b09/dataclasses-0.7-py3-none-any.whl\n",
      "Requirement already satisfied: asn1crypto>=0.21.0 in c:\\users\\valerie\\anaconda3\\lib\\site-packages (from cryptography>=2.3->pyopenssl==19.0.0->deeppavlov) (0.24.0)\n",
      "Requirement already satisfied: cffi!=1.11.3,>=1.8 in c:\\users\\valerie\\anaconda3\\lib\\site-packages (from cryptography>=2.3->pyopenssl==19.0.0->deeppavlov) (1.11.5)\n",
      "Requirement already satisfied: pycparser in c:\\users\\valerie\\anaconda3\\lib\\site-packages (from cffi!=1.11.3,>=1.8->cryptography>=2.3->pyopenssl==19.0.0->deeppavlov) (2.18)\n",
      "Building wheels for collected packages: pytelegrambotapi, overrides, uvicorn, nltk, starlette, docopt\n",
      "  Running setup.py bdist_wheel for pytelegrambotapi: started\n",
      "  Running setup.py bdist_wheel for pytelegrambotapi: finished with status 'done'\n",
      "  Stored in directory: C:\\Users\\Valerie\\AppData\\Local\\pip\\Cache\\wheels\\ae\\69\\d7\\26f1fb04ac4d4c95bff643cea765a8e91c4348da25b4744e08\n",
      "  Running setup.py bdist_wheel for overrides: started\n",
      "  Running setup.py bdist_wheel for overrides: finished with status 'done'\n",
      "  Stored in directory: C:\\Users\\Valerie\\AppData\\Local\\pip\\Cache\\wheels\\8d\\52\\86\\e5a83b1797e7d263b458d2334edd2704c78508b3eea9323718\n",
      "  Running setup.py bdist_wheel for uvicorn: started\n",
      "  Running setup.py bdist_wheel for uvicorn: finished with status 'done'\n",
      "  Stored in directory: C:\\Users\\Valerie\\AppData\\Local\\pip\\Cache\\wheels\\19\\e5\\d1\\a50d405d3bb18fac538ef9606ed9b6cd5efb6e06b6de834507\n",
      "  Running setup.py bdist_wheel for nltk: started\n",
      "  Running setup.py bdist_wheel for nltk: finished with status 'done'\n",
      "  Stored in directory: C:\\Users\\Valerie\\AppData\\Local\\pip\\Cache\\wheels\\da\\8c\\38\\a8a36581975f8d03275c49960019f955e4d19fd14ae7e42d3d\n",
      "  Running setup.py bdist_wheel for starlette: started\n",
      "  Running setup.py bdist_wheel for starlette: finished with status 'done'\n",
      "  Stored in directory: C:\\Users\\Valerie\\AppData\\Local\\pip\\Cache\\wheels\\fd\\a0\\1d\\17eb20c5742e3236799a7883e56325823d57fcd8ce2a0c9348\n",
      "  Running setup.py bdist_wheel for docopt: started\n",
      "  Running setup.py bdist_wheel for docopt: finished with status 'done'\n",
      "  Stored in directory: C:\\Users\\Valerie\\AppData\\Local\\pip\\Cache\\wheels\\9b\\04\\dd\\7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\n",
      "Successfully built pytelegrambotapi overrides uvicorn nltk starlette docopt\n",
      "Installing collected packages: requests, pytelegrambotapi, tqdm, numpy, h5py, pymorphy2-dicts-ru, scipy, Cython, overrides, pandas, starlette, dataclasses, pydantic, fastapi, dawg-python, pymorphy2-dicts, docopt, pymorphy2, pyopenssl, joblib, scikit-learn, keras-preprocessing, keras-applications, keras, click, h11, websockets, uvicorn, rusenttokenize, nltk, fuzzywuzzy, deeppavlov\n",
      "  Found existing installation: requests 2.19.1\n",
      "    Uninstalling requests-2.19.1:\n",
      "      Successfully uninstalled requests-2.19.1\n",
      "  Found existing installation: tqdm 4.23.4\n",
      "    Uninstalling tqdm-4.23.4:\n",
      "      Successfully uninstalled tqdm-4.23.4\n",
      "  Found existing installation: numpy 1.14.5\n",
      "    Uninstalling numpy-1.14.5:\n",
      "      Successfully uninstalled numpy-1.14.5\n",
      "  Found existing installation: h5py 2.8.0\n",
      "    Uninstalling h5py-2.8.0:\n",
      "      Successfully uninstalled h5py-2.8.0\n",
      "  Found existing installation: scipy 1.1.0\n",
      "    Uninstalling scipy-1.1.0:\n",
      "      Successfully uninstalled scipy-1.1.0\n",
      "  Found existing installation: Cython 0.28.3\n",
      "    Uninstalling Cython-0.28.3:\n",
      "      Successfully uninstalled Cython-0.28.3\n",
      "  Found existing installation: pandas 0.23.3\n",
      "    Uninstalling pandas-0.23.3:\n",
      "      Successfully uninstalled pandas-0.23.3\n",
      "  Found existing installation: pyOpenSSL 18.0.0\n",
      "    Uninstalling pyOpenSSL-18.0.0:\n",
      "      Successfully uninstalled pyOpenSSL-18.0.0\n",
      "  Found existing installation: scikit-learn 0.19.1\n",
      "    Uninstalling scikit-learn-0.19.1:\n",
      "      Successfully uninstalled scikit-learn-0.19.1\n",
      "  Found existing installation: click 6.7\n",
      "    Uninstalling click-6.7:\n",
      "      Successfully uninstalled click-6.7\n",
      "  Found existing installation: nltk 3.3\n",
      "    Uninstalling nltk-3.3:\n",
      "      Successfully uninstalled nltk-3.3\n",
      "Successfully installed Cython-0.29.12 click-7.0 dataclasses-0.7 dawg-python-0.7.2 deeppavlov-0.7.1 docopt-0.6.2 fastapi-0.38.1 fuzzywuzzy-0.17.0 h11-0.8.1 h5py-2.9.0 joblib-0.14.0 keras-2.2.4 keras-applications-1.0.8 keras-preprocessing-1.1.0 nltk-3.2.5 numpy-1.16.4 overrides-1.9 pandas-0.24.2 pydantic-0.32.2 pymorphy2-0.8 pymorphy2-dicts-2.4.393442.3710985 pymorphy2-dicts-ru-2.4.404381.4453942 pyopenssl-19.0.0 pytelegrambotapi-3.6.6 requests-2.22.0 rusenttokenize-0.0.5 scikit-learn-0.21.2 scipy-1.3.0 starlette-0.12.8 tqdm-4.32.2 uvicorn-0.9.0 websockets-8.1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "thinc 6.10.2 requires pathlib<2.0.0,>=1.0.0, which is not installed.\n",
      "spacy 2.0.11 requires pathlib, which is not installed.\n",
      "distributed 1.22.0 requires msgpack, which is not installed.\n",
      "spacy 2.0.11 has requirement regex==2017.4.5, but you'll have regex 2017.11.9 which is incompatible.\n",
      "pyrxnlp 0.0.0 has requirement requests==2.9.1, but you'll have requests 2.22.0 which is incompatible.\n",
      "You are using pip version 10.0.1, however version 19.3.1 is available.\n",
      "You should consider upgrading via the 'python -m pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "! pip install deeppavlov"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Q5cA1hCADs5g"
   },
   "source": [
    "Install requirements for BERT-based classification model trained to detect insults in [Social Commentary](https://www.kaggle.com/c/detecting-insults-in-social-commentary):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 956
    },
    "colab_type": "code",
    "id": "OkUU9OTHDrtY",
    "outputId": "55f9cc52-8994-42d2-b204-da4672e3c120"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "email-validator not installed, email fields will be treated as str.\n",
      "To install, run: pip install email-validator\n",
      "2019-11-18 15:41:15.934 INFO in 'deeppavlov.core.common.file'['file'] at line 30: Interpreting 'insults_kaggle_bert' as '/usr/local/lib/python3.6/dist-packages/deeppavlov/configs/classifiers/insults_kaggle_bert.json'\n",
      "Collecting tensorflow==1.14.0\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/de/f0/96fb2e0412ae9692dbf400e5b04432885f677ad6241c088ccc5fe7724d69/tensorflow-1.14.0-cp36-cp36m-manylinux1_x86_64.whl (109.2MB)\n",
      "\u001b[K     |████████████████████████████████| 109.2MB 101kB/s \n",
      "\u001b[?25hRequirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (1.12.0)\n",
      "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (1.15.0)\n",
      "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (0.33.6)\n",
      "Requirement already satisfied: gast>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (0.2.2)\n",
      "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (3.10.0)\n",
      "Collecting tensorboard<1.15.0,>=1.14.0\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/91/2d/2ed263449a078cd9c8a9ba50ebd50123adf1f8cfbea1492f9084169b89d9/tensorboard-1.14.0-py3-none-any.whl (3.1MB)\n",
      "\u001b[K     |████████████████████████████████| 3.2MB 38.6MB/s \n",
      "\u001b[?25hRequirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (0.8.0)\n",
      "Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (0.8.1)\n",
      "Requirement already satisfied: keras-preprocessing>=1.0.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (1.1.0)\n",
      "Collecting tensorflow-estimator<1.15.0rc0,>=1.14.0rc0\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/3c/d5/21860a5b11caf0678fbc8319341b0ae21a07156911132e0e71bffed0510d/tensorflow_estimator-1.14.0-py2.py3-none-any.whl (488kB)\n",
      "\u001b[K     |████████████████████████████████| 491kB 30.5MB/s \n",
      "\u001b[?25hRequirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (1.1.0)\n",
      "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (1.11.2)\n",
      "Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (0.1.8)\n",
      "Requirement already satisfied: keras-applications>=1.0.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (1.0.8)\n",
      "Requirement already satisfied: numpy<2.0,>=1.14.5 in /usr/local/lib/python3.6/dist-packages (from tensorflow==1.14.0) (1.16.4)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.1->tensorflow==1.14.0) (41.4.0)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.1.1)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (0.16.0)\n",
      "Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras-applications>=1.0.6->tensorflow==1.14.0) (2.9.0)\n",
      "Installing collected packages: tensorboard, tensorflow-estimator, tensorflow\n",
      "  Found existing installation: tensorboard 1.15.0\n",
      "    Uninstalling tensorboard-1.15.0:\n",
      "      Successfully uninstalled tensorboard-1.15.0\n",
      "  Found existing installation: tensorflow-estimator 1.15.1\n",
      "    Uninstalling tensorflow-estimator-1.15.1:\n",
      "      Successfully uninstalled tensorflow-estimator-1.15.1\n",
      "  Found existing installation: tensorflow 1.15.0\n",
      "    Uninstalling tensorflow-1.15.0:\n",
      "      Successfully uninstalled tensorflow-1.15.0\n",
      "Successfully installed tensorboard-1.14.0 tensorflow-1.14.0 tensorflow-estimator-1.14.0\n",
      "Collecting git+https://github.com/deepmipt/bert.git@feat/multi_gpu\n",
      "  Cloning https://github.com/deepmipt/bert.git (to revision feat/multi_gpu) to /tmp/pip-req-build-uvahdowo\n",
      "  Running command git clone -q https://github.com/deepmipt/bert.git /tmp/pip-req-build-uvahdowo\n",
      "  Running command git checkout -b feat/multi_gpu --track origin/feat/multi_gpu\n",
      "  Switched to a new branch 'feat/multi_gpu'\n",
      "  Branch 'feat/multi_gpu' set up to track remote branch 'feat/multi_gpu' from 'origin'.\n",
      "Building wheels for collected packages: bert-dp\n",
      "  Building wheel for bert-dp (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for bert-dp: filename=bert_dp-1.0-cp36-none-any.whl size=23580 sha256=e60b9029137ade8c6051b8ad6866fcf9af580a9c5ece5e82031cd2a62cc101c3\n",
      "  Stored in directory: /tmp/pip-ephem-wheel-cache-1q8teks0/wheels/1e/41/94/886107eaf932532594886fd8bfc9cb9d4db632e94add49d326\n",
      "Successfully built bert-dp\n",
      "Installing collected packages: bert-dp\n",
      "Successfully installed bert-dp-1.0\n"
     ]
    }
   ],
   "source": [
    "! python -m deeppavlov install insults_kaggle_bert"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VPcVqL1yEuab"
   },
   "source": [
    "Download and interact with pre-trained model with CLI:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "VI_aYO38E0PN",
    "outputId": "b392df43-35c1-4b64-eb4e-f9970279ad89"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "email-validator not installed, email fields will be treated as str.\n",
      "To install, run: pip install email-validator\n",
      "2019-11-18 15:42:02.410 INFO in 'deeppavlov.core.common.file'['file'] at line 30: Interpreting 'insults_kaggle_bert' as '/usr/local/lib/python3.6/dist-packages/deeppavlov/configs/classifiers/insults_kaggle_bert.json'\n",
      "2019-11-18 15:42:03.505 INFO in 'deeppavlov.core.data.utils'['utils'] at line 64: Downloading from http://files.deeppavlov.ai/deeppavlov_data/bert/cased_L-12_H-768_A-12.zip?config=insults_kaggle_bert to /root/.deeppavlov/downloads/cased_L-12_H-768_A-12.zip\n",
      "100% 404M/404M [01:36<00:00, 4.20MB/s]\n",
      "2019-11-18 15:43:39.720 INFO in 'deeppavlov.core.data.utils'['utils'] at line 216: Extracting /root/.deeppavlov/downloads/cased_L-12_H-768_A-12.zip archive into /root/.deeppavlov/downloads/bert_models\n",
      "2019-11-18 15:43:44.100 INFO in 'deeppavlov.core.data.utils'['utils'] at line 64: Downloading from http://files.deeppavlov.ai/deeppavlov_data/classifiers/insults_kaggle_v3.tar.gz?config=insults_kaggle_bert to /root/.deeppavlov/models/insults_kaggle_v3.tar.gz\n",
      "100% 804M/804M [02:40<00:00, 5.02MB/s]\n",
      "2019-11-18 15:46:24.169 INFO in 'deeppavlov.core.data.utils'['utils'] at line 216: Extracting /root/.deeppavlov/models/insults_kaggle_v3.tar.gz archive into /root/.deeppavlov/models/classifiers\n",
      "2019-11-18 15:46:32.532 INFO in 'deeppavlov.core.data.utils'['utils'] at line 64: Downloading from http://files.deeppavlov.ai/datasets/insults_data.tar.gz?config=insults_kaggle_bert to /root/.deeppavlov/insults_data.tar.gz\n",
      "100% 682k/682k [00:00<00:00, 770kB/s]\n",
      "2019-11-18 15:46:33.418 INFO in 'deeppavlov.core.data.utils'['utils'] at line 216: Extracting /root/.deeppavlov/insults_data.tar.gz archive into /root/.deeppavlov/downloads\n",
      "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
      "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
      "[nltk_data] Downloading package perluniprops to /root/nltk_data...\n",
      "[nltk_data]   Unzipping misc/perluniprops.zip.\n",
      "[nltk_data] Downloading package nonbreaking_prefixes to\n",
      "[nltk_data]     /root/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/nonbreaking_prefixes.zip.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/bert_dp/tokenization.py:125: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.\n",
      "\n",
      "2019-11-18 15:46:36.12 INFO in 'deeppavlov.core.data.simple_vocab'['simple_vocab'] at line 112: [loading vocabulary from /root/.deeppavlov/models/classifiers/insults_kaggle_v3/classes.dict]\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/deeppavlov/core/models/tf_model.py:38: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/deeppavlov/core/models/tf_model.py:223: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/deeppavlov/core/models/tf_model.py:223: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
      "\n",
      "Using TensorFlow backend.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/deeppavlov/core/models/tf_model.py:194: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/deeppavlov/models/bert/bert_classifier.py:83: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
      "\n",
      "2019-11-18 15:46:36.157984: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA\n",
      "2019-11-18 15:46:36.160817: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz\n",
      "2019-11-18 15:46:36.160984: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2ceaa00 executing computations on platform Host. Devices:\n",
      "2019-11-18 15:46:36.161008: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/deeppavlov/models/bert/bert_classifier.py:160: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/bert_dp/modeling.py:178: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.\n",
      "\n",
      "WARNING:tensorflow:\n",
      "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
      "For more information, please see:\n",
      "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
      "  * https://github.com/tensorflow/addons\n",
      "  * https://github.com/tensorflow/io (for I/O related ops)\n",
      "If you depend on functionality not listed there, please file an issue.\n",
      "\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/bert_dp/modeling.py:366: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/bert_dp/modeling.py:680: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use keras.layers.dense instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/deeppavlov/core/models/tf_model.py:235: The name tf.train.AdadeltaOptimizer is deprecated. Please use tf.compat.v1.train.AdadeltaOptimizer instead.\n",
      "\n",
      "2019-11-18 15:46:48.527071: W tensorflow/core/framework/allocator.cc:107] Allocation of 89075712 exceeds 10% of system memory.\n",
      "2019-11-18 15:46:48.573293: W tensorflow/core/framework/allocator.cc:107] Allocation of 89075712 exceeds 10% of system memory.\n",
      "2019-11-18 15:46:49.563478: W tensorflow/core/framework/allocator.cc:107] Allocation of 89075712 exceeds 10% of system memory.\n",
      "2019-11-18 15:46:49.582380: W tensorflow/core/framework/allocator.cc:107] Allocation of 89075712 exceeds 10% of system memory.\n",
      "2019-11-18 15:46:49.831177: W tensorflow/core/framework/allocator.cc:107] Allocation of 89075712 exceeds 10% of system memory.\n",
      "2019-11-18 15:46:50.987038: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/deeppavlov/models/bert/bert_classifier.py:96: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n",
      "2019-11-18 15:46:54.295 INFO in 'deeppavlov.core.models.tf_model'['tf_model'] at line 52: [loading model from /root/.deeppavlov/models/classifiers/insults_kaggle_v3/model]\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/deeppavlov/core/models/tf_model.py:55: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.\n",
      "\n",
      "x::"
     ]
    }
   ],
   "source": [
    "! python -m deeppavlov interact -d insults_kaggle_bert"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "n0AHB7nAxIHx"
   },
   "source": [
    "Interact with text classification model with DeepPavlov Python API:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zGYf5yshE5re"
   },
   "outputs": [],
   "source": [
    "from deeppavlov import build_model, configs\n",
    "\n",
    "model = build_model(configs.classifiers.insults_kaggle_bert, download=False) # download=True if model is not downloaded yet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "dvAr9jdUzRj_",
    "outputId": "f362d045-de26-496d-b2e9-525f0b68c96a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Not Insult', 'Insult']"
      ]
     },
     "execution_count": 5,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model(['hey, how are you?', 'You are so stupid!'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pEWXw2YNzqh2"
   },
   "source": [
    "### Dataformat for classification\n",
    "\n",
    "Let's check training data for  insults classification model. We can get data path from model configuration file from section `dataset_reader`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mrVwZWadz52L"
   },
   "outputs": [],
   "source": [
    "import json\n",
    "from pprint import pprint\n",
    "model_config = json.load(open(configs.classifiers.insults_kaggle_bert))\n",
    "\n",
    "pprint(model_config['dataset_reader'])\n",
    "pprint(model_config['metadata']['variables'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BrILmNkN35nD"
   },
   "source": [
    "there are three .csv files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "BF8-DrRR2Q-T",
    "outputId": "9a50c3fa-1fd8-4591-c086-c1a540c356be"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test.csv  train.csv  valid.csv\n"
     ]
    }
   ],
   "source": [
    "! ls ~/.deeppavlov/downloads/insults_data/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "60OCJIOZ2MYd"
   },
   "outputs": [],
   "source": [
    "! head ~/.deeppavlov/downloads/insults_data/train.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XMXaNhQz5CMo"
   },
   "source": [
    "If you want to train model on your data you need to create configuration file and set up `data_path` to folder with train.csv, valid.csv, test.csv and change `MODEL_PATH` where to save trained model. Details in [documentation](http://docs.deeppavlov.ai/en/master/features/models/classifiers.html#how-to-train-on-other-datasets).\n",
    "\n",
    "Train model with CLI:\n",
    "```\n",
    "! python -m deeppavlov train config_name\n",
    "```\n",
    "or in Python\n",
    "```\n",
    "from deeppavlov import train_model\n",
    "model = train_model(model_config)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PHC71FmA4UAx"
   },
   "source": [
    "## BERT for tagging (Named Entity Recognition)\n",
    "\n",
    "BERT model can be used for tagging tasks such like Named Entity Recognition and Part of Speech tagging.\n",
    "We train only one dense layer on top of the output from the last BERT Transformer layer for each token. You can optionally add CRF layer on top the dense layer like in most common architecture BiLSTM + CRF for tagging.\n",
    "\n",
    "Named Entity Recognition:\n",
    "\n",
    "For example, we want to extract persons' and organizations' names from the text. Then for the input text:\n",
    "\n",
    "    Yan Goodfellow works for Google Brain\n",
    "\n",
    "a NER model needs to provide the following sequence of tags:\n",
    "\n",
    "    B-PER I-PER    O     O   B-ORG  I-ORG\n",
    "\n",
    "Where *B-* and *I-* prefixes stand for the beginning and inside of the entity, while *O* stands for out of tag or no tag. Markup with the prefix scheme is called *BIO markup*. This markup is introduced for distinguishing of consequent entities with similar types.\n",
    "\n",
    "Here is how input is preprocessed for tagging:\n",
    "\n",
    "<img src=\"https://github.com/deepmipt/dp_tutorials/blob/master/img/BERT_NER.png?raw=1\" width=\"75%\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HaMfNCkT3GDb"
   },
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Wb1IICCG2nRe"
   },
   "outputs": [],
   "source": [
    "! python -m deeppavlov interact ner_ontonotes_bert -d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8OCQoQ9w2rK8"
   },
   "source": [
    "Data for Named Enitity Recognition task is usually stored in CoNLL files.\n",
    "Typical CoNLL file with NER data contains lines with pairs of tokens (word/punctuation symbol) and tags, separated by a whitespace. In many cases additional information such as POS tags included between  Different documents are separated by lines **started** with **-DOCSTART-** token. Different sentences are separated by an empty line. Example\n",
    "\n",
    "    -DOCSTART- -X- -X- O\n",
    "\n",
    "    EU NNP B-NP B-ORG\n",
    "    rejects VBZ B-VP O\n",
    "    German JJ B-NP B-MISC\n",
    "    call NN I-NP O\n",
    "    to TO B-VP O\n",
    "    boycott VB I-VP O\n",
    "    British JJ B-NP B-MISC\n",
    "    lamb NN I-NP O\n",
    "    . . O O\n",
    "\n",
    "    Peter NNP B-NP B-PER\n",
    "    Blackburn NNP I-NP I-PER\n",
    "    \n",
    "    \n",
    "If you want to train model on your own data you can convert it to this CoNLL format or implement your version of `dataset_reader`. As for classification task model can be trained with CLI:\n",
    "```\n",
    "! python -m deeppavlov train config_name\n",
    "```\n",
    "or in Python\n",
    "```\n",
    "from deeppavlov import train_model\n",
    "model = train_model(model_config)\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "BERT_example.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}