bert_finetuning_qa.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Copia de BERT for QA",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "meO7ZaISZfZ1",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jorgeramirez/bert-qa-colab/blob/master/bert_finetuning_qa.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5s3Q_8_6LI6X",
        "colab_type": "text"
      },
      "source": [
        "# Changes\n",
        "This is a modified version of [this notebook](https://github.com/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb), adapted for QA on the SQuAD dataset using [bert-qa](https://github.com/chiayewken/bert-qa). It also includes some scripts for running predictions on a custom dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rkTLZ3I4_7c_",
        "colab_type": "text"
      },
      "source": [
        "# BERT End to End (Fine-tuning + Predicting)  with Cloud TPU for SQuAD"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1wtjs1QDb3DX",
        "colab_type": "text"
      },
      "source": [
        "## Overview\n",
        "\n",
        "**BERT**, or **B**idirectional **E**mbedding **R**epresentations from **T**ransformers, is a new method of pre-training language representations which obtains state-of-the-art results on a wide array of Natural Language Processing (NLP) tasks. The academic paper can be found here: https://arxiv.org/abs/1810.04805.\n",
        "\n",
        "This Colab demonstates using a free Colab Cloud TPU to fine-tune sentence and sentence-pair classification tasks built on top of pretrained BERT models and \n",
        "run predictions on tuned model. The colab demonsrates loading pretrained BERT models from both [TF Hub](https://www.tensorflow.org/hub) and checkpoints.\n",
        "\n",
        "**Note:**  You will need a GCP (Google Compute Engine) account and a GCS (Google Cloud \n",
        "Storage) bucket for this Colab to run.\n",
        "\n",
        "Please follow the [Google Cloud TPU quickstart](https://cloud.google.com/tpu/docs/quickstart) for how to create GCP account and GCS bucket. You have [$300 free credit](https://cloud.google.com/free/) to get started with any GCP product. You can learn more about Cloud TPU at https://cloud.google.com/tpu/docs.\n",
        "\n",
        "This notebook is hosted on GitHub. To view it in its original repository, after opening the notebook, select **File > View on GitHub**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ld-JXlueIuPH",
        "colab_type": "text"
      },
      "source": [
        "## Instructions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "POkof5uHaQ_c",
        "colab_type": "text"
      },
      "source": [
        "<h3><a href=\"https://cloud.google.com/tpu/\"><img valign=\"middle\" src=\"https://raw.githubusercontent.com/GoogleCloudPlatform/tensorflow-without-a-phd/master/tensorflow-rl-pong/images/tpu-hexagon.png\" width=\"50\"></a>  &nbsp;&nbsp;Train on TPU</h3>\n",
        "\n",
        "   1. Create a Cloud Storage bucket for your TensorBoard logs at http://console.cloud.google.com/storage and fill in the BUCKET parameter in the \"Parameters\" section below.\n",
        " \n",
        "   1. On the main menu, click Runtime and select **Change runtime type**. Set \"TPU\" as the hardware accelerator.\n",
        "   1. Click Runtime again and select **Runtime > Run All** (Watch out: the \"Colab-only auth for this notebook and the TPU\" cell requires user input). You can also run the cells manually with Shift-ENTER."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UdMmwCJFaT8F",
        "colab_type": "text"
      },
      "source": [
        "## Set up your TPU environment\n",
        "\n",
        "In this section, you perform the following tasks:\n",
        "\n",
        "*   Set up a Colab TPU running environment\n",
        "*   Verify that you are connected to a TPU device\n",
        "*   Upload your credentials to TPU to access your GCS bucket."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "191zq3ZErihP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# run all of this to prepare for training bert-qa\n",
        "import datetime\n",
        "import json\n",
        "import os\n",
        "import pprint\n",
        "import random\n",
        "import string\n",
        "import sys\n",
        "import tensorflow as tf\n",
        "\n",
        "assert 'COLAB_TPU_ADDR' in os.environ, 'ERROR: Not connected to a TPU runtime; please see the first cell in this notebook for instructions!'\n",
        "TPU_ADDRESS = 'grpc://' + os.environ['COLAB_TPU_ADDR']\n",
        "print('TPU address is', TPU_ADDRESS)\n",
        "\n",
        "from google.colab import auth\n",
        "auth.authenticate_user()\n",
        "with tf.Session(TPU_ADDRESS) as session:\n",
        "  print('TPU devices:')\n",
        "  pprint.pprint(session.list_devices())\n",
        "\n",
        "  # Upload credentials to TPU.\n",
        "  with open('/content/adc.json', 'r') as f:\n",
        "    auth_info = json.load(f)\n",
        "  tf.contrib.cloud.configure_gcs(session, credentials=auth_info)\n",
        "  # Now credentials are set for all future sessions on this TPU."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HUBP35oCDmbF",
        "colab_type": "text"
      },
      "source": [
        "## Prepare and import BERT modules\n",
        "​\n",
        "With your environment configured, you can now prepare and import the BERT modules. The following step clones the source code from GitHub and import the modules from the source. Alternatively, you can install BERT using pip (!pip install bert-tensorflow)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7wzwke0sxS6W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import sys\n",
        "\n",
        "!test -d bert_repo || git clone https://github.com/google-research/bert bert_repo\n",
        "if not 'bert_repo' in sys.path:\n",
        "  sys.path += ['bert_repo']\n",
        "\n",
        "# import python modules defined by BERT\n",
        "import modeling\n",
        "import optimization\n",
        "import run_classifier\n",
        "import run_classifier_with_tfhub\n",
        "import tokenization\n",
        "\n",
        "# import tfhub \n",
        "import tensorflow_hub as hub"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IHqSFW5n7DAT",
        "colab_type": "text"
      },
      "source": [
        "## Prepare for training\n",
        "Run the following lines to download dependencies for training bert-qa."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c0WXxJw1qo_P",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "BUCKET_NAME=\"gs://bert-qa-demo\"\n",
        "\n",
        "# set the bert directory from the bucket\n",
        "BERT_BASE_DIR= f\"{BUCKET_NAME}/bert_base\"\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UkbyeyQoNorc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "! git clone https://github.com/chiayewken/bert-qa.git\n",
        "! wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json\n",
        "! wget https://raw.githubusercontent.com/allenai/bi-att-flow/master/squad/evaluate-v1.1.py\n",
        "! wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json\n",
        "! mkdir squad_dir\n",
        "\n",
        "! mv train-v1.1.json squad_dir/\n",
        "! mv dev-v1.1.json squad_dir & mv evaluate-v1.1.py squad_dir\n",
        "! ls squad_dir\n",
        "! wget https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip\n",
        "! unzip uncased_L-12_H-768_A-12.zip -d bert_base \n",
        "! mv bert_base/uncased_L-12_H-768_A-12/* bert_base\n",
        "\n",
        "! gsutil cp -r bert_base $BUCKET_NAME"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oXms5KLesMrX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# we start the training process with the following line\n",
        "\n",
        "!cd bert-qa && python run_squad.py \\\n",
        "  --vocab_file=$BERT_BASE_DIR/vocab.txt \\\n",
        "  --bert_config_file=$BERT_BASE_DIR/bert_config.json \\\n",
        "  --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \\\n",
        "  --do_train=True \\\n",
        "  --train_file=../squad_dir/train-v1.1.json \\\n",
        "  --do_predict=True \\\n",
        "  --predict_file=../squad_dir/dev-v1.1.json \\\n",
        "  --train_batch_size=32 \\\n",
        "  --learning_rate=3e-5 \\\n",
        "  --num_train_epochs=2.0 \\\n",
        "  --max_seq_length=384 \\\n",
        "  --doc_stride=128 \\\n",
        "  --use_tpu=True   \\\n",
        "  --tpu_name=$TPU_ADDRESS \\\n",
        "  --output_dir=$BUCKET_NAME"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oBG1zw8iIGZk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# let's evaluate the predictions\n",
        "\n",
        "!mkdir squad\n",
        "!gsutil cp $BUCKET_NAME/predictions.json squad/predictions.json\n",
        "\n",
        "!cd bert-qa && python ../squad_dir/evaluate-v1.1.py ../squad_dir/dev-v1.1.json  ../squad/predictions.json\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ybIn5UD2-e4N",
        "colab_type": "text"
      },
      "source": [
        "## Prepare custom dataset for predictions\n",
        "\n",
        "We should format our custom dataset based on the format found in [dev-v1.1.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json). When running `run_squad.py` in prediction mode the dev file just needs to contain the following:\n",
        "\n",
        "```json\n",
        "{\n",
        "    data: [\n",
        "        {\n",
        "            title: \"\",\n",
        "            paragraphs: [\n",
        "                {\n",
        "                    context: \"<our text>\", \n",
        "                    qas: [{id: \"<we can use some random UUID here\", question: \"<our question>\"}, ...]\n",
        "                }\n",
        "            ]\n",
        "        }\n",
        "    ]\n",
        "}\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oo9wv0U4SLaJ",
        "colab_type": "code",
        "cellView": "both",
        "colab": {}
      },
      "source": [
        "import pandas as pd\n",
        "import json\n",
        "import uuid\n",
        "import io\n",
        "from google.colab import files\n",
        "\n",
        "\n",
        "\n",
        "def gen_question_entry(q):\n",
        "  return {\n",
        "      \"id\": str(uuid.uuid4()),\n",
        "      \"question\": q\n",
        "  }\n",
        "\n",
        "def to_prediction_format(item, questions):\n",
        "  output = {\n",
        "      \"title\": item.title,\n",
        "      \"paragraphs\": [\n",
        "          {\n",
        "              \"context\": item.text,\n",
        "              \"qas\": [gen_question_entry(q) for q in questions]\n",
        "          }\n",
        "      ]\n",
        "  }\n",
        "  return output\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "def run(input_file, output_file, questions):\n",
        "  uploaded = files.upload() # upload the CSV file, it expects the columns: title, text\n",
        "\n",
        "  # read the INPUT_FILE\n",
        "  df = pd.read_csv(io.BytesIO(uploaded[input_file]))\n",
        "  data = []\n",
        "\n",
        "  for _, item in df.iterrows():\n",
        "    formated_entry = to_prediction_format(item, questions)\n",
        "    data.append(formated_entry)\n",
        "\n",
        "  with open(output_file, \"w\") as f:\n",
        "    json.dump({\"data\": data}, f)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pJGNMxndLcdh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "\n",
        "run(\"MY_FILE.csv\", \"./my_file_qa.json\", [\n",
        "     \"MY QUESTION?\"\n",
        " ]\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SzS0uAW_F0rK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# copy the generated file to the bucket\n",
        "!gsutil cp ./my_file_qa.json  $BUCKET_NAME/my_file_qa.json"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CT5j4ZlIWH64",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# let's inspect the formatted file\n",
        "!cat my_file_qa.json | python -m json.tool"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9FasJYQZYC9a",
        "colab_type": "text"
      },
      "source": [
        "## Run prediction on the custom dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fsy-m_2xYHJv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!cd bert-qa && python run_squad.py \\\n",
        "  --vocab_file=$BERT_BASE_DIR/vocab.txt \\\n",
        "  --bert_config_file=$BERT_BASE_DIR/bert_config.json \\\n",
        "  --init_checkpoint=$BUCKET_NAME/model.ckpt-5474 \\\n",
        "  --do_train=False \\\n",
        "  --train_file=../squad_dir/train-v1.1.json \\\n",
        "  --do_predict=True \\\n",
        "  --predict_file=../my_file_qa.json \\\n",
        "  --train_batch_size=32 \\\n",
        "  --learning_rate=3e-5 \\\n",
        "  --num_train_epochs=2.0 \\\n",
        "  --max_seq_length=384 \\\n",
        "  --doc_stride=128 \\\n",
        "  --use_tpu=True   \\\n",
        "  --tpu_name=$TPU_ADDRESS \\\n",
        "  --output_dir=$BUCKET_NAME/     "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YbNDY7OZa08B",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# let's inspect the answers\n",
        "\n",
        "!gsutil cp  $BUCKET_NAME/predictions.json predictions.json\n",
        "\n",
        "!cat predictions.json | python -m json.tool\n"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}