Assignment_4.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Assignment4_def.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "machine_shape": "hm"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "my_6Ebtj4La-"
      },
      "source": [
        "# Notebook initialization\n",
        "- Fix random seeds for reproducibile results. This cannot be done entirely in keras due to GPU intrinsically non-deterministic operations (https://keras.io/getting_started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development)\n",
        "\n",
        "- Enable Weights&biases, a platform for experiment logging. (keep it disabled if no login is available)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5tel_4LXsf8q"
      },
      "source": [
        "#@title { form-width: \"31%\" }\n",
        "import os\n",
        "import random\n",
        "import math\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "\n",
        "# fix random seeds\n",
        "seed_value = 42 #@param {type:\"integer\"}\n",
        "\n",
        "os.environ['PYTHONHASHSEED']=str(seed_value)\n",
        "random.seed(seed_value)\n",
        "np.random.seed(seed_value)\n",
        "\n",
        "tf.compat.v1.set_random_seed(seed_value)\n",
        "\n",
        "session_conf = tf.compat.v1.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)\n",
        "sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n",
        "tf.compat.v1.keras.backend.set_session(sess)\n",
        "\n",
        "# Weigths and Biases API int\n",
        "ENABLE_WANDB = False        #@param {type:\"boolean\"}\n",
        "if ENABLE_WANDB:\n",
        "    !pip install wandb\n",
        "    !wandb login wandb_api_token\n",
        "    import wandb"
      ],
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Ftf0cUbfCq1"
      },
      "source": [
        "# Dataset preparation\n",
        "\n",
        "- The dataset is downloaded by the provided script  \n",
        "- Sentences are preprocessed  \n",
        "- The dataset is converted to word embeddings  \n",
        "- A data iterator is defined for usage by the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BspxZcRjW0NG"
      },
      "source": [
        "#@title Download { form-width: \"31%\" }\n",
        "\n",
        "import os\n",
        "import requests\n",
        "import zipfile\n",
        "\n",
        "def save_response_content(response, destination):\n",
        "    CHUNK_SIZE = 32768\n",
        "\n",
        "    with open(destination, \"wb\") as f:\n",
        "        for chunk in response.iter_content(CHUNK_SIZE):\n",
        "            if chunk: # filter out keep-alive new chunks\n",
        "                f.write(chunk)\n",
        "\n",
        "def download_data(data_path):\n",
        "    toy_data_path = os.path.join(data_path, 'fever_data.zip')\n",
        "    toy_data_url_id = \"1wArZhF9_SHW17WKNGeLmX-QTYw9Zscl1\"\n",
        "    toy_url = \"https://docs.google.com/uc?export=download\"\n",
        "\n",
        "    if not os.path.exists(data_path):\n",
        "        os.makedirs(data_path)\n",
        "\n",
        "    if not os.path.exists(toy_data_path):\n",
        "        print(\"Downloading FEVER data splits...\")\n",
        "        with requests.Session() as current_session:\n",
        "            response = current_session.get(toy_url,\n",
        "                                   params={'id': toy_data_url_id},\n",
        "                                   stream=True)\n",
        "        save_response_content(response, toy_data_path)\n",
        "        print(\"Download completed!\")\n",
        "\n",
        "        print(\"Extracting dataset...\")\n",
        "        with zipfile.ZipFile(toy_data_path) as loaded_zip:\n",
        "            loaded_zip.extractall(data_path)\n",
        "        print(\"Extraction completed!\")\n",
        "\n",
        "download_data('dataset')"
      ],
      "execution_count": 40,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x1EFmSkijGmA"
      },
      "source": [
        "## Preprocessing\n",
        "\n",
        "Light preprocessing is done to each sentence:\n",
        "- the dataset comes from wikipedia english but has IPA phonetic alphabet symbols: we remove every non-ascii symbols to filter them out;  \n",
        "- bracket symbols: `[\"-LRB-\", \"-LSB-\", \"-RRB-\", \"-RSB-\"]` are extensively used, we remove them as stopwords;  \n",
        "- every word is lowercased so to match the selected GloVe word embedding representations;  \n",
        "- SUPPORT/REFUTES labels are converted to binary.\n",
        "\n",
        "Every claim and evidence sentence is processed in the same way.  \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 541
        },
        "id": "E-Rf7nIxc-30",
        "outputId": "b7d16e23-0d5c-4833-af47-fec5c17ed48c"
      },
      "source": [
        "#@title { form-width: \"31%\" }\n",
        "import pandas as pd\n",
        "# implement preprocessor pipeline, show its effect on a single sentence\n",
        "\n",
        "import re\n",
        "from functools import reduce\n",
        "\n",
        "GOOD_SYMBOLS_RE = re.compile('[^0-9a-zA-Z]')\n",
        "STOPWORDS = [\"-LRB-\", \"-LSB-\", \"-RRB-\", \"-RSB-\"]\n",
        "\n",
        "def lower(text):\n",
        "    \"\"\"\n",
        "    Transforms given text to lower case.\n",
        "    Example:\n",
        "    Input: 'I really like New York city'\n",
        "    Output: 'i really like new your city'\n",
        "    \"\"\"\n",
        "\n",
        "    return text.lower()\n",
        "\n",
        "def filter_out_uncommon_symbols(text):\n",
        "    \"\"\"\n",
        "    Removes any special character that is not in the\n",
        "    good symbols list (check regular expression)\n",
        "    \"\"\"\n",
        "    \n",
        "    return GOOD_SYMBOLS_RE.sub(' ', text)\n",
        "\n",
        "def remove_stopwords(text):\n",
        "    return ' '.join([x for x in text.split() if x and x not in STOPWORDS])\n",
        "\n",
        "PREPROCESSING_PIPELINE = [\n",
        "                          remove_stopwords,\n",
        "                          filter_out_uncommon_symbols,\n",
        "                          lower\n",
        "                          ]\n",
        "\n",
        "# Anchor method\n",
        "\n",
        "def text_prepare(text, filter_methods=None):\n",
        "    \"\"\"\n",
        "    Applies a list of pre-processing functions in sequence (reduce).\n",
        "    Note that the order is important here!\n",
        "    \"\"\"\n",
        "\n",
        "    filter_methods = filter_methods if filter_methods is not None else PREPROCESSING_PIPELINE\n",
        "\n",
        "    return reduce(lambda txt, f: f(txt), filter_methods, text)\n",
        "\n",
        "dir = \"dataset/\"\n",
        "names = [\"N\", \"Claim\",\t\"Evidence\",\t\"ID\",\t\"Label\"]\n",
        "\n",
        "# create pandas dataframes\n",
        "df_train = pd.read_csv(dir+\"train_pairs.csv\")\n",
        "df_val = pd.read_csv(dir+\"val_pairs.csv\")\n",
        "df_test = pd.read_csv(dir+\"test_pairs.csv\")\n",
        "\n",
        "# converts string label to binary class\n",
        "df_train['Label'] = df_train['Label'].map(lambda x: 1 if x=='SUPPORTS' else 0)\n",
        "df_val['Label'] = df_val['Label'].map(lambda x: 1 if x=='SUPPORTS' else 0)\n",
        "df_test['Label'] = df_test['Label'].map(lambda x: 1 if x=='SUPPORTS' else 0)\n",
        "\n",
        "df_train_strings = df_train.copy()\n",
        "df_val_strings = df_val.copy()\n",
        "df_test_strings = df_test.copy()\n",
        "\n",
        "df_train.drop(df_train.columns[0], axis='columns', inplace=True)\n",
        "df_val.drop(df_val.columns[0], axis='columns', inplace=True)\n",
        "df_test.drop(df_test.columns[0], axis='columns', inplace=True)\n",
        "\n",
        "evidence = df_train[\"Evidence\"][0]\n",
        "print('Original evidence text:')\n",
        "print(evidence,end='\\n\\n')\n",
        "print('Uncommon symbols and stopwords removal:')\n",
        "print(text_prepare(evidence),end='\\n\\n')\n",
        "\n",
        "# apply the text_prepare pipeline to each sentence\n",
        "# drops firt element of evidences, that is: the sentence pair id\n",
        "df_train[\"Claim\"] = df_train[\"Claim\"].map(lambda s: text_prepare(s).split())\n",
        "df_train[\"Evidence\"] = df_train[\"Evidence\"].map(lambda s: text_prepare(s).split()[1:])\n",
        "df_val[\"Claim\"] = df_val[\"Claim\"].map(lambda s: text_prepare(s).split())\n",
        "df_val[\"Evidence\"] = df_val[\"Evidence\"].map(lambda s: text_prepare(s).split()[1:])\n",
        "df_test[\"Claim\"] = df_test[\"Claim\"].map(lambda s: text_prepare(s).split())\n",
        "df_test[\"Evidence\"] = df_test[\"Evidence\"].map(lambda s: text_prepare(s).split()[1:])\n",
        "\n",
        "df_train"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Original evidence text:\n",
            "2\tHemsworth has also appeared in the science fiction action film Star Trek -LRB- 2009 -RRB- , the thriller adventure A Perfect Getaway -LRB- 2009 -RRB- , the horror comedy The Cabin in the Woods -LRB- 2012 -RRB- , the dark-fantasy action film Snow White and the Huntsman -LRB- 2012 -RRB- , the war film Red Dawn -LRB- 2012 -RRB- , and the biographical sports drama film Rush -LRB- 2013 -RRB- .\tStar Trek\tStar Trek (film)\tA Perfect Getaway\tA Perfect Getaway\tThe Cabin in the Woods\tThe Cabin in the Woods\tSnow White and the Huntsman\tSnow White and the Huntsman\tRed Dawn\tRed Dawn (2012 film)\tRush\tRush (2013 film)\n",
            "\n",
            "Uncommon symbols and stopwords removal:\n",
            "2 hemsworth has also appeared in the science fiction action film star trek 2009   the thriller adventure a perfect getaway 2009   the horror comedy the cabin in the woods 2012   the dark fantasy action film snow white and the huntsman 2012   the war film red dawn 2012   and the biographical sports drama film rush 2013   star trek star trek  film  a perfect getaway a perfect getaway the cabin in the woods the cabin in the woods snow white and the huntsman snow white and the huntsman red dawn red dawn  2012 film  rush rush  2013 film \n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Claim</th>\n",
              "      <th>Evidence</th>\n",
              "      <th>ID</th>\n",
              "      <th>Label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>[chris, hemsworth, appeared, in, a, perfect, g...</td>\n",
              "      <td>[hemsworth, has, also, appeared, in, the, scie...</td>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>[roald, dahl, is, a, writer]</td>\n",
              "      <td>[roald, dahl, langpron, ro, ld, d, l, u, l, d,...</td>\n",
              "      <td>7</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>[roald, dahl, is, a, governor]</td>\n",
              "      <td>[roald, dahl, langpron, ro, ld, d, l, u, l, d,...</td>\n",
              "      <td>8</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>[ireland, has, relatively, low, lying, mountains]</td>\n",
              "      <td>[the, island, s, geography, comprises, relativ...</td>\n",
              "      <td>9</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>[ireland, does, not, have, relatively, low, ly...</td>\n",
              "      <td>[the, island, s, geography, comprises, relativ...</td>\n",
              "      <td>10</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>121735</th>\n",
              "      <td>[april, was, the, month, anderson, silva, was,...</td>\n",
              "      <td>[anderson, da, silva, de, s, siwv, born, april...</td>\n",
              "      <td>229440</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>121736</th>\n",
              "      <td>[anderson, silva, is, an, american, brazilian,...</td>\n",
              "      <td>[anderson, da, silva, de, s, siwv, born, april...</td>\n",
              "      <td>229443</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>121737</th>\n",
              "      <td>[anderson, silva, is, incapable, of, being, a,...</td>\n",
              "      <td>[anderson, da, silva, de, s, siwv, born, april...</td>\n",
              "      <td>229444</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>121738</th>\n",
              "      <td>[anderson, silva, was, born, on, the, month, o...</td>\n",
              "      <td>[anderson, da, silva, de, s, siwv, born, april...</td>\n",
              "      <td>229445</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>121739</th>\n",
              "      <td>[anderson, silva, was, born, on, the, day, of,...</td>\n",
              "      <td>[anderson, da, silva, de, s, siwv, born, april...</td>\n",
              "      <td>229448</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>121740 rows × 4 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                    Claim  ... Label\n",
              "0       [chris, hemsworth, appeared, in, a, perfect, g...  ...     1\n",
              "1                            [roald, dahl, is, a, writer]  ...     1\n",
              "2                          [roald, dahl, is, a, governor]  ...     0\n",
              "3       [ireland, has, relatively, low, lying, mountains]  ...     1\n",
              "4       [ireland, does, not, have, relatively, low, ly...  ...     0\n",
              "...                                                   ...  ...   ...\n",
              "121735  [april, was, the, month, anderson, silva, was,...  ...     1\n",
              "121736  [anderson, silva, is, an, american, brazilian,...  ...     0\n",
              "121737  [anderson, silva, is, incapable, of, being, a,...  ...     0\n",
              "121738  [anderson, silva, was, born, on, the, month, o...  ...     1\n",
              "121739  [anderson, silva, was, born, on, the, day, of,...  ...     0\n",
              "\n",
              "[121740 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rgkg3IyJmDry"
      },
      "source": [
        "## Embedding and vocabularies\n",
        "Here we download the selected size GloVe pre-trained word embeddings.  \n",
        "This is then used to incrementally build Train, Validation and Test vocabularies, respectively v2, v3 and v4.  \n",
        "Corresponding `embedding_matrix_v2`, `embedding_matrix_v3` , `embedding_matrix_v4` will be passed to train, validation and test models to keep indices consistency.   \n",
        "\n",
        "The code proceeds as follows:\n",
        "\n",
        "repeat 1-6 for each one of train, val, test different vocabularies\n",
        "1. build separate vocabularies\n",
        "2. merge with previous vocabulary with consistent indices\n",
        "3. parse txt data to categorical (for embedding layer)\n",
        "4. build inverse vocabulary\n",
        "5. embed OOV terms with random vectors\n",
        "6. concatenate pad and unk vectors to embedding"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HNqpaxOojirY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e918d365-3364-4367-e0f2-0c75562cccb1"
      },
      "source": [
        "#@title { form-width: \"31%\" }\n",
        "\n",
        "import gensim\n",
        "import gensim.downloader as gloader\n",
        "from collections import OrderedDict, Counter\n",
        "from itertools import chain\n",
        "from functools import reduce\n",
        "\n",
        "from tqdm import tqdm\n",
        "\n",
        "def build_unique_voc(words, special_tokens=[]):\n",
        "    wordset = set()\n",
        "    wordset = wordset.union(*[set([word for word in wordlist]) for wordlist in words])\n",
        "    w2i = OrderedDict()\n",
        "    i2w = OrderedDict()\n",
        "    for i,w in enumerate(chain(special_tokens, wordset)):\n",
        "        w2i[w] = i\n",
        "        i2w[i] = w\n",
        "    return w2i, i2w\n",
        "    \n",
        "def text_to_categorical(df, w2i):\n",
        "    \"\"\"Converts DataFrame words to categorical for use in the RNNs\"\"\"\n",
        "    df['Claim'] = df['Claim'].map(lambda s: [w2i.get(w, w2i[UNK]) for w in s])\n",
        "    df['Evidence'] = df['Evidence'].map(lambda s: [w2i.get(w, w2i[UNK]) for w in s])\n",
        "\n",
        "def load_embedding_model(embedding_dimension=50):\n",
        "    \"\"\"Loads GloVe with specified embedding dimension.\"\"\"\n",
        "    download_path = \"glove-wiki-gigaword-{}\".format(embedding_dimension)\n",
        "    try:\n",
        "        emb_model = gloader.load(download_path)\n",
        "    except ValueError as e:\n",
        "        print(\"Error downloading GloVe\")\n",
        "        raise e\n",
        "    return emb_model\n",
        "\n",
        "def merge_voc(old_voc, add_voc):\n",
        "    \"\"\"Merges vocabularies keeping consistent indices.\"\"\"\n",
        "    voc = old_voc.copy()\n",
        "    added_counter = 0\n",
        "    oov_terms = []\n",
        "    for i, word in enumerate(add_voc.keys()):\n",
        "        if word not in old_voc.keys():\n",
        "            oov_terms.append(word)\n",
        "            voc[word] = added_counter + len(old_voc)\n",
        "            added_counter += 1\n",
        "    return voc, oov_terms\n",
        "\n",
        "def embedd_OOV_terms(embedding_model, oov_terms, co_occurrence_matrix, w2i, i2w, rnd_OOV = False):\n",
        "    \"\"\"Embedd OOV words by weighted average of co-occurring neighbors.\"\"\"\n",
        "    for i, word in enumerate(oov_terms):\n",
        "        if rnd_OOV:\n",
        "            oov_vec = np.random.rand(embedding_dimension)\n",
        "        else:\n",
        "            oov_vec = np.zeros(embedding_dimension)\n",
        "            for count_row in co_occurrence_matrix[w2i[word]]:\n",
        "                weights_acc = 0\n",
        "                for count, index in zip(count_row.data, count_row.indices):\n",
        "                    if i2w[index] not in oov_terms:\n",
        "                        weights_acc += count\n",
        "                        oov_vec += count*embedding_model[i2w[index]]\n",
        "\n",
        "            oov_vec/=weights_acc\n",
        "        embedding_model.add(word, oov_vec)\n",
        "\n",
        "    return embedding_model\n",
        "\n",
        "# download pretrained GloVe embedding\n",
        "embedding_dimension = 50 #@param [50, 100, 300] {type:\"raw\"}\n",
        "print(\"Downloading Glove embedding with dimension:\", embedding_dimension)\n",
        "print(\"Be \",int(np.sqrt((embedding_dimension//50 - 1)))*\"very \",\"patient :)\",sep='')\n",
        "embedding_model = load_embedding_model(embedding_dimension)\n",
        "\n",
        "# allows \"PAD\" to have index zero, crucial for consistency\n",
        "PAD, UNK = '<pad>', '<unk>'\n",
        "pad_unk_vec = np.zeros((2, embedding_dimension)) # pad\n",
        "pad_unk_vec[1] = np.random.rand(embedding_dimension) # unk\n",
        "v1 = {e:i for i,e in enumerate(chain([PAD, UNK], embedding_model.vocab.keys()))}\n",
        "\n",
        "\n",
        "#build vocabularies and embedding matrices\n",
        "\n",
        "# v2 (glove + train)\n",
        "print(\"Building train vocabulary...\")\n",
        "w2i_train, _ = build_unique_voc(df_train[\"Claim\"].append(df_train[\"Evidence\"]))\n",
        "v2, oov1 = merge_voc(v1, w2i_train)\n",
        "print(f\"Found {len(oov1)} oov words: {len(oov1)/len(w2i_train):.2%}\")\n",
        "text_to_categorical(df_train, v2)\n",
        "inv2 = {v:k for k,v in v2.items()}\n",
        "\n",
        "embeding_model = embedd_OOV_terms(embedding_model, oov1, None, v2, inv2, rnd_OOV=True)\n",
        "embedding_matrix_v2 = np.concatenate((pad_unk_vec, embedding_model.vectors.copy()))\n",
        "\n",
        "# v3 (glove + train + val)\n",
        "print(\"Building validation vocabulary...\")\n",
        "w2i_val, _ = build_unique_voc(df_val[\"Claim\"].append(df_val[\"Evidence\"]))\n",
        "v3, oov2 = merge_voc(v2, w2i_val)\n",
        "print(f\"Found {len(oov2)} oov words: {len(oov2)/len(w2i_val):.2%}\")\n",
        "text_to_categorical(df_val, v3)\n",
        "inv3 = {v:k for k,v in v3.items()}\n",
        "\n",
        "embeding_model = embedd_OOV_terms(embedding_model, oov2, None, v3, inv3, rnd_OOV=True)\n",
        "embedding_matrix_v3 = np.concatenate((pad_unk_vec, embedding_model.vectors.copy()))\n",
        "\n",
        "# v4 (glove + train + val + test)\n",
        "print(\"Building test vocabulary...\")\n",
        "w2i_test, _ = build_unique_voc(df_test[\"Claim\"].append(df_test[\"Evidence\"]))\n",
        "v4, oov3 = merge_voc(v3, w2i_test)\n",
        "print(f\"Found {len(oov3)} oov words: {len(oov3)/len(w2i_test):.2%}\")\n",
        "text_to_categorical(df_test, v4)\n",
        "inv4 = {v:k for k,v in v4.items()}\n",
        "\n",
        "embeding_model = embedd_OOV_terms(embedding_model, oov3, None, v4, inv4, rnd_OOV=True)\n",
        "embedding_matrix_v4 = np.concatenate((pad_unk_vec, embedding_model.vectors.copy()))\n"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading Glove embedding with dimension: 50\n",
            "Be patient :)\n",
            "Building train vocabulary...\n",
            "Found 2311 oov words: 7.15%\n",
            "Building validation vocabulary...\n",
            "Found 193 oov words: 2.17%\n",
            "Building test vocabulary...\n",
            "Found 235 oov words: 2.37%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yTmwkRZAzLNi"
      },
      "source": [
        "## Data iterators\n",
        "A data iterator provides batches of `(claim, evidence), label` to the model.  \n",
        "We build train and validation ones."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wRhqipuIVqNG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "65f9e3d7-4637-4fd4-de69-f30e753adc23"
      },
      "source": [
        "# @title { form-width: \"30%\" }\n",
        "# helper class to iterate the data\n",
        "class DataIterator:\n",
        "    def __init__(self, df, sequence_len, batch_size):\n",
        "        self.X = df[['Claim', 'Evidence']]\n",
        "        self.Y = df['Label'].to_numpy()\n",
        "        assert self.X.shape[0] == self.Y.shape[0]\n",
        "        self.num_sentence = self.X.shape[0]\n",
        "        self.batch_size = batch_size\n",
        "        # pad sentences\n",
        "        self.data_list = []\n",
        "        for i in range(self.num_sentence):\n",
        "            c = np.zeros((sequence_len))\n",
        "            e = np.zeros((sequence_len))\n",
        "            label = self.Y[i]\n",
        "            l0 = min(len(self.X.iloc[i][0]), sequence_len)\n",
        "            l1 = min(len(self.X.iloc[i][1]), sequence_len)\n",
        "            c[:l0] = self.X.iloc[i][0][:l0]\n",
        "            e[:l1] = self.X.iloc[i][1][:l1]\n",
        "            self.data_list.append((c, e, label))\n",
        "        self.shuffle()\n",
        "\n",
        "    def shuffle(self):\n",
        "        self.current = 0\n",
        "        random.shuffle(self.data_list)\n",
        "        # batch the data\n",
        "        num_batches = math.ceil(self.num_sentence/self.batch_size)\n",
        "        self.batches_c = []\n",
        "        self.batches_e = []\n",
        "        self.batches_y = []\n",
        "        for i in range(num_batches):\n",
        "            batchc = []\n",
        "            batche = []\n",
        "            batchy = []\n",
        "            for j in range(self.batch_size):\n",
        "                if i*self.batch_size+j >= self.num_sentence:\n",
        "                    break\n",
        "                batchc.append(self.data_list[i*self.batch_size+j][0])\n",
        "                batche.append(self.data_list[i*self.batch_size+j][1])\n",
        "                batchy.append(self.data_list[i*self.batch_size+j][2])\n",
        "            self.batches_c.append(np.array(batchc))\n",
        "            self.batches_e.append(np.array(batche))\n",
        "            self.batches_y.append(np.array(batchy))\n",
        "\n",
        "    def __iter__(self):\n",
        "        return self\n",
        "\n",
        "    def __next__(self):\n",
        "        if self.current >= len(self.batches_c):\n",
        "            raise StopIteration\n",
        "        claim = self.batches_c[self.current]\n",
        "        evidence = self.batches_e[self.current]\n",
        "        y = self.batches_y[self.current]\n",
        "        claim = tf.cast(claim, tf.float32)\n",
        "        evidence = tf.cast(evidence, tf.float32)\n",
        "        y = tf.cast(y, tf.float32)\n",
        "        self.current += 1\n",
        "        return claim, evidence, y\n",
        " \n",
        "# hyperparameters\n",
        "\n",
        "sequence_len = 216 # marks the longest sentence token count\n",
        "batch_size =   128 #@param {type:\"integer\"}\n",
        "\n",
        "print(\"Building Data Iterators\")\n",
        "train_data = DataIterator(df_train, sequence_len, batch_size)\n",
        "val_data = DataIterator(df_val, sequence_len, batch_size)\n",
        "print(\"Built\")"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Building Data Iterators\n",
            "Built\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-DpKJXdS4RH8"
      },
      "source": [
        "# Model\n",
        "We define a `BaseModel` class that implements common attributes and methods for all possible models,  \n",
        "`sentence_embedder` and `embedding_merger` required arguments allow for model customization:  \n",
        "the required embedding and merging strategies are implemented as `keras.layers` and passed to the constructor at instantiation time, to allow for building each feasible combination.  \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PDMHrrmcQ1vi"
      },
      "source": [
        "#@title Model definition { form-width: \"31%\" }\n",
        "\n",
        "from sklearn.metrics import f1_score\n",
        "from sklearn.metrics.pairwise import cosine_similarity\n",
        "from tensorflow.keras.models import Model\n",
        "from tensorflow.keras.layers import (Input, LayerNormalization, LSTM, Dense, \n",
        "                                     Bidirectional, Embedding, \n",
        "                                     concatenate, Add, Average, Flatten, Dropout)\n",
        "\n",
        "################################################################################\n",
        "class BaseModel(tf.keras.Model):\n",
        "    def __init__(self, batch_size, sequence_len, embedding_matrix,\n",
        "                 sentence_embedder, embedding_merger, train_word_emb = True,**kwargs):\n",
        "      \n",
        "        super(BaseModel, self).__init__(**kwargs)\n",
        "\n",
        "        word_voc = embedding_matrix.shape[0]\n",
        "        embedding_dim = embedding_matrix.shape[1]\n",
        "\n",
        "        self.trainable = train_word_emb\n",
        "\n",
        "        self.input_layer_claim = Input(batch_input_shape=(batch_size, sequence_len),\n",
        "                                       name=\"claim_input\")\n",
        "        self.input_layer_evidence = Input(batch_input_shape=(batch_size, sequence_len),\n",
        "                                          name=\"evidence_input\")\n",
        "        \n",
        "        self.word_embedding = Embedding(word_voc, embedding_dim,\n",
        "                            weights=[embedding_matrix], trainable=self.trainable,\n",
        "                            mask_zero=True, name=\"word_embedding\")\n",
        "        \n",
        "        self.sentence_embedder = sentence_embedder\n",
        "        self.embedding_merger = embedding_merger\n",
        "\n",
        "        self.classifier = Dense(2, activation=\"softmax\", name=\"classifier\")\n",
        "\n",
        "\n",
        "    def call(self, c, e):\n",
        "        claim_word_embedding = self.word_embedding(c)\n",
        "        evidence_word_embedding = self.word_embedding(e)\n",
        "        \n",
        "        claim_sentence_embedding = self.sentence_embedder(claim_word_embedding)\n",
        "        evidence_sentence_embedding = self.sentence_embedder(evidence_word_embedding)\n",
        "        merged_embedding = self.embedding_merger(claim_sentence_embedding,\n",
        "                                            evidence_sentence_embedding)\n",
        "                                            \n",
        "        output = self.classifier(merged_embedding)\n",
        "        return output\n",
        "\n",
        "    def loss_function(self, y, predictions):\n",
        "        sce = tf.keras.losses.sparse_categorical_crossentropy(\n",
        "                y, predictions, from_logits=False)\n",
        "        return tf.reduce_mean(sce)\n",
        "\n",
        "    def update_metrics(self, y, predictions, loss_obj, acc_obj, f1_obj):\n",
        "        # loss\n",
        "        current_loss = self.loss_function(y, predictions)\n",
        "        loss_obj.update_state(current_loss)\n",
        "\n",
        "        pred_np = np.argmax(predictions, axis=-1)\n",
        "        # accuracy\n",
        "        acc_obj.update_state(y, pred_np)\n",
        "        # f1 macro\n",
        "        y_np = y.numpy().astype(np.int32)\n",
        "        f1 = f1_score(y_np.flatten(), pred_np.flatten(), \n",
        "                        average='macro', zero_division=0, \n",
        "                        labels=[0,1])\n",
        "        f1_obj.update_state(f1)\n",
        "\n",
        "    # consistently copies embedding weights from one model to another\n",
        "    def copy_weights_from(self, other):\n",
        "        for layer, other_layer in zip(self.layers, other.layers):\n",
        "            if layer.name == \"word_embedding\":\n",
        "                from_weights = other_layer.get_weights()[0];\n",
        "                new_weights = layer.get_weights()[0][from_weights.shape[0]:]\n",
        "                layer.set_weights([np.vstack((from_weights, new_weights))])\n",
        "            else:\n",
        "                layer.set_weights(other_layer.get_weights())\n",
        "\n",
        "    def summary(self):\n",
        "        model = Model(inputs=[self.input_layer_claim, self.input_layer_evidence],\n",
        "                      outputs=self.call(self.input_layer_claim, self.input_layer_evidence))\n",
        "        model.summary()\n",
        "\n",
        "################################################################################\n",
        "########################  CUSTOM SENTENCE EMBEDDERS  ###########################\n",
        "################################################################################\n",
        "\n",
        "class BOVEmbeddeing(tf.keras.layers.Layer):\n",
        "    def __init__(self, **kwargs):\n",
        "        super(BOVEmbeddeing, self).__init__(**kwargs)\n",
        "\n",
        "    def call(self, x): \n",
        "        return tf.reduce_mean(x, axis=1)\n",
        "\n",
        "class MLPEmbedding(tf.keras.layers.Layer):\n",
        "    def __init__(self, embedding_dim, **kwargs):\n",
        "        super(MLPEmbedding, self).__init__(**kwargs)\n",
        "        self.flatten = Flatten()\n",
        "        self.d1 = Dense(embedding_dim)\n",
        "\n",
        "    def call(self, x):\n",
        "        flattened = self.flatten(x)\n",
        "        return self.d1(flattened)\n",
        "\n",
        "class RNNEmbedding(tf.keras.layers.Layer):\n",
        "    def __init__(self, latent_dim, average_all_outputs, **kwargs):\n",
        "        super(RNNEmbedding, self).__init__(**kwargs)\n",
        "        self.average_all_outputs = average_all_outputs\n",
        "        self.lstm = Bidirectional(LSTM(\n",
        "            latent_dim, return_state=False, return_sequences=self.average_all_outputs))\n",
        "        \n",
        "    def call(self, x): \n",
        "        if self.average_all_outputs:\n",
        "            return tf.reduce_mean(self.lstm(x), axis=1)\n",
        "        else:\n",
        "            return self.lstm(x)\n",
        "\n",
        "################################################################################\n",
        "#########################  CUSTOM EMBEDDING MERGERS  ###########################\n",
        "################################################################################\n",
        "\n",
        "class Merge(tf.keras.layers.Layer):\n",
        "    def __init__(self, cosine, **kwargs):\n",
        "        super(Merge, self).__init__(**kwargs)\n",
        "        self.cosine = cosine\n",
        "        self.dot = tf.keras.layers.Dot((1, 1))\n",
        "\n",
        "    def compute(self, a, b):\n",
        "        pass\n",
        "\n",
        "    def call(self, a, b):\n",
        "        if not self.cosine:\n",
        "            return self.compute(a, b)\n",
        "        cs = self.dot([a, b])/(tf.norm(a)*tf.norm(b)+1e-9)\n",
        "        computed = self.compute(a, b)\n",
        "        return concatenate([computed, cs])\n",
        "\n",
        "class ConcatMerge(Merge):\n",
        "    def __init__(self, cosine, **kwargs):\n",
        "        super(ConcatMerge, self).__init__(cosine, **kwargs)\n",
        "\n",
        "    def compute(self, a, b): \n",
        "        return concatenate([a, b])\n",
        "\n",
        "class SumMerge(Merge):\n",
        "    def __init__(self, cosine, **kwargs):\n",
        "        super(SumMerge, self).__init__(cosine, **kwargs)\n",
        "        self.add_layer = Add()\n",
        "\n",
        "    def compute(self, a, b): \n",
        "        return self.add_layer([a, b])\n",
        "\n",
        "class AvgMerge(Merge):\n",
        "    def __init__(self, cosine, **kwargs):\n",
        "        super(AvgMerge, self).__init__(cosine, **kwargs)\n",
        "        self.avg_layer = Average()\n",
        "\n",
        "    def compute(self, a, b):\n",
        "        return self.avg_layer([a, b])\n"
      ],
      "execution_count": 44,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "atnzCMFuVCjI"
      },
      "source": [
        "## Training\n",
        "We train"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XZkPrYQH1tFE",
        "outputId": "391e3265-947d-4309-df32-7edb9365f344"
      },
      "source": [
        "#@title { form-width: \"31%\" }\n",
        "\n",
        "RNN_latent_dim = 64#@param {type:\"integer\"}\n",
        "\n",
        "RNNlast = RNNEmbedding(latent_dim=RNN_latent_dim, average_all_outputs=False, name=\"RNNEmbedding-last\")\n",
        "RNNavg = RNNEmbedding(latent_dim=RNN_latent_dim, average_all_outputs=True, name=\"RNNEmbedding-avg\")\n",
        "MLP = MLPEmbedding(embedding_dim=50, name=\"MLPEmbedding\")\n",
        "BOV = BOVEmbeddeing(name=\"BOVEmbedding\")\n",
        "\n",
        "concat_merge = ConcatMerge(cosine=False, name=\"ConcatMerge\")\n",
        "concat_merge_cosine = ConcatMerge(cosine=True, name=\"ConcatMerge-cos\")\n",
        "\n",
        "sum_merge = SumMerge(cosine=False, name=\"SumMerge\")\n",
        "sum_merge_cosine = SumMerge(cosine=True, name=\"SumMerge-cos\")\n",
        "\n",
        "avg_merge = AvgMerge(cosine=False, name=\"AvgMerge\")\n",
        "avg_merge_cosine = AvgMerge(cosine=True, name=\"AvgMerge-cos\")\n",
        "\n",
        "################################################################################\n",
        "\n",
        "sentence_embedder = RNNlast #@param ['RNNlast', 'RNNavg', 'MLP', 'BOV'] {type:\"raw\"}\n",
        "embedding_merger = concat_merge #@param ['concat_merge', 'concat_merge_cosine','sum_merge','sum_merge_cosine','avg_merge','avg_merge_cosine'] {type:\"raw\"}\n",
        "\n",
        "max_epochs = 100\n",
        "learning_rate = 5e-4 #@param {type:\"number\"}\n",
        "\n",
        "\n",
        "@tf.function\n",
        "def train_step_graph_fn(optimizer, model, c, e, y):\n",
        "    with tf.GradientTape() as tape:\n",
        "        predictions = model(c, e)\n",
        "        loss = model.loss_function(y, predictions)\n",
        "    gradients = tape.gradient(loss, model.trainable_variables)\n",
        "    optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
        "    return predictions\n",
        "\n",
        "@tf.function\n",
        "def val_step_graph_fn(model, c, e):\n",
        "    predictions = model(c,e)\n",
        "    return predictions\n",
        "\n",
        "# train model\n",
        "model_name = f\"{sentence_embedder.name}__{embedding_merger.name}\"\n",
        "model = BaseModel(batch_size, sequence_len, embedding_matrix_v2, \n",
        "                  sentence_embedder, embedding_merger, name=model_name)\n",
        "model.summary()\n",
        "# val model\n",
        "val_model = BaseModel(batch_size, sequence_len, embedding_matrix_v3,\n",
        "                        sentence_embedder, embedding_merger, name=model_name)\n",
        "# best model\n",
        "best_model = BaseModel(batch_size, sequence_len, embedding_matrix_v4,\n",
        "                        sentence_embedder, embedding_merger, name=model_name)\n",
        "# build the graph model\n",
        "train_data.shuffle()\n",
        "c, e, _ = train_data.__next__()\n",
        "val_model(c, e)\n",
        "best_model(c, e)\n",
        "\n",
        "if ENABLE_WANDB:\n",
        "    wandb.config.batch_size = batch_size\n",
        "    wandb.config.latent_dim = latent_dim\n",
        "    wandb.config.learning_rate = learning_rate \n",
        "    wandb.config.emdedding_dim = embedding_dimension\n",
        "    wandb.config.model = model.name\n",
        "    wandb_experiment_name = f\"{model.name}_b{batch_size}_lr{learning_rate:.0e}\"\n",
        "    wandb.init(project=\"NLP04\", name=wandb_experiment_name)\n",
        "\n",
        "# metrics\n",
        "train_loss_obj = tf.keras.metrics.Mean(name='train_loss')\n",
        "train_f1_obj = tf.keras.metrics.Mean(name='train_f1')\n",
        "train_acc_obj = tf.keras.metrics.Accuracy(name='train_accuracy')\n",
        "val_loss_obj = tf.keras.metrics.Mean(name='val_loss')\n",
        "val_f1_obj = tf.keras.metrics.Mean(name='val_f1')\n",
        "val_acc_obj = tf.keras.metrics.Accuracy(name='val_accuracy')\n",
        "\n",
        "# optimizer\n",
        "optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)\n",
        "\n",
        "# train loop\n",
        "max_val_acc = -1\n",
        "not_improving = 0\n",
        "max_iter_not_improv = 3\n",
        "\n",
        "for epoch in range(max_epochs):\n",
        "    train_loss_obj.reset_states()\n",
        "    train_f1_obj.reset_states()\n",
        "    train_acc_obj.reset_states()\n",
        "\n",
        "    train_data.shuffle()\n",
        "    for c, e, y in train_data:\n",
        "        predictions = train_step_graph_fn(optimizer, model, c, e, y)\n",
        "        model.update_metrics(y, predictions, train_loss_obj, train_acc_obj, train_f1_obj)\n",
        "\n",
        "    print(\"{}.  \\t[TRAINING]\\t  loss: {}  \\t accuracy: {} \\t f1-macro: {}\".format(epoch, \n",
        "                                                      train_loss_obj.result(),\n",
        "                                                      train_acc_obj.result(),\n",
        "                                                      train_f1_obj.result()))\n",
        "    if ENABLE_WANDB:\n",
        "        wandb.log({\n",
        "            'train_loss': train_loss_obj.result(),\n",
        "            'train_accuracy': train_acc_obj.result(),\n",
        "            'train_f1': train_f1_obj.result()\n",
        "        }, step=epoch)\n",
        "\n",
        "    # validation\n",
        "    if True:# epoch%5 == 4:\n",
        "        val_loss_obj.reset_states()\n",
        "        val_acc_obj.reset_states()\n",
        "        val_f1_obj.reset_states()\n",
        "\n",
        "        val_model.copy_weights_from(model)\n",
        "        val_data.shuffle()\n",
        "        for c, e, y in val_data:\n",
        "            predictions = val_step_graph_fn(val_model, c, e)\n",
        "            val_model.update_metrics(y, predictions, val_loss_obj, val_acc_obj, val_f1_obj)\n",
        "\n",
        "        print(\"     \\t[VALIDATION]\\t   loss: {}  \\t  accuracy: {} \\t  f1-macro: {}\".format(\n",
        "                                                      val_loss_obj.result(),\n",
        "                                                      val_acc_obj.result(),\n",
        "                                                      val_f1_obj.result()))\n",
        "        if ENABLE_WANDB:\n",
        "            wandb.log({\n",
        "                'val_loss': val_loss_obj.result(),\n",
        "                'val_accuracy': val_acc_obj.result(),\n",
        "                'val_f1': val_f1_obj.result()\n",
        "            }, step=epoch)\n",
        "\n",
        "        # early stopping\n",
        "        if val_acc_obj.result() > max_val_acc:\n",
        "            best_model.copy_weights_from(val_model)\n",
        "            max_val_acc = val_acc_obj.result()\n",
        "            not_improving = 0\n",
        "        else:\n",
        "            not_improving += 1\n",
        "            print(\"VALIDATION ACCURACY NOT IMPROVING, STRIKE:\", not_improving,\"!!\")\n",
        "            if not_improving >= max_iter_not_improv:\n",
        "                print(\"Validation accuracy not improving for\", max_iter_not_improv,\n",
        "                      \"successive computations.\")\n",
        "                print(\"YOU ARE OUT !!!\")\n",
        "                print(\"Best model accuracy:\", max_val_acc.numpy())\n",
        "                break"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "claim_input (InputLayer)        [(128, 216)]         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "evidence_input (InputLayer)     [(128, 216)]         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "word_embedding (Embedding)      (128, 216, 50)       20115650    claim_input[0][0]                \n",
            "                                                                 evidence_input[0][0]             \n",
            "__________________________________________________________________________________________________\n",
            "RNNEmbedding-last (RNNEmbedding (128, 128)           58880       word_embedding[0][0]             \n",
            "                                                                 word_embedding[1][0]             \n",
            "__________________________________________________________________________________________________\n",
            "ConcatMerge (ConcatMerge)       (128, 256)           0           RNNEmbedding-last[0][0]          \n",
            "                                                                 RNNEmbedding-last[1][0]          \n",
            "__________________________________________________________________________________________________\n",
            "classifier (Dense)              (128, 2)             514         ConcatMerge[0][0]                \n",
            "==================================================================================================\n",
            "Total params: 20,175,044\n",
            "Trainable params: 20,175,044\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n",
            "0.  \t[TRAINING]\t  loss: 0.45497244596481323  \t accuracy: 0.8018563985824585 \t f1-macro: 0.669087827205658\n",
            "     \t[VALIDATION]\t   loss: 0.5646363496780396  \t  accuracy: 0.7087229490280151 \t  f1-macro: 0.6845734715461731\n",
            "1.  \t[TRAINING]\t  loss: 0.3605252504348755  \t accuracy: 0.8503778576850891 \t f1-macro: 0.7796408534049988\n",
            "     \t[VALIDATION]\t   loss: 0.5055816769599915  \t  accuracy: 0.748778760433197 \t  f1-macro: 0.7399455904960632\n",
            "2.  \t[TRAINING]\t  loss: 0.32202574610710144  \t accuracy: 0.8670938014984131 \t f1-macro: 0.8109845519065857\n",
            "     \t[VALIDATION]\t   loss: 0.5649393200874329  \t  accuracy: 0.7411025762557983 \t  f1-macro: 0.7283143997192383\n",
            "VALIDATION ACCURACY NOT IMPROVING, STRIKE: 1 !!\n",
            "3.  \t[TRAINING]\t  loss: 0.29556143283843994  \t accuracy: 0.8778873085975647 \t f1-macro: 0.8294010162353516\n",
            "     \t[VALIDATION]\t   loss: 0.5497254729270935  \t  accuracy: 0.7535240650177002 \t  f1-macro: 0.7465249300003052\n",
            "4.  \t[TRAINING]\t  loss: 0.2760891914367676  \t accuracy: 0.8858304619789124 \t f1-macro: 0.8430062532424927\n",
            "     \t[VALIDATION]\t   loss: 0.5781392455101013  \t  accuracy: 0.7514305710792542 \t  f1-macro: 0.7429275512695312\n",
            "VALIDATION ACCURACY NOT IMPROVING, STRIKE: 1 !!\n",
            "5.  \t[TRAINING]\t  loss: 0.25931844115257263  \t accuracy: 0.8931082487106323 \t f1-macro: 0.853691041469574\n",
            "     \t[VALIDATION]\t   loss: 0.621522843837738  \t  accuracy: 0.7411025762557983 \t  f1-macro: 0.7305501699447632\n",
            "VALIDATION ACCURACY NOT IMPROVING, STRIKE: 2 !!\n",
            "6.  \t[TRAINING]\t  loss: 0.2463339865207672  \t accuracy: 0.8985131978988647 \t f1-macro: 0.8622032999992371\n",
            "     \t[VALIDATION]\t   loss: 0.646205723285675  \t  accuracy: 0.7429169416427612 \t  f1-macro: 0.732873797416687\n",
            "VALIDATION ACCURACY NOT IMPROVING, STRIKE: 3 !!\n",
            "Validation accuracy not improving for 3 successive computations.\n",
            "YOU ARE OUT !!!\n",
            "Best model accuracy: 0.75352407\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mIfvbQuEWfGH"
      },
      "source": [
        "## Evaluation on test"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PPdFWxujYonp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7fd4e13a-c260-48e7-b1dc-8eb9128a8adf"
      },
      "source": [
        "#@title { form-width: \"31%\" }\n",
        "\n",
        "test_loss_obj = tf.keras.metrics.Mean(name='test_loss')\n",
        "test_f1_obj = tf.keras.metrics.Mean(name='test_f1')\n",
        "test_acc_obj = tf.keras.metrics.Accuracy(name='test_accuracy')\n",
        "\n",
        "test_data = DataIterator(df_test, sequence_len, batch_size)\n",
        "\n",
        "test_data.shuffle()\n",
        "for c, e, y in test_data:\n",
        "    predictions = val_step_graph_fn(best_model, c, e)\n",
        "    best_model.update_metrics(y, predictions, test_loss_obj, test_acc_obj, test_f1_obj)\n",
        "\n",
        "test_log = \"\\nTEST loss: {}  \\t accuracy: {} \\t f1-macro: {}\\n\".format(\n",
        "                                                test_loss_obj.result(),\n",
        "                                                test_acc_obj.result(),\n",
        "                                                test_f1_obj.result())\n",
        "if ENABLE_WANDB:\n",
        "    wandb.log({\"Test\": wandb.Html(\n",
        "        \"<pre>\"+test_log+\"<pre>\", inject=False)})\n",
        "print(test_log)"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "TEST loss: 0.555887758731842  \t accuracy: 0.7259702086448669 \t f1-macro: 0.7153576612472534\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AHllSCq_n5zi"
      },
      "source": [
        "# BERT\n",
        "\n",
        "We play with transfer learning via a pre-trained BERT model to explore the state-of-the-art in sentence embedding.  \n",
        "We select a simple functional model and implement the data iterator as a versatile `keras.utils.Sequence`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "czRYQZakueEc"
      },
      "source": [
        "!pip install -q -U sentence-transformers\n",
        "from keras.utils import Sequence\n",
        "from sentence_transformers import SentenceTransformer\n",
        "\n",
        "#Semantic Textual Similarity trained model\n",
        "bert_model = SentenceTransformer('distilbert-base-nli-mean-tokens')\n",
        "\n",
        "class BERTSequence(Sequence):\n",
        "    def __init__(self, batch_size, df):\n",
        "        self.df = df\n",
        "        self.claims = df[\"Claim\"]\n",
        "        self.evidences = df[\"Evidence\"]\n",
        "        self.labels = df[\"Label\"]\n",
        "        self.batch_size = batch_size\n",
        "\n",
        "    def __len__(self):\n",
        "        return math.ceil(len(self.df)/self.batch_size)\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        processed_c = bert_model.encode(list(self.claims[idx * self.batch_size : (idx + 1) * self.batch_size]))\n",
        "        processed_e = bert_model.encode(list(self.evidences[idx * self.batch_size : (idx + 1) * self.batch_size]))\n",
        "        batch_y = self.labels[idx * self.batch_size : (idx + 1) * self.batch_size].to_numpy()\n",
        "        return (processed_c, processed_e), batch_y\n",
        "\n",
        "batch_size = 512\n",
        "train_sequence = BERTSequence(batch_size, df_train_strings)\n",
        "val_sequence = BERTSequence(batch_size, df_val_strings)"
      ],
      "execution_count": 51,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 769
        },
        "id": "QLmpcZuVujUv",
        "outputId": "67a70b83-6840-4ca4-ef4d-91f6f0a05811"
      },
      "source": [
        "# BERT classification model\n",
        "\n",
        "\n",
        "input_c = Input(shape=(768)) # BERT sentence embedded claims\n",
        "input_e = Input(shape=(768)) # BERT sentence embedded evidences\n",
        "merged = concat_merge(input_c, input_e)\n",
        "output = Dense(2, activation=\"softmax\")(merged)\n",
        "\n",
        "model = tf.keras.Model(inputs=(input_c, input_e), outputs=output, name=\"BERT\")\n",
        "model.summary()\n",
        "\n",
        "learning_rate = 5e-4\n",
        " \n",
        "model.compile(\n",
        "    loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=False),\n",
        "    optimizer=tf.keras.optimizers.Adam(learning_rate=learning_rate),\n",
        "    metrics=[\"accuracy\"]\n",
        ")\n",
        " \n",
        "history = model.fit(train_sequence, batch_size=batch_size,\n",
        "                    epochs=2, verbose=True, shuffle=True,\n",
        "                    validation_data=val_sequence)"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"BERT\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_15 (InputLayer)           [(None, 768)]        0                                            \n",
            "__________________________________________________________________________________________________\n",
            "input_16 (InputLayer)           [(None, 768)]        0                                            \n",
            "__________________________________________________________________________________________________\n",
            "ConcatMerge (ConcatMerge)       multiple             0           input_15[0][0]                   \n",
            "                                                                 input_16[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "dense_26 (Dense)                (None, 2)            3074        ConcatMerge[5][0]                \n",
            "==================================================================================================\n",
            "Total params: 3,074\n",
            "Trainable params: 3,074\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n",
            "Epoch 1/2\n",
            "238/238 [==============================] - 433s 2s/step - loss: 0.4920 - accuracy: 0.8017 - val_loss: 0.5230 - val_accuracy: 0.7445\n",
            "Epoch 2/2\n",
            "238/238 [==============================] - 432s 2s/step - loss: 0.4111 - accuracy: 0.8294 - val_loss: 0.4917 - val_accuracy: 0.7468\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cyw9erOg6cZx",
        "outputId": "c5f0948c-b0b0-4c18-c785-9b012c7db1ca"
      },
      "source": [
        "test_sequence = BERTSequence(batch_size, df_test_strings)\n",
        "model.evaluate(test_sequence)"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "15/15 [==============================] - 22s 1s/step - loss: 0.5887 - accuracy: 0.7161\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.5887011289596558, 0.7160940170288086]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 55
        }
      ]
    }
  ]
}