2_BERT_Fine_tuning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "2_BERT-Fine-tuning.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.1"
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": true,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": true,
      "toc_position": {},
      "toc_section_display": true,
      "toc_window_display": true
    },
    "varInspector": {
      "cols": {
        "lenName": 16,
        "lenType": 16,
        "lenVar": 40
      },
      "kernels_config": {
        "python": {
          "delete_cmd_postfix": "",
          "delete_cmd_prefix": "del ",
          "library": "var_list.py",
          "varRefreshCmd": "print(var_dic_list())"
        },
        "r": {
          "delete_cmd_postfix": ") ",
          "delete_cmd_prefix": "rm(",
          "library": "var_list.r",
          "varRefreshCmd": "cat(var_dic_list()) "
        }
      },
      "types_to_exclude": [
        "module",
        "function",
        "builtin_function_or_method",
        "instance",
        "_Feature"
      ],
      "window_display": false
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/julrods/aggressive-tweet-analyzer/blob/main/2_BERT_Fine_tuning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "toc": true,
        "id": "aTClOoMqcUmn"
      },
      "source": [
        "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
        "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Environment\" data-toc-modified-id=\"Environment-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Environment</a></span><ul class=\"toc-item\"><li><span><a href=\"#Libraries\" data-toc-modified-id=\"Libraries-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Libraries</a></span></li><li><span><a href=\"#Functions\" data-toc-modified-id=\"Functions-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Functions</a></span></li></ul></li><li><span><a href=\"#BERT-model-setup\" data-toc-modified-id=\"BERT-model-setup-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>BERT model setup</a></span></li><li><span><a href=\"#Fine-tuning-the-pre-trained-BERT-model\" data-toc-modified-id=\"Fine-tuning-the-pre-trained-BERT-model-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Fine-tuning the pre-trained BERT model</a></span><ul class=\"toc-item\"><li><span><a href=\"#Aggression-dataset---4-epochs\" data-toc-modified-id=\"Aggression-dataset---4-epochs-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Aggression dataset - 4 epochs</a></span><ul class=\"toc-item\"><li><span><a href=\"#Loading-the-data-from-the-pickle-files\" data-toc-modified-id=\"Loading-the-data-from-the-pickle-files-3.1.1\"><span class=\"toc-item-num\">3.1.1&nbsp;&nbsp;</span>Loading the data from the pickle files</a></span></li><li><span><a href=\"#Splitting-into-train,-test-and-validation-set\" data-toc-modified-id=\"Splitting-into-train,-test-and-validation-set-3.1.2\"><span class=\"toc-item-num\">3.1.2&nbsp;&nbsp;</span>Splitting into train, test and validation set</a></span></li><li><span><a href=\"#Training-the-model\" data-toc-modified-id=\"Training-the-model-3.1.3\"><span class=\"toc-item-num\">3.1.3&nbsp;&nbsp;</span>Training the model</a></span></li><li><span><a href=\"#Evaluating-the-model\" data-toc-modified-id=\"Evaluating-the-model-3.1.4\"><span class=\"toc-item-num\">3.1.4&nbsp;&nbsp;</span>Evaluating the model</a></span><ul class=\"toc-item\"><li><span><a href=\"#With-train-data\" data-toc-modified-id=\"With-train-data-3.1.4.1\"><span class=\"toc-item-num\">3.1.4.1&nbsp;&nbsp;</span>With train data</a></span></li><li><span><a href=\"#With-test-data\" data-toc-modified-id=\"With-test-data-3.1.4.2\"><span class=\"toc-item-num\">3.1.4.2&nbsp;&nbsp;</span>With test data</a></span></li></ul></li></ul></li><li><span><a href=\"#Aggression-dataset---1-epoch\" data-toc-modified-id=\"Aggression-dataset---1-epoch-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>Aggression dataset - 1 epoch</a></span><ul class=\"toc-item\"><li><span><a href=\"#Training-the-model\" data-toc-modified-id=\"Training-the-model-3.2.1\"><span class=\"toc-item-num\">3.2.1&nbsp;&nbsp;</span>Training the model</a></span></li><li><span><a href=\"#Evaluating-the-model\" data-toc-modified-id=\"Evaluating-the-model-3.2.2\"><span class=\"toc-item-num\">3.2.2&nbsp;&nbsp;</span>Evaluating the model</a></span><ul class=\"toc-item\"><li><span><a href=\"#With-train-data\" data-toc-modified-id=\"With-train-data-3.2.2.1\"><span class=\"toc-item-num\">3.2.2.1&nbsp;&nbsp;</span>With train data</a></span></li><li><span><a href=\"#With-test-data\" data-toc-modified-id=\"With-test-data-3.2.2.2\"><span class=\"toc-item-num\">3.2.2.2&nbsp;&nbsp;</span>With test data</a></span></li></ul></li></ul></li><li><span><a href=\"#Kaggle-dataset---1-epoch\" data-toc-modified-id=\"Kaggle-dataset---1-epoch-3.3\"><span class=\"toc-item-num\">3.3&nbsp;&nbsp;</span>Kaggle dataset - 1 epoch</a></span><ul class=\"toc-item\"><li><span><a href=\"#Loading-the-data-from-the-pickle-files\" data-toc-modified-id=\"Loading-the-data-from-the-pickle-files-3.3.1\"><span class=\"toc-item-num\">3.3.1&nbsp;&nbsp;</span>Loading the data from the pickle files</a></span></li><li><span><a href=\"#Splitting-into-train,-test-and-validation-set\" data-toc-modified-id=\"Splitting-into-train,-test-and-validation-set-3.3.2\"><span class=\"toc-item-num\">3.3.2&nbsp;&nbsp;</span>Splitting into train, test and validation set</a></span></li><li><span><a href=\"#Training-the-model\" data-toc-modified-id=\"Training-the-model-3.3.3\"><span class=\"toc-item-num\">3.3.3&nbsp;&nbsp;</span>Training the model</a></span></li><li><span><a href=\"#Evaluating-the-model\" data-toc-modified-id=\"Evaluating-the-model-3.3.4\"><span class=\"toc-item-num\">3.3.4&nbsp;&nbsp;</span>Evaluating the model</a></span><ul class=\"toc-item\"><li><span><a href=\"#With-Kaggle-train-data\" data-toc-modified-id=\"With-Kaggle-train-data-3.3.4.1\"><span class=\"toc-item-num\">3.3.4.1&nbsp;&nbsp;</span>With Kaggle train data</a></span></li><li><span><a href=\"#With-Kaggle-test-data\" data-toc-modified-id=\"With-Kaggle-test-data-3.3.4.2\"><span class=\"toc-item-num\">3.3.4.2&nbsp;&nbsp;</span>With Kaggle test data</a></span></li></ul></li></ul></li><li><span><a href=\"#Kaggle-dataset---10-epochs\" data-toc-modified-id=\"Kaggle-dataset---10-epochs-3.4\"><span class=\"toc-item-num\">3.4&nbsp;&nbsp;</span>Kaggle dataset - 10 epochs</a></span><ul class=\"toc-item\"><li><span><a href=\"#Training-the-model\" data-toc-modified-id=\"Training-the-model-3.4.1\"><span class=\"toc-item-num\">3.4.1&nbsp;&nbsp;</span>Training the model</a></span></li><li><span><a href=\"#Evaluating-the-model\" data-toc-modified-id=\"Evaluating-the-model-3.4.2\"><span class=\"toc-item-num\">3.4.2&nbsp;&nbsp;</span>Evaluating the model</a></span><ul class=\"toc-item\"><li><span><a href=\"#With-Kaggle-train-data\" data-toc-modified-id=\"With-Kaggle-train-data-3.4.2.1\"><span class=\"toc-item-num\">3.4.2.1&nbsp;&nbsp;</span>With Kaggle train data</a></span></li><li><span><a href=\"#With-Kaggle-test-data\" data-toc-modified-id=\"With-Kaggle-test-data-3.4.2.2\"><span class=\"toc-item-num\">3.4.2.2&nbsp;&nbsp;</span>With Kaggle test data</a></span></li><li><span><a href=\"#Plotting-the-accuracy-and-loss-curves\" data-toc-modified-id=\"Plotting-the-accuracy-and-loss-curves-3.4.2.3\"><span class=\"toc-item-num\">3.4.2.3&nbsp;&nbsp;</span>Plotting the accuracy and loss curves</a></span></li><li><span><a href=\"#Evaluating-with-Aggression-train-data\" data-toc-modified-id=\"Evaluating-with-Aggression-train-data-3.4.2.4\"><span class=\"toc-item-num\">3.4.2.4&nbsp;&nbsp;</span>Evaluating with Aggression train data</a></span></li><li><span><a href=\"#Evaluating-with-Aggression-test-data\" data-toc-modified-id=\"Evaluating-with-Aggression-test-data-3.4.2.5\"><span class=\"toc-item-num\">3.4.2.5&nbsp;&nbsp;</span>Evaluating with Aggression test data</a></span></li></ul></li></ul></li><li><span><a href=\"#Racism-dataset---1-epoch\" data-toc-modified-id=\"Racism-dataset---1-epoch-3.5\"><span class=\"toc-item-num\">3.5&nbsp;&nbsp;</span>Racism dataset - 1 epoch</a></span><ul class=\"toc-item\"><li><span><a href=\"#Loading-the-data-from-the-pickle-files\" data-toc-modified-id=\"Loading-the-data-from-the-pickle-files-3.5.1\"><span class=\"toc-item-num\">3.5.1&nbsp;&nbsp;</span>Loading the data from the pickle files</a></span></li><li><span><a href=\"#Splitting-into-train,-test-and-validation-set\" data-toc-modified-id=\"Splitting-into-train,-test-and-validation-set-3.5.2\"><span class=\"toc-item-num\">3.5.2&nbsp;&nbsp;</span>Splitting into train, test and validation set</a></span></li><li><span><a href=\"#Training-the-model\" data-toc-modified-id=\"Training-the-model-3.5.3\"><span class=\"toc-item-num\">3.5.3&nbsp;&nbsp;</span>Training the model</a></span></li><li><span><a href=\"#Evaluating-the-model\" data-toc-modified-id=\"Evaluating-the-model-3.5.4\"><span class=\"toc-item-num\">3.5.4&nbsp;&nbsp;</span>Evaluating the model</a></span><ul class=\"toc-item\"><li><span><a href=\"#With-Racism-train-data\" data-toc-modified-id=\"With-Racism-train-data-3.5.4.1\"><span class=\"toc-item-num\">3.5.4.1&nbsp;&nbsp;</span>With Racism train data</a></span></li><li><span><a href=\"#With-Racism-test-data\" data-toc-modified-id=\"With-Racism-test-data-3.5.4.2\"><span class=\"toc-item-num\">3.5.4.2&nbsp;&nbsp;</span>With Racism test data</a></span></li><li><span><a href=\"#Evaluating-with-Aggression-train-data\" data-toc-modified-id=\"Evaluating-with-Aggression-train-data-3.5.4.3\"><span class=\"toc-item-num\">3.5.4.3&nbsp;&nbsp;</span>Evaluating with Aggression train data</a></span></li><li><span><a href=\"#Evaluating-with-Aggression-test-data\" data-toc-modified-id=\"Evaluating-with-Aggression-test-data-3.5.4.4\"><span class=\"toc-item-num\">3.5.4.4&nbsp;&nbsp;</span>Evaluating with Aggression test data</a></span></li></ul></li></ul></li></ul></li><li><span><a href=\"#Conclusion\" data-toc-modified-id=\"Conclusion-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Conclusion</a></span></li></ul></div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OLGEF4orcUmr"
      },
      "source": [
        "# BERT Fine-tuning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M8zMxgaj-kfz"
      },
      "source": [
        "In this notebook I have fine tuned the pre-trained BERT base model (uncased). My intention was to train a binary classifier (class 0 = not bullying / class 1 = bullying) on the 8 datasets that I preprocessed and tokenized. \n",
        "\n",
        "After fine-tuning the model with one dataset I obtained satisfactory results. However, as I kept fine-tuning with more datasets, the score of the model decreased. \n",
        "\n",
        "My conclusion is that to distinguish between different types of bullying behaviors (racism, sexism, aggression, etc) we would need to build a multiclass classifier, not a binary classifier. Due to time constraints for this project, I decided to train the model with one dataset only to ensure the model has a good score. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3vRX1i6WPD7Z"
      },
      "source": [
        "## Environment"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yymChyEhPccH"
      },
      "source": [
        "### Libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d-4PXEPEPnkn"
      },
      "source": [
        "!pip install transformers"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zSCmn831Pesa"
      },
      "source": [
        "# Base libraries\n",
        "import os\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.ticker as ticker\n",
        "import pickle\n",
        "\n",
        "# ML and DL libraries\n",
        "import keras\n",
        "from keras.models import Model\n",
        "from keras.callbacks import ModelCheckpoint\n",
        "from keras.models import load_model\n",
        "import keras.backend as K\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import confusion_matrix,f1_score,classification_report\n",
        "from sklearn.utils import shuffle\n",
        "from transformers import BertTokenizer, TFBertModel, BertConfig, TFBertForSequenceClassification"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JZvvUxu0b-mD",
        "outputId": "3da2fb50-df1f-4cc0-81ca-17765407c07a"
      },
      "source": [
        "# Connecting to google drive to be able to use files that are stored there\n",
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/gdrive; to attempt to forcibly remount, call drive.mount(\"/content/gdrive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yL1eRChsCCse"
      },
      "source": [
        "### Functions"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2GJtH0GuCEnB"
      },
      "source": [
        "def load_vectors(dataset_name):\n",
        "  \"\"\"\n",
        "  Input: name of the dataset\n",
        "  Output: dataset tokenized into 3 vectors (input_ids, attention_masks and labels) ready to be inputed into the model \n",
        "  \"\"\"\n",
        "  \n",
        "  # Create the paths\n",
        "  pickle_inp_path = f'/content/gdrive/MyDrive/Cyber-bullying-project/data/3_tokenized_data/bert_inp_{dataset_name}.pkl'\n",
        "  pickle_mask_path = f'/content/gdrive/MyDrive/Cyber-bullying-project/data/3_tokenized_data/bert_mask_{dataset_name}.pkl'\n",
        "  pickle_label_path = f'/content/gdrive/MyDrive/Cyber-bullying-project/data/3_tokenized_data/bert_label_{dataset_name}.pkl'\n",
        "\n",
        "  # Load the files\n",
        "  input_ids = pickle.load(open(pickle_inp_path, 'rb'))\n",
        "  attention_masks = pickle.load(open(pickle_mask_path, 'rb'))\n",
        "  labels = pickle.load(open(pickle_label_path, 'rb'))\n",
        "\n",
        "  return input_ids, attention_masks, labels"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Uy1nMl5SD_aU"
      },
      "source": [
        "def train_test_val(input_ids, attention_masks, labels):\n",
        "  \"\"\" \n",
        "  Input: 3 vectors (input_ids, attention_masks and labels)\n",
        "  Output: 9 vectors (the 3 input vectors split into train, test and validation)\n",
        "  \"\"\"\n",
        "  train_inp, test_inp, train_mask, test_mask, train_label, test_label = train_test_split(input_ids,\n",
        "                                                                                         attention_masks,\n",
        "                                                                                         labels,\n",
        "                                                                                         test_size = 0.2,\n",
        "                                                                                         random_state = 42,\n",
        "                                                                                         stratify = labels)\n",
        "  \n",
        "  train_inp, val_inp, train_mask, val_mask, train_label, val_label = train_test_split(train_inp,\n",
        "                                                                                      train_mask,\n",
        "                                                                                      train_label,\n",
        "                                                                                      test_size = 0.2,\n",
        "                                                                                      random_state = 42,\n",
        "                                                                                      stratify = train_label)\n",
        "  \n",
        "  return train_inp, val_inp, test_inp, train_mask, val_mask, test_mask, train_label, val_label, test_label"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RDCrMzkjSoBK"
      },
      "source": [
        "def bert_setup():\n",
        "  \"\"\" \n",
        "  Loads BERT for sequence classification; sets the loss, metric and optimizer; compiles the model\n",
        "  \"\"\"\n",
        "  \n",
        "  # Load BERT base model\n",
        "  base_model = TFBertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)\n",
        "  \n",
        "  # Define loss, metric and optimizer\n",
        "  loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n",
        "  metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')\n",
        "  optimizer = tf.keras.optimizers.Adam(learning_rate=2e-5,\n",
        "                                       epsilon=1e-08)\n",
        "  # Compile model\n",
        "  base_model.compile(loss = loss, optimizer = optimizer, metrics = [metric])\n",
        "  \n",
        "  return base_model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LnXYEXIlSoBL"
      },
      "source": [
        "def train_model(base_model, new_model_name, old_model_name=None, batch_size=32, epochs=4):\n",
        "  \"\"\" \n",
        "  Input: BERT model compiled; name we want to give to the new model; name of the model weights we want to load (optional); batch size; epochs\n",
        "  Output: a trained model, history variable\n",
        "  \"\"\"\n",
        "\n",
        "  # Create a new model from the base model and set the save path for the new model\n",
        "  new_model = base_model\n",
        "  new_model_save_path = f'/content/gdrive/MyDrive/Cyber-bullying-project/models/{new_model_name}.h5'\n",
        "  print(f'\\nNew model will be saved at {new_model_save_path}')\n",
        "\n",
        "  # Defining the callbacks variable\n",
        "  callbacks = [tf.keras.callbacks.ModelCheckpoint(filepath = new_model_save_path,\n",
        "                                                  save_weights_only = True,\n",
        "                                                  monitor = 'val_loss',\n",
        "                                                  mode = 'min',\n",
        "                                                  save_best_only = True)]\n",
        "\n",
        "  # If old_model_name is defined, load the weights of that model\n",
        "  if old_model_name: \n",
        "    old_model_save_path = f'/content/gdrive/MyDrive/Cyber-bullying-project/models/{old_model_name}.h5'\n",
        "    new_model.load_weights(old_model_save_path)\n",
        "    print(f'\\nTraining on previous model {old_model_name}')\n",
        "  else:\n",
        "    print('\\nTraining new model')\n",
        "  \n",
        "  # Train the model\n",
        "  print('\\nTraining starting...\\n\\n')\n",
        "  history = new_model.fit(x = [train_inp, train_mask],\n",
        "                           y = train_label,\n",
        "                           batch_size = batch_size,\n",
        "                           epochs = epochs,\n",
        "                           validation_data = ([val_inp, val_mask], val_label), \n",
        "                           callbacks = callbacks)\n",
        "  \n",
        "  # Save the history\n",
        "  history_save_path = f'/content/gdrive/MyDrive/Cyber-bullying-project/models/histories/{new_model_name}_hist.pkl'\n",
        "  print(f'\\nFinished training. Saving history at {history_save_path} ...')\n",
        "  pickle.dump((history.history), open(history_save_path,'wb'))\n",
        "\n",
        "  return new_model, history.history"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AR4ZIh6PdF3n"
      },
      "source": [
        "def plot_history(history, metric='loss'):\n",
        "  \"\"\" \n",
        "  Input: history, metric to plot\n",
        "  Output: history plot\n",
        "  \"\"\"\n",
        "\n",
        "  # If history is a string, load it into a variable\n",
        "  if type(history) == str:\n",
        "    history_path = f'/content/gdrive/MyDrive/Cyber-bullying-project/models/histories/{history}.pkl'\n",
        "    hist = pickle.load(open(history_path, 'rb'))\n",
        "\n",
        "  # If history is a defined variable, we don't need to load it again\n",
        "  else:\n",
        "    hist = history\n",
        "\n",
        "  # Plot the history: \n",
        "  plt.plot(np.arange(len(hist[f'{metric}'])) + 0.5, hist[f'{metric}'], 'b.', label=f'Training {metric}')\n",
        "  plt.plot(np.arange(len(hist[f'val_{metric}'])) + 1, hist[f'val_{metric}'], 'r-', label=f'Validation {metric}')\n",
        "  plt.gca().xaxis.set_major_locator(ticker.MaxNLocator(integer=True))\n",
        "  plt.legend(fontsize=14)\n",
        "  plt.xlabel('Epochs')\n",
        "  plt.ylabel(f'{metric.capitalize()}')\n",
        "  plt.grid(True) \n",
        "\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kyPH_-OxSwmo"
      },
      "source": [
        "def evaluate_model(model_name, inputs, mask, labels, base_model=None):\n",
        "  \"\"\" \n",
        "  Input: model_name (a defined variable or a string with the name of a model we want to load); \n",
        "  evaluation data split into 3 vectors (inputs, mask and labels);\n",
        "  if the model_name is a string, we need a base model to load the weights into)\n",
        "\n",
        "  Output: predictions, predicted labels, f1 score, classification report, confusion matrix\n",
        "  \"\"\"\n",
        "  \n",
        "  # Load the model weights if model_name is a string\n",
        "  if type(model_name) == str:\n",
        "    model_save_path = f'/content/gdrive/MyDrive/Cyber-bullying-project/models/{model_name}.h5'\n",
        "    base_model.load_weights(model_save_path)\n",
        "    trained_model = base_model\n",
        "  \n",
        "  # If model_name is a defined variable, we don't need to load the weights\n",
        "  else:\n",
        "    trained_model = model_name\n",
        "  \n",
        "  # Make predictions\n",
        "  preds = trained_model.predict([inputs, mask],\n",
        "                                batch_size=32)\n",
        "  \n",
        "  # Find the predicted labels\n",
        "  pred_labels = [np.argmax(pred) for pred in preds[0]]\n",
        "  \n",
        "  # Store evaluation metrics and reports in variables\n",
        "  f1 = f1_score(labels, pred_labels)\n",
        "  class_report = classification_report(labels, pred_labels)\n",
        "  conf_matrix = confusion_matrix(labels, pred_labels)\n",
        "\n",
        "  return preds, pred_labels, f1, class_report, conf_matrix"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E-BclTlh3mXt"
      },
      "source": [
        "def print_eval(f1, class_report, conf_matrix):\n",
        "  \"\"\" \n",
        "  Input: f1 score, classification report, confusion matrix\n",
        "  Output: print the inputs with titles\n",
        "  \"\"\"\n",
        "\n",
        "  print('F1 score:', f1)\n",
        "  print('Classification Report:')\n",
        "  print(class_report)\n",
        "  print('Confusion Matrix:')\n",
        "  print(conf_matrix)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HOXbd0P2dyrT"
      },
      "source": [
        "## BERT model setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PswEQPYNeCuV",
        "outputId": "0405bbe7-779e-4ee8-d801-13659deaebb2"
      },
      "source": [
        "# set the base model \n",
        "base_model = bert_setup()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "All model checkpoint layers were used when initializing TFBertForSequenceClassification.\n",
            "\n",
            "Some layers of TFBertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jl9dnsQLZGLn"
      },
      "source": [
        "## Fine-tuning the pre-trained BERT model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GWO2j4B728KG"
      },
      "source": [
        "### Aggression dataset - 4 epochs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AHK0Q85vhfTJ"
      },
      "source": [
        "#### Loading the data from the pickle files"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ft5_W2L1hfTM"
      },
      "source": [
        "# Load the inpud_ids, attention_masks and labels for the aggression dataset\n",
        "input_ids, attention_masks, labels = load_vectors('aggression')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LVpXEzfchfTM"
      },
      "source": [
        "#### Splitting into train, test and validation set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wsqzlbm_hfTM"
      },
      "source": [
        "train_inp, val_inp, test_inp, train_mask, val_mask, test_mask, train_label, val_label, test_label = train_test_val(\n",
        "    input_ids, attention_masks, labels)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8zqXWwmAIwmq"
      },
      "source": [
        "#### Training the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5UhsDOXRIwm1",
        "outputId": "26c546b1-0982-4254-ed86-1ad03da0bf67"
      },
      "source": [
        "# Train the model and save the weights\n",
        "aggression_model_new = train_model(base_model=base_model, new_model_name='aggression_model_new')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "New model will be saved at /content/gdrive/MyDrive/Cyber-bullying-project/models/aggression_model_new.h5\n",
            "\n",
            "Training new model\n",
            "\n",
            "Training starting...\n",
            "\n",
            "\n",
            "Epoch 1/4\n",
            "WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n",
            "WARNING:tensorflow:AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f578ffb1d70>> and will run it as-is.\n",
            "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
            "Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method\n",
            "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
            "WARNING: AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f578ffb1d70>> and will run it as-is.\n",
            "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
            "Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method\n",
            "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
            "WARNING:tensorflow:AutoGraph could not transform <function wrap at 0x7f57ab166170> and will run it as-is.\n",
            "Cause: while/else statement not yet supported\n",
            "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
            "WARNING: AutoGraph could not transform <function wrap at 0x7f57ab166170> and will run it as-is.\n",
            "Cause: while/else statement not yet supported\n",
            "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
            "WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/array_ops.py:5049: calling gather (from tensorflow.python.ops.array_ops) with validate_indices is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "The `validate_indices` argument has no effect. Indices are always validated on CPU and never validated on GPU.\n",
            "WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n",
            "WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n",
            "2318/2318 [==============================] - ETA: 0s - loss: 0.1670 - accuracy: 0.9365WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n",
            "WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n",
            "2318/2318 [==============================] - 1583s 661ms/step - loss: 0.1670 - accuracy: 0.9365 - val_loss: 0.1392 - val_accuracy: 0.9460\n",
            "Epoch 2/4\n",
            "2318/2318 [==============================] - 1535s 662ms/step - loss: 0.1225 - accuracy: 0.9530 - val_loss: 0.1595 - val_accuracy: 0.9427\n",
            "Epoch 3/4\n",
            "2318/2318 [==============================] - 1534s 662ms/step - loss: 0.0787 - accuracy: 0.9700 - val_loss: 0.1698 - val_accuracy: 0.9361\n",
            "Epoch 4/4\n",
            "2318/2318 [==============================] - 1534s 662ms/step - loss: 0.0429 - accuracy: 0.9846 - val_loss: 0.2620 - val_accuracy: 0.9347\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tP8tOGkDsvM0"
      },
      "source": [
        "We can see that we have the best results for the validation set (lowest loss and highest accuracy) in the first epoch. After that, the model starts overfitting, as the results improve for the training set but they get worse for the validation set. Therefore in the next iteration we will limit the training to one epoch and see if we have better results with the test set. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cah3mrqHIwm1"
      },
      "source": [
        "#### Evaluating the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s-agL_yUIwm1"
      },
      "source": [
        "##### With train data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9owu2aWxIwm2"
      },
      "source": [
        "# Generate predictions, predicted labels, F1 score, classification report and confusion matrix with train data\n",
        "preds_train, pred_labels_train, f1_train, class_report_train, conf_matrix_train = evaluate_model(aggression_model_new, train_inp, train_mask, train_label)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3cptXQ_MIwm2",
        "outputId": "e5649d6b-f1fc-45fb-c9b3-516949ffe2bc"
      },
      "source": [
        "# Print the evaluation metric and reports\n",
        "print_eval(f1_train, class_report_train, conf_matrix_train)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "F1 score: 0.980590629239278\n",
            "Classification Report:\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       1.00      1.00      1.00     64691\n",
            "           1       0.97      0.99      0.98      9461\n",
            "\n",
            "    accuracy                           0.99     74152\n",
            "   macro avg       0.98      0.99      0.99     74152\n",
            "weighted avg       1.00      0.99      1.00     74152\n",
            "\n",
            "Confusion Matrix:\n",
            "[[64383   308]\n",
            " [   64  9397]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t7LMYyaLIwm2"
      },
      "source": [
        "##### With test data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wb3V23-RIwm2"
      },
      "source": [
        "# Generate predictions, predicted labels, F1 score, classification report and confusion matrix with test data\n",
        "preds_test, pred_labels_test, f1_test, class_report_test, conf_matrix_test = evaluate_model(aggression_model_new, test_inp, test_mask, test_label)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DsXzBw0fIwm2",
        "outputId": "b392da13-0602-4f3a-ccdb-3a9ffa64cb45"
      },
      "source": [
        "# Print the evaluation metric and reports\n",
        "print_eval(f1_test, class_report_test, conf_matrix_test)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "F1 score: 0.7467543138866065\n",
            "Classification Report:\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.97      0.96      0.96     20217\n",
            "           1       0.73      0.77      0.75      2956\n",
            "\n",
            "    accuracy                           0.93     23173\n",
            "   macro avg       0.85      0.86      0.85     23173\n",
            "weighted avg       0.94      0.93      0.93     23173\n",
            "\n",
            "Confusion Matrix:\n",
            "[[19360   857]\n",
            " [  684  2272]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ygGKa9-hEHC"
      },
      "source": [
        "Insight: the metrics are almost perfect in the evaluation with the train dataset, which is a sign of overfitting. In the evaluation with the test dataset we have a lower precision and recall, especially for the class 1. We will now train the model with only one epoch. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q-aEK0IPz2ld"
      },
      "source": [
        "### Aggression dataset - 1 epoch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kErP1xBUz2le"
      },
      "source": [
        "#### Training the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yylkFQxkz2le",
        "outputId": "0dc3bb9c-92f4-44c5-8718-897418126c47"
      },
      "source": [
        "aggression_model_1epoch, history = train_model(\n",
        "    base_model=base_model, new_model_name='aggression_model_1epoch', epochs=1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "New model will be saved at /content/gdrive/MyDrive/Cyber-bullying-project/models/aggression_model_1epoch.h5\n",
            "\n",
            "Training new model\n",
            "\n",
            "Training starting...\n",
            "\n",
            "\n",
            "WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n",
            "WARNING:tensorflow:AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f28e1a43d70>> and will run it as-is.\n",
            "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
            "Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method\n",
            "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
            "WARNING: AutoGraph could not transform <bound method Socket.send of <zmq.sugar.socket.Socket object at 0x7f28e1a43d70>> and will run it as-is.\n",
            "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
            "Cause: module, class, method, function, traceback, frame, or code object was expected, got cython_function_or_method\n",
            "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
            "WARNING:tensorflow:AutoGraph could not transform <function wrap at 0x7f28fcbf5170> and will run it as-is.\n",
            "Cause: while/else statement not yet supported\n",
            "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
            "WARNING: AutoGraph could not transform <function wrap at 0x7f28fcbf5170> and will run it as-is.\n",
            "Cause: while/else statement not yet supported\n",
            "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
            "WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/array_ops.py:5049: calling gather (from tensorflow.python.ops.array_ops) with validate_indices is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "The `validate_indices` argument has no effect. Indices are always validated on CPU and never validated on GPU.\n",
            "WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n",
            "WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n",
            "2318/2318 [==============================] - ETA: 0s - loss: 0.1677 - accuracy: 0.9371WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n",
            "WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n",
            "2318/2318 [==============================] - 1639s 684ms/step - loss: 0.1677 - accuracy: 0.9371 - val_loss: 0.1395 - val_accuracy: 0.9461\n",
            "\n",
            "Finished training. Saving history at /content/gdrive/MyDrive/Cyber-bullying-project/models/histories/aggression_model_1epoch_hist.pkl ...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UM-rsXXSz2lg"
      },
      "source": [
        "#### Evaluating the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9zXp1wj7z2lg"
      },
      "source": [
        "##### With train data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CzqI3FIkz2lg"
      },
      "source": [
        "preds_train, pred_labels_train, f1_train, class_report_train, conf_matrix_train = evaluate_model(\n",
        "    aggression_model_1epoch, train_inp, train_mask, train_label)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GbsmUrafz2lg",
        "outputId": "8bb1f23b-39e1-4851-e6c4-45ca628b22ba"
      },
      "source": [
        "print_eval(f1_train, class_report_train, conf_matrix_train)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "F1 score: 0.830719131614654\n",
            "Classification Report:\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.97      0.98      0.98     64691\n",
            "           1       0.85      0.81      0.83      9461\n",
            "\n",
            "    accuracy                           0.96     74152\n",
            "   macro avg       0.91      0.89      0.90     74152\n",
            "weighted avg       0.96      0.96      0.96     74152\n",
            "\n",
            "Confusion Matrix:\n",
            "[[63380  1311]\n",
            " [ 1808  7653]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WOAD63Hdz2lh"
      },
      "source": [
        "##### With test data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lgs_CPbgz2lh"
      },
      "source": [
        "preds_test, pred_labels_test, f1_test, class_report_test, conf_matrix_test = evaluate_model(\n",
        "    aggression_model_1epoch, test_inp, test_mask, test_label)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "anh_o1mtz2lh",
        "outputId": "efaf5833-51e1-4446-ed1d-061d53b44398"
      },
      "source": [
        "print_eval(f1_test, class_report_test, conf_matrix_test)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "F1 score: 0.7775453277545327\n",
            "Classification Report:\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.96      0.97      0.97     20217\n",
            "           1       0.80      0.75      0.78      2956\n",
            "\n",
            "    accuracy                           0.94     23173\n",
            "   macro avg       0.88      0.86      0.87     23173\n",
            "weighted avg       0.94      0.94      0.94     23173\n",
            "\n",
            "Confusion Matrix:\n",
            "[[19667   550]\n",
            " [  726  2230]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "urJWanAGiUgx"
      },
      "source": [
        "Insight: \n",
        "- With this model we have less overfitting, as the evaluation metrics with train split are closer to the evaluation with the test split. \n",
        "- The precision and F1 score are better than in the 4-epoch model. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tq62wr6OA3e5"
      },
      "source": [
        "### Kaggle dataset - 1 epoch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qO9D-o5l2k60"
      },
      "source": [
        "We will now train the last model (the one trained on the aggression dataset with only one epoch) with the Kaggle dataset. We will start with one epoch since this is what worked best with the Aggression dataset. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H5irrYs2A3e6"
      },
      "source": [
        "#### Loading the data from the pickle files"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fSGw7dd0A3e6"
      },
      "source": [
        "# Load the inpud_ids, attention_masks and labels for the Kaggle dataset\n",
        "input_ids, attention_masks, labels = load_vectors('kaggle')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MPlzesQ_A3e6"
      },
      "source": [
        "#### Splitting into train, test and validation set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z9JO4cToA3e6"
      },
      "source": [
        "train_inp, val_inp, test_inp, train_mask, val_mask, test_mask, train_label, val_label, test_label = train_test_val(\n",
        "    input_ids, attention_masks, labels)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qR0L52cDA3e6"
      },
      "source": [
        "#### Training the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XvHcMjtgA3e6",
        "outputId": "3add1c27-2a27-43a5-a129-6a91663930d2"
      },
      "source": [
        "kaggle_model_1epoch, history = train_model(\n",
        "    base_model=base_model, \n",
        "    old_model_name='aggression_model_1epoch', \n",
        "    new_model_name='kaggle_model_1epoch', \n",
        "    epochs=1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "New model will be saved at /content/gdrive/MyDrive/Cyber-bullying-project/models/kaggle_model_1epoch.h5\n",
            "\n",
            "Training on previous model aggression_model_1epoch\n",
            "\n",
            "Training starting...\n",
            "\n",
            "\n",
            "176/176 [==============================] - ETA: 0s - loss: 0.4239 - accuracy: 0.8105WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n",
            "WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n",
            "176/176 [==============================] - 122s 694ms/step - loss: 0.4239 - accuracy: 0.8105 - val_loss: 0.3969 - val_accuracy: 0.8239\n",
            "\n",
            "Finished training. Saving history at /content/gdrive/MyDrive/Cyber-bullying-project/models/histories/kaggle_model_1epoch_hist.pkl ...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q1gjTIj1A3e7"
      },
      "source": [
        "#### Evaluating the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uuiuNQBCA3e7"
      },
      "source": [
        "##### With Kaggle train data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vwQB0Zg1A3e7"
      },
      "source": [
        "preds_train_kaggle, pred_labels_train_kaggle, f1_train_kaggle, class_report_train_kaggle, conf_matrix_train_kaggle = evaluate_model(\n",
        "    kaggle_model_1epoch, train_inp, train_mask, train_label)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "m6bzSgrdA3e7",
        "outputId": "7b20b641-1f6a-43c2-a53c-f145c195b9f9"
      },
      "source": [
        "print_eval(f1_train_kaggle, class_report_train_kaggle, conf_matrix_train_kaggle)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "F1 score: 0.8232265446224255\n",
            "Classification Report:\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.91      0.93      0.92      3835\n",
            "           1       0.85      0.80      0.82      1796\n",
            "\n",
            "    accuracy                           0.89      5631\n",
            "   macro avg       0.88      0.87      0.87      5631\n",
            "weighted avg       0.89      0.89      0.89      5631\n",
            "\n",
            "Confusion Matrix:\n",
            "[[3574  261]\n",
            " [ 357 1439]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gKL4kgtgA3e7"
      },
      "source": [
        "##### With Kaggle test data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mq99w-ANA3e7"
      },
      "source": [
        "preds_test_kaggle, pred_labels_test_kaggle, f1_test_kaggle, class_report_test_kaggle, conf_matrix_test_kaggle = evaluate_model(\n",
        "    kaggle_model_1epoch, test_inp, test_mask, test_label)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FPoKSWeEA3e7",
        "outputId": "cb56cf5a-9cc8-476d-8933-587e50c7fb5f"
      },
      "source": [
        "print_eval(f1_test_kaggle, class_report_test_kaggle, conf_matrix_test_kaggle)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "F1 score: 0.7281105990783411\n",
            "Classification Report:\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.87      0.89      0.88      1199\n",
            "           1       0.75      0.70      0.73       561\n",
            "\n",
            "    accuracy                           0.83      1760\n",
            "   macro avg       0.81      0.80      0.80      1760\n",
            "weighted avg       0.83      0.83      0.83      1760\n",
            "\n",
            "Confusion Matrix:\n",
            "[[1070  129]\n",
            " [ 166  395]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "33qRZrYU3Dpu"
      },
      "source": [
        "Insight: all the metrics are slightly lower than in the evaluation of the aggression dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c4cQ-IZSDCt6"
      },
      "source": [
        "### Kaggle dataset - 10 epochs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IeNbrRwU3PTg"
      },
      "source": [
        "We will now train the model with the same data in 9 more epochs, to see if we can improve its evaluation metrics. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eyHPe607DCt7"
      },
      "source": [
        "#### Training the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RbdZWUc-DCt7",
        "outputId": "dbb6de78-3a7f-436a-c025-6dfbf7e36fef"
      },
      "source": [
        "kaggle_model_10epochs, history = train_model(\n",
        "    base_model=base_model, \n",
        "    old_model_name='kaggle_model_1epoch', \n",
        "    new_model_name='kaggle_model_10epochs', \n",
        "    epochs=9)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "New model will be saved at /content/gdrive/MyDrive/Cyber-bullying-project/models/kaggle_model_10epochs.h5\n",
            "\n",
            "Training on previous model kaggle_model_1epoch\n",
            "\n",
            "Training starting...\n",
            "\n",
            "\n",
            "Epoch 1/9\n",
            "176/176 [==============================] - 120s 683ms/step - loss: 0.2915 - accuracy: 0.8718 - val_loss: 0.4574 - val_accuracy: 0.8253\n",
            "Epoch 2/9\n",
            "176/176 [==============================] - 119s 675ms/step - loss: 0.1587 - accuracy: 0.9410 - val_loss: 0.5656 - val_accuracy: 0.7891\n",
            "Epoch 3/9\n",
            "176/176 [==============================] - 119s 676ms/step - loss: 0.0738 - accuracy: 0.9755 - val_loss: 0.7335 - val_accuracy: 0.8040\n",
            "Epoch 4/9\n",
            "176/176 [==============================] - 119s 678ms/step - loss: 0.0491 - accuracy: 0.9822 - val_loss: 0.8115 - val_accuracy: 0.8139\n",
            "Epoch 5/9\n",
            "176/176 [==============================] - 119s 676ms/step - loss: 0.0345 - accuracy: 0.9877 - val_loss: 0.9669 - val_accuracy: 0.8061\n",
            "Epoch 6/9\n",
            "176/176 [==============================] - 119s 676ms/step - loss: 0.0295 - accuracy: 0.9909 - val_loss: 0.8954 - val_accuracy: 0.8153\n",
            "Epoch 7/9\n",
            "176/176 [==============================] - 119s 675ms/step - loss: 0.0247 - accuracy: 0.9920 - val_loss: 0.9058 - val_accuracy: 0.8089\n",
            "Epoch 8/9\n",
            "176/176 [==============================] - 119s 675ms/step - loss: 0.0171 - accuracy: 0.9943 - val_loss: 1.0830 - val_accuracy: 0.7891\n",
            "Epoch 9/9\n",
            "176/176 [==============================] - 119s 674ms/step - loss: 0.0261 - accuracy: 0.9911 - val_loss: 0.8858 - val_accuracy: 0.8089\n",
            "\n",
            "Finished training. Saving history at /content/gdrive/MyDrive/Cyber-bullying-project/models/histories/kaggle_model_10epochs_hist.pkl ...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DwxBoFGu3oq9"
      },
      "source": [
        "Insight: we can see that the validation loss has increased with every epoch. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xVBS6YdUDCt7"
      },
      "source": [
        "#### Evaluating the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hU4QJdZzDCt7"
      },
      "source": [
        "##### With Kaggle train data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xLM3h5FUDCt7"
      },
      "source": [
        "preds_train_kaggle, pred_labels_train_kaggle, f1_train_kaggle, class_report_train_kaggle, conf_matrix_train_kaggle = evaluate_model(\n",
        "    kaggle_model_1epoch, train_inp, train_mask, train_label)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mW8HwMQ9DCt7",
        "outputId": "64acd12d-38d5-4586-e294-d134b6bffdf2"
      },
      "source": [
        "print_eval(f1_train_kaggle, class_report_train_kaggle, conf_matrix_train_kaggle)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "F1 score: 0.9944227551589515\n",
            "Classification Report:\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       1.00      1.00      1.00      3835\n",
            "           1       1.00      0.99      0.99      1796\n",
            "\n",
            "    accuracy                           1.00      5631\n",
            "   macro avg       1.00      1.00      1.00      5631\n",
            "weighted avg       1.00      1.00      1.00      5631\n",
            "\n",
            "Confusion Matrix:\n",
            "[[3828    7]\n",
            " [  13 1783]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CoVVOv5VDCt8"
      },
      "source": [
        "##### With Kaggle test data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PXWfEhzgDCt8"
      },
      "source": [
        "preds_test_kaggle, pred_labels_test_kaggle, f1_test_kaggle, class_report_test_kaggle, conf_matrix_test_kaggle = evaluate_model(\n",
        "    kaggle_model_1epoch, test_inp, test_mask, test_label)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JH9GMqVrDCt8",
        "outputId": "918d8a0e-7989-42b3-f0cd-2a0d31ec6059"
      },
      "source": [
        "print_eval(f1_test_kaggle, class_report_test_kaggle, conf_matrix_test_kaggle)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "F1 score: 0.7174721189591077\n",
            "Classification Report:\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.86      0.89      0.88      1199\n",
            "           1       0.75      0.69      0.72       561\n",
            "\n",
            "    accuracy                           0.83      1760\n",
            "   macro avg       0.80      0.79      0.80      1760\n",
            "weighted avg       0.82      0.83      0.83      1760\n",
            "\n",
            "Confusion Matrix:\n",
            "[[1070  129]\n",
            " [ 175  386]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r1Lu8uHa31Ed"
      },
      "source": [
        "Inisghts: \n",
        "- There is a big difference between the evaluations with the train and test splits. The metrics are almost perfect in training, which means that the model is overfitting. \n",
        "- The metrics of the evaluation data are worse than in the previous iteration with only one epoch. The recall for class 1 has decreased, which makes the F1 score decrease as well. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Spyn-PQaDCt8"
      },
      "source": [
        "##### Plotting the accuracy and loss curves"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yvoP_wzGC75Q"
      },
      "source": [
        "# Load the history of the 1-epoch model\n",
        "path_kaggle_1epoch = f'/content/gdrive/MyDrive/Cyber-bullying-project/models/histories/kaggle_model_1epoch_hist.pkl'\n",
        "kaggle_1epoch = pickle.load(open(path_kaggle_1epoch, 'rb'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iY4qW4ymC7ua"
      },
      "source": [
        "# Load the history of the 9-epoch model\n",
        "path_kaggle_10epochs = f'/content/gdrive/MyDrive/Cyber-bullying-project/models/histories/kaggle_model_10epochs_hist.pkl'\n",
        "kaggle_10epochs = pickle.load(open(path_kaggle_10epochs, 'rb'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XXwZ4h_aY9H0"
      },
      "source": [
        "# Concatenate the two histories into one dataframe to obtain the 10-epoch model history\n",
        "history_df = pd.concat([pd.DataFrame(kaggle_1epoch), pd.DataFrame(kaggle_10epochs)]).reset_index(drop=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "id": "Fll0cF6aZOCj",
        "outputId": "09aace68-1aff-49e0-e139-e4f9316d3cb2"
      },
      "source": [
        "plot_history(history_df, metric='loss')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZyN9R7A8c/XWGZsY7vGNhnV4FoTqalkypIoChVaKKK00KVUt6tISSgUlbLdZCu6qUQlU5EllRa0yFIoW4XJ0pj53j9+xzSmweA85zkz5/t+veZ1znnOc57n+zszc77ntzy/n6gqxhhjIlcBvwMwxhjjL0sExhgT4SwRGGNMhLNEYIwxEc4SgTHGRLiCfgdwosqVK6cJCQl+h+GLP/74g2LFivkdhm8ivfxg74GV/+TL/+mnn+5U1X/k9FyeSwQJCQmsXLnS7zB8kZKSQnJyst9h+CbSyw/2Hlj5T778IrLpaM9Z05AxxkQ4SwTGGBPhLBEYY0yEs0RgjDERzhKBMcZEuDw3auh49uzZw/bt20lLS/M7lKCLjY1l7dq1fofhG7/KX6hQIcqXL0/JkiVDfm5jQiFfJYI9e/awbds2KleuTExMDCLid0hBtXfvXkqUKOF3GL7xo/yqyv79+9myZQuAJQOTL+WrpqHt27dTuXJlihYtmu+SgPGHiFC0aFEqV67M9u3b/Q7H+CUtDV54AXbs8DsST+SrRJCWlkZMTIzfYZh8KCYmJl82N5pcOHQIOneGnj2hVStITfU7oqDLV4kAsJqA8YT9XUWo9HS48UaYPRu6doVVq6BTJ5cc8pF8lwiMMSYoMjKge3eYPh0efxwmT4Zx4+Ctt+DOOyEfre6YrzqLjTEmKDIy4NZbYcoUGDQIBgxw23v1go0bXWJISPhrex5nNYJ8qlu3blx++eUn9Jrk5GTuuOMOjyL6y8MPP0ydOnU8P48xJ0UV7rrLdQ7/+9/wn/8c+fyjj7o+g/vug2nT/IkxyKxG4LPjtT137dqVyZMnn/BxR48ejZ5g1XXOnDkUKlTohM9lTL6hCv36wdix0L8/PPIIZP8fLVAAJk2CrVvhppugcmVo2tSfeIPEEoHPfv7558z7b775JrfccssR27KPgkpLS8vVh3VsbOwJx1KmTJkTfo0x+YYq3H8/PPWUqxE88cTfk8BhRYrAa6/BBRfAlVfCkiVQq1Zo4w0iaxo6iqVLYehQd+ulChUqZP6UKlXqiG0HDhygVKlSTJ8+nUsuuYTy5cvz/PPPs2vXLjp37kyVKlWIiYmhdu3aTJo06YjjZm8aSk5Opnfv3jzwwAOUK1eO8uXL079/fzIyMo7YJ2vTUEJCAkOGDKFXr16ULFmSKlWqMHz48CPO891339G0aVOio6OpUaMG8+bNo3jx4idUi8nIyOCRRx4hPj6eIkWKULduXV5//fUj9hk8eDC1a9emSJEiVKhQgRtvvDHzuQ8//JDzzjuP4sWLExsbS+PGjfn6669zfX5jAHj4YRg2zPUNjBp19CRwWOnS8PbbEB0NrVtDli9weY0lghwsXQrNmrmmwWbNvE8Gx3P//ffTu3dvVqxYwZVXXsmBAwc4++yzefPNN1m9ejV9+vShV69eLFy48JjHefnllylYsCAff/wxzzzzDKNGjWLmzJnHfM1TTz1F3bp1+eyzzxgwYAD33nsvSwNvSEZGBldddRUFCxZk2bJlTJ48mUGDBnHw4METKt/o0aMZPnw4w4YN46uvvuKqq66iffv2rFq1CoDZs2czYsQIRo4cyffff8+bb75J48aNATh06BDt2rXjwgsv5IsvvmD58uX07duXqKioE4rBRLhHH4XBg+Hmm12zUG6HC1etCm++CTt3wuWX591rDFQ1T/00bNhQj2bNmjVHfe5EPPaYalSUKrjbxx4LymGP65VXXlH3K3E2bNiggI4YMUJVVffs2XPU11577bXavXv3zMddu3bVNm3aZD5u2rSpnnfeeUe8pnnz5ke8pmnTpnr77bdnPq5atap26tTpiNeceeaZ+sgjj6iq6vz58zUqKko3b96c+fySJUsU0EmTJh011oceekhr166d+bhSpUo6aNCgI/Zp2rSpXnfddaqqOnLkSK1evbru2rXrb8fatWuXApqSknLU8wVLsP6+TsWiRYv8DsFXnpR/+HD3z37DDaqHDp3cMd56S7VAAdXWrVXT0oIbXxanUn5gpR7lc9VqBDlITobChSEqyt36vTJeo0aNjnicnp7Oo48+Sr169ShbtizFixdnzpw5/Pjjj8c8Tr169Y54XKlSpeNOm3Cs13zzzTdUqlSJypUrZz5/zjnnUKBA7v+s9uzZw9atW7nggguO2H7hhReyZs0aAK6++moOHDhA3bp16d69O6+88kpmraNMmTJ069aNSy+9lDZt2vDkk08e930wJtOYMXDPPXDNNTBxovunPxmtW8Ozz8K8edC7d567xsASQQ6SkmDhQjdgYOFC99hP2RerPtxMcs8997Bw4UJWrVrFlVdeyZ9//nnM42TvZBaRI/oIgvWaYDk8oio+Pp5vv/2WUaNGUbJkSfr160fDhg35448/AJg0aRLLly/noosuYu7cudSoUYMFCxaEJEaThz33HPTpA1ddBVOnQsFTHDvTsyc88IAbdvr448GJMUQ8SwQiMlFEtotIjr124owRkXUi8qWInO1VLCcjKckNIPA7CeRk8eLFXHHFFdxwww2cddZZnHHGGXz33Xchj6NmzZps3bqVrVu3Zm5buXLlCSWKkiVLUqlSJZYsWXLE9sWLF1MryyiM6OhoWrVqxVNPPcUnn3zC6tWrj3hN/fr1GTBgQObi3lOmTDmFkpl8b+JEuO02164/YwYEa9j0kCHQpYtLCC+/HJxjhoCXw0cnA88A/z3K85cBiYGfc4FnA7fmOKpXr87MmTNZvHgx5cqV4+mnn2bDhg00aNAgpHG0aNGCGjVq0LVrV0aMGMH+/fv517/+RcGCBU9obp577rmHgQMHkpiYSMOGDZk6dSofffQRn332GQCTJ0/m0KFD1KlTh7i4OGbOnEmhQoVITExkw4YNPP/887Rt25bKlSuzfv16vvzyS2677Tavim3yupdegh494NJL4ZVXXPtvsIi4JHP4GoNKleDii4N3fI94lghU9UMRSTjGLu2A/wY6MZaJSCkRqaiqeXcMVog8+OCDbNiwgcsuu4yYmBi6devGddddl9mmHioFChTgtddeo0ePHjRu3JiEhARGjhxJ+/btiY6OzvVx7rrrLvbu3cu9997Ltm3bqFGjBrNnz6Z+/foAlCpVimHDhrF27VrS0tKoVasWc+bMoVq1amzbto3vvvuOq6++mp07dxIXF8d1113HgHxy6b8JspkzoVs39+H82mtu6GewFSkCc+bAhRe6ZqclS6B27eCfJ5iO1oscjB8gAfj6KM+9CVyY5fFCoNHxjhmKUUPh6lijhsLFqlWrFNCVK1cG/dh+lz8c/r5s1NCik3/xnDluGGCTJqqpqUGL6ag2blStUEE1Pl51y5agHNKrUUN54spiEekJ9ASIi4sjJSUlx/1iY2PZu3dvCCMLrfT09LAr3xtvvEHRokU544wz+PHHH3nggQeoW7cuiYmJQY/V7/IfOHDgqH97oZKamup7DH462fKXXbqU2gMHsrdmTb687z7SP/kk+MHloPigQTTo04d9ycmsGjWK9KJFT+l4nv3+j5YhgvHDsWsEzwOdszz+Fqh4vGNajSC8TJkyRRMTEzU6OlorVqyonTt31l9++cWTc/ld/nD4+7IawaITf9H8+aqFC6s2aqT6++9Bj+m45s1zNZFWrU75GoP8eB3BXODGwOih84Ddav0Dec6NN97Id999x/79+9m6dSvTpk0jLi7O77CMcRYudHMB1aoF77wDJzEH1ym77DJ3jcH8+W6kUhheY+BZ05CITAeSgXIishl4CCgEoKrPAfOA1sA6YB9wk1exGGMi0IcfQtu2cOaZ8O67bm4gv9xyC2za5KaySEhw01uHES9HDXU+zvMK3O7V+Y0xEWzpUmjTBk47zdUKypXzOyJ3heqmTfDgg26Oouuv9zuiTHZlsTF5waxZUKUKNZ54Aj791O9owtsnn7hF5itWhPffh/Ll/Y7IEYEJE9zQ1ZtvdrGFCUsExoS7p592C6YXL075RYugUSM491y3hu7+/X5HF14+/xxatoSyZd0HbcWKfkd0pMKF3TUG1atD+/YQJtOlWyIwJlypuqkK7roL2rWDzz/n41decYlh796/Vsfq3x/WrfM7Wv999RW0aAElS7okUKWK3xHlrFQpNzld0aJusrosU7T4xRKBMeEoLc190A8d6hZMf/VViIkhvXhxuOMOWL0aFi2C5s1h9GhITHTNIXPnQnq639GH3tq17r2IjnZJICHB74iO7bTT4K234LffXF+Gz9cHWSLIJ7IvCJ+bBeLvuOMOkoMwx3aoFqO/9dZbj1h1Ld/64w9XA5gyBQYNckMPs0+PLOLmR581y3VADhrkvhG3awenn+5Gp2zb5kv4Iff9924FKRHXMXzGGX5HlDsNGri5jr76Cq6+2iV/n1gi8Fnbtm1p1qxZjs+tXbsWEeGdd9454eP279+fDz744FTDO8LGjRsREVauXOn5uSLWjh1wySWwYAGMHw8DBx5/taxKldx+mzb91f784IMQHw+dO8NHH4Xl2PWgWL/evV+HDrmaQI0afkd0Ylq1guefd79vH68xsETgs+7du7No0SI2btz4t+cmTJhA1apVad68+Qkft3jx4pQtWzYIEYbXufK1DRvcYuhffuk+0G+55cReX7Cgm+Ts3Xfhm2/g9tvdmroXXQT16rmaRZhNUXJKNm1ySWDfPnjvvby7eHz37m5d3AkTXE3OB5YIfNamTRvi4uL+tvh8WloaL730EjfffDOqSvfu3albty4xMTEkJibyxBNPHHPe/+zNNenp6fTv35/SpUtTunRp+vbtS3q2tuT58+fTpEkTSpcuTZkyZbj00ktZu3Zt5vPVqlUD3CpkIpLZrJT9XMdbjP5wzWL27Nm0aNGCokWLUqtWLd59990Teu8OHjxI3759iYuLIzo6mvPOO4/Fixcf8R7eddddVKpUiSJFihAfH899992X+fycOXOoV68eMTExlClThqZNm7LNr+aUVavg/PPd2rfvveeaeE5FjRrw1FOwZQu8+KKbb793b1d7uP32sBmtctK2bHFJYPdul/iyraSX5wwaBDfc4BLCf482c7938sSkc6ekb1/3TxZKZ50Fo0blateCBQvStWtXJk+ezEMPPZS5zOMbb7zBzp07uemmm8jIyKBy5cpMnjyZhIQEVqxYQc+ePSlbtizdu3fP1XlGjhzJCy+8wAsvvEC9evUYO3YsL7/8Mmef/dd6QH/88Qd9+/alXr167N+/nyFDhnDFFVewZs0aChcuzIoVK2jcuDHz58+nfv36FD7KPO6HF6N/7rnnaNSoEVOnTqV9+/Z8+umnnHXWWZn7/fvf/2b48OGMGzeOIUOG0KlTJzZt2kTx4sVzVaZ7772XWbNmMXHiRE4//XSefPJJWrVqxffff0/FihUZM2YMr732GjNmzCAhIYHNmzfz7bffAvDLL7/QqVMnhg4dSocOHUhNTWXZsmW5Om/Qvf++mwahVCnXxh3Mb7bFirlvnDffDCtWuFrBhAkwbhw0aeKSQ/v2wZ2T32s//+ySwI4dLmmeHVZrWp0cEZewt2xxv6/KlV2/R4hYjSAMdO/enR9//JH33nsvc9uECRNo2bIl8fHxFCpUiMGDB9OwYUMSEhK45ppruPXWW5k+fXquzzFq1CjuvfderrnmGmrWrMno0aOpUKHCEft06NCBDh06kJiYSL169Zg0aRIbNmxgxYoVAPzjH/8AoGzZslSoUIEyZcrkeK4RI0bQv39/unTpQvXq1Rk8eDBNmjRhxIgRR+x39913c8UVV5CYmMhjjz3Gr7/+yqpcJu0//viDZ599lmHDhtGmTRv++c9/8txzzxEXF8fYsWMB2LRpE9WrV6dJkyacdtppnH/++dx0k5vJZOvWraSlpdGxY0cSEhKoU6cOPXr0CP08STNnunbi006Djz/2rnlD5K9rD7ZsgeHD3W3nzq4v4cEHIQ+s9Vzot9/c6KAtW1yzV+PGfocUPIevMahZ0yXnr74K2anzf40gl9/M/ZSYmEjTpk2ZOHEiLVu2ZOvWrSxYsIAZM2Zk7vPcc88xfvx4fvrpJ/bv309aWhpVq1bN1fF3797Nzz//TFKWdTcLFCjAueeey08//ZS57YcffuA///kPy5cvZ8eOHWRkZJCRkXFCi8EfazH6efPmHbGtXpbqfKVKlQDYvn17rs7zww8/kJaWdsR5oqKiSEpKylygp1u3brRo0YLq1avTsmVLWrduzWWXXUaBAgWoX78+zZs3p06dOrRs2ZLmzZvTsWPHzGQXEqNHuxprkybw+uuhmwunbFl37cG//uUmYhs3Dh57zA1VveIKV0to3hwKhNn3xF27qN+/v6sRvP2260/Jb2Jj3TUG553nrjFYtszVDjwWZr/pyNW9e3f+97//8euvvzJ58mTKlClDu0A78cyZM+nbty9dunRhwYIFrFq1it69ex93sfoTdfnll7Njxw6ef/55li9fzueff07BggWDdp7sy1cWyrJO7OHnTmS94+Od5+yzz2bjxo0MHTqUjIwMunbtSosWLcjIyCAqKop33nmHd955h3r16jFhwgQSExP54osvTvn8x6UK993nksBVV7kRI35MiFagwF/XHqxfDwMGuFrJpZe6kUcjR8Kvv3ofx4EDbqjrd9+55qt333XXTbz4ooth4EC3yPxFF1H0p59cvE2beh+XX+LjXTLYvdtdY7Bnj+enzP81gjyiY8eO3HnnnUydOpWJEydy4403Zn5QLl68mHPPPZdevXpRokQJwH0jzq3Y2FgqVqzIsmXLuOSSSwC3DsWKFSuoGLgEf9euXXzzzTeMGzeOiwNrrH722WccOnQo8ziH+wSydzJnlXUx+qzDYrMvRn+qzjjjDAoXLsySJUs4IzBuPD09naVLl9KlS5fM/UqUKEHHjh3p2LEj3bp147zzzmPdunVUr14dESEpKYmkpCQGDhxI7dq1mTlzZuYSmZ5IS3Pr5f73v3DrrfDMM3+/RsAPCQmuVvDQQzB7tqsl9O/vmow6dXK1hHPO+fvrDh1yH1i7d8Pvv+d8/3jPHTx47NhE3NXC5cvz9ZAh1DuJUXR5Tv36Lhm2bg0dO7qLz7J8cQo2SwRhIiYmhi5duvDwww/z22+/HdEJXL16dSZPnpz57XXGjBl88MEHlD6Bb5F9+vRh6NChVK9enbp16zJu3Dh+/vnnzERQunRpypUrxwsvvEB8fDxbtmzhnnvuoWDBv/5EypcvT0xMDAsWLCAhIYHo6Ghic5jf/XiL0QdDsWLFuO222xgwYADlypWjWrVqPPXUU2zbto3evXsD8OSTT1KxYkXOOussChUqxLRp0yhZsiRVqlRh2bJlvPfee1x66aXExcXx+eef89NPPwU1Wf1Naqr7p16wwM1E+e9/H/8agVArUgS6dHE/X37pOpdfesn1LdSt6zqfs36I//HH8Y9ZrJhr8ihVyt2WK+cu+jr8OOtz2e/HxkKJEpnNVL9G0upsLVu6a0m6d3dXl0+Y4NmpLBGEkR49evDss89y/vnn889//jNze69evVi1ahU9evRAVenQoQP9+vVj4sSJuT52v379+OWXX+jRowcAN9xwA9ddd13m8NACBQowc+ZM7rrrLurUqcOZZ57JyJEj6dChQ+YxChYsyJgxYxg8eDCDBg2iSZMmOS6bd7zF6INl2LBhANx00038/vvvNGjQgPnz52cmtxIlSjB8+HC+//57RIQGDRrw9ttvU7RoUWJjY1myZAlPP/00v//+O/Hx8fznP//heq+mBt6xw1XzP/0UXnjB1QrC3eFrD4YNc8ng1VfdtQqVKx/7gzv744L2MXPSbr7ZXS8xeLCrtV10kTfnOdrSZeH6Y0tVRi6/y3/Sf18//KB65pmq0dGqr79+SjHYUpWL/A4h9DIyVLt2VQVdc//9J30Y8vri9cbkWZ995tp509LcNQLnn+93RCavEXFNRFFR7K1Z05NT2KghY7zy3ntudEvhwrB4sSUBc/IKF4YJE9h32mmeHN4SgTFemD7d1QQSEtyyiVn6fIwJN5YIjAm2p55yo26SktzMnyG4IMiYU5HvEoHm1+l2ja9y9XeVkQH33OOu2G3f3g0TLVXK++CMOUX5KhEUKlSI/baGq/HA/v37j7gS+m/+/BO6doURI9zFV7NmudWyjMkD8lUiKF++PFu2bGHfvn1WMzBBoars27ePLVu2UL58+Zx32rvXzdEzdSoMGRI+Vwsbk0v5avhoyZIlgb9mlsxvDhw4QHQEf8v0q/yFChUiLi4u8+/rCNu3u07hVavclZ833xzy+Iw5VfkqEYBLBjn+w+YDKSkpNGjQwO8wfBN25f/hBzdB29at8L//QSSsp2zypXyXCIwJiU8/dTWBQ4fchWJZpvg2Jq/JV30ExoTEu+9CcrLrDF6yxJKAyfMsERhzIl5+2dUEqlVzF4p5dMm/MaFkicCY3FCFJ5+E6693K2N9+KFbCN6YfMDTRCAirUTkWxFZJyL35fD8aSKySEQ+F5EvRaS1l/EYc8IOHYIZM9zauP36ufUE5s+3C8VMvuJZIhCRKGAscBlQC+gsItlX/XgQmKWqDYBOwDiv4jHmhOzd66aKOPNMt8D77t3w3HMuKUTwEF6TP3k5aqgxsE5V1wOIyAygHbAmyz4KHB7rGQts9TAeY45v82YYM8ZN+7t7t1tYfswYNzQ03BZzNyZIxKsrcEWkI9BKVXsEHt8AnKuqd2TZpyLwDlAaKAY0V9VPczhWT6AnQFxcXMMZM2Z4EnO4S01NpXjx4n6H4Rsvy19s3TriZ82i/PvvI6rsuOgifrrmGvaG2ayh9jdg5T/Z8l988cWfqmqjHJ882oo1p/oDdARezPL4BuCZbPv8C+gXuJ+Eqy0UONZxj7VCWX4XkaszZRH08mdkqL79tmrz5qqgWqyYap8+quvXB/c8QWR/A4v8DsFXp1J+fFqhbAsQn+VxlcC2rLoDrQBUdamIRAPlgO0exmUi3cGDbr2AkSPh66+hYkV4/HHo2RNKl/Y7OmNCzstGz0+ARBGpJiKFcZ3Bc7Pt8yPQDEBE/glEAzs8jMlEsl9/haFD3WIxN93ktk2eDBs3woABlgRMxPKsRqCqh0TkDmABEAVMVNXVIjIYV0WZC/QDXhCRu3Edx90CVRhjgmf9ehg1yk0Kt28ftGgBU6a4WxG/ozPGd57ONaSq84B52bYNzHJ/DXCBlzGYCLZ8uVsfYM4cNy10585u0Zj69f2OzJiwYpPOmfwlPR3eeMMlgCVLIDbWrRp25522ZKQxR2GJwOQP+/a55p4nn4R166BqVdccdPPNUKKE39EZE9YsEZi8bds2GDsWxo2DXbvgnHNg5ky3ZnBB+/M2JjfsP8XkTWvXum//L73khoO2bevmAmrSxDqAjTlBlghM3qEKKSmu/f+tt6BIEejWDe6+G2rU8Ds6Y/IsSwQmb1izhrNvuw2+/RbKlYOHHoLeveFoC8obY3LNEoEJf198Ac2bE52e7mYAvfFGiInxOypj8g2bTtGEt5Ur4eKLITqaz8eMgV69LAkYE2SWCEz4WrYMmjVz1wJ8+CH7q1TxOyJj8iVLBCY8ffSRmwKifHm3LGS1an5HZEy+ZYnAhJ+FC6FVK6hSBT74AOLjj/8aY8xJs0Rgwsv8+W41sNNPd0NFbYF4YzxnicCEjzfegHbtoGZNWLQI4uL8jsiYiGCJwISH2bPdtBD168P777trBYwxIWGJwPhv+nS49lpo3Bjee88WiDEmxCwRGH9NmQLXXQcXXggLFkDJkn5HZEzEsURg/DN+vFsysnlzmDcPihf3OyJjIpIlAuOPZ55xVwlfdhnMnQtFi/odkTERyxKBCb2RI92KYVde6ZaRjI72OyJjIpolAhNajz4K/fvDNdfArFluKmljjK8sEZjQUHVTRz/4IFx/Pbz8MhQq5HdUxhhsGmoTCqpw//0wbJhbQ3j8eIiK8jsqY0yAJQLjLVW3gtjo0XDbba6TuIBVRI0JJ/YfabyTkQG33+6SQJ8+bpF5SwLGhB37rzTeSE+HW26BZ5+FAQPgqadsUXljwpQlAhN8hw65ReUnToSBA2HoUEsCxoQx6yMwwZWW5qaMeOUVN1T0gQf8jsgYcxyWCEzwHDzoJo97/XUYMQL69fM7ImNMLnjaNCQirUTkWxFZJyL3HWWfa0RkjYisFpFpXsZjPHTggJtG+vXX4emnLQkYk4d4ViMQkShgLNAC2Ax8IiJzVXVNln0SgfuBC1T1NxEp71U8xkP79rkFZRYuhOefh549/Y7IGHMCvKwRNAbWqep6Vf0TmAG0y7bPLcBYVf0NQFW3exiP8UJqKrRu7RaTmTTJkoAxeZCXfQSVgZ+yPN4MnJttn+oAIrIEiAIeVtX52Q8kIj2BngBxcXGkpKR4EW/YS01NDauyR6WmUu+++yi5di1rH3iA7VWrunWGPRJu5fdDpL8HVn5vyp+rRCAixYD9qpohItWBmsDbqpoWhPMnAslAFeBDEamrqr9n3UlVxwPjARo1aqTJycmneNq8KSUlhbAp+2+/waWXwrffwqxZ1OrQgVoenzKsyu+TSH8PrPzelD+3TUMfAtEiUhl4B7gBmHyc12wB4rM8rhLYltVmYK6qpqnqBuA7XGIw4WznTrjkEvjiCzeNdIcOfkdkjDkFuU0Eoqr7gPbAOFW9Gqh9nNd8AiSKSDURKQx0AuZm2+d/uNoAIlIO11S0PpcxGT9s2wYXXwzffONGCF1xhd8RGWNOUa4TgYgkAdcBbwW2HXP6SFU9BNwBLADWArNUdbWIDBaRtoHdFgC7RGQNsAi4R1V3nWghTIhs3QrJybB+Pbz5JrRq5XdExpggyG1ncV/cMM/XAh/mp+M+uI9JVecB87JtG5jlvgL/CvyYcPbNN+7b/y+/wPz50KSJ3xEZY4IkV4lAVT8APgAQkQLATp2SKNUAAA+2SURBVFW9y8vATJg4cMDNFTR0qFtc/p13ICnJ76iMMUGUq6YhEZkmIiUDo4e+BtaIyD3ehmZ8t2gR1K8Pgwe7pSXXrrUkYEw+lNs+glqquge4EngbqIYbOWTyo5073eyhl1ziZhJdsACmToW4OL8jM8Z4ILeJoJCIFMIlgrmB6wfUu7CML1RhyhSoWdOtKXz//fD119Cypd+RGWM8lNtE8DywESiGu+irKrDHq6CMD7791tUAunWDGjXg88/hsccgJsbvyIwxHstVIlDVMapaWVVbq7MJuNjj2EwoHDzo+gDq1XMf/s89Bx99BHXq+B2ZMSZEcjvFRCzwEHBRYNMHwGBgt0dxmVD48EPo1csNDe3UyS0nWaGC31EZY0Ist01DE4G9wDWBnz3AJK+CMh7btQu6d4emTd3w0HnzYPp0SwLGRKjcXlB2hqpmnVBmkIis8iIg4yFV1wl8991u0rh774WHHoKiRf2OzBjjo9zWCPaLyIWHH4jIBcB+b0Iynvj+e2jRAm64Ac44Az77DIYNsyRgjMl1jeBW4L+BvgKA34Cu3oRkgurPP2H4cHjkEShSBMaNc4vHRB1zqihjTATJ7RQTXwD1RaRk4PEeEekLfOllcOYULV7sPvTXroWrr4ZRo6BSJb+jMsaEmRNaqlJV9wSuMAabKC58/fYb3HKLmxhu3z43U+isWZYEjDE5OpU1iyVoUZjgUHWjf2rWdOsH9+8Pq1dDmzZ+R2aMCWOnsmaxTTERTn74AXr3drODnnOOmx/orLP8jsoYkwccMxGIyF5y/sAXwOYeCAdpaTBihLs6uFAhePppuO026ww2xuTaMROBqpYIVSDmJHz8sbsy+OuvoX17GD0aqlTxOypjTB5zKn0Exi+//+6+9V9wAeze7dYOnj3bkoAx5qRYIshLVGHmTNcZPH68u0J4zRpo2/b4rzXGmKM4lc5iEyoZGbB0KXXvvx+WL4ezz4a33oKGDf2OzBiTD1giCFeq8NVXMG0azJgBmzYRGxPjLgq7/XYoaL86Y0xw2KdJuFm/3l0LMG2aa/aJioJLL4UhQ1hapgxNWrf2O0JjTD5jiSAc/PKLu/J32jTX9APuquBx46BjR/jHPwBIT0nxL0ZjTL5licAvu3fDnDnuw//9910/QP36bkbQTp3gtNP8jtAYEyEiJhEsXQopKZCcDElJPgWxf7/r5J02zS0Gc/AgnH46PPAAdO4MtWr5FJgxJpJFRCJYuhSaNXMzMhcuDAsXhjAZHDrkTjhtGrz2GuzdC3FxcOut0KWLmw5CbNomY4x/IiIRpKS4JJCe7m5TUjxOBKou+0yb5tr+d+yA2Fg3FXSXLq5aYlNAGGPCREQkguRkVxM4XCNITvboRIeHe06fDps2QXQ0XHGF+/C/7DK3MIwxxoQZTxOBiLQCRgNRwIuq+vhR9usAvAqco6orgx1HUpJrnfGkj2DDhr+Ge65e7b7pt2jhVgRr1w5KlgziyYwxJvg8SwQiEgWMBVoAm4FPRGSuqq7Jtl8JoA+w3KtYwH34By0BbNv213DPZcvctgsugLFjXfNPYLinMcbkBV7WCBoD61R1PYCIzADaAWuy7fcIMAy4x8NYTt3h4Z7Tp7vqRUYG1KsHjz/uhntWrep3hMYYc1K8TASVgZ+yPN4MnJt1BxE5G4hX1bdE5KiJQER6Aj0B4uLiSPHhwqrTpk7l9AkT2F+xItu7dGHbJZewr1o19+SGDe7HY6mpqb6UPVxEevnB3gMrvzfl962zWEQKAE8C3Y63r6qOB8YDNGrUSJM96+09hho14JZbiGncmKoi+PH9PyUlBV/KHiYivfxg74GV35vye5kItgDxWR5XCWw7rARQB0gRN46+AjBXRNp60WF8yipWdD/GGJPPeLkewSdAoohUE5HCQCdg7uEnVXW3qpZT1QRVTQCWAeGZBIwxJh/zLBGo6iHgDmABsBaYpaqrRWSwiNhKKsYYEyY87SNQ1XnAvGzbBh5l32QvYzHGGJMzW6rSGGMinCUCY4yJcJYIjDEmwlkiMMaYCGeJwBhjIpwlAmOMiXCWCIwxJsJZIjDGmAhnicAYYyKcJQJjjIlwlgiMMSbCWSIwxpgIZ4kgRJYuhaFD3a0xxoQT31YoiyRLl0KzZvDnn1C4sFvyOCnJ76iMMcaxGkEIpKS4JJCe7m4jeMlVY0wYskQQAsnJriYQFeVuI3jJVWNMGLKmoRBISnLNQSkpLglYs5AxJpxYIgiRpCRLAMaY8GRNQ8YYE+EsERhjTISzRGCMMRHOEoExxkQ4SwTGGBPhLBEYY0yEs0RgjDERzhKBMcZEOEsExhgT4SwRGGNMhPM0EYhIKxH5VkTWich9OTz/LxFZIyJfishCEanqZTzGGGP+zrNEICJRwFjgMqAW0FlEamXb7XOgkarWA14FnvAqHmOMMTnzskbQGFinqutV9U9gBtAu6w6qukhV9wUeLgOqeBiPMcaYHHg5+2hl4KcsjzcD5x5j/+7A2zk9ISI9gZ4AcXFxpEToyi6pqakRW3aw8oO9B1Z+b8ofFtNQi8j1QCOgaU7Pq+p4YDxAo0aNNDlCV3ZJSUkhUssOVn6w98DK7035vUwEW4D4LI+rBLYdQUSaA/8GmqrqQQ/jMcYYkwMv+wg+ARJFpJqIFAY6AXOz7iAiDYDngbaqut3DWIwxxhyFZ4lAVQ8BdwALgLXALFVdLSKDRaRtYLfhQHHgFRFZJSJzj3I4Y4wxHvG0j0BV5wHzsm0bmOV+cy/Pb4wx5vjsyuIIsXQpDB3qbo0xJquwGDVkvLV0KTRrBn/+CYULw8KFkJTkd1TGmHBhNYIIkJLikkB6uruN4GHYxpgcWCKIAMnJriYQFeVuI3gYtjEmB9Y0FAGSklxzUEqKSwLWLGSMycoSQYRISrIEYIzJmTUNGWNMhLNEYIwxEc4SgTHGRDhLBMYYE+EsERhjTISzRGCMMRHOEoExxkQ4SwTGGBPhLBEYY0yEs0RgQsKmwTYmfNkUE8ZzNg22MeHNagTGczYNtjHhzRKB8ZxNg21MeLOmIeM5mwbbmPBmicCEhN/TYC9daonImKOxRGDyPeusNubYrI/A5HvWWW3MsVkiMPleOHRW23UUJpxZ05DJ9/zurA6Xpim/+0n8Pr/fwrn8lghMRPCzszqnpqlIS0Z+n99v4V5+axoyxmPh0DTldz+J3+cHf5vnwqH8x2I1AmM85nfTFPyVjA5/Iw11MvL7/H5/I/e7/MfjaSIQkVbAaCAKeFFVH8/2fBHgv0BDYBdwrapu9DImY/zg93UUficjv8/vd/Oc3+U/Hs8SgYhEAWOBFsBm4BMRmauqa7Ls1h34TVXPFJFOwDDgWq9iMiaShUMy8uv84fCN3O/3/1i8rBE0Btap6noAEZkBtAOyJoJ2wMOB+68Cz4iIqKp6GJcxJsKE+zdyv4lXn7ki0hFopao9Ao9vAM5V1Tuy7PN1YJ/Ngcc/BPbZme1YPYGeAHFxcQ1nzJjhSczhLjU1leLFi/sdhm8ivfxg74GV/+TLf/HFF3+qqo1yei5PdBar6nhgPECjRo00Odx6WkIkJSWFSC07WPnB3gMrvzfl93L46BYgPsvjKoFtOe4jIgWBWFynsTHGmBDxMhF8AiSKSDURKQx0AuZm22cu0DVwvyPwvvUPGGNMaHnWNKSqh0TkDmABbvjoRFVdLSKDgZWqOheYALwkIuuAX3HJwhhjTAh52kegqvOAedm2Dcxy/wBwtZcxGGOMOTabYsIYYyKcZ8NHvSIiO4BNfsfhk3LAzuPulX9FevnB3gMr/8mXv6qq/iOnJ/JcIohkIrLyaOOAI0Gklx/sPbDye1N+axoyxpgIZ4nAGGMinCWCvGW83wH4LNLLD/YeWPk9YH0ExhgT4axGYIwxEc4SgTHGRDhLBHmAiMSLyCIRWSMiq0Wkj98x+UFEokTkcxF50+9YQk1ESonIqyLyjYisFZGImlFfRO4O/O1/LSLTRSTa75i8JCITRWR7YKr+w9vKiMi7IvJ94LZ0sM5niSBvOAT0U9VawHnA7SJSy+eY/NAHWOt3ED4ZDcxX1ZpAfSLofRCRysBdQCNVrYObuyy/z0s2GWiVbdt9wEJVTQQWBh4HhSWCPEBVf1bVzwL39+I+BCr7G1VoiUgVoA3wot+xhJqIxAIX4SZpRFX/VNXf/Y0q5AoCMYHp6osCW32Ox1Oq+iFuIs6s2gFTAvenAFcG63yWCPIYEUkAGgDL/Y0k5EYB9wIZfgfig2rADmBSoGnsRREp5ndQoaKqW4ARwI/Az8BuVX3H36h8EaeqPwfu/wLEBevAlgjyEBEpDswG+qrqHr/jCRURuRzYrqqf+h2LTwoCZwPPqmoD4A+C2CwQ7gJt4e1wCbESUExErvc3Kn8F1m0J2th/SwR5hIgUwiWBl1V1jt/xhNgFQFsR2QjMAC4Rkan+hhRSm4HNqnq4FvgqLjFEiubABlXdoappwBzgfJ9j8sM2EakIELjdHqwDWyLIA0REcO3Da1X1Sb/jCTVVvV9Vq6hqAq6T8H1VjZhvhKr6C/CTiNQIbGoGrPExpFD7EThPRIoG/heaEUGd5VlkXdGxK/B6sA5siSBvuAC4AfdNeFXgp7XfQZmQuhN4WUS+BM4CHvM5npAJ1IReBT4DvsJ9buXrqSZEZDqwFKghIptFpDvwONBCRL7H1ZIeD9r5bIoJY4yJbFYjMMaYCGeJwBhjIpwlAmOMiXCWCIwxJsJZIjDGmAhnicCYABFJzzI8d5WIBO3qXRFJyDqTpDHhpKDfARgTRvar6ll+B2FMqFmNwJjjEJGNIvKEiHwlIitE5MzA9gQReV9EvhSRhSJyWmB7nIi8JiJfBH4OT4cQJSIvBObVf0dEYgL73xVYa+JLEZnhUzFNBLNEYMxfYrI1DV2b5bndqloXeAY3EyrA08AUVa0HvAyMCWwfA3ygqvVxcwKtDmxPBMaqam3gd6BDYPt9QIPAcW71qnDGHI1dWWxMgIikqmrxHLZvBC5R1fWByf9+UdWyIrITqKiqaYHtP6tqORHZAVRR1YNZjpEAvBtYVAQRGQAUUtUhIjIfSAX+B/xPVVM9LqoxR7AagTG5o0e5fyIOZrmfzl99dG2AsbjawyeBxVeMCRlLBMbkzrVZbpcG7n/MX0smXgd8FLi/ELgNMtdZjj3aQUWkABCvqouAAUAs8LdaiTFesm8exvwlRkRWZXk8X1UPDyEtHZj58yDQObDtTtyqYffgVhC7KbC9DzA+MGNkOi4p/EzOooCpgWQhwJgIXIbS+Mz6CIw5jkAfQSNV3el3LMZ4wZqGjDEmwlmNwBhjIpzVCIwxJsJZIjDGmAhnicAYYyKcJQJjjIlwlgiMMSbC/R9/v3je27OaWgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_4HrxB7SfM9-"
      },
      "source": [
        "Inight: the validation loss increases steadily after the first epoch, as the training loss decreases and the model starts overfitting. Therefore the best model is the one trained with only 1 epoch, so we will keep that one. We will reevaluate it with Aggression data, to see if the training with the Kaggle dataset has changed how the model predicts aggressive comments. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5RY_kPPzA3e7"
      },
      "source": [
        "##### Evaluating with Aggression train data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D2u1k3WsWjIb"
      },
      "source": [
        "input_ids_aggression, attention_masks_aggression, labels_aggression = load_vectors('aggression')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vKIxfO3sW4d0"
      },
      "source": [
        "train_inp_aggression, val_inp_aggression, test_inp_aggression, train_mask_aggression, val_mask_aggression, test_mask_aggression, train_label_aggression, val_label_aggression, test_label_aggression = train_test_val(\n",
        "    input_ids_aggression, attention_masks_aggression, labels_aggression)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LYamxcQqXG9I"
      },
      "source": [
        "preds_train_aggression, pred_labels_train_aggression, f1_train_aggression, class_report_train_aggression, conf_matrix_train_aggression = evaluate_model(\n",
        "    'kaggle_model_1epoch', train_inp_aggression, train_mask_aggression, train_label_aggression, base_model=base_model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k4DWhm6fCA2S",
        "outputId": "1b76719c-7606-4177-ba7a-525b9bafca6b"
      },
      "source": [
        "print_eval(f1_train_aggression, class_report_train_aggression, conf_matrix_train_aggression)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "F1 score: 0.7045273540403112\n",
            "Classification Report:\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.94      0.98      0.96     64691\n",
            "           1       0.84      0.61      0.70      9461\n",
            "\n",
            "    accuracy                           0.93     74152\n",
            "   macro avg       0.89      0.80      0.83     74152\n",
            "weighted avg       0.93      0.93      0.93     74152\n",
            "\n",
            "Confusion Matrix:\n",
            "[[63579  1112]\n",
            " [ 3711  5750]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TB0FDIBXCA2a"
      },
      "source": [
        "##### Evaluating with Aggression test data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YzaZ03viYHf5"
      },
      "source": [
        "preds_test_aggression, pred_labels_test_aggression, f1_test_aggression, class_report_test_aggression, conf_matrix_test_aggression = evaluate_model(\n",
        "    'kaggle_model_1epoch', test_inp_aggression, test_mask_aggression, test_label_aggression, base_model=base_model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "keSEs_9AYHgB",
        "outputId": "86dbfe29-828a-4ce1-dfe8-d619fe89e863"
      },
      "source": [
        "print_eval(f1_test_aggression, class_report_test_aggression, conf_matrix_test_aggression)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "F1 score: 0.6633584755855498\n",
            "Classification Report:\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.94      0.98      0.96     20217\n",
            "           1       0.80      0.57      0.66      2956\n",
            "\n",
            "    accuracy                           0.93     23173\n",
            "   macro avg       0.87      0.77      0.81     23173\n",
            "weighted avg       0.92      0.93      0.92     23173\n",
            "\n",
            "Confusion Matrix:\n",
            "[[19806   411]\n",
            " [ 1285  1671]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DGX7oxJn5BFM"
      },
      "source": [
        "Insight: the recall for class 1 has decreased a lot in both evaluations. \n",
        "\n",
        "We will discard this model for now. The next step is to train the aggression model (1 epoch) with the Racism dataset to see if we have the same issue. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CQVMBjMYgV_Z"
      },
      "source": [
        "### Racism dataset - 1 epoch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5noASwNegV_Z"
      },
      "source": [
        "#### Loading the data from the pickle files"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "93piv32SgV_Z"
      },
      "source": [
        "input_ids, attention_masks, labels = load_vectors('racism')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GuURX1NLgV_Z"
      },
      "source": [
        "#### Splitting into train, test and validation set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1B5ug1ZlgV_Z"
      },
      "source": [
        "train_inp, val_inp, test_inp, train_mask, val_mask, test_mask, train_label, val_label, test_label = train_test_val(\n",
        "    input_ids, attention_masks, labels)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "00-lW6zGgV_Z"
      },
      "source": [
        "#### Training the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GrywJEcbgV_Z",
        "outputId": "eebb6458-d97b-4ed0-cc2a-d9ef9fb85d95"
      },
      "source": [
        "racism_model_1epoch, racism_1epoch_history = train_model(\n",
        "    base_model=base_model, \n",
        "    old_model_name='aggression_model_1epoch', \n",
        "    new_model_name='racism_model_1epoch', \n",
        "    epochs=1)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "New model will be saved at /content/gdrive/MyDrive/Cyber-bullying-project/models/racism_model_1epoch.h5\n",
            "\n",
            "Training on previous model aggression_model_1epoch\n",
            "\n",
            "Training starting...\n",
            "\n",
            "\n",
            "WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n",
            "WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n",
            "WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n",
            "WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n",
            "270/270 [==============================] - ETA: 0s - loss: 0.1990 - accuracy: 0.9144WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n",
            "WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n",
            "270/270 [==============================] - 362s 1s/step - loss: 0.1990 - accuracy: 0.9144 - val_loss: 0.1773 - val_accuracy: 0.9184\n",
            "\n",
            "Finished training. Saving history at /content/gdrive/MyDrive/Cyber-bullying-project/models/histories/racism_model_1epoch_hist.pkl ...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fvFhEJ84gV_a"
      },
      "source": [
        "#### Evaluating the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aRjhu7B3gV_a"
      },
      "source": [
        "##### With Racism train data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "80Nf992mgV_a"
      },
      "source": [
        "preds_train_racism, pred_labels_train_racism, f1_train_racism, class_report_train_racism, conf_matrix_train_racism = evaluate_model(\n",
        "    racism_model_1epoch, train_inp, train_mask, train_label)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W56dnUfvgV_a",
        "outputId": "1bf371de-7ea1-46cb-a18e-73159c303d91"
      },
      "source": [
        "print_eval(f1_train_racism, class_report_train_racism, conf_matrix_train_racism)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "F1 score: 0.7047434656340754\n",
            "Classification Report:\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.93      0.99      0.96      7359\n",
            "           1       0.90      0.58      0.70      1261\n",
            "\n",
            "    accuracy                           0.93      8620\n",
            "   macro avg       0.92      0.78      0.83      8620\n",
            "weighted avg       0.93      0.93      0.92      8620\n",
            "\n",
            "Confusion Matrix:\n",
            "[[7282   77]\n",
            " [ 533  728]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7i0k8s_ygV_a"
      },
      "source": [
        "##### With Racism test data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2eHBK2twgV_a"
      },
      "source": [
        "preds_test_racism, pred_labels_test_racism, f1_test_racism, class_report_test_racism, conf_matrix_test_racism = evaluate_model(\n",
        "    racism_model_1epoch, test_inp, test_mask, test_label)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O-4-_jVegV_a",
        "outputId": "f549bc0c-e6b1-4906-9bdd-b77eced09bbd"
      },
      "source": [
        "print_eval(f1_test_racism, class_report_test_racism, conf_matrix_test_racism)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "F1 score: 0.6518282988871223\n",
            "Classification Report:\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.92      0.99      0.95      2301\n",
            "           1       0.87      0.52      0.65       394\n",
            "\n",
            "    accuracy                           0.92      2695\n",
            "   macro avg       0.90      0.75      0.80      2695\n",
            "weighted avg       0.92      0.92      0.91      2695\n",
            "\n",
            "Confusion Matrix:\n",
            "[[2271   30]\n",
            " [ 189  205]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_H5rGFBRo9F6"
      },
      "source": [
        "##### Evaluating with Aggression train data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PMIYDcA6o9F7"
      },
      "source": [
        "preds_train_aggression, pred_labels_train_aggression, f1_train_aggression, class_report_train_aggression, conf_matrix_train_aggression = evaluate_model(\n",
        "    'racism_model_1epoch', train_inp_aggression, train_mask_aggression, train_label_aggression, base_model=base_model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qUcAXn6Go9F7",
        "outputId": "d8575d73-3e83-4c46-82ff-6ec20f7761c0"
      },
      "source": [
        "print_eval(f1_train_aggression, class_report_train_aggression, conf_matrix_train_aggression)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "F1 score: 0.03481624758220503\n",
            "Classification Report:\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.87      1.00      0.93     64691\n",
            "           1       0.47      0.02      0.03      9461\n",
            "\n",
            "    accuracy                           0.87     74152\n",
            "   macro avg       0.67      0.51      0.48     74152\n",
            "weighted avg       0.82      0.87      0.82     74152\n",
            "\n",
            "Confusion Matrix:\n",
            "[[64500   191]\n",
            " [ 9290   171]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YbQ1_Uwgo9F7"
      },
      "source": [
        "##### Evaluating with Aggression test data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y7t9g9kUo9F7"
      },
      "source": [
        "preds_test_aggression, pred_labels_test_aggression, f1_test_aggression, class_report_test_aggression, conf_matrix_test_aggression = evaluate_model(\n",
        "    'racism_model_1epoch', test_inp_aggression, test_mask_aggression, test_label_aggression, base_model=base_model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WBBR1pjWo9F8",
        "outputId": "5bbe3f35-6e63-41fe-f5ef-7fda048b97eb"
      },
      "source": [
        "print_eval(f1_test_aggression, class_report_test_aggression, conf_matrix_test_aggression)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "F1 score: 0.030045721750489876\n",
            "Classification Report:\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.87      1.00      0.93     20217\n",
            "           1       0.43      0.02      0.03      2956\n",
            "\n",
            "    accuracy                           0.87     23173\n",
            "   macro avg       0.65      0.51      0.48     23173\n",
            "weighted avg       0.82      0.87      0.82     23173\n",
            "\n",
            "Confusion Matrix:\n",
            "[[20157    60]\n",
            " [ 2910    46]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jJeNyFjp50Zx"
      },
      "source": [
        "Insight: in this case the evaluation with the aggression dataset yields very poor results with a recall of 0.02 for class 1. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KJo5zjRy6dVr"
      },
      "source": [
        "## Conclusion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UMpSYdZJ6imu"
      },
      "source": [
        "To distinguish between different types of bullying behaviors (racism, sexism, aggression, etc) we would need to build a multiclass classifier, not a binary classifier. Due to time constraints, for this project we will keep the best model trained on a single dataset, the aggression_model_1epoch. "
      ]
    }
  ]
}