parlai_gutenberg_experiments.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nicolay-r/book-persona-retriever/blob/master/parlai_gutenberg_experiments.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Literature Dialogue Response Task (LDR) 📚 • [![twitter](https://img.shields.io/twitter/url/https/shields.io.svg?style=social)](https://x.com/nicolayr_/status/1801009815784677862)\n",
        "![](https://img.shields.io/badge/Python-3.8-lightgreen.svg)\n",
        "[![twitter](https://img.shields.io/twitter/url/https/shields.io.svg?style=social)](https://x.com/nicolayr_/status/1801009815784677862)\n",
        "[![Youtube badge](https://img.shields.io/badge/-Youtube-Cc4c4c?style=flat-square&logo=Youtube&logoColor=white&link=https://twitter.com/nicolayr_)](https://youtu.be/UQQsXfZyjjc)\n",
        "\n",
        "[![](https://markdown-videos-api.jorgenkh.no/youtube/UQQsXfZyjjc)](https://youtu.be/UQQsXfZyjjc)\n",
        "\n",
        "> ⚠️ **Disclaimer**: this repository setups the task for the predefined `train` and `valid` splits. In order to replicate studies on different splits you have to manually update the related parts.\n",
        "> We believe that ParlAI supports task initialization in Cross-Validation mode, however it goes beyond the capabilities of this project version.\n",
        "\n",
        "This repository represent a supplementary material for the [`nicolay-r/book-persona-retreiver`](https://github.com/nicolay-r/book-persona-retriever) experiments organization 🧪 mentioneed in paper\n",
        "[Personality Profiling for Literary Character Dialogue Agents with Human Level Attributes (**pre-print**)](https://www.dropbox.com/scl/fi/0c2axh97hadolwphgu7it/rusnachenko2024personality.pdf?rlkey=g2yyzv01th2rjt4o1oky0q8zc&st=omssztha&dl=1)\n",
        "that has been accepted for *Long Paper* track at [LOD-2024](https://lod2024.icas.events/).\n"
      ],
      "metadata": {
        "id": "4Lk62Z4o1LpX"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "T1ktkLcs4NUp"
      },
      "outputs": [],
      "source": [
        "# select python version\n",
        "!sudo apt-get install python3.8 --fix-missing\n",
        "!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 1\n",
        "!sudo update-alternatives --config python3\n",
        "# check python version\n",
        "!python --version\n",
        "# install pip for new python\n",
        "!sudo apt-get install python3.8-distutils\n",
        "!wget https://bootstrap.pypa.io/pip/get-pip.py\n",
        "!sudo python get-pip.py"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!python3 --version"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "us6pny_Zsql9",
        "outputId": "e6bc3650-515c-4a07-914e-1966b2b74d8c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Python 3.8.18\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "H-QGeENzUbyT"
      },
      "outputs": [],
      "source": [
        "!rm -rf parlai_bookchar_task\n",
        "!git clone https://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@github.com/nicolay-r/parlai_bookchar_task.git"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3Z4RITZaQ1rg"
      },
      "outputs": [],
      "source": [
        "!pip install parlai pytorch-pretrained-bert\n",
        "# Install py-rouge metrics\n",
        "!pip install py-rouge\n",
        "!python -c \"import nltk; nltk.download('punkt')\"\n",
        "!pip install transformers"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "H9l435Ob00Y5"
      },
      "outputs": [],
      "source": [
        "!rm -rf \"/usr/local/lib/python3.10/dist-packages/data/GutenbertBookChars\"\n",
        "!cd parlai_bookchar_task && ./setup_colab.sh"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Dataset downloading\n",
        "\n",
        "Here is the link for the [dataset-v4](https://drive.google.com/drive/folders/1Xz71KeBUurVWNs5XOaqCRA7BkYRt_bBM?usp=sharing) which is expected to be downloaded and then locally copied into `./parlai/`"
      ],
      "metadata": {
        "id": "oho_zhNG15rF"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l-PZVWYumbk1"
      },
      "source": [
        "This is a **BERT-Bi-Ranker** application.\n",
        "\n",
        "It is supposed to be pretrained first on the ConvAI2 data!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nb-GA7iiVmb2"
      },
      "outputs": [],
      "source": [
        "!parlai train_model -t gutenbergbookchars -m bert_ranker/bi_encoder_ranker --batchsize 20 -veps 1 --num-epochs 10 \\\n",
        "--save-after-valid True --log_every_n_steps 500 --tensorboard_log True --model_file ./parlai_bert/bert_biencoder_test --fp16 True --truncate 360 \\\n",
        "--candidates batch --dict-tokenizer bpe --dict-lower True --history-size -1 --optimizer adam -lr 5e-05 --data-parallel True"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iPnVaS6amjHi"
      },
      "source": [
        "Random selection application"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2d1YrncXmmHJ"
      },
      "outputs": [],
      "source": [
        "!parlai eval_model -m random_candidate -t gutenbergbookchars:Spectrum"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Li4Sncqxmy3z"
      },
      "source": [
        "Mem neural network application\n",
        "`-mf` denotes the **model file** to load/save the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZGl_-Kunm1kb"
      },
      "outputs": [],
      "source": [
        "!parlai train_model -m memnn -t gutenbergbookchars --model_file ./parlai/memnn-origin -veps 1 -eps 20 \\\n",
        "--save-after-valid True --log_every_n_steps 5000 --tensorboard_log True --batchsize 128 -lr 2 \\\n",
        "  --dynamic-batching full --truncate 320"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jiR-p-KpXD1a"
      },
      "outputs": [],
      "source": [
        "!parlai train_model -m memnn --init-model ./parlai/memnn-origin  -t gutenbergbookchars:Spectrum --model_file ./parlai/memnn-spectrum -veps 1 -eps 20 \\\n",
        "--save-after-valid True --log_every_n_steps 5000 --tensorboard_log True --batchsize 128 -lr 0.5"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "WYQLpSsv2z4u"
      },
      "outputs": [],
      "source": [
        "!parlai train_model -m tfidf_retriever -t gutenbergbookchars -mf ./parlai/gutenbertbookchars_tfidf \\\n",
        " -eps 1 --datatype train:ordered  --tensorboard_log True"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9c2llKlCdvko"
      },
      "source": [
        "# IR baseline\n",
        "\n",
        "**`NOTE:`**` the non-trained version is worse, so it is better to refer a pretrained zoo`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "13R7v8fo5Y_R"
      },
      "outputs": [],
      "source": [
        "!parlai eval_model -m ir_baseline -t gutenbergbookchars"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MBopfgdRbGQS"
      },
      "outputs": [],
      "source": [
        "!parlai eval_model -m ir_baseline -t gutenbergbookchars -mf zoo:wikipedia_full/tfidf_retriever/model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1q6oGZg1eFJX"
      },
      "outputs": [],
      "source": [
        "!parlai eval_model -m ir_baseline -t gutenbergbookchars:Spectrum -mf zoo:wikipedia_full/tfidf_retriever/model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yR-xCF-x-kCI"
      },
      "source": [
        "# IR-baseline model (dict)\n",
        "IR-baseline model, trained with the dict vocabulary.\n",
        "We limit the `-eps` to `5` according to the preliminary analysis here:\n",
        "https://docs.google.com/spreadsheets/d/1-_lJ-wfSlscyM1un1DdMw_xsuuD8U6GR_MmL1_iF0uY/edit#gid=0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8gYQwUQH5k6S"
      },
      "outputs": [],
      "source": [
        "!parlai train_model -m ir_baseline -t gutenbergbookchars \\\n",
        "  --dict-file ./parlai/gutenbergbookchars.dict -veps 1 -eps 5 \\\n",
        "  --model-file ./parlai/ir_baseline_dict --tensorboard_log True"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai train_model -m ir_baseline -t gutenbergbookchars:Spectrum \\\n",
        "  --init-model ./parlai/ir_baseline_dict \\\n",
        "  --dict-file ./parlai/gutenbergbookchars_spectrum.dict -veps 1 -eps 5 \\\n",
        "  --model-file ./parlai/ir_baseline_spectrum_dict --tensorboard_log True"
      ],
      "metadata": {
        "id": "b-NA8YVSOAV1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -m ir_baseline -mf parlai/ir_baseline_dict -t gutenbergbookchars --metrics all"
      ],
      "metadata": {
        "id": "vu4kWO3WMkO0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -m ir_baseline -mf parlai/ir_baseline_dict -t gutenbergbookchars:Spectrum --metrics all"
      ],
      "metadata": {
        "id": "-tYZOvAhPvlb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -m ir_baseline -mf parlai/ir_baseline_spectrum_dict -t gutenbergbookchars:Spectrum --metrics all"
      ],
      "metadata": {
        "id": "FMWllJv_RCu9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -m ir_baseline -mf parlai/ir_baseline_spectrum_dict -t gutenbergbookchars --metrics all"
      ],
      "metadata": {
        "id": "9lbGRtuSRoIE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Save content onto GDRIVE"
      ],
      "metadata": {
        "id": "2lmcZMcUXB0q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "cp -r /content/parlai/model_poly_* /content/gdrive/MyDrive/work-NewCastle/my-studies/dataset-v4.1/parlai/"
      ],
      "metadata": {
        "id": "IiKjROfYRyDm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fUmWFlhjB9HD"
      },
      "source": [
        "# Transformers\n",
        "\n",
        "ConvAI2 application of these models:\n",
        "To the certain extent correct, but we keep only information about persona\n",
        "without mentioning the exact type of the persona (at the moment, dataset v3)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install tensorrt"
      ],
      "metadata": {
        "id": "en36vCBSqf3y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5IJQ8RQRBy63"
      },
      "outputs": [],
      "source": [
        "!parlai eval_model -mf zoo:pretrained_transformers/model_poly/model -t gutenbergbookchars \\\n",
        " --eval-candidates inline --batchsize 20 --text-truncate 360 --dynamic-batching full"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -mf zoo:pretrained_transformers/model_poly/model -t gutenbergbookchars:Spectrum \\\n",
        " --eval-candidates inline --batchsize 20  --text-truncate 360 --dynamic-batching full"
      ],
      "metadata": {
        "id": "RKvjgRdOp13J"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oDO5cxnkCO9h"
      },
      "outputs": [],
      "source": [
        "!parlai eval_model -mf zoo:pretrained_transformers/model_bi/model -t gutenbergbookchars:Spectrum \\\n",
        " --eval-candidates inline --batchsize 20 --text-truncate 360 --dynamic-batching full"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UFbtEqoizDVQ"
      },
      "source": [
        "# Fine-tunning pretrained ConvAI2 models\n",
        "Fine-tunning model on the original dataset **without human level attributes**\n",
        "\n",
        "https://parl.ai/projects/polyencoder/\n",
        "\n",
        "Follow this tread in order to launch fine-tunning:\n",
        "\n",
        "https://github.com/facebookresearch/ParlAI/issues/2931"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ugEZh4r1y5UF"
      },
      "outputs": [],
      "source": [
        "!parlai train_model \\\n",
        "    --init-model zoo:pretrained_transformers/model_bi/model \\\n",
        "    --batchsize 32 -t gutenbergbookchars \\\n",
        "    --model transformer/biencoder --eval-batchsize 6 \\\n",
        "    --warmup_updates 100 --lr-scheduler-patience 0 \\\n",
        "    --lr-scheduler-decay 0.4 -lr 5e-05 --data-parallel True \\\n",
        "    --history-size 20 --label-truncate 72 --text-truncate 360 \\\n",
        "    --num-epochs 3.0 --max_train_time 200000 -veps 0.5 -vme 8000 \\\n",
        "    --validation-metric accuracy --validation-metric-mode max \\\n",
        "    --save-after-valid True --log_every_n_secs 20 --candidates batch \\\n",
        "    --dict-tokenizer bpe --dict-lower True --optimizer adamax \\\n",
        "    --output-scaling 0.06 \\\n",
        "     --variant xlm --reduction-type mean --share-encoders False \\\n",
        "     --learn-positional-embeddings True --n-layers 12 --n-heads 12 \\\n",
        "     --ffn-size 3072 --attention-dropout 0.1 --relu-dropout 0.0 --dropout 0.1 \\\n",
        "     --n-positions 1024 --embedding-size 768 --activation gelu \\\n",
        "     --embeddings-scale False --n-segments 2 --learn-embeddings True \\\n",
        "     --share-word-embeddings False --dict-endtoken __start__ --fp16 True \\\n",
        "     --model-file ./parlai/model_bi_finetuned --tensorboard_log True"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nGQaUPrDgUNe"
      },
      "outputs": [],
      "source": [
        "!parlai train_model \\\n",
        "  --init-model zoo:pretrained_transformers/poly_model_huge_reddit/model \\\n",
        "  -t gutenbergbookchars \\\n",
        "  --model transformer/polyencoder --batchsize 20 --eval-batchsize 10 \\\n",
        "  --warmup_updates 100 --lr-scheduler-patience 0 --lr-scheduler-decay 0.4 \\\n",
        "  -lr 5e-05 --data-parallel True --history-size 20 --label-truncate 72 \\\n",
        "  --text-truncate 360 --num-epochs 8.0 --max_train_time 200000 -veps 0.5 \\\n",
        "  -vme 8000 --validation-metric accuracy --validation-metric-mode max \\\n",
        "  --save-after-valid True --log_every_n_secs 20 --candidates batch --fp16 True \\\n",
        "  --dict-tokenizer bpe --dict-lower True --optimizer adamax --output-scaling 0.06 \\\n",
        "  --variant xlm --reduction-type mean --share-encoders False \\\n",
        "  --learn-positional-embeddings True --n-layers 12 --n-heads 12 --ffn-size 3072 \\\n",
        "  --attention-dropout 0.1 --relu-dropout 0.0 --dropout 0.1 --n-positions 1024 \\\n",
        "  --embedding-size 768 --activation gelu --embeddings-scale False --n-segments 2 \\\n",
        "  --learn-embeddings True --polyencoder-type codes --poly-n-codes 64 \\\n",
        "  --poly-attention-type basic --dict-endtoken __start__ \\\n",
        "  --model-file ./parlai/model_poly_finetuned --tensorboard_log True"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uJzx--sWVzDM"
      },
      "source": [
        "# Eval fine-tuned model on dataset with spectrums\n",
        "\n",
        "Important: mention version of the model with `.checkpoint`\n",
        "\n",
        "https://github.com/facebookresearch/ParlAI/issues/2904"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aOdzhteMQd5H"
      },
      "outputs": [],
      "source": [
        "!parlai eval_model -mf ./parlai/model_bi_finetuned.checkpoint -t gutenbergbookchars:Spectrum \\\n",
        "  --eval-candidates inline --batchsize 20"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DkLlS6OrWy77"
      },
      "outputs": [],
      "source": [
        "!parlai eval_model -mf ./parlai/model_poly_finetuned.checkpoint -t gutenbergbookchars:Spectrum \\\n",
        "  --eval-candidates inline --batchsize 20"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7uqUMWPdTiYH"
      },
      "outputs": [],
      "source": [
        "!parlai train_model \\\n",
        "    --init-model ./parlai/model_bi_finetuned.checkpoint \\\n",
        "    --batchsize 32 -t gutenbergbookchars:Spectrum \\\n",
        "    --model transformer/biencoder --eval-batchsize 6 \\\n",
        "    --warmup_updates 100 --lr-scheduler-patience 0 \\\n",
        "    --lr-scheduler-decay 0.4 -lr 5e-05 --data-parallel True \\\n",
        "    --history-size 20 --label-truncate 72 --text-truncate 360 \\\n",
        "    --num-epochs 6.0 --max_train_time 200000 -veps 0.5 -vme 8000 \\\n",
        "    --validation-metric accuracy --validation-metric-mode max \\\n",
        "    --save-after-valid True --log_every_n_secs 20 --candidates batch \\\n",
        "    --dict-tokenizer bpe --dict-lower True --optimizer adamax \\\n",
        "    --output-scaling 0.06 \\\n",
        "     --variant xlm --reduction-type mean --share-encoders False \\\n",
        "     --learn-positional-embeddings True --n-layers 12 --n-heads 12 \\\n",
        "     --ffn-size 3072 --attention-dropout 0.1 --relu-dropout 0.0 --dropout 0.1 \\\n",
        "     --n-positions 1024 --embedding-size 768 --activation gelu \\\n",
        "     --embeddings-scale False --n-segments 2 --learn-embeddings True \\\n",
        "     --share-word-embeddings False --dict-endtoken __start__ --fp16 True \\\n",
        "     --model-file ./parlai/model_bi_spectrums_finetuned --tensorboard_log True"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai train_model \\\n",
        "  --init-model ./parlai/model_poly_finetuned.checkpoint \\\n",
        "  -t gutenbergbookchars \\\n",
        "  --model transformer/polyencoder --batchsize 20 --eval-batchsize 10 \\\n",
        "  --warmup_updates 100 --lr-scheduler-patience 0 --lr-scheduler-decay 0.4 \\\n",
        "  -lr 5e-05 --data-parallel True --history-size 20 --label-truncate 72 \\\n",
        "  --text-truncate 360 --num-epochs 8.0 --max_train_time 200000 -veps 0.5 \\\n",
        "  -vme 8000 --validation-metric accuracy --validation-metric-mode max \\\n",
        "  --save-after-valid True --log_every_n_secs 20 --candidates batch --fp16 True \\\n",
        "  --dict-tokenizer bpe --dict-lower True --optimizer adamax --output-scaling 0.06 \\\n",
        "  --variant xlm --reduction-type mean --share-encoders False \\\n",
        "  --learn-positional-embeddings True --n-layers 12 --n-heads 12 --ffn-size 3072 \\\n",
        "  --attention-dropout 0.1 --relu-dropout 0.0 --dropout 0.1 --n-positions 1024 \\\n",
        "  --embedding-size 768 --activation gelu --embeddings-scale False --n-segments 2 \\\n",
        "  --learn-embeddings True --polyencoder-type codes --poly-n-codes 64 \\\n",
        "  --poly-attention-type basic --dict-endtoken __start__ \\\n",
        "  --model-file ./parlai/model_poly_spectrums_finetuned --tensorboard_log True"
      ],
      "metadata": {
        "id": "Wk5IFkOOEdE6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5LFjFTriu7h5"
      },
      "outputs": [],
      "source": [
        "!parlai eval_model -mf ./parlai/model_bi_spectrums_finetuned.checkpoint -t gutenbergbookchars:Spectrum \\\n",
        "  --eval-candidates inline --batchsize 20"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AP-toGcOuEhd"
      },
      "source": [
        "Just in case, check what happens with this model when it is without traits"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qu_-9Qk7uDKM"
      },
      "outputs": [],
      "source": [
        "!parlai eval_model -mf ./parlai/model_bi_spectrums_finetuned.checkpoint -t gutenbergbookchars --batchsize 20"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fQzc99TBS7F4"
      },
      "source": [
        "# Generative Models"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1FHvgHdzUbCV"
      },
      "source": [
        "### GPT-2 Small"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kdhZmicuK2oR"
      },
      "outputs": [],
      "source": [
        "!parlai train_model -m hugging_face/gpt2 --add-special-tokens True \\\n",
        "--add-start-token True --gpt2-size small -t gutenbergbookchars -bs 24 \\\n",
        "-mf parlai/gpt-2-small-no-hla -veps 0.5 --tensorboard_log True --num-epochs 6 \\\n",
        " --fp16 True --text-truncate 360 --dynamic-batching full \\\n",
        " --validation-metric f1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GAGnFK0USqnZ"
      },
      "outputs": [],
      "source": [
        "!parlai train_model -m hugging_face/gpt2 --add-special-tokens True \\\n",
        "--add-start-token True --gpt2-size small -t gutenbergbookchars:Spectrum -bs 8 \\\n",
        "-mf parlai/gpt-2-small-spectrum -veps 0.5 --tensorboard_log True --num-epochs 6 \\\n",
        " --fp16 True --text-truncate 360 --dynamic-batching full \\\n",
        "--validation-metric f1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x7HCwaInUMyZ"
      },
      "source": [
        "### GPT-2 Medium"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IwbL7ReXUPgN"
      },
      "outputs": [],
      "source": [
        "!parlai train_model -m hugging_face/gpt2 --add-special-tokens True \\\n",
        "--add-start-token True --gpt2-size medium -t gutenbergbookchars -bs 6 \\\n",
        "-mf parlai/gpt-2-medium-no-hla -veps 0.5 --tensorboard_log True --num-epochs 3 \\\n",
        " --sval True --fp16 True --text-truncate 360 --dynamic-batching full \\\n",
        " --validation-metric f1"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OzUX_V1hUPWW"
      },
      "outputs": [],
      "source": [
        "!parlai train_model -m hugging_face/gpt2 --add-special-tokens True \\\n",
        "--add-start-token True --gpt2-size medium -t gutenbergbookchars:Spectrum -bs 6 \\\n",
        "-mf parlai/gpt-2-medium-spectrum -veps 0.5 --tensorboard_log True --num-epochs 2 \\\n",
        " --fp16 True --text-truncate 360 --dynamic-batching full \\\n",
        "--validation-metric f1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OVrgU09oUJSg"
      },
      "source": [
        "## Infer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zAFlnhZYhyBp"
      },
      "outputs": [],
      "source": [
        "!parlai eval_model -mf parlai/gpt-2-small-no-hla -t gutenbergbookchars:Spectrum -bs 34 \\\n",
        "  --fp16 True --dynamic-batching full \\\n",
        "  --metrics ppl,f1,accuracy,rouge,bleu --report-filename \"gpt-2-small-no-hla-report.json\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cGXRkZEqoiY5"
      },
      "outputs": [],
      "source": [
        "# act as the pre-trained version on non-HLA.\n",
        "!parlai eval_model -mf parlai/gpt-2-small-spectrum -t gutenbergbookchars:Spectrum -bs 34 \\\n",
        "  --fp16 True --dynamic-batching full \\\n",
        "  --metrics ppl,f1,accuracy,rouge,bleu --report-filename \"gpt-2-small-hla-report.json\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PNH841xHmvx-"
      },
      "outputs": [],
      "source": [
        "# act as the pre-trained version on non-HLA.\n",
        "!parlai eval_model -mf parlai/gpt-2-medium-no-hla \\\n",
        "  -t gutenbergbookchars:Spectrum -bs 10 --fp16 True --dynamic-batching full \\\n",
        "  --metrics ppl,f1,accuracy,rouge,bleu --report-filename \"gpt-2-medium-no-hla-report.json\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lubqIF-NKHw0"
      },
      "outputs": [],
      "source": [
        "!parlai eval_model -mf parlai/gpt-2-medium-spectrum \\\n",
        "  -t gutenbergbookchars:Spectrum -bs 10 \\\n",
        "  --fp16 True --dynamic-batching full --metrics ppl,f1,accuracy,rouge,bleu \\\n",
        "  --report-filename \"gpt-2-medium-hla-report.json\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cpZDX1uysfPb"
      },
      "outputs": [],
      "source": [
        "!parlai eval_model -m hugging_face/gpt2 -t gutenbergbookchars \\\n",
        "  -bs 10 --fp16 True --dynamic-batching full --metrics ppl,f1,accuracy,rouge,bleu \\\n",
        "  --report-filename \"gpt-2-medium-report.json\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "golZbejLsh9j"
      },
      "outputs": [],
      "source": [
        "!parlai eval_model -m hugging_face/gpt2 --gpt2-size medium \\\n",
        "  -t gutenbergbookchars:Spectrum -bs 10 --fp16 True \\\n",
        "  --dynamic-batching full --metrics ppl,f1,accuracy,rouge,bleu"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "ppd1fA1EDB4t"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kY79S1QKrE9F"
      },
      "source": [
        "# Test Zone"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "22HUocWPrCZ4"
      },
      "outputs": [],
      "source": [
        "!parlai eval_model --model fixed_response --task dailydialog --fixed-response \"how may i help you ?\" --metrics rouge"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "CCm_jvtt3CGK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -mf ./parlai/model_bi_finetuned.checkpoint -t gutenbergbookchars:S1 \\\n",
        "  --eval-candidates inline --batchsize 20 --report-filename \"S1-bi-model.json\""
      ],
      "metadata": {
        "id": "vKg7U-ey3Cds"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -mf ./parlai/model_bi_finetuned.checkpoint -t gutenbergbookchars:S2 \\\n",
        "  --eval-candidates inline --batchsize 20 --report-filename \"S2-bi-model.json\""
      ],
      "metadata": {
        "id": "VfwcC4NN3L1K"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -mf ./parlai/model_bi_finetuned.checkpoint -t gutenbergbookchars:S3 \\\n",
        "  --eval-candidates inline --batchsize 20 --report-filename \"S3-bi-model.json\""
      ],
      "metadata": {
        "id": "D9YN57GG3NgZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -mf ./parlai/model_bi_finetuned.checkpoint -t gutenbergbookchars:S4 \\\n",
        "  --eval-candidates inline --batchsize 20 --report-filename \"S4-bi-model.json\""
      ],
      "metadata": {
        "id": "vXxAIJ_v3YZi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -mf ./parlai/model_bi_finetuned.checkpoint -t gutenbergbookchars:S5 \\\n",
        "  --eval-candidates inline --batchsize 20 --report-filename \"S5-bi-model.json\""
      ],
      "metadata": {
        "id": "ZDzJP8Jy3Zit"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "nMqMMXT16E-K"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -mf ./parlai/model_poly_finetuned.checkpoint -t gutenbergbookchars:S1 \\\n",
        "  --eval-candidates inline --batchsize 20 --report-filename \"S1-poly-model.json\""
      ],
      "metadata": {
        "id": "mdo_xYhC6FS9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -mf ./parlai/model_poly_finetuned.checkpoint -t gutenbergbookchars:S2 \\\n",
        "  --eval-candidates inline --batchsize 20 --report-filename \"S2-poly-model.json\""
      ],
      "metadata": {
        "id": "vUjqZFr66MUC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -mf ./parlai/model_poly_finetuned.checkpoint -t gutenbergbookchars:S3 \\\n",
        "  --eval-candidates inline --batchsize 20 --report-filename \"S3-poly-model.json\""
      ],
      "metadata": {
        "id": "44Qzxeek6bJg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -mf ./parlai/model_poly_finetuned.checkpoint -t gutenbergbookchars:S4 \\\n",
        "  --eval-candidates inline --batchsize 20 --report-filename \"S4-poly-model.json\""
      ],
      "metadata": {
        "id": "ohRUd6E36dIE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -mf ./parlai/model_poly_finetuned.checkpoint -t gutenbergbookchars:S5 \\\n",
        "  --eval-candidates inline --batchsize 20 --report-filename \"S5-poly-model.json\""
      ],
      "metadata": {
        "id": "EG-glqYP6eyC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -mf parlai/gpt-2-small-spectrum \\\n",
        "  -t gutenbergbookchars:S1 -bs 10 \\\n",
        "  --fp16 True --dynamic-batching full --metrics ppl,f1,accuracy,rouge,bleu \\\n",
        "  --report-filename \"s1-gpt2-small.json\""
      ],
      "metadata": {
        "id": "b9AQ3W4S7ioN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -mf parlai/gpt-2-small-spectrum \\\n",
        "  -t gutenbergbookchars:S2 -bs 10 \\\n",
        "  --fp16 True --dynamic-batching full --metrics ppl,f1,accuracy,rouge,bleu \\\n",
        "  --report-filename \"s2-gpt2-small.json\""
      ],
      "metadata": {
        "id": "E1ETfPsE7r8y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -mf parlai/gpt-2-small-spectrum \\\n",
        "  -t gutenbergbookchars:S3 -bs 10 \\\n",
        "  --fp16 True --dynamic-batching full --metrics ppl,f1,accuracy,rouge,bleu \\\n",
        "  --report-filename \"s3-gpt2-small.json\""
      ],
      "metadata": {
        "id": "LWUTKHBN8Azx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -mf parlai/gpt-2-small-spectrum \\\n",
        "  -t gutenbergbookchars:S4 -bs 10 \\\n",
        "  --fp16 True --dynamic-batching full --metrics ppl,f1,accuracy,rouge,bleu \\\n",
        "  --report-filename \"s4-gpt2-small.json\""
      ],
      "metadata": {
        "id": "LSpj10X-8BTG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!parlai eval_model -mf parlai/gpt-2-small-spectrum \\\n",
        "  -t gutenbergbookchars:S5 -bs 10 \\\n",
        "  --fp16 True --dynamic-batching full --metrics ppl,f1,accuracy,rouge,bleu \\\n",
        "  --report-filename \"s5-gpt2-small.json\""
      ],
      "metadata": {
        "id": "RPzjFbXl8BtR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "m0Duunze6MA_"
      }
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "machine_shape": "hm",
      "provenance": [],
      "gpuType": "V100",
      "authorship_tag": "ABX9TyOd8Df1OzRBut6GqrcRCU6a",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}