08_4_Developing a Sequence-to-Sequence Model__M1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network\" target=\"_blank\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\">\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sequence-to-Sequence RNN Models: Translation Task\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **60** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this hands-on guide, you will explore the fundamentals of sequence-to-sequence models and learn how to implement an RNN-based model for a translation task using PyTorch.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## __Table of Contents__\n",
    "\n",
    "<ol>\n",
    "    <li><a href=\"#Objectives\">Objectives</a></li>\n",
    "    <li>\n",
    "        <a href=\"#Setup\">Setup</a>\n",
    "        <ol>\n",
    "            <li><a href=\"#Installing-required-libraries\">Installing required libraries</a></li>\n",
    "            <li><a href=\"#Importing-required-libraries\">Importing required libraries</a></li>\n",
    "        </ol>\n",
    "    </li>\n",
    "    <li>\n",
    "        <a href=\"#Background\">Background</a>\n",
    "        <ol>\n",
    "            <li><a href=\"#History-of-sequence-to-sequence-models\">History of sequence-to-sequence models</a></li>\n",
    "            <li><a href=\"#Introduction-to-RNNs\">Introduction to RNNs</a></li>\n",
    "            <li><a href=\"#Sequence-to-sequence-architecture\">Sequence-to-sequence architecture</a></li>    \n",
    "        </ol>\n",
    "    <li><a href=\"#Encoder-implementation-in-PyTorch\">Encoder implementation in PyTorch</a></li>\n",
    "    <li><a href=\"#Decoder-implementation-in-PyTorch\">Decoder implementation in PyTorch</a></li>\n",
    "    <li><a href=\"#Sequence-to-sequence-model-implementation-in-PyTorch\">Sequence-to-sequence model implementation in PyTorch</a>\n",
    "    <li><a href=\"#Training-model-in-PyTorch\">Training model in PyTorch</a></li>\n",
    "    <li><a href=\"#Evaluating-model-in-PyTorch\">Evaluating model in PyTorch</a></li>\n",
    "    <li><a href=\"#Data-preprocessing\">Data preprocessing</a></li>\n",
    "    <li><a href=\"#Training-the-model\">Training the model</a></li>\n",
    "    <ol>\n",
    "        <li><a href=\"#Initializations\">Initializations</a></li>\n",
    "        <li><a href=\"#Training\">Training</a></li>\n",
    "    </ol>\n",
    "    <li><a href=\"#Model-inference\">Model inference</a></li>\n",
    "    <li><a href=\"#BLEU-score-metric-for-evaluation\">BLEU score metric for evaluation</a></li>\n",
    "    <li><a href=\"#Exercises\">Exercises</a></li>\n",
    "\n",
    "</ol>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n",
    "After completing this lab, you will be able to:\n",
    "\n",
    " - Comprehend recurrent neural networks (RNN) architecture\n",
    " - Create an Encoder-Decoder model for a translation task\n",
    " - Train and evaluate the model\n",
    " - Create a generator for the translation task\n",
    " - Explain concepts related to Perplexity and BLEU score and use them for evaluating translations\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing required libraries\n",
    "\n",
    "All the required libraries are pre-installed in the Skills Network Labs environment. However, if you run this notebook commands in a different Jupyter environment (e.g. Watson Studio or Ananconda), you will need to install these libraries by removing the `#` sign before `!mamba` in the code cell below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# All Libraries required for this lab are listed below. The libraries pre-installed on Skills Network Labs are commented.\n",
    "# !mamba install -qy numpy==1.21.4 seaborn==0.9.0\n",
    "# Note: If your environment doesn't support \"!mamba install\", use \"!pip install\"\n",
    "# The working version of each package is commented in front of each package name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following required libraries are __not__ pre-installed in the Skills Network Labs environment. __You will need to run the following cell__ to install them:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting torchtext==0.15.1\n",
      "  Downloading torchtext-0.15.1-cp311-cp311-manylinux1_x86_64.whl.metadata (7.4 kB)\n",
      "Requirement already satisfied: tqdm in /opt/conda/lib/python3.11/site-packages (from torchtext==0.15.1) (4.66.4)\n",
      "Requirement already satisfied: requests in /opt/conda/lib/python3.11/site-packages (from torchtext==0.15.1) (2.31.0)\n",
      "Collecting torch==2.0.0 (from torchtext==0.15.1)\n",
      "  Downloading torch-2.0.0-cp311-cp311-manylinux1_x86_64.whl.metadata (24 kB)\n",
      "Collecting numpy (from torchtext==0.15.1)\n",
      "  Downloading numpy-2.1.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (62 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.0/62.0 kB\u001b[0m \u001b[31m9.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting torchdata==0.6.0 (from torchtext==0.15.1)\n",
      "  Downloading torchdata-0.6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (892 bytes)\n",
      "Collecting filelock (from torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading filelock-3.16.1-py3-none-any.whl.metadata (2.9 kB)\n",
      "Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchtext==0.15.1) (4.12.2)\n",
      "Collecting sympy (from torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading sympy-1.13.3-py3-none-any.whl.metadata (12 kB)\n",
      "Collecting networkx (from torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading networkx-3.4.2-py3-none-any.whl.metadata (6.3 kB)\n",
      "Requirement already satisfied: jinja2 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchtext==0.15.1) (3.1.3)\n",
      "Collecting nvidia-cuda-nvrtc-cu11==11.7.99 (from torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)\n",
      "Collecting nvidia-cuda-runtime-cu11==11.7.99 (from torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
      "Collecting nvidia-cuda-cupti-cu11==11.7.101 (from torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
      "Collecting nvidia-cudnn-cu11==8.5.0.96 (from torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
      "Collecting nvidia-cublas-cu11==11.10.3.66 (from torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
      "Collecting nvidia-cufft-cu11==10.9.0.58 (from torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)\n",
      "Collecting nvidia-curand-cu11==10.2.10.91 (from torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
      "Collecting nvidia-cusolver-cu11==11.4.0.1 (from torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading nvidia_cusolver_cu11-11.4.0.1-2-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
      "Collecting nvidia-cusparse-cu11==11.7.4.91 (from torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading nvidia_cusparse_cu11-11.7.4.91-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)\n",
      "Collecting nvidia-nccl-cu11==2.14.3 (from torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl.metadata (1.8 kB)\n",
      "Collecting nvidia-nvtx-cu11==11.7.91 (from torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)\n",
      "Collecting triton==2.0.0 (from torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading triton-2.0.0-1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.0 kB)\n",
      "Requirement already satisfied: urllib3>=1.25 in /opt/conda/lib/python3.11/site-packages (from torchdata==0.6.0->torchtext==0.15.1) (2.2.1)\n",
      "Requirement already satisfied: setuptools in /opt/conda/lib/python3.11/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.0->torchtext==0.15.1) (69.5.1)\n",
      "Requirement already satisfied: wheel in /opt/conda/lib/python3.11/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.0->torchtext==0.15.1) (0.43.0)\n",
      "Collecting cmake (from triton==2.0.0->torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading cmake-3.31.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.5 kB)\n",
      "Collecting lit (from triton==2.0.0->torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading lit-18.1.8-py3-none-any.whl.metadata (2.5 kB)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.11/site-packages (from requests->torchtext==0.15.1) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.11/site-packages (from requests->torchtext==0.15.1) (3.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.11/site-packages (from requests->torchtext==0.15.1) (2024.8.30)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.11/site-packages (from jinja2->torch==2.0.0->torchtext==0.15.1) (2.1.5)\n",
      "Collecting mpmath<1.4,>=1.1.0 (from sympy->torch==2.0.0->torchtext==0.15.1)\n",
      "  Downloading mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)\n",
      "Downloading torchtext-0.15.1-cp311-cp311-manylinux1_x86_64.whl (2.0 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.0/2.0 MB\u001b[0m \u001b[31m102.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading torch-2.0.0-cp311-cp311-manylinux1_x86_64.whl (619.9 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m619.9/619.9 MB\u001b[0m \u001b[31m824.0 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading torchdata-0.6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.6/4.6 MB\u001b[0m \u001b[31m101.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m317.1/317.1 MB\u001b[0m \u001b[31m1.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl (11.8 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m11.8/11.8 MB\u001b[0m \u001b[31m50.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m0:01\u001b[0m\n",
      "\u001b[?25hDownloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.0/21.0 MB\u001b[0m \u001b[31m28.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m849.3/849.3 kB\u001b[0m \u001b[31m66.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m557.1/557.1 MB\u001b[0m \u001b[31m714.4 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux2014_x86_64.whl (168.4 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m168.4/168.4 MB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl (54.6 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m54.6/54.6 MB\u001b[0m \u001b[31m12.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading nvidia_cusolver_cu11-11.4.0.1-2-py3-none-manylinux1_x86_64.whl (102.6 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m102.6/102.6 MB\u001b[0m \u001b[31m1.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading nvidia_cusparse_cu11-11.7.4.91-py3-none-manylinux1_x86_64.whl (173.2 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m173.2/173.2 MB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl (177.1 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m177.1/177.1 MB\u001b[0m \u001b[31m833.7 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl (98 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m98.6/98.6 kB\u001b[0m \u001b[31m15.1 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading triton-2.0.0-1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (63.3 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m63.3/63.3 MB\u001b[0m \u001b[31m11.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading numpy-2.1.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.3 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m16.3/16.3 MB\u001b[0m \u001b[31m35.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading filelock-3.16.1-py3-none-any.whl (16 kB)\n",
      "Downloading networkx-3.4.2-py3-none-any.whl (1.7 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m78.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading sympy-1.13.3-py3-none-any.whl (6.2 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.2/6.2 MB\u001b[0m \u001b[31m52.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading mpmath-1.3.0-py3-none-any.whl (536 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m536.2/536.2 kB\u001b[0m \u001b[31m45.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading cmake-3.31.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (27.8 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m27.8/27.8 MB\u001b[0m \u001b[31m23.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading lit-18.1.8-py3-none-any.whl (96 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m96.4/96.4 kB\u001b[0m \u001b[31m10.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: mpmath, lit, sympy, nvidia-nvtx-cu11, nvidia-nccl-cu11, nvidia-cusparse-cu11, nvidia-curand-cu11, nvidia-cufft-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, numpy, networkx, filelock, cmake, nvidia-cusolver-cu11, nvidia-cudnn-cu11, triton, torch, torchdata, torchtext\n",
      "Successfully installed cmake-3.31.1 filelock-3.16.1 lit-18.1.8 mpmath-1.3.0 networkx-3.4.2 numpy-2.1.3 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-cupti-cu11-11.7.101 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 nvidia-cufft-cu11-10.9.0.58 nvidia-curand-cu11-10.2.10.91 nvidia-cusolver-cu11-11.4.0.1 nvidia-cusparse-cu11-11.7.4.91 nvidia-nccl-cu11-2.14.3 nvidia-nvtx-cu11-11.7.91 sympy-1.13.3 torch-2.0.0 torchdata-0.6.0 torchtext-0.15.1 triton-2.0.0\n",
      "Requirement already satisfied: torch==2.0.0 in /opt/conda/lib/python3.11/site-packages (2.0.0)\n",
      "Requirement already satisfied: filelock in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (3.16.1)\n",
      "Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (4.12.2)\n",
      "Requirement already satisfied: sympy in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (1.13.3)\n",
      "Requirement already satisfied: networkx in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (3.4.2)\n",
      "Requirement already satisfied: jinja2 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (3.1.3)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (11.7.101)\n",
      "Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (8.5.0.96)\n",
      "Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (11.10.3.66)\n",
      "Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (10.9.0.58)\n",
      "Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (10.2.10.91)\n",
      "Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (11.4.0.1)\n",
      "Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (11.7.4.91)\n",
      "Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (2.14.3)\n",
      "Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (11.7.91)\n",
      "Requirement already satisfied: triton==2.0.0 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0) (2.0.0)\n",
      "Requirement already satisfied: setuptools in /opt/conda/lib/python3.11/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.0) (69.5.1)\n",
      "Requirement already satisfied: wheel in /opt/conda/lib/python3.11/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.0) (0.43.0)\n",
      "Requirement already satisfied: cmake in /opt/conda/lib/python3.11/site-packages (from triton==2.0.0->torch==2.0.0) (3.31.1)\n",
      "Requirement already satisfied: lit in /opt/conda/lib/python3.11/site-packages (from triton==2.0.0->torch==2.0.0) (18.1.8)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.11/site-packages (from jinja2->torch==2.0.0) (2.1.5)\n",
      "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/lib/python3.11/site-packages (from sympy->torch==2.0.0) (1.3.0)\n",
      "Collecting spacy==3.7.2\n",
      "  Downloading spacy-3.7.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (25 kB)\n",
      "Collecting spacy-legacy<3.1.0,>=3.0.11 (from spacy==3.7.2)\n",
      "  Downloading spacy_legacy-3.0.12-py2.py3-none-any.whl.metadata (2.8 kB)\n",
      "Collecting spacy-loggers<2.0.0,>=1.0.0 (from spacy==3.7.2)\n",
      "  Downloading spacy_loggers-1.0.5-py3-none-any.whl.metadata (23 kB)\n",
      "Collecting murmurhash<1.1.0,>=0.28.0 (from spacy==3.7.2)\n",
      "  Downloading murmurhash-1.0.11-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.0 kB)\n",
      "Collecting cymem<2.1.0,>=2.0.2 (from spacy==3.7.2)\n",
      "  Downloading cymem-2.0.10-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.4 kB)\n",
      "Collecting preshed<3.1.0,>=3.0.2 (from spacy==3.7.2)\n",
      "  Downloading preshed-3.0.9-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.2 kB)\n",
      "Collecting thinc<8.3.0,>=8.1.8 (from spacy==3.7.2)\n",
      "  Downloading thinc-8.2.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (15 kB)\n",
      "Collecting wasabi<1.2.0,>=0.9.1 (from spacy==3.7.2)\n",
      "  Downloading wasabi-1.1.3-py3-none-any.whl.metadata (28 kB)\n",
      "Collecting srsly<3.0.0,>=2.4.3 (from spacy==3.7.2)\n",
      "  Downloading srsly-2.4.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (20 kB)\n",
      "Collecting catalogue<2.1.0,>=2.0.6 (from spacy==3.7.2)\n",
      "  Downloading catalogue-2.0.10-py3-none-any.whl.metadata (14 kB)\n",
      "Collecting weasel<0.4.0,>=0.1.0 (from spacy==3.7.2)\n",
      "  Downloading weasel-0.3.4-py3-none-any.whl.metadata (4.7 kB)\n",
      "Collecting typer<0.10.0,>=0.3.0 (from spacy==3.7.2)\n",
      "  Downloading typer-0.9.4-py3-none-any.whl.metadata (14 kB)\n",
      "Collecting smart-open<7.0.0,>=5.2.1 (from spacy==3.7.2)\n",
      "  Downloading smart_open-6.4.0-py3-none-any.whl.metadata (21 kB)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /opt/conda/lib/python3.11/site-packages (from spacy==3.7.2) (4.66.4)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /opt/conda/lib/python3.11/site-packages (from spacy==3.7.2) (2.31.0)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /opt/conda/lib/python3.11/site-packages (from spacy==3.7.2) (2.10.0)\n",
      "Requirement already satisfied: jinja2 in /opt/conda/lib/python3.11/site-packages (from spacy==3.7.2) (3.1.3)\n",
      "Requirement already satisfied: setuptools in /opt/conda/lib/python3.11/site-packages (from spacy==3.7.2) (69.5.1)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.11/site-packages (from spacy==3.7.2) (24.0)\n",
      "Collecting langcodes<4.0.0,>=3.2.0 (from spacy==3.7.2)\n",
      "  Downloading langcodes-3.5.0-py3-none-any.whl.metadata (29 kB)\n",
      "Requirement already satisfied: numpy>=1.19.0 in /opt/conda/lib/python3.11/site-packages (from spacy==3.7.2) (2.1.3)\n",
      "Collecting language-data>=1.2 (from langcodes<4.0.0,>=3.2.0->spacy==3.7.2)\n",
      "  Downloading language_data-1.3.0-py3-none-any.whl.metadata (4.3 kB)\n",
      "Requirement already satisfied: annotated-types>=0.6.0 in /opt/conda/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy==3.7.2) (0.7.0)\n",
      "Requirement already satisfied: pydantic-core==2.27.0 in /opt/conda/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy==3.7.2) (2.27.0)\n",
      "Requirement already satisfied: typing-extensions>=4.12.2 in /opt/conda/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy==3.7.2) (4.12.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy==3.7.2) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy==3.7.2) (3.7)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy==3.7.2) (2.2.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy==3.7.2) (2024.8.30)\n",
      "Collecting blis<0.8.0,>=0.7.8 (from thinc<8.3.0,>=8.1.8->spacy==3.7.2)\n",
      "  Downloading blis-0.7.11-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.4 kB)\n",
      "Collecting confection<1.0.0,>=0.0.1 (from thinc<8.3.0,>=8.1.8->spacy==3.7.2)\n",
      "  Downloading confection-0.1.5-py3-none-any.whl.metadata (19 kB)\n",
      "Collecting numpy>=1.19.0 (from spacy==3.7.2)\n",
      "  Downloading numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.0/61.0 kB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting click<9.0.0,>=7.1.1 (from typer<0.10.0,>=0.3.0->spacy==3.7.2)\n",
      "  Downloading click-8.1.7-py3-none-any.whl.metadata (3.0 kB)\n",
      "Collecting cloudpathlib<0.17.0,>=0.7.0 (from weasel<0.4.0,>=0.1.0->spacy==3.7.2)\n",
      "  Downloading cloudpathlib-0.16.0-py3-none-any.whl.metadata (14 kB)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.11/site-packages (from jinja2->spacy==3.7.2) (2.1.5)\n",
      "Collecting marisa-trie>=1.1.0 (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy==3.7.2)\n",
      "  Downloading marisa_trie-1.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.0 kB)\n",
      "Downloading spacy-3.7.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.6 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.6/6.6 MB\u001b[0m \u001b[31m91.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading catalogue-2.0.10-py3-none-any.whl (17 kB)\n",
      "Downloading cymem-2.0.10-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (218 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m218.8/218.8 kB\u001b[0m \u001b[31m25.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading langcodes-3.5.0-py3-none-any.whl (182 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m183.0/183.0 kB\u001b[0m \u001b[31m20.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading murmurhash-1.0.11-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (134 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.3/134.3 kB\u001b[0m \u001b[31m15.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading preshed-3.0.9-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (157 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m157.2/157.2 kB\u001b[0m \u001b[31m22.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading smart_open-6.4.0-py3-none-any.whl (57 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.0/57.0 kB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading spacy_legacy-3.0.12-py2.py3-none-any.whl (29 kB)\n",
      "Downloading spacy_loggers-1.0.5-py3-none-any.whl (22 kB)\n",
      "Downloading srsly-2.4.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (490 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m490.9/490.9 kB\u001b[0m \u001b[31m49.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading thinc-8.2.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (920 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m920.2/920.2 kB\u001b[0m \u001b[31m63.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.3 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m18.3/18.3 MB\u001b[0m \u001b[31m91.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading typer-0.9.4-py3-none-any.whl (45 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading wasabi-1.1.3-py3-none-any.whl (27 kB)\n",
      "Downloading weasel-0.3.4-py3-none-any.whl (50 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.1/50.1 kB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading blis-0.7.11-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.2 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m10.2/10.2 MB\u001b[0m \u001b[31m102.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m0:01\u001b[0m\n",
      "\u001b[?25hDownloading click-8.1.7-py3-none-any.whl (97 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m97.9/97.9 kB\u001b[0m \u001b[31m12.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading cloudpathlib-0.16.0-py3-none-any.whl (45 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.0/45.0 kB\u001b[0m \u001b[31m5.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading confection-0.1.5-py3-none-any.whl (35 kB)\n",
      "Downloading language_data-1.3.0-py3-none-any.whl (5.4 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.4/5.4 MB\u001b[0m \u001b[31m103.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading marisa_trie-1.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.4 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m87.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: cymem, wasabi, spacy-loggers, spacy-legacy, smart-open, numpy, murmurhash, marisa-trie, cloudpathlib, click, catalogue, typer, srsly, preshed, language-data, blis, langcodes, confection, weasel, thinc, spacy\n",
      "  Attempting uninstall: numpy\n",
      "    Found existing installation: numpy 2.1.3\n",
      "    Uninstalling numpy-2.1.3:\n",
      "      Successfully uninstalled numpy-2.1.3\n",
      "Successfully installed blis-0.7.11 catalogue-2.0.10 click-8.1.7 cloudpathlib-0.16.0 confection-0.1.5 cymem-2.0.10 langcodes-3.5.0 language-data-1.3.0 marisa-trie-1.2.1 murmurhash-1.0.11 numpy-1.26.4 preshed-3.0.9 smart-open-6.4.0 spacy-3.7.2 spacy-legacy-3.0.12 spacy-loggers-1.0.5 srsly-2.4.8 thinc-8.2.5 typer-0.9.4 wasabi-1.1.3 weasel-0.3.4\n",
      "Requirement already satisfied: torchdata==0.6.0 in /opt/conda/lib/python3.11/site-packages (0.6.0)\n",
      "Requirement already satisfied: urllib3>=1.25 in /opt/conda/lib/python3.11/site-packages (from torchdata==0.6.0) (2.2.1)\n",
      "Requirement already satisfied: requests in /opt/conda/lib/python3.11/site-packages (from torchdata==0.6.0) (2.31.0)\n",
      "Requirement already satisfied: torch==2.0.0 in /opt/conda/lib/python3.11/site-packages (from torchdata==0.6.0) (2.0.0)\n",
      "Requirement already satisfied: filelock in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (3.16.1)\n",
      "Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (4.12.2)\n",
      "Requirement already satisfied: sympy in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (1.13.3)\n",
      "Requirement already satisfied: networkx in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (3.4.2)\n",
      "Requirement already satisfied: jinja2 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (3.1.3)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (11.7.101)\n",
      "Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (8.5.0.96)\n",
      "Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (11.10.3.66)\n",
      "Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (10.9.0.58)\n",
      "Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (10.2.10.91)\n",
      "Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (11.4.0.1)\n",
      "Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (11.7.4.91)\n",
      "Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (2.14.3)\n",
      "Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (11.7.91)\n",
      "Requirement already satisfied: triton==2.0.0 in /opt/conda/lib/python3.11/site-packages (from torch==2.0.0->torchdata==0.6.0) (2.0.0)\n",
      "Requirement already satisfied: setuptools in /opt/conda/lib/python3.11/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.0->torchdata==0.6.0) (69.5.1)\n",
      "Requirement already satisfied: wheel in /opt/conda/lib/python3.11/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.0->torchdata==0.6.0) (0.43.0)\n",
      "Requirement already satisfied: cmake in /opt/conda/lib/python3.11/site-packages (from triton==2.0.0->torch==2.0.0->torchdata==0.6.0) (3.31.1)\n",
      "Requirement already satisfied: lit in /opt/conda/lib/python3.11/site-packages (from triton==2.0.0->torch==2.0.0->torchdata==0.6.0) (18.1.8)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.11/site-packages (from requests->torchdata==0.6.0) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.11/site-packages (from requests->torchdata==0.6.0) (3.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.11/site-packages (from requests->torchdata==0.6.0) (2024.8.30)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.11/site-packages (from jinja2->torch==2.0.0->torchdata==0.6.0) (2.1.5)\n",
      "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/lib/python3.11/site-packages (from sympy->torch==2.0.0->torchdata==0.6.0) (1.3.0)\n",
      "Collecting nltk==3.8.1\n",
      "  Downloading nltk-3.8.1-py3-none-any.whl.metadata (2.8 kB)\n",
      "Requirement already satisfied: click in /opt/conda/lib/python3.11/site-packages (from nltk==3.8.1) (8.1.7)\n",
      "Collecting joblib (from nltk==3.8.1)\n",
      "  Downloading joblib-1.4.2-py3-none-any.whl.metadata (5.4 kB)\n",
      "Collecting regex>=2021.8.3 (from nltk==3.8.1)\n",
      "  Downloading regex-2024.11.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (40 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.5/40.5 kB\u001b[0m \u001b[31m3.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: tqdm in /opt/conda/lib/python3.11/site-packages (from nltk==3.8.1) (4.66.4)\n",
      "Downloading nltk-3.8.1-py3-none-any.whl (1.5 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.5/1.5 MB\u001b[0m \u001b[31m65.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading regex-2024.11.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (792 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m792.7/792.7 kB\u001b[0m \u001b[31m52.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading joblib-1.4.2-py3-none-any.whl (301 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m301.8/301.8 kB\u001b[0m \u001b[31m32.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: regex, joblib, nltk\n",
      "Successfully installed joblib-1.4.2 nltk-3.8.1 regex-2024.11.6\n",
      "Collecting matplotlib\n",
      "  Downloading matplotlib-3.9.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)\n",
      "Collecting contourpy>=1.0.1 (from matplotlib)\n",
      "  Downloading contourpy-1.3.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.4 kB)\n",
      "Collecting cycler>=0.10 (from matplotlib)\n",
      "  Downloading cycler-0.12.1-py3-none-any.whl.metadata (3.8 kB)\n",
      "Collecting fonttools>=4.22.0 (from matplotlib)\n",
      "  Downloading fonttools-4.55.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (164 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m164.5/164.5 kB\u001b[0m \u001b[31m16.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting kiwisolver>=1.3.1 (from matplotlib)\n",
      "  Downloading kiwisolver-1.4.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.3 kB)\n",
      "Requirement already satisfied: numpy>=1.23 in /opt/conda/lib/python3.11/site-packages (from matplotlib) (1.26.4)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.11/site-packages (from matplotlib) (24.0)\n",
      "Collecting pillow>=8 (from matplotlib)\n",
      "  Downloading pillow-11.0.0-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (9.1 kB)\n",
      "Collecting pyparsing>=2.3.1 (from matplotlib)\n",
      "  Downloading pyparsing-3.2.0-py3-none-any.whl.metadata (5.0 kB)\n",
      "Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/lib/python3.11/site-packages (from matplotlib) (2.9.0)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.11/site-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)\n",
      "Downloading matplotlib-3.9.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (8.3 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.3/8.3 MB\u001b[0m \u001b[31m110.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading contourpy-1.3.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (326 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m326.2/326.2 kB\u001b[0m \u001b[31m37.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading cycler-0.12.1-py3-none-any.whl (8.3 kB)\n",
      "Downloading fonttools-4.55.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.9 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.9/4.9 MB\u001b[0m \u001b[31m115.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading kiwisolver-1.4.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.4 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.4/1.4 MB\u001b[0m \u001b[31m77.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading pillow-11.0.0-cp311-cp311-manylinux_2_28_x86_64.whl (4.4 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.4/4.4 MB\u001b[0m \u001b[31m88.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading pyparsing-3.2.0-py3-none-any.whl (106 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m106.9/106.9 kB\u001b[0m \u001b[31m12.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: pyparsing, pillow, kiwisolver, fonttools, cycler, contourpy, matplotlib\n",
      "Successfully installed contourpy-1.3.1 cycler-0.12.1 fonttools-4.55.0 kiwisolver-1.4.7 matplotlib-3.9.3 pillow-11.0.0 pyparsing-3.2.0\n",
      "Collecting en-core-web-sm==3.7.1\n",
      "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl (12.8 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.8/12.8 MB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: spacy<3.8.0,>=3.7.2 in /opt/conda/lib/python3.11/site-packages (from en-core-web-sm==3.7.1) (3.7.2)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.0.12)\n",
      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.0.5)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.0.11)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.0.10)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.0.9)\n",
      "Requirement already satisfied: thinc<8.3.0,>=8.1.8 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (8.2.5)\n",
      "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.1.3)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.4.8)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.0.10)\n",
      "Requirement already satisfied: weasel<0.4.0,>=0.1.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.3.4)\n",
      "Requirement already satisfied: typer<0.10.0,>=0.3.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.9.4)\n",
      "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (6.4.0)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (4.66.4)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.31.0)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.10.0)\n",
      "Requirement already satisfied: jinja2 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.1.3)\n",
      "Requirement already satisfied: setuptools in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (69.5.1)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (24.0)\n",
      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.5.0)\n",
      "Requirement already satisfied: numpy>=1.19.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.26.4)\n",
      "Requirement already satisfied: language-data>=1.2 in /opt/conda/lib/python3.11/site-packages (from langcodes<4.0.0,>=3.2.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.3.0)\n",
      "Requirement already satisfied: annotated-types>=0.6.0 in /opt/conda/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.7.0)\n",
      "Requirement already satisfied: pydantic-core==2.27.0 in /opt/conda/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.27.0)\n",
      "Requirement already satisfied: typing-extensions>=4.12.2 in /opt/conda/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (4.12.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.7)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.2.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2024.8.30)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /opt/conda/lib/python3.11/site-packages (from thinc<8.3.0,>=8.1.8->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.7.11)\n",
      "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /opt/conda/lib/python3.11/site-packages (from thinc<8.3.0,>=8.1.8->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.1.5)\n",
      "Requirement already satisfied: click<9.0.0,>=7.1.1 in /opt/conda/lib/python3.11/site-packages (from typer<0.10.0,>=0.3.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (8.1.7)\n",
      "Requirement already satisfied: cloudpathlib<0.17.0,>=0.7.0 in /opt/conda/lib/python3.11/site-packages (from weasel<0.4.0,>=0.1.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.16.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.11/site-packages (from jinja2->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.1.5)\n",
      "Requirement already satisfied: marisa-trie>=1.1.0 in /opt/conda/lib/python3.11/site-packages (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.2.1)\n",
      "Installing collected packages: en-core-web-sm\n",
      "Successfully installed en-core-web-sm-3.7.1\n",
      "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the package via spacy.load('en_core_web_sm')\n",
      "Collecting de-core-news-sm==3.7.0\n",
      "  Downloading https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-3.7.0/de_core_news_sm-3.7.0-py3-none-any.whl (14.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.6/14.6 MB\u001b[0m \u001b[31m115.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: spacy<3.8.0,>=3.7.0 in /opt/conda/lib/python3.11/site-packages (from de-core-news-sm==3.7.0) (3.7.2)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.0.12)\n",
      "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.0.5)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.0.11)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.0.10)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.0.9)\n",
      "Requirement already satisfied: thinc<8.3.0,>=8.1.8 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (8.2.5)\n",
      "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.1.3)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.4.8)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.0.10)\n",
      "Requirement already satisfied: weasel<0.4.0,>=0.1.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.3.4)\n",
      "Requirement already satisfied: typer<0.10.0,>=0.3.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.9.4)\n",
      "Requirement already satisfied: smart-open<7.0.0,>=5.2.1 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (6.4.0)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (4.66.4)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.31.0)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.10.0)\n",
      "Requirement already satisfied: jinja2 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.1.3)\n",
      "Requirement already satisfied: setuptools in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (69.5.1)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (24.0)\n",
      "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.5.0)\n",
      "Requirement already satisfied: numpy>=1.19.0 in /opt/conda/lib/python3.11/site-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.26.4)\n",
      "Requirement already satisfied: language-data>=1.2 in /opt/conda/lib/python3.11/site-packages (from langcodes<4.0.0,>=3.2.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.3.0)\n",
      "Requirement already satisfied: annotated-types>=0.6.0 in /opt/conda/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.7.0)\n",
      "Requirement already satisfied: pydantic-core==2.27.0 in /opt/conda/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.27.0)\n",
      "Requirement already satisfied: typing-extensions>=4.12.2 in /opt/conda/lib/python3.11/site-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (4.12.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.7)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.2.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.11/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2024.8.30)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /opt/conda/lib/python3.11/site-packages (from thinc<8.3.0,>=8.1.8->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.7.11)\n",
      "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /opt/conda/lib/python3.11/site-packages (from thinc<8.3.0,>=8.1.8->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.1.5)\n",
      "Requirement already satisfied: click<9.0.0,>=7.1.1 in /opt/conda/lib/python3.11/site-packages (from typer<0.10.0,>=0.3.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (8.1.7)\n",
      "Requirement already satisfied: cloudpathlib<0.17.0,>=0.7.0 in /opt/conda/lib/python3.11/site-packages (from weasel<0.4.0,>=0.1.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.16.0)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.11/site-packages (from jinja2->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.1.5)\n",
      "Requirement already satisfied: marisa-trie>=1.1.0 in /opt/conda/lib/python3.11/site-packages (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.2.1)\n",
      "Installing collected packages: de-core-news-sm\n",
      "Successfully installed de-core-news-sm-3.7.0\n",
      "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the package via spacy.load('de_core_news_sm')\n"
     ]
    }
   ],
   "source": [
    "!pip install torchtext==0.15.1\n",
    "!pip install torch==2.0.0\n",
    "!pip install spacy==3.7.2\n",
    "!pip install torchdata==0.6.0\n",
    "!pip install portalocker>=2.0.0 #2.7.0\n",
    "!pip install nltk==3.8.1\n",
    "!pip install -U matplotlib\n",
    "\n",
    "!python -m spacy download en_core_web_sm\n",
    "!python -m spacy download de_core_news_sm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing required libraries\n",
    "\n",
    "_It is recommended that you import all required libraries in one place (here):_\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchtext.data.utils import get_tokenizer\n",
    "from torchtext.vocab import build_vocab_from_iterator\n",
    "from torchtext.datasets import multi30k, Multi30k\n",
    "from typing import Iterable, List\n",
    "from torch.nn.utils.rnn import pad_sequence\n",
    "from torch.utils.data import DataLoader\n",
    "from torchdata.datapipes.iter import IterableWrapper, Mapper\n",
    "import torchtext\n",
    "from torchtext.vocab import build_vocab_from_iterator\n",
    "from nltk.translate.bleu_score import sentence_bleu\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "\n",
    "import numpy as np\n",
    "import random\n",
    "import math\n",
    "import time\n",
    "from tqdm import tqdm\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "# You can also use this section to suppress warnings generated by your code:\n",
    "def warn(*args, **kwargs):\n",
    "    pass\n",
    "import warnings\n",
    "warnings.warn = warn\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background\n",
    "\n",
    "Sequence-to-sequence (Seq2seq) models have revolutionized various natural language processing (NLP) tasks, such as machine translation, text summarization, and chatbots. These models employ Recurrent Neural Networks (RNNs) to process variable-length input sequences and generate variable-length output sequences.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### History of sequence-to-sequence models\n",
    "\n",
    "Sequence-to-sequence models were introduced as an extension of traditional feedforward neural networks.\n",
    "Researchers realized the need for models that could handle variable-length input and output sequences, such as machine translation.\n",
    "The pioneering work of Sutskever et al. (2014) introduced the use of RNNs for seq2seq models.\n",
    "\n",
    "Here are some main objectives of seq2seq models:\n",
    "- Translation: Translating a sequence from one domain to another (e.g., English to French).\n",
    "- Question answering: Generating a natural language response given an input sentence (e.g., chatbots).\n",
    "- Summarization: Summarizing a long document into a shorter sequence of sentences.\n",
    "And many more applications that involve sequence generation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction to RNNs\n",
    "\n",
    "RNNs are a class of neural networks designed to process sequential data.\n",
    "They maintain an internal memory($h_t$) to capture information from previous steps and use it for current predictions.\n",
    "RNNs have a recurrent connection that allows information to flow from one step to the next.\n",
    "Recurrent Neural Networks (RNNs) operate on sequences and utilize previous states to influence the current state. Here's the general formulation of a simple RNN:\n",
    "\n",
    "\n",
    "Given:\n",
    "\n",
    "-$ \\mathbf{x}_t $: input vector at time step $t$\n",
    "\n",
    "-$ \\mathbf{h}_{t-1} $: hidden state vector from the previous time step\n",
    "\n",
    "-$ \\mathbf{W}_x $ and $ \\mathbf{W}_h $: weight matrices for the input and hidden state, respectively\n",
    "\n",
    "-$ \\mathbf{b} $: bias vector\n",
    "\n",
    "-$\\sigma$: activation function (often a sigmoid or tanh)\n",
    "\n",
    "The update equations for the hidden state $ \\mathbf{h}_t $ and the output $ \\mathbf{y}_t $ are as follows:\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\mathbf{h}_t &= \\sigma(\\mathbf{W}_x \\cdot \\mathbf{x}_t + \\mathbf{W}_h \\cdot \\mathbf{h}_{t-1} + \\mathbf{b})\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "It can be seen that the hidden state function depends on the previous hidden state as well as the input at time t, which is why it has a collective memory of previous time steps.\n",
    "\n",
    "For the output (if you're making a prediction at each time step):\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\mathbf{y}_t &= \\text{softmax}(\\mathbf{W}_o \\cdot \\mathbf{h}_t + \\mathbf{b}_o)\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "Where:\n",
    "\n",
    "$ \\mathbf{W}_o $: weight matrix for the output AND $ \\mathbf{b}_o$: bias vector for the output\n",
    "\n",
    "\n",
    "\n",
    "Depending on the specific task, an RNN cell can either produce an output from $h_t$ or solely transfer it to the succeeding cell, serving as internal memory. While the architecture's ability to retain memory might seem elusive at first glance, let's elucidate this by implementing a simple RNN to handle the following data mechanism:\n",
    "\n",
    "![a title](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-AI0205EN-SkillsNetwork/Screenshot%202023-10-19%20at%2011.29.23%E2%80%AFAM.png)\n",
    "\n",
    "\n",
    "The diagram showcases a state machine or transition model with three distinct states, depicted by the prominent purple circles. Each state is distinctly labeled with a value for $ h $: $ h = -1 $, $ h = 0 $, and $ h = 1 $.\n",
    "\n",
    "1. **State $ h = -1 $**:\n",
    "   - Maintains itself when $ x = 1 $ (illustrated by the yellow loop).\n",
    "   - Proceeds to the $ h = 0$ state upon receiving $ x = -1$ (highlighted by the red arrow).\n",
    "\n",
    "2. **State $ h = 0 $**:\n",
    "   - Moves to the $h = -1 $ state when $ x = 1$ (illustrated by the red arrow).\n",
    "   - Advances to the $ h = 1 $ state with $ x = -1$ (marked by the red arrow).\n",
    "\n",
    "3. **State $h = 1 $**:\n",
    "   - Sustains its position when $ x = -1 $ (indicated by the yellow loop).\n",
    "   - Transitions to the $ h = 0 $ state upon receiving $ x = 1 $ (signified by the red arrow).\n",
    "\n",
    "To encapsulate, the diagram effectively portrays transitions among three states based on the input $ x $. Contingent on the prevailing state and the input $ x $, the state machine either transitions to a different state or remains stationary.\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "You can represent the previously mentioned state machine using the layer detailed below. Use $tanh$ as the $h$ value should fall between [-1, 1]. Note that you have excluded the output for simplification:\n",
    "\n",
    "$$\\begin{align*}\n",
    "W_{xh} & = -10.0 \\\\\\\\\\\\\\\\\\\\\n",
    "W_{hh} & = 10.0 \\\\\n",
    "b_h & = 0.0 \\\\\n",
    "x_t & = 1 \\\\\n",
    "h_{\\text{prev}} & = 0.0 \\\\\n",
    "h_t & = \\tanh(x_t \\cdot W_{xh} + h_{\\text{prev}} \\cdot W_{hh} + b_h)\n",
    "\\end{align*}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    " W_xh=torch.tensor(-10.0)\n",
    " W_hh=torch.tensor(10.0)\n",
    " b_h=torch.tensor(0.0)\n",
    " x_t=1\n",
    " h_prev=torch.tensor(-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider the following sequence $x_t$ for  $t=0,1,..,7$,\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "X=[1,1,-1,-1,1,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assuming that you start from the intial state $h = 0$,  with the above input vector $x$, the state vector $h$ should look like this:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "H=[-1,-1,0,1,0,-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t= 1\n",
      "h_t-1 -1\n",
      "x_t 1\n",
      "h_t -1.0\n",
      "\n",
      "\n",
      "t= 2\n",
      "h_t-1 -1.0\n",
      "x_t 1\n",
      "h_t -1.0\n",
      "\n",
      "\n",
      "t= 3\n",
      "h_t-1 -1.0\n",
      "x_t -1\n",
      "h_t 0.0\n",
      "\n",
      "\n",
      "t= 4\n",
      "h_t-1 0.0\n",
      "x_t -1\n",
      "h_t 1.0\n",
      "\n",
      "\n",
      "t= 5\n",
      "h_t-1 1.0\n",
      "x_t 1\n",
      "h_t 0.0\n",
      "\n",
      "\n",
      "t= 6\n",
      "h_t-1 0.0\n",
      "x_t 1\n",
      "h_t -1.0\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Initialize an empty list to store the predicted state values\n",
    "H_hat = []\n",
    "# Loop through each data point in the input sequence X\n",
    "t=1\n",
    "for x in X:\n",
    "    # Assign the current data point to x_t\n",
    "    print(\"t=\",t)\n",
    "    x_t = x\n",
    "    # Print the value of the previous state (h at time t-1)\n",
    "    print(\"h_t-1\", h_prev.item())\n",
    "\n",
    "    # Compute the current state (h at time t) using the RNN formula with tanh activation\n",
    "    h_t = torch.tanh(x_t * W_xh + h_prev * W_hh + b_h)\n",
    "\n",
    "    # Update h_prev to the current state value for the next iteration\n",
    "    h_prev = h_t\n",
    "\n",
    "    # Print the current input value (x at time t)\n",
    "    print(\"x_t\", x_t)\n",
    "\n",
    "    # Print the computed state value (h at time t)\n",
    "    print(\"h_t\", h_t.item())\n",
    "    print(\"\\n\")\n",
    "\n",
    "    # Append the current state value to the H_hat list after converting it to integer\n",
    "    H_hat.append(int(h_t.item()))\n",
    "    t+=1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "You can evaluate the accuracy of the predicted state ```H_hat``` by comparing it to the actual state ```H```. In RNNs, the state $ h_t $ is utilized to predict an output sequence $y_t $ based on the given input sequence $ x_t $.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-1, -1, 0, 1, 0, -1]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "H_hat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-1, -1, 0, 1, 0, -1]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "H"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While you have pre-defined the $W_{xh}$ and $W_{hh}$  and $b_h$, in practice these values need to be identified through training on data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice, modifications and enhancements, such as Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), are often used to address issues like the vanishing gradient problem in basic RNNs.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An LSTM cell has three main components: an input gate, a forget gate, and an output gate.\n",
    "- The **input gate** controls how much new information should be stored in the cell's memory. It looks at the current input and the previous hidden state and decides which parts of the new input to remember.\n",
    "- The **forget gate** determines what information should be discarded or forgotten from the cell's memory. It considers the current input and the previous hidden state and decides which parts of the previous memory are no longer relevant.\n",
    "- The **output gate** determines what information should be outputted from the cell. It looks at the current input and the previous hidden state and decides which parts of the cell's memory to include in the output.\n",
    "\n",
    "The key idea behind LSTM cells is that they have a separate memory state that can selectively retain or forget information over time. This helps them handle long-range dependencies and remember important information from earlier steps in a sequence.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sequence-to-sequence architecture\n",
    "\n",
    "Seq2seq models have an Encoder-Decoder structure. The encoder encodes the input sequence into a fixed-dimensional representation, often called the context vector($h_t$). The decoder generates the output sequence based on the encoded context vector.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look closer into the encoder and decoder boxes in the video below. Translation is a typical sequence-to-sequence task. The input is a sequence of words in the original language(\"I love to travel\"), while output is its translation in the destination language(\"J'adore voyager\"). As shown in the video, input is fed into the decoder part, one word after another. Each RNN cell receives a word($x_t$) and has an internal memory($h_t$). After processing the input and $h_t$, RNN cell passes an updated context vector($h_{t+1}$) to the next RNN cell. When the end of sentence is reached, the context vector is passed to the decoder part. Decoder cells are also RNN cells that receive context vector and generate the output word by word. Each RNN receives the generated word as well as the updated context vector from its previous cell and generates the next word($y_t$). This architecture allows for generating text without length restrictions.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<video width=\"640\" height=\"480\"\n",
    "src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-AI0205EN-SkillsNetwork/Translation_RNN.mp4\"\n",
    "controls>\n",
    "</video>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoder implementation in PyTorch\n",
    "\n",
    "To implement the encoder part using Pytorch, you will create the sub-class of the torch.nn.Module class and define the __init__() and __forward__() method.\n",
    "\n",
    "Let's first define the parameters that are used in __init__() function:\n",
    "- The `vocab_len` is nothing but the number of unique words present in the vocabulary. After pre-processing the data, you can count the number of unique words in your vocabulary and use that count here. This will be the dimension of the model input.\n",
    "- The embedding_dim is the output dimension of the embedding vector you need. A good practice is to use 256-512 for sample demo app like you are building here.\n",
    "- LSTM can indeed be stacked, allowing for multiple layers. In the initial implementation, you will use only one layer. However, to accommodate future flexibility, you will pass the parameter `n_layers` to specify the number of layers in the LSTM.\n",
    "- `hid_dim` is the dimensionality of the hidden and cell states.\n",
    "- `dropout` is the amount of dropout to use. This is a regularization parameter to prevent overfitting.\n",
    "\n",
    "Now, let's look into the layers:\n",
    "- The Embedding layer takes the input data and outputs the embedding vector, hence the dimension of those needs to be defined as `vocab_len` and `embedding_dim`.\n",
    "- The LSTM Layer takes the `embedding_dim` as the input data and creates total 3 outputs: `hidden`, `cell` and `output`. Here you need to define the number of neurons you need in LSTM, which is defined using the `hid_dim`.\n",
    "\n",
    "\n",
    "In the __forward__() function, the Embedding layer is defined that utilizes the `vocab_len` to internally convert the input_batch into a one-hot representation. Next, the LSTM layer receives the embedded input and outputs three vectors: Output, Hidden and cell. As for the encoder, you don't require the output vector from the LSTM as you only pass the context vector(`hidden`+`cell`) to the decoder block. Therefore, forward() only returns hidden and cell.\n",
    "\n",
    "Note: When using an LSTM, you have an additional cell state. However, if you were using a GRU, you would only have the hidden state.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n",
    "    def __init__(self, vocab_len, emb_dim, hid_dim, n_layers, dropout_prob):\n",
    "        super().__init__()\n",
    "\n",
    "        self.hid_dim = hid_dim\n",
    "        self.n_layers = n_layers\n",
    "\n",
    "        self.embedding = nn.Embedding(vocab_len, emb_dim)\n",
    "\n",
    "        self.lstm = nn.LSTM(emb_dim, hid_dim, n_layers, dropout = dropout_prob)\n",
    "        self.dropout = nn.Dropout(dropout_prob)\n",
    "\n",
    "    def forward(self, input_batch):\n",
    "        #input_batch = [src len, batch size]\n",
    "        embed = self.dropout(self.embedding(input_batch))\n",
    "        embed = embed.to(device)\n",
    "        #outputs = [src len, batch size, hid dim * n directions]\n",
    "        #hidden = [n layers * n directions, batch size, hid dim]\n",
    "        #cell = [n layers * n directions, batch size, hid dim]\n",
    "        outputs, (hidden, cell) = self.lstm(embed)\n",
    "\n",
    "        return hidden, cell\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you are ready to create an encoder instance to see how it works:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_len = 8\n",
    "emb_dim = 10\n",
    "hid_dim=8\n",
    "n_layers=1\n",
    "dropout_prob=0.5\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
    "encoder_t = Encoder(vocab_len, emb_dim, hid_dim, n_layers, dropout_prob).to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see a simple example where the encoder forward method transforms the `src` sentence into a `hidden` and `cell` states. tensor([[0],[3],[4],[2],[1]]) is equal to `src` = 0,3,4,2,1 in which each number represents a token in the `src` vocabulary. For instance, 0:`<bos>`,3:\"Das\", 4:\"ist\",2:\"schön\", 1:`<eos>`. Note that here you have batch size of 1.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of input(src) tensor: torch.Size([5, 1])\n",
      "Hidden tensor from encoder: tensor([[[-0.0167,  0.4281, -0.1922,  0.0549, -0.1653, -0.1185,  0.0299,\n",
      "          -0.1351]]], grad_fn=<StackBackward0>) \n",
      "Cell tensor from encoder: tensor([[[-0.0393,  0.6798, -0.3463,  0.0975, -0.3017, -0.3406,  0.0870,\n",
      "          -0.3092]]], grad_fn=<StackBackward0>)\n"
     ]
    }
   ],
   "source": [
    "src_batch = torch.tensor([[0,3,4,2,1]])\n",
    "# you need to transpose the input tensor as the encoder LSTM is in Sequence_first mode by default\n",
    "src_batch = src_batch.t().to(device)\n",
    "print(\"Shape of input(src) tensor:\", src_batch.shape)\n",
    "hidden_t , cell_t = encoder_t(src_batch)\n",
    "print(\"Hidden tensor from encoder:\",hidden_t ,\"\\nCell tensor from encoder:\", cell_t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The encoder takes the entire source sequence as input, which consists of a sequence of words or tokens. The encoder LSTM processes the entire input sequence and updates its hidden states at each time step. The hidden states of the LSTM network act as a form of memory and capture the contextual information of the input sequence. After processing the entire input sequence, the final hidden state of the encoder LSTM captures the summarized representation of the input sequence's context. This final hidden state is sometimes referred to as the \"context vector\".\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoder implementation in PyTorch\n",
    "\n",
    "To have a better understanding of the internal mechanism of the decoder part, let's take a closer look into it:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<video width=\"640\" height=\"480\"\n",
    "       src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-AI0205EN-SkillsNetwork/decoder_RNN.mp4\"\n",
    "       controls>\n",
    "</video>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The decoder class inherits from nn.Module, which is a base class for all neural network modules in PyTorch.\n",
    "The constructor (__init__ method) initializes the parameters and layers of the decoder.\n",
    "- `output_dim` is the number of possible output values(target vocab length).\n",
    "- `emb_dim` is the dimensionality of the embedding layer.\n",
    "- `hid_dim` is the dimensionality of the hidden state in the LSTM.\n",
    "- `n_layers` is the number of layers in the LSTM.\n",
    "- `dropout` is the dropout probability.\n",
    "\n",
    "The decoder contains the following layers:\n",
    "- `embedding`: An embedding layer that maps the output values to dense vectors of size emb_dim.\n",
    "- `lstm`: An LSTM layer that takes the embedded input and produces hidden states of size hid_dim.\n",
    "-  `fc_out`: A linear layer that maps the LSTM output to the output dimension output_dim.\n",
    "- `softmax`: A log-softmax activation function applied to the output to obtain a probability distribution over the output values.\n",
    "- `dropout`: A dropout layer that applies dropout to the embedded input.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Decoder(nn.Module):\n",
    "    def __init__(self, output_dim, emb_dim, hid_dim, n_layers, dropout):\n",
    "        super().__init__()\n",
    "\n",
    "        self.output_dim = output_dim\n",
    "        self.hid_dim = hid_dim\n",
    "        self.n_layers = n_layers\n",
    "\n",
    "\n",
    "        self.embedding = nn.Embedding(output_dim, emb_dim)\n",
    "        self.lstm = nn.LSTM(emb_dim, hid_dim, n_layers, dropout = dropout)\n",
    "        self.fc_out = nn.Linear(hid_dim, output_dim)\n",
    "        self.softmax = nn.LogSoftmax(dim=1)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "    def forward(self, input, hidden, cell):\n",
    "\n",
    "\n",
    "        #input = [batch size]\n",
    "\n",
    "        #hidden = [n layers * n directions, batch size, hid dim]\n",
    "        #cell = [n layers * n directions, batch size, hid dim]\n",
    "\n",
    "        #n directions in the decoder will both always be 1, therefore:\n",
    "        #hidden = [n layers, batch size, hid dim]\n",
    "        #context = [n layers, batch size, hid dim]\n",
    "\n",
    "        input = input.unsqueeze(0)\n",
    "        #input = [1, batch size]\n",
    "\n",
    "        embedded = self.dropout(self.embedding(input))\n",
    "        #embedded = [1, batch size, emb dim]\n",
    "\n",
    "        output, (hidden, cell) = self.lstm(embedded, (hidden, cell))\n",
    "        #output = [seq len, batch size, hid dim * n directions]\n",
    "        #hidden = [n layers * n directions, batch size, hid dim]\n",
    "        #cell = [n layers * n directions, batch size, hid dim]\n",
    "\n",
    "        #seq len and n directions will always be 1 in the decoder, therefore:\n",
    "        #output = [1, batch size, hid dim]\n",
    "        #hidden = [n layers, batch size, hid dim]\n",
    "        #cell = [n layers, batch size, hid dim]\n",
    "        prediction_logit = self.fc_out(output.squeeze(0))\n",
    "        prediction = self.softmax(prediction_logit)\n",
    "        #prediction = [batch size, output dim]\n",
    "\n",
    "\n",
    "        return prediction, hidden, cell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can create a decoder instance. The output dimension is set as the target vocab length.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_dim = 6\n",
    "emb_dim=10\n",
    "hid_dim = 8\n",
    "n_layers=1\n",
    "dropout=0.5\n",
    "decoder_t = Decoder(output_dim, emb_dim, hid_dim, n_layers, dropout).to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have instances of both encoder and decoder, you are ready to connect them (the red box in the diagram below). First, let's see how you can pass the Hidden and Cell (the pink cell within the red box) from encoder (the green boxes container) to decoder (the orange boxes container). Looking at the diagram, you can see that the decoder also receives an input which is the previous word that it has predicted. For the first decoder cell, this input is `<bos>` token. Each decoder cell outputs a prediction and updates the cell and state to pass to the next decoder cell. prediction is a probability distribution over possible target tokens (length of target vocab).\n",
    "\n",
    "![connection](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-AI0205EN-SkillsNetwork/ED_connection.JPG)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction: tensor([[-1.5288, -1.6982, -1.7631, -1.9680, -1.9547, -1.9151]],\n",
      "       grad_fn=<LogSoftmaxBackward0>) \n",
      "Hidden: tensor([[[ 0.0143,  0.0774,  0.0247,  0.1473, -0.4244,  0.0311,  0.0442,\n",
      "           0.0841]]], grad_fn=<StackBackward0>) \n",
      "Cell: tensor([[[ 0.1032,  0.2182,  0.0537,  0.4129, -0.5258,  0.0652,  0.2073,\n",
      "           0.3299]]], grad_fn=<StackBackward0>)\n"
     ]
    }
   ],
   "source": [
    "input_t = torch.tensor([0]).to(device) #<bos>\n",
    "input_t.shape\n",
    "prediction, hidden, cell = decoder_t(input_t, hidden_t , cell_t)\n",
    "print(\"Prediction:\", prediction, '\\nHidden:',hidden,'\\nCell:', cell)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Encoder-decoder connection\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright! You learned how to create encoder and decoder modules and how to pass input to them. Now you need to create the connection so that the model can process (`src`,`trg`) pairs and generate the translation. suppose that `trg` is tensor ([[0],[2],[3],[5],[1]]) which is equal to sequence 0,2,3,5,1 in which each number represents a token in the target vocabulary. For instance, 0:`<bos>`,2:\"this\", 3:\"is\",5:\"beautiful\", 1:`<eos>`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[[ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000]],\n",
      "\n",
      "        [[-1.4360, -1.7722, -1.7827, -1.9793, -1.9747, -1.9171]],\n",
      "\n",
      "        [[-1.3303, -2.1302, -1.7360, -1.9873, -1.8773, -1.8936]],\n",
      "\n",
      "        [[-1.3632, -2.0700, -1.8923, -1.9223, -1.9382, -1.7316]],\n",
      "\n",
      "        [[-1.2384, -2.1523, -1.9634, -1.9585, -1.9096, -1.8058]]],\n",
      "       grad_fn=<CopySlices>) torch.Size([5, 1, 6])\n"
     ]
    }
   ],
   "source": [
    "\n",
    "#trg = [trg len, batch size]\n",
    "#teacher_forcing_ratio is probability to use teacher forcing\n",
    "#e.g. if teacher_forcing_ratio is 0.75 you use ground-truth inputs 75% of the time\n",
    "teacher_forcing_ratio = 0.5\n",
    "trg = torch.tensor([[0],[2],[3],[5],[1]]).to(device)\n",
    "\n",
    "\n",
    "batch_size = trg.shape[1]\n",
    "trg_len = trg.shape[0]\n",
    "trg_vocab_size = decoder_t.output_dim\n",
    "\n",
    "#tensor to store decoder outputs\n",
    "outputs_t = torch.zeros(trg_len, batch_size, trg_vocab_size).to(device)\n",
    "\n",
    "#send to device\n",
    "\n",
    "hidden_t = hidden_t.to(device)\n",
    "cell_t = cell_t.to(device)\n",
    "\n",
    "\n",
    "#first input to the decoder is the <bos> tokens\n",
    "input = trg[0,:]\n",
    "\n",
    "\n",
    "for t in range(1, trg_len):\n",
    "\n",
    "    #you loop through the trg len and generate tokens\n",
    "    #decoder receives previous generated token, cell and hidden\n",
    "    # decoder outputs it prediction(probablity distribution for the next token) and updates hidden and cell\n",
    "    output_t, hidden_t, cell_t = decoder_t(input, hidden_t, cell_t)\n",
    "\n",
    "    #place predictions in a tensor holding predictions for each token\n",
    "    outputs_t[t] = output_t\n",
    "\n",
    "    #decide if you are going to use teacher forcing or not\n",
    "    teacher_force = random.random() < teacher_forcing_ratio\n",
    "\n",
    "    #get the highest predicted token from your predictions\n",
    "    top1 = output_t.argmax(1)\n",
    "\n",
    "\n",
    "    #if teacher forcing, use actual next token as next input\n",
    "    #if not, use predicted token\n",
    "    #input = trg[t] if teacher_force else top1\n",
    "    input = trg[t] if teacher_force else top1\n",
    "\n",
    "print(outputs_t,outputs_t.shape )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The size of output tensor is (trg_len, batch_size, trg_vocab_size). This is because for each `trg` token (length of `trg`) the model outputs a probability distribution over all possible tokens(trg vocab length). Therefore, to generate the predicted tokens or translation of the `src` sentence, you need to get the maximum probability for each token:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0],\n",
      "        [0],\n",
      "        [0],\n",
      "        [0],\n",
      "        [0]])\n"
     ]
    }
   ],
   "source": [
    "# Note that you need to get the argmax from the second dimension as **outputs** is an array of **output** tensors\n",
    "pred_tokens = outputs_t.argmax(2)\n",
    "print(pred_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is no surprise that the translation is not correct (trg = tensor([[0],[2],[3],[5],[1]]) as the model has not yet gone through any training.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's put together all the code for connecting the encoder and decoder in a seq2seq class for better usability.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sequence-to-sequence model implementation in PyTorch\n",
    "Let's connect encoder and decoder components to create the seq2seq model.\n",
    "\n",
    "You define the seq2seq class that inherits from nn.Module, which is the base class for all neural network modules in PyTorch.\n",
    "Inputs are:\n",
    "- `encoder` and `decoder` are instances of the encoder and decoder networks that you have already defined.\n",
    "- `device` specifies the device (e.g., CPU or GPU) on which the computations will be performed.\n",
    "- `trg_vocab` represents the vocabulary of the target language. It is used to determine the size of the output vocabulary.\n",
    "\n",
    "**forward** method defines the forward pass of the seq2seq model. It takes three arguments: `src`, `trg`, and `teacher_forcing_ratio`.:\n",
    "\n",
    "- `src` represents the source sequences, and `trg` represents the target sequences.\n",
    "- `teacher_forcing_ratio` is a probability that determines whether teacher forcing will be used during training only. Teacher forcing is a technique where the true target sequence is fed as input to the decoder at each time step, instead of using the predicted output from the previous time step.\n",
    "\n",
    "The **forward** method initializes some variables needed for the forward pass, such as `batch_size`, `trg_len`, and `trg_vocab_size`. It also creates an empty tensor called `outputs` to store the decoder outputs for each time step.\n",
    "\n",
    "The `hidden` and `cell` states of the encoder are obtained by calling the encoder (src) method. These states are then used as the initial states for the decoder.\n",
    "\n",
    "The input to the decoder at the first time step is the <bos> token of the target sequences.\n",
    "\n",
    "The decoder is iterated over for each time step in the target sequences (`for t in range(1, trg_len)`). The input, along with the previous hidden and cell states, is passed to the decoder, and it produces an output tensor. The `output` tensor is stored in the `outputs` tensor.\n",
    "\n",
    "At each time step, there is a decision made whether to use teacher forcing or not based on the teacher_forcing_ratio probability. If teacher forcing is used, the true next token from the target sequences (`trg[t]`) is used as the input for the next time step. Otherwise, the predicted token from the previous time step (`top1 = output.argmax(1)`) is used.\n",
    "\n",
    "Finally, the `outputs` tensor containing the predicted outputs for each time step is returned.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Seq2Seq(nn.Module):\n",
    "    def __init__(self, encoder, decoder, device,trg_vocab):\n",
    "        super().__init__()\n",
    "\n",
    "        self.encoder = encoder\n",
    "        self.decoder = decoder\n",
    "        self.device = device\n",
    "        self.trg_vocab = trg_vocab\n",
    "\n",
    "        assert encoder.hid_dim == decoder.hid_dim, \\\n",
    "            \"Hidden dimensions of encoder and decoder must be equal!\"\n",
    "        assert encoder.n_layers == decoder.n_layers, \\\n",
    "            \"Encoder and decoder must have equal number of layers!\"\n",
    "\n",
    "    def forward(self, src, trg, teacher_forcing_ratio = 0.5):\n",
    "        #src = [src len, batch size]\n",
    "        #trg = [trg len, batch size]\n",
    "        #teacher_forcing_ratio is probability to use teacher forcing\n",
    "        #e.g. if teacher_forcing_ratio is 0.75 you use ground-truth inputs 75% of the time\n",
    "\n",
    "\n",
    "        batch_size = trg.shape[1]\n",
    "        trg_len = trg.shape[0]\n",
    "        trg_vocab_size = self.decoder.output_dim\n",
    "\n",
    "        #tensor to store decoder outputs\n",
    "        outputs = torch.zeros(trg_len, batch_size, trg_vocab_size).to(self.device)\n",
    "\n",
    "        #last hidden state of the encoder is used as the initial hidden state of the decoder\n",
    "        hidden, cell = self.encoder(src)\n",
    "        hidden = hidden.to(device)\n",
    "        cell = cell.to(device)\n",
    "\n",
    "\n",
    "        #first input to the decoder is the <bos> tokens\n",
    "        input = trg[0,:]\n",
    "\n",
    "        for t in range(1, trg_len):\n",
    "\n",
    "            #insert input token embedding, previous hidden and previous cell states\n",
    "            #receive output tensor (predictions) and new hidden and cell states\n",
    "            output, hidden, cell = self.decoder(input, hidden, cell)\n",
    "\n",
    "            #place predictions in a tensor holding predictions for each token\n",
    "            outputs[t] = output\n",
    "\n",
    "            #decide if you are going to use teacher forcing or not\n",
    "            teacher_force = random.random() < teacher_forcing_ratio\n",
    "\n",
    "            #get the highest predicted token from your predictions\n",
    "            top1 = output.argmax(1)\n",
    "\n",
    "\n",
    "            #if teacher forcing, use actual next token as next input\n",
    "            #if not, use predicted token\n",
    "            #input = trg[t] if teacher_force else top1\n",
    "            input = trg[t] if teacher_force else top1\n",
    "\n",
    "\n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training model in PyTorch\n",
    "Now that the model is defined, you define a train function the seq2seq model. Let's go through the code and understand its components:\n",
    "\n",
    "1. `train(model, iterator, optimizer, criterion, clip)` takes five arguments:\n",
    "\n",
    "   - `model` is the model that will be trained.\n",
    "   - `iterator` is an iterable object that provides the training data in batches.\n",
    "   - `optimizer` is the optimization algorithm used to update the model's parameters.\n",
    "   - `criterion` is the loss function that measures the model's performance.\n",
    "   - `clip` is a value used to clip the gradients to prevent them from becoming too large during backpropagation.\n",
    "\n",
    "2. The function starts by setting the model to training mode with `model.train()`. This is necessary to enable certain layers (e.g., dropout) that behave differently during training and evaluation.\n",
    "\n",
    "3. It initializes a variable `epoch_loss` to keep track of the accumulated loss during the epoch.\n",
    "\n",
    "4. The function iterates over the training data provided by the `iterator`. Each iteration retrieves a batch of input sequences (`src`) and target sequences (`trg`).\n",
    "\n",
    "5. The input sequences (`src`) and target sequences (`trg`) are moved to the appropriate device (e.g., GPU) using `src = src.to(device)` and `trg = trg.to(device)`.\n",
    "\n",
    "6. The gradients of the model's parameters are cleared using `optimizer.zero_grad()` to prepare for the new batch.\n",
    "\n",
    "7. The model is then called with `output = model(src, trg)` to obtain the model's predictions for the target sequences.\n",
    "\n",
    "8. The `output` tensor has dimensions `[trg len, batch size, output dim]`. To calculate the loss, the tensor is reshaped to `[trg len - 1, batch size, output dim]` to remove the initial `<bos>` token, which is not used for calculating the loss.\n",
    "\n",
    "9. The target sequences (`trg`) are also reshaped to `[trg len - 1]` by removing the initial `<bos>` token and making it a contiguous tensor. This matches the shape of the reshaped `output` tensor.\n",
    "\n",
    "10. The loss between the reshaped `output` and `trg` tensors is calculated using the specified `criterion`.\n",
    "\n",
    "11. The gradients of the loss with respect to the model's parameters are computed using `loss.backward()`.\n",
    "\n",
    "12. The gradients are then clipped to a maximum value specified by `clip` using `torch.nn.utils.clip_grad_norm_(model.parameters(), clip)`. This prevents the gradients from becoming too large, which can cause issues during optimization.\n",
    "\n",
    "13. The optimizer's `step()` method is called to update the model's parameters using the computed gradients.\n",
    "\n",
    "14. The current batch loss (`loss.item()`) is added to the `epoch_loss` variable.\n",
    "\n",
    "15. After all the batches have been processed, the function returns the average loss per batch for the entire epoch, calculated as `epoch_loss / len(list(iterator))`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(model, iterator, optimizer, criterion, clip):\n",
    "\n",
    "    model.train()\n",
    "\n",
    "    epoch_loss = 0\n",
    "\n",
    "    # Wrap iterator with tqdm for progress logging\n",
    "    train_iterator = tqdm(iterator, desc=\"Training\", leave=False)\n",
    "\n",
    "    for i, (src,trg) in enumerate(iterator):\n",
    "\n",
    "        src = src.to(device)\n",
    "        trg = trg.to(device)\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        output = model(src, trg)\n",
    "\n",
    "        #trg = [trg len, batch size]\n",
    "        #output = [trg len, batch size, output dim]\n",
    "\n",
    "        output_dim = output.shape[-1]\n",
    "\n",
    "        output = output[1:].view(-1, output_dim)\n",
    "\n",
    "        trg = trg[1:].contiguous().view(-1)\n",
    "\n",
    "        #trg = [(trg len - 1) * batch size]\n",
    "        #output = [(trg len - 1) * batch size, output dim]\n",
    "\n",
    "        loss = criterion(output, trg)\n",
    "\n",
    "        loss.backward()\n",
    "\n",
    "        torch.nn.utils.clip_grad_norm_(model.parameters(), clip)\n",
    "\n",
    "        optimizer.step()\n",
    "\n",
    "        # Update tqdm progress bar with the current loss\n",
    "        train_iterator.set_postfix(loss=loss.item())\n",
    "\n",
    "        epoch_loss += loss.item()\n",
    "\n",
    "\n",
    "    return epoch_loss / len(list(iterator))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating model in PyTorch\n",
    "You also need to define a function to evaluate the model. Let's go through the code and understand its components:\n",
    "\n",
    "1. `evaluate(model, iterator, criterion)` takes three arguments:\n",
    "   - `model` is the neural network model that will be evaluated.\n",
    "   - `iterator` is an iterable object that provides the evaluation data in batches.\n",
    "   - `criterion` is the loss function that measures the model's performance.\n",
    "* Note that evaluate function do not perform any optimization on the model.\n",
    "\n",
    "2. The function starts by setting the model to evaluation mode with `model.eval()`.\n",
    "\n",
    "3. It initializes a variable `epoch_loss` to keep track of the accumulated loss during the evaluation.\n",
    "\n",
    "4. The function enters a `with torch.no_grad()` block, which ensures that no gradients are computed during the evaluation. This saves memory and speeds up the evaluation process since gradients are not needed for parameter updates.\n",
    "\n",
    "5. The function iterates over the evaluation data provided by the `iterator`. Each iteration retrieves a batch of input sequences (`src`) and target sequences (`trg`).\n",
    "\n",
    "6. The input sequences (`src`) and target sequences (`trg`) are moved to the appropriate device (e.g., GPU) using `src = src.to(device)` and `trg = trg.to(device)`.\n",
    "\n",
    "7. The model is then called with `output = model(src, trg, 0)` to obtain the model's predictions for the target sequences. The third argument `0` is passed to indicate that teacher forcing is turned off during evaluation.  During evaluation, teacher forcing is typically turned off to evaluate the model's ability to generate sequences based on its own predictions.\n",
    "\n",
    "8. The `output` tensor has dimensions `[trg len, batch size, output dim]`. To calculate the loss, the tensor is reshaped to `[trg len - 1, batch size, output dim]` to remove the initial `<bos>` (beginning of sequence) token, which is not used for calculating the loss.\n",
    "\n",
    "9. The target sequences (`trg`) are also reshaped to `[trg len - 1]` by removing the initial `<bos>` token and making it a contiguous tensor. This matches the shape of the reshaped `output` tensor.\n",
    "\n",
    "10. The loss between the reshaped `output` and `trg` tensors is calculated using the specified `criterion`.\n",
    "\n",
    "11. The current batch loss (`loss.item()`) is added to the `epoch_loss` variable.\n",
    "\n",
    "12. After all the batches have been processed, the function returns the average loss per batch for the entire evaluation, calculated as `epoch_loss / len(list(iterator))`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(model, iterator, criterion):\n",
    "\n",
    "    model.eval()\n",
    "\n",
    "    epoch_loss = 0\n",
    "\n",
    "    # Wrap iterator with tqdm for progress logging\n",
    "    valid_iterator = tqdm(iterator, desc=\"Training\", leave=False)\n",
    "\n",
    "    with torch.no_grad():\n",
    "\n",
    "        for i, (src,trg) in enumerate(iterator):\n",
    "\n",
    "            src = src.to(device)\n",
    "            trg = trg.to(device)\n",
    "\n",
    "            output = model(src, trg, 0) #turn off teacher forcing\n",
    "\n",
    "            #trg = [trg len, batch size]\n",
    "            #output = [trg len, batch size, output dim]\n",
    "\n",
    "            output_dim = output.shape[-1]\n",
    "\n",
    "            output = output[1:].view(-1, output_dim)\n",
    "\n",
    "            trg = trg[1:].contiguous().view(-1)\n",
    "\n",
    "\n",
    "            #trg = [(trg len - 1) * batch size]\n",
    "            #output = [(trg len - 1) * batch size, output dim]\n",
    "\n",
    "            loss = criterion(output, trg)\n",
    "            # Update tqdm progress bar with the current loss\n",
    "            valid_iterator.set_postfix(loss=loss.item())\n",
    "\n",
    "            epoch_loss += loss.item()\n",
    "\n",
    "    return epoch_loss / len(list(iterator))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preprocessing\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, you will fetch a language translation dataset called Multi30k, collate it (tokenization, numericalization, and adding BOS/EOS and padding) and create iterable batches of src and trg tensors.\n",
    "\n",
    "This leverages the predefined collate_fn to efficiently curate and ready batches for training the transformer model. The primary aim is to delve deeper into the intricacies of the RNN encoder and decoder components.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A \"Multi30K_de_en_dataloader.py\" file has been created that contains all the transformation processes on data. Here, you only download the file:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-12-02 18:46:52--  https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-AI0205EN-SkillsNetwork/Multi30K_de_en_dataloader.py\n",
      "Resolving cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)... 169.63.118.104, 169.63.118.104\n",
      "Connecting to cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)|169.63.118.104|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 4890 (4.8K) [application/x-python]\n",
      "Saving to: ‘Multi30K_de_en_dataloader.py’\n",
      "\n",
      "Multi30K_de_en_data 100%[===================>]   4.78K  --.-KB/s    in 0s      \n",
      "\n",
      "2024-12-02 18:46:52 (954 MB/s) - ‘Multi30K_de_en_dataloader.py’ saved [4890/4890]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget 'https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-AI0205EN-SkillsNetwork/Multi30K_de_en_dataloader.py'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run it:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run Multi30K_de_en_dataloader.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There you go! You only need to call the function `get_translation_dataloaders(batch_size = N,flip=True)` with an arbitrary batch size `N` and setting flip to True in order for the LSTM encoder receive input sequence in reversed order. This can help the training.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataloader, valid_dataloader = get_translation_dataloaders(batch_size = 4)#,flip=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check the `src` and `trg` tensors:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[    2,     2,     2,     2],\n",
       "         [    3,  5510,  5510, 12642],\n",
       "         [    1,     3,     3,     8],\n",
       "         [    1,     1,     1,  1701],\n",
       "         [    1,     1,     1,     3]]),\n",
       " tensor([[   2,    2,    2,    2],\n",
       "         [   3, 6650,  216,    6],\n",
       "         [   1, 4623,  110, 3398],\n",
       "         [   1,  259, 3913,  202],\n",
       "         [   1,  172, 1650,  109],\n",
       "         [   1, 9953, 3823,   37],\n",
       "         [   1,  115,   71,    3],\n",
       "         [   1,  692, 2808,    1],\n",
       "         [   1, 3428, 2187,    1],\n",
       "         [   1,    5,    5,    1],\n",
       "         [   1,    3,    3,    1]]))"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "src, trg = next(iter(train_dataloader))\n",
    "src,trg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also get the english and german strings using `index_to_eng` and `index_to_german` functions provided in the .py file:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "________________\n",
      "german\n",
      "<bos> Personen mit schwarzen Hüten in der Innenstadt . <eos>\n",
      "<bos> Eine Gruppe Menschen protestiert in einer Stadt . <eos>\n",
      "<bos> Eine Gruppe teilt ihre politischen Ansichten mit . <eos>\n",
      "<bos> Mehrere Personen sitzen an einem felsigen Strand . <eos>\n",
      "________________\n",
      "english\n",
      "<bos> People in black hats gathered together downtown . <eos> <pad> <pad> <pad>\n",
      "<bos> A group of people protesting in a city . <eos> <pad> <pad>\n",
      "<bos> A group is letting their political opinion be known . <eos> <pad>\n",
      "<bos> A group of people are sitting on a rocky beach . <eos>\n",
      "________________\n",
      "german\n",
      "<bos> Zwei sitzende Personen mit Hüten und Sonnenbrillen . <eos>\n",
      "<bos> Ein kleiner Junge mit Hut beim Angeln . <eos>\n",
      "<bos> Diese zwei Frauen haben Spaß im Giorgio's . <eos>\n",
      "<bos> Zwei kleine Kinder schlafen auf dem Sofa . <eos>\n",
      "________________\n",
      "english\n",
      "<bos> Two people sitting in hats and shades . <eos> <pad> <pad> <pad>\n",
      "<bos> A young boy in a hat is fishing by himself . <eos>\n",
      "<bos> These two women is at Giorgio 's having fun . <eos> <pad>\n",
      "<bos> Two young children are asleep on a couch . <eos> <pad> <pad>\n",
      "________________\n",
      "german\n",
      "<bos> Zwei junge Mädchen marschieren in einem Umzug . <eos>\n",
      "<bos> Eine Frau läuft vor einer gestreiften Wand . <eos>\n",
      "<bos> Ein Mann fährt Jet-Ski auf dem Ozean . <eos>\n",
      "<bos> Die städtischen Straßenbahnen an einem sonnigen Tag . <eos>\n",
      "________________\n",
      "english\n",
      "<bos> Two young girls walk in a parade . <eos> <pad> <pad> <pad> <pad>\n",
      "<bos> A woman is running in front of a striped wall . <eos> <pad>\n",
      "<bos> A man rides a jet ski across the ocean . <eos> <pad> <pad>\n",
      "<bos> The urban trolly 's of a city on a sunny day . <eos>\n"
     ]
    }
   ],
   "source": [
    "data_itr = iter(train_dataloader)\n",
    "# moving forward in the dataset to reach sequences of longer length for illustration purpose. (Remember the dataset is sorted on sequence len for optimal padding)\n",
    "for n in range(1000):\n",
    "    german, english= next(data_itr)\n",
    "\n",
    "for n in range(3):\n",
    "    german, english=next(data_itr)\n",
    "    german=german.T\n",
    "    english=english.T\n",
    "    print(\"________________\")\n",
    "    print(\"german\")\n",
    "    for g in german:\n",
    "        print(index_to_german(g))\n",
    "    print(\"________________\")\n",
    "    print(\"english\")\n",
    "    for e in english:\n",
    "        print(index_to_eng(e))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Note: When working with PyTorch tensors that represent data, it's important to understand the conventions around representing sequences. In most cases, the rows (the first dimension) in a PyTorch tensor represent individual samples, while the columns (the second dimension) represent features or time steps in the case of sequences. When dealing with sequences in PyTorch, it's common to use functions like `pad_sequence` to ensure that all sequences have the same length. Surprisingly, the padding operation is applied along the second dimension (columns), even though sequences are typically represented in the first dimension (rows). This can be confusing at first due to the way batches of sequences are represented. In many sequence-related tasks in PyTorch, especially when working with recurrent models like RNNs, LSTMs, and GRUs, batches of sequences are usually represented with the shape [sequence_length, batch_size, feature_size], where `sequence_length` refers to the length of the longest sequence within the batch(here it is equevalent to `src_len` or `trg_len`). If you check the src tensor above, you can see that the first word of of all sentences are in the first line, the second word of all sentences are in the second line, etc. That is why the first dimension is the length of the sequence.\n",
    "\n",
    "    When you use `pad_sequence`, it adds padding to the sequences in a batch so that they all have the same length, matching the length of the longest sequence. Since sequences are represented in the first dimension, the padding is applied along that dimension. As a result, the output tensor from `pad_sequence` will have the format [sequence_length, batch_size]. (Check the output for `src` and `trg` from the above cell.) This convention is commonly used because models like LSTMs expect the data to be in this format. However, if you're accustomed to working with more traditional tabular data in PyTorch, it can initially cause confusion. It's important to be aware of this convention to avoid potential errors and understand how to properly prepare and format sequence data for your models.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training the model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initializations\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code sets the random seed for various libraries and modules. This is done to make the results reproducible:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "SEED = 1234\n",
    "random.seed(SEED)\n",
    "np.random.seed(SEED)\n",
    "torch.manual_seed(SEED)\n",
    "torch.cuda.manual_seed(SEED)\n",
    "torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "Now, define an instance of the model:\n",
    "\n",
    "- `enc = Encoder(INPUT_DIM, ENC_EMB_DIM, HID_DIM, N_LAYERS, ENC_DROPOUT)`: This line creates an instance of the `Encoder` class, which represents the encoder component of the Seq2Seq model. The `Encoder` class takes the input dimension, embedding dimension, hidden dimension, number of layers, and dropout probability as arguments.\n",
    "\n",
    "- `dec = Decoder(OUTPUT_DIM, DEC_EMB_DIM, HID_DIM, N_LAYERS, DEC_DROPOUT)`: This line creates an instance of the `Decoder` class, which represents the decoder component of the Seq2Seq model. The `Decoder` class takes the output dimension, embedding dimension, hidden dimension, number of layers, and dropout probability as arguments.\n",
    "\n",
    "- `model = Seq2Seq(enc, dec, device,trg_vocab = vocab_transform['en']).to(device)`: This line creates an instance of the `Seq2Seq` class, which represents the entire Seq2Seq model. The `Seq2Seq` class takes the encoder, decoder, and device (e.g., CPU or GPU) as arguments. It combines the encoder and decoder to form the complete Seq2Seq architecture.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "INPUT_DIM = len(vocab_transform['de'])\n",
    "OUTPUT_DIM = len(vocab_transform['en'])\n",
    "ENC_EMB_DIM = 128 #256\n",
    "DEC_EMB_DIM = 128 #256\n",
    "HID_DIM = 256 #512\n",
    "N_LAYERS = 1 #2\n",
    "ENC_DROPOUT = 0.3 #0.5\n",
    "DEC_DROPOUT = 0.3 #0.5\n",
    "\n",
    "enc = Encoder(INPUT_DIM, ENC_EMB_DIM, HID_DIM, N_LAYERS, ENC_DROPOUT)\n",
    "dec = Decoder(OUTPUT_DIM, DEC_EMB_DIM, HID_DIM, N_LAYERS, DEC_DROPOUT)\n",
    "\n",
    "model = Seq2Seq(enc, dec, device,trg_vocab = vocab_transform['en']).to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`def init_weights(m)`defines a function named `init_weights` that takes a module `m` as input. The purpose of this function is to initialize the weights of the neural network module.\n",
    "\n",
    "The next line `for name, param in m.named_parameters():` starts a loop that iterates over the named parameters of the module `m`. Each parameter is accessed as `param` and its corresponding name is accessed as `name`.\n",
    "\n",
    "`nn.init.uniform_(param.data, -0.08, 0.08)`initializes the parameter's data with values drawn from a uniform distribution between `-0.08` and `0.08`. The `nn.init.uniform_` function is provided by the PyTorch library and is used to initialize the weights of neural network parameters.\n",
    "\n",
    "Finally, `model.apply(init_weights)` applies the `init_weights` function to the `model` instance. This ensures that the weights of all the parameters in the model are initialized using the specified uniform distribution.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Seq2Seq(\n",
       "  (encoder): Encoder(\n",
       "    (embedding): Embedding(19214, 128)\n",
       "    (lstm): LSTM(128, 256, dropout=0.3)\n",
       "    (dropout): Dropout(p=0.3, inplace=False)\n",
       "  )\n",
       "  (decoder): Decoder(\n",
       "    (embedding): Embedding(10837, 128)\n",
       "    (lstm): LSTM(128, 256, dropout=0.3)\n",
       "    (fc_out): Linear(in_features=256, out_features=10837, bias=True)\n",
       "    (softmax): LogSoftmax(dim=1)\n",
       "    (dropout): Dropout(p=0.3, inplace=False)\n",
       "  )\n",
       "  (trg_vocab): Vocab()\n",
       ")"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def init_weights(m):\n",
    "    for name, param in m.named_parameters():\n",
    "        nn.init.uniform_(param.data, -0.08, 0.08)\n",
    "\n",
    "model.apply(init_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code defines a function `count_parameters` that counts the number of trainable parameters in a given model. It then prints the count of trainable parameters in a formatted string.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has 7,422,165 trainable parameters\n"
     ]
    }
   ],
   "source": [
    "def count_parameters(model):\n",
    "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
    "\n",
    "print(f'The model has {count_parameters(model):,} trainable parameters')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell sets up the optimizer and loss function for training the model.\n",
    "\n",
    "1. `optimizer = optim.Adam(model.parameters())`: This line creates an instance of the Adam optimizer and passes the model's parameters (`model.parameters()`) as the parameters to be optimized. The Adam optimizer is a popular optimization algorithm commonly used for training deep neural networks. It adjusts the model's parameters based on the gradients computed during backpropagation to minimize the loss function.\n",
    "\n",
    "2. `PAD_IDX = vocab_transform['en'].get_stoi()['<pad>']`: This line retrieves the index of the `<pad>` token in the target vocabulary.\n",
    "\n",
    "3. `criterion = nn.CrossEntropyLoss(ignore_index=PAD_IDX)`: This line creates an instance of the CrossEntropyLoss criterion. The CrossEntropyLoss is a commonly used loss function for multi-class classification tasks. In this case, it is used for training the model to predict the next word in the translated sequence. The `ignore_index` parameter is set to `PAD_IDX`, which indicates that the loss should be ignored for any predictions where the target is the padding token. This is useful to exclude padding tokens from contributing to the loss during training.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = optim.Adam(model.parameters())\n",
    "\n",
    "PAD_IDX = vocab_transform['en'].get_stoi()['<pad>']\n",
    "\n",
    "criterion = nn.CrossEntropyLoss(ignore_index = PAD_IDX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following helper function provides a convenient way to calculate the elapsed time in minutes and seconds given the start and end times. It will be used to measure the time taken for each epoch during training or any other time-related calculations.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please be aware that training the model using CPUs can be a time-consuming process. If you don't have access to GPUs, you can jump to \"loading the saved model\" and proceed with loading the pre-trained model using the provided code. The model has been trained for five epochs and saved for your convenience.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start the training epochs:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training:   0%|          | 0/7250 [31:01<?, ?it/s, loss=4.76]"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[32], line 18\u001b[0m\n\u001b[1;32m     14\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m epoch \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mrange\u001b[39m(N_EPOCHS):\n\u001b[1;32m     16\u001b[0m     start_time \u001b[38;5;241m=\u001b[39m time\u001b[38;5;241m.\u001b[39mtime()\n\u001b[0;32m---> 18\u001b[0m     train_loss \u001b[38;5;241m=\u001b[39m \u001b[43mtrain\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmodel\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtrain_dataloader\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43moptimizer\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcriterion\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mCLIP\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     19\u001b[0m     train_ppl \u001b[38;5;241m=\u001b[39m math\u001b[38;5;241m.\u001b[39mexp(train_loss)\n\u001b[1;32m     20\u001b[0m     valid_loss \u001b[38;5;241m=\u001b[39m evaluate(model, valid_dataloader, criterion)\n",
      "Cell \u001b[0;32mIn[19], line 32\u001b[0m, in \u001b[0;36mtrain\u001b[0;34m(model, iterator, optimizer, criterion, clip)\u001b[0m\n\u001b[1;32m     27\u001b[0m \u001b[38;5;66;03m#trg = [(trg len - 1) * batch size]\u001b[39;00m\n\u001b[1;32m     28\u001b[0m \u001b[38;5;66;03m#output = [(trg len - 1) * batch size, output dim]\u001b[39;00m\n\u001b[1;32m     30\u001b[0m loss \u001b[38;5;241m=\u001b[39m criterion(output, trg)\n\u001b[0;32m---> 32\u001b[0m \u001b[43mloss\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbackward\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     34\u001b[0m torch\u001b[38;5;241m.\u001b[39mnn\u001b[38;5;241m.\u001b[39mutils\u001b[38;5;241m.\u001b[39mclip_grad_norm_(model\u001b[38;5;241m.\u001b[39mparameters(), clip)\n\u001b[1;32m     36\u001b[0m optimizer\u001b[38;5;241m.\u001b[39mstep()\n",
      "File \u001b[0;32m/opt/conda/lib/python3.11/site-packages/torch/_tensor.py:487\u001b[0m, in \u001b[0;36mTensor.backward\u001b[0;34m(self, gradient, retain_graph, create_graph, inputs)\u001b[0m\n\u001b[1;32m    477\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m has_torch_function_unary(\u001b[38;5;28mself\u001b[39m):\n\u001b[1;32m    478\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m handle_torch_function(\n\u001b[1;32m    479\u001b[0m         Tensor\u001b[38;5;241m.\u001b[39mbackward,\n\u001b[1;32m    480\u001b[0m         (\u001b[38;5;28mself\u001b[39m,),\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    485\u001b[0m         inputs\u001b[38;5;241m=\u001b[39minputs,\n\u001b[1;32m    486\u001b[0m     )\n\u001b[0;32m--> 487\u001b[0m \u001b[43mtorch\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mautograd\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mbackward\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m    488\u001b[0m \u001b[43m    \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mgradient\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mretain_graph\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcreate_graph\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43minputs\u001b[49m\n\u001b[1;32m    489\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/opt/conda/lib/python3.11/site-packages/torch/autograd/__init__.py:200\u001b[0m, in \u001b[0;36mbackward\u001b[0;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)\u001b[0m\n\u001b[1;32m    195\u001b[0m     retain_graph \u001b[38;5;241m=\u001b[39m create_graph\n\u001b[1;32m    197\u001b[0m \u001b[38;5;66;03m# The reason we repeat same the comment below is that\u001b[39;00m\n\u001b[1;32m    198\u001b[0m \u001b[38;5;66;03m# some Python versions print out the first line of a multi-line function\u001b[39;00m\n\u001b[1;32m    199\u001b[0m \u001b[38;5;66;03m# calls in the traceback and some print out the last line\u001b[39;00m\n\u001b[0;32m--> 200\u001b[0m \u001b[43mVariable\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_execution_engine\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun_backward\u001b[49m\u001b[43m(\u001b[49m\u001b[43m  \u001b[49m\u001b[38;5;66;43;03m# Calls into the C++ engine to run the backward pass\u001b[39;49;00m\n\u001b[1;32m    201\u001b[0m \u001b[43m    \u001b[49m\u001b[43mtensors\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mgrad_tensors_\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mretain_graph\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcreate_graph\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m    202\u001b[0m \u001b[43m    \u001b[49m\u001b[43mallow_unreachable\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43maccumulate_grad\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m)\u001b[49m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "torch.cuda.empty_cache()\n",
    "\n",
    "N_EPOCHS = 3 #run the training for at least 5 epochs\n",
    "CLIP = 1\n",
    "\n",
    "best_valid_loss = float('inf')\n",
    "best_train_loss = float('inf')\n",
    "train_losses = []\n",
    "valid_losses = []\n",
    "\n",
    "train_PPLs = []\n",
    "valid_PPLs = []\n",
    "\n",
    "for epoch in range(N_EPOCHS):\n",
    "\n",
    "    start_time = time.time()\n",
    "\n",
    "    train_loss = train(model, train_dataloader, optimizer, criterion, CLIP)\n",
    "    train_ppl = math.exp(train_loss)\n",
    "    valid_loss = evaluate(model, valid_dataloader, criterion)\n",
    "    valid_ppl = math.exp(valid_loss)\n",
    "\n",
    "\n",
    "    end_time = time.time()\n",
    "\n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "\n",
    "\n",
    "    if valid_loss < best_valid_loss:\n",
    "\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'RNN-TR-model.pt')\n",
    "\n",
    "    train_losses.append(train_loss)\n",
    "    train_PPLs.append(train_ppl)\n",
    "    valid_losses.append(valid_loss)\n",
    "    valid_PPLs.append(valid_ppl)\n",
    "\n",
    "    print(f'Epoch: {epoch+1:02} | Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train PPL: {train_ppl:7.3f}')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. PPL: {valid_ppl:7.3f}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize the model train and validation losses over the training epochs:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "\n",
    "# Create a list of epoch numbers\n",
    "epochs = [epoch+1 for epoch in range(N_EPOCHS)]\n",
    "\n",
    "# Create the figure and axes\n",
    "fig, ax1 = plt.subplots(figsize=(10, 6))\n",
    "ax2 = ax1.twinx()\n",
    "\n",
    "# Plotting the training and validation loss\n",
    "ax1.plot(epochs, train_losses, label='Train Loss', color='blue')\n",
    "ax1.plot(epochs, valid_losses, label='Validation Loss', color='orange')\n",
    "ax1.set_xlabel('Epochs')\n",
    "ax1.set_ylabel('Loss')\n",
    "ax1.set_title('Training and Validation Loss/PPL')\n",
    "\n",
    "# Plotting the training and validation perplexity\n",
    "ax2.plot(epochs, train_PPLs, label='Train PPL', color='green')\n",
    "ax2.plot(epochs, valid_PPLs, label='Validation PPL', color='red')\n",
    "ax2.set_ylabel('Perplexity')\n",
    "\n",
    "# Adjust the y-axis scaling for PPL plot\n",
    "ax2.set_ylim(bottom=min(min(train_PPLs), min(valid_PPLs)) - 10, top=max(max(train_PPLs), max(valid_PPLs)) + 10)\n",
    "\n",
    "# Set the legend\n",
    "lines1, labels1 = ax1.get_legend_handles_labels()\n",
    "lines2, labels2 = ax2.get_legend_handles_labels()\n",
    "lines = lines1 + lines2\n",
    "labels = labels1 + labels2\n",
    "ax1.legend(lines, labels, loc='upper right')\n",
    "\n",
    "\n",
    "# Show the plot\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be seen that the loss and perplexity are decreasing as model gets trained. The validation loss starts to stabilize and then grow at Epoch 9, which suggests you do not need to continue training the model to avoid overtraining.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the saved model\n",
    "If you want to skip training and load the pre-trained model that is provided, go ahead and uncomment the following cell:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-12-02 19:19:09--  https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-AI0201EN-Coursera/RNN-TR-model.pt\n",
      "Resolving cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)... 169.63.118.104, 169.63.118.104\n",
      "Connecting to cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)|169.63.118.104|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 29692282 (28M) [binary/octet-stream]\n",
      "Saving to: ‘RNN-TR-model.pt’\n",
      "\n",
      "RNN-TR-model.pt     100%[===================>]  28.32M  45.5MB/s    in 0.6s    \n",
      "\n",
      "2024-12-02 19:19:10 (45.5 MB/s) - ‘RNN-TR-model.pt’ saved [29692282/29692282]\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "!wget 'https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMSkillsNetwork-AI0201EN-Coursera/RNN-TR-model.pt'\n",
    "model.load_state_dict(torch.load('RNN-TR-model.pt',map_location=torch.device('cpu')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model inference\n",
    "\n",
    "\n",
    "Next, create a generator function that generates translations for input source sentences:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn.functional as F\n",
    "\n",
    "def generate_translation(model, src_sentence, src_vocab, trg_vocab, max_len=50):\n",
    "    model.eval()  # Set the model to evaluation mode\n",
    "\n",
    "    with torch.no_grad():\n",
    "        src_tensor = text_transform[SRC_LANGUAGE](src_sentence).view(-1, 1).to(device)\n",
    "\n",
    "        # Pass the source tensor through the encoder\n",
    "        hidden, cell = model.encoder(src_tensor)\n",
    "\n",
    "        # Create a tensor to store the generated translation\n",
    "        # get_stoi() maps tokens to indices\n",
    "        trg_indexes = [trg_vocab.get_stoi()['<bos>']]  # Start with <bos> token\n",
    "\n",
    "        # Convert the initial token to a PyTorch tensor\n",
    "        trg_tensor = torch.LongTensor(trg_indexes).unsqueeze(1)  # Add batch dimension\n",
    "\n",
    "        # Move the tensor to the same device as the model\n",
    "        trg_tensor = trg_tensor.to(model.device)\n",
    "\n",
    "\n",
    "        # Generate the translation\n",
    "        for _ in range(max_len):\n",
    "\n",
    "            # Pass the target tensor and the previous hidden and cell states through the decoder\n",
    "            output, hidden, cell = model.decoder(trg_tensor[-1], hidden, cell)\n",
    "\n",
    "            # Get the predicted next token\n",
    "            pred_token = output.argmax(1)[-1].item()\n",
    "\n",
    "            # Append the predicted token to the translation\n",
    "            trg_indexes.append(pred_token)\n",
    "\n",
    "\n",
    "            # If the predicted token is the <eos> token, stop generating\n",
    "            if pred_token == trg_vocab.get_stoi()['<eos>']:\n",
    "                break\n",
    "\n",
    "            # Convert the predicted token to a PyTorch tensor\n",
    "            trg_tensor = torch.LongTensor(trg_indexes).unsqueeze(1)  # Add batch dimension\n",
    "\n",
    "            # Move the tensor to the same device as the model\n",
    "            trg_tensor = trg_tensor.to(model.device)\n",
    "\n",
    "        # Convert the generated tokens to text\n",
    "        # get_itos() maps indices to tokens\n",
    "        trg_tokens = [trg_vocab.get_itos()[i] for i in trg_indexes]\n",
    "\n",
    "        # Remove the <sos> and <eos> from the translation\n",
    "        if trg_tokens[0] == '<bos>':\n",
    "            trg_tokens = trg_tokens[1:]\n",
    "        if trg_tokens[-1] == '<eos>':\n",
    "            trg_tokens = trg_tokens[:-1]\n",
    "\n",
    "        # Return the translation list as a string\n",
    "\n",
    "        translation = \" \".join(trg_tokens)\n",
    "\n",
    "        return translation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you can check the model's output for a sample sentence:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "An Asian man is on the sidewalk .\n"
     ]
    }
   ],
   "source": [
    "# model.load_state_dict(torch.load('RNN-TR-model.pt'))\n",
    "\n",
    "# Actual translation: Asian man sweeping the walkway.\n",
    "src_sentence = 'Ein asiatischer Mann kehrt den Gehweg.'\n",
    "\n",
    "\n",
    "generated_translation = generate_translation(model, src_sentence=src_sentence, src_vocab=vocab_transform['de'], trg_vocab=vocab_transform['en'], max_len=12)\n",
    "#generated_translation = \" \".join(generated_translation_list).replace(\"<bos>\", \"\").replace(\"<eos>\", \"\")\n",
    "print(generated_translation)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fantastic! You have created a translation model that can generate german-english translations pretty accurate, huh?\n",
    "\n",
    "You can play with the model parameters and hyperparameters to improve the model performance.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BLEU score metric for evaluation\n",
    "While peplexity serves as a general metric to evaluate the performance of language model in predicting the correct next token, BLEU score is helpful in evaluating the quality of the final generated translation.\n",
    "Validating the results using BLEU score is helpful when there is more than a single valid translation for a sentence as you can include many translation versions in the reference list and compare the generated translation with different versions of translations.\n",
    "\n",
    "The BLEU (Bilingual Evaluation Understudy) score is a metric commonly used to evaluate the quality of machine-generated translations by comparing them to one or more reference translations. It measures the similarity between the generated translation and the reference translations based on n-gram matching.\n",
    "\n",
    "The BLEU score is calculated using the following formulas:\n",
    "\n",
    "1. **Precision**:\n",
    "   - Precision measures the proportion of n-grams in the generated translation that appear in the reference translations.\n",
    "   - Precision is calculated for each n-gram order (1 to N) and then combined using a geometric mean.\n",
    "   - The precision for a particular n-gram order is calculated as:\n",
    "   \n",
    "   $$\\text{Precision}_n(t) = \\frac{\\text{CountClip}_n(t)}{\\text{Count}_n(t)}$$\n",
    "   \n",
    "   where:\n",
    "     - $\\text{CountClip}_n(t)$ is the count of n-grams in the generated translation that appear in any reference translation, clipped by the maximum count of that n-gram in any single reference translation.\n",
    "     - $\\text{Count}_n(t)$ is the count of n-grams in the generated translation.\n",
    "\n",
    "2. . **Brevity penalty**:\n",
    "   - The brevity penalty accounts for the fact that shorter translations tend to have higher precision scores.\n",
    "   - It encourages translations that are closer in length to the reference translations.\n",
    "   - The brevity penalty is calculated as:\n",
    "   \n",
    "  $$\\text{BP} = \\begin{cases} 1 & \\text{if } c > r \\\\\\\\\\\\\\\\\\\\ e^{(1 - \\frac{r}{c})} & \\text{if } c \\leq r \\end{cases}$$\n",
    "   \n",
    "   where:\n",
    "     - $c$ is the total length of the generated translation.\n",
    "     - $r$ is the total length of the reference translations.\n",
    "\n",
    "3. **BLEU score**:\n",
    "   - The BLEU score is the geometric mean of the precisions, weighted by the brevity penalty.\n",
    "   - It is calculated as:\n",
    "   \n",
    "   $$\\text{BLEU} = \\text{BP} \\cdot \\exp(\\sum_{n=1}^{N}w_n \\log(\\text{Precision}_n(t)))$$\n",
    "   \n",
    "   where:\n",
    "     - $N$ is the maximum n-gram order.\n",
    "     - $w_n$ is the weight assigned to the precision at n-gram order $n$, commonly set as $\\frac{1}{N}$ for equal weights.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_bleu_score(generated_translation, reference_translations):\n",
    "    # Convert the generated translations and reference translations into the expected format for sentence_bleu\n",
    "    references = [reference.split() for reference in reference_translations]\n",
    "    hypothesis = generated_translation.split()\n",
    "\n",
    "    # Calculate the BLEU score\n",
    "    bleu_score = sentence_bleu(references, hypothesis)\n",
    "\n",
    "    return bleu_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's calculate the BLEU score for a sample sentence:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BLEU Score: 0.5\n"
     ]
    }
   ],
   "source": [
    "reference_translations = [\n",
    "    \"Asian man sweeping the walkway .\",\n",
    "    \"An asian man sweeping the walkway .\",\n",
    "    \"An Asian man sweeps the sidewalk .\",\n",
    "    \"An Asian man is sweeping the sidewalk .\",\n",
    "    \"An asian man is sweeping the walkway .\",\n",
    "    \"Asian man sweeping the sidewalk .\"\n",
    "]\n",
    "\n",
    "bleu_score = calculate_bleu_score(generated_translation, reference_translations)\n",
    "print(\"BLEU Score:\", bleu_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1 - Translate a German sentence to English.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original German text: Menschen gehen auf der Straße\n",
      "Translated English text: People are walking on the street .\n"
     ]
    }
   ],
   "source": [
    "# Define the German text to be translated\n",
    "german_text = \"Menschen gehen auf der Straße\"\n",
    "\n",
    "# The function should be defined to accept the text, the model, source and target vocabularies, and the device as parameters.\n",
    "english_translation = generate_translation(\n",
    "    model, \n",
    "    src_sentence=german_text, \n",
    "    src_vocab=vocab_transform['de'], \n",
    "    trg_vocab=vocab_transform['en'], \n",
    "    max_len=50\n",
    ")\n",
    "\n",
    "# Display the original and translated text\n",
    "print(f\"Original German text: {german_text}\")\n",
    "print(f\"Translated English text: {english_translation}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "german_text = \"Menschen gehen auf der Straße\"\n",
    "\n",
    "# The function should be defined to accept the text, the model, source and target vocabularies, and the device as parameters.\n",
    "english_translation = generate_translation(\n",
    "    model, \n",
    "    src_sentence=german_text, \n",
    "    src_vocab=vocab_transform['de'], \n",
    "    trg_vocab=vocab_transform['en'], \n",
    "    max_len=50\n",
    ")\n",
    "\n",
    "# Display the original and translated text\n",
    "print(f\"Original German text: {german_text}\")\n",
    "print(f\"Translated English text: {english_translation}\")\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Fateme Akbari](https://www.linkedin.com/in/fatemeakbari/) is a PhD candidate in Information Systems at McMaster University with demonstrated research experience in Machine Learning and NLP.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{## Change log}\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{|Date (YYYY-MM-DD)|Version|Changed By|Change Description||-|-|-|-||2020-07-17|0.1|Sam|Create Lab Template|}\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "© Copyright IBM Corporation. All rights reserved.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  },
  "prev_pub_hash": "50b168a2230d253ace1c41749630f15ead999c4adeff60aa79c91ff90b902a92"
 },
 "nbformat": 4,
 "nbformat_minor": 4
}