Transformers_Part-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b7380bc2",
   "metadata": {},
   "source": [
    "# Transformers Part 1:\n",
    "\n",
    "In this notebook, we explore the task of text generation using transformers from `huggingface`.\n",
    "\n",
    "__Agenda__:\n",
    "\n",
    "1. load a pretrained model from `huggingface`.\n",
    "2. get a simple prompt.\n",
    "3. try out different decoding strategies. \n",
    "\n",
    "\n",
    "__Note about decoding__:<br>\n",
    "The process of selecting output tokens to generate text is known as __decoding__. Modifying a decoding strategy does not change the values of any trainable parameters. However, it can have a noticeable impact on the quality of the generated output. It can help reduce repetition in the text and make it more coherent.\n",
    "\n",
    "Let's try to load a model and generate some text. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "784df73b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoModelForCausalLM, AutoTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "05a286bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2a13b3afe60d4e3c98f202b1ba994459",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)lve/main/config.json:   0%|          | 0.00/665 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fd9d3a0d5ad3477fa31e172487cb2d20",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)olve/main/vocab.json:   0%|          | 0.00/1.04M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "70cac99ebad04353a61449f968a07bfe",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)olve/main/merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5fd0d8e1574d423d92fdc919ba7d7cb9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)/main/tokenizer.json:   0%|          | 0.00/1.36M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ca739dbed5df41048957279885c94020",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading pytorch_model.bin:   0%|          | 0.00/548M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1029b876c2b64bae9703fbb3ed119fb2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading (…)neration_config.json:   0%|          | 0.00/124 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# which model\n",
    "checkpoint=\"gpt2\"\n",
    "\n",
    "# load the tokenizer\n",
    "tokenizer=AutoTokenizer.from_pretrained(checkpoint)\n",
    "\n",
    "# load the model\n",
    "model=AutoModelForCausalLM.from_pretrained(checkpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "782e0626",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GenerationConfig {\n",
       "  \"_from_model_config\": true,\n",
       "  \"bos_token_id\": 50256,\n",
       "  \"eos_token_id\": 50256,\n",
       "  \"transformers_version\": \"4.27.4\"\n",
       "}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.generation_config"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa3bf59a",
   "metadata": {},
   "source": [
    "The default generation tactic is greedy search and the default config of the model limits the size of the output to a max of 20 tokens. \n",
    "\n",
    "Now, greedy search is the simplest decoding strategy and it is useful for generating short spans of texts. But, when used to generate longer texts, greedy search can start producing highly repetitive results. \n",
    "\n",
    "Let's start with an example of __greedy search__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "44ce1e95",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt=\"This year I am going to play soccer and try out surfing.\"\n",
    "\n",
    "# tokenize input\n",
    "inputs=tokenizer(prompt, return_tensors=\"pt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "db238553",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[ 1212,   614,   314,   716,  1016,   284,   711, 11783,   290,  1949,\n",
       "           503, 36254,    13]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "847669b7",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[\"This year I am going to play soccer and try out surfing. I'm going to try to be a good swimmer and try to be a good swimmer. I'm going to try to be a good swimmer and try to be a good swimmer. I'm going to try to be a good swimmer and try to be a good swimmer. I'm going to try to be a good swimmer and try to be a good swimmer. I'm going to try to be a good swimmer and try to be a good swimmer.\"]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outputs=model.generate(**inputs, max_new_tokens=100)\n",
    "tokenizer.batch_decode(outputs, skip_special_tokens=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8d24fb7",
   "metadata": {},
   "source": [
    "See, for a longer text, __greedy search__ gives a repetitive result. Okay, let's see a bunch of other decoding strategies.\n",
    "\n",
    "__Contrastive Search__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "27c582f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[\"This year I am going to play soccer and try out surfing. I'm going to try to be a good swimmer and try to be a good swimmer. I'm going to try to be a good swimmer and try to be a good swimmer. I'm going to try to be a good swimmer and try to be a good swimmer. I'm going to try to be a good swimmer and try to be a good swimmer. I'm going to try to be a good swimmer and try to be a good swimmer.\"]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outputs=model.generate(**inputs, penalty_alpha=0.6, top_k=4, max_new_tokens=100)\n",
    "tokenizer.batch_decode(outputs, skip_special_tokens=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba1b0637",
   "metadata": {},
   "source": [
    "The generated text is grammatically correct, but makes little sense."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b2a24de",
   "metadata": {},
   "source": [
    "__Mutinomial Sampling__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "0187cf4f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['This year I am going to play soccer and try out surfing. So for this year, we are going to be doing swim classes and it will be a tough one. Hopefully I\\'m ready, or maybe I\\'m not…maybe I\\'m not. Right now I\\'m looking forward to getting stronger and trying out for more.\"\\n\\nBut as he got ready to enter the field on Saturday, his teammates didn\\'t look for comfort in the field, especially with him on his back.\\n\\nThe team manager, Jeff Larentowicz, told reporters after practice in']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outputs=model.generate(**inputs, do_sample=True,max_new_tokens=100)\n",
    "tokenizer.batch_decode(outputs, skip_special_tokens=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5864f587",
   "metadata": {},
   "source": [
    "It seems that the model is hallucinating now. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfa4a7f2",
   "metadata": {},
   "source": [
    "__Beam Search__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "1047fe46",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[\"This year I am going to play soccer and try out surfing. I'm going to try out surfing. I'm going to try out surfing. I'm going to try out surfing. I'm going to try out surfing. I'm going to try out surfing. I'm going to try out surfing. I'm going to try out surfing. I'm going to try out surfing. I'm going to try out surfing. I'm going to try out surfing. I'm going to try out surfing. I'm going to try out surfing. I'm going to\"]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outputs=model.generate(**inputs,num_beams=3,  max_new_tokens=100)\n",
    "tokenizer.batch_decode(outputs, skip_special_tokens=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8edc7fbb",
   "metadata": {},
   "source": [
    "Ok, repetitive."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b6d7902",
   "metadata": {},
   "source": [
    "__Beam Search Mutinomial Sampling__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "92dd6dcf",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['This year I am going to play soccer and try out surfing.\\n\\n\"I don\\'t know if I can go to the Olympics or not. I don\\'t know if I\\'m going to be able to play in the Olympics. I don\\'t know if I\\'m going to be able to compete in the World Cup or not. I don\\'t know if I\\'m going to be able to compete in the World Cup. I don\\'t know if I\\'m going to be able to compete in the World Cup. I don\\'t know if I\\'m going to be']"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outputs=model.generate(**inputs,num_beams=3, do_sample=True, max_new_tokens=100)\n",
    "tokenizer.batch_decode(outputs, skip_special_tokens=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a094c4b",
   "metadata": {},
   "source": [
    "It's get repetitive after a point, but makes sense in the starting. \n",
    "\n",
    "Finally, let's see __Diverse beam search decoding__."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "6311fdbd",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[\"This year I am going to play soccer and try out surfing. I'm going to try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and try out surfing and\"]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "outputs=model.generate(**inputs,num_beams=6,num_beam_groups=3, max_new_tokens=100)\n",
    "tokenizer.batch_decode(outputs, skip_special_tokens=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee27b359",
   "metadata": {},
   "source": [
    "Ok!!! In this notebook, we tried different decoding strategies for transformer models and saw the varying results produced by each strategy. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nlp",
   "language": "python",
   "name": "nlp"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}