Y002_tokenizer_文本生成.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e3ce13cd-d3bf-48cd-ad8f-92212d3c4ce0",
   "metadata": {},
   "source": [
    "# 文本生成"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3c737a8-0b70-4bda-b08f-8c3b7c63192c",
   "metadata": {},
   "source": [
    "## 初始環境設定"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4b113b8a-c6ff-43f8-a097-30d7d48629fd",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/work/g00cjz00/github/20240115_RAG'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "HOME = str(Path.home())\n",
    "Add_Binarry_Path=HOME+'/.local/bin'\n",
    "os.environ['PATH']=os.environ['PATH']+':'+Add_Binarry_Path\n",
    "current_foldr=!pwd\n",
    "current_foldr=current_foldr[0]\n",
    "current_foldr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a67c5f9-c606-4408-a786-7cb2055ba180",
   "metadata": {},
   "source": [
    "## 確認CUDA版本, 以及否能使用GPU\n",
    "若無gpu 請點選右側->已連線->變更執行階段類型->T4 Gpu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5518bf7-cef3-4743-b295-c98d5f21408e",
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi\n",
    "import torch\n",
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f532364c-4c38-4a17-aaec-38de13cfac6d",
   "metadata": {},
   "source": [
    "## 安裝套件\n",
    "安裝完成後建議, 點選上方選單, 直接階段->重新啟動工作階段, 確保 library重置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92e0c628-3302-4eee-bd11-5588a95e6006",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install cohere gdown kaleido langchain openai pyngrok pypdf python-dotenv sentence-transformers tiktoken -q\n",
    "!pip install accelerate bitsandbytes hf_transfer huggingface_hub optimum transformers -q "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3106d537-7083-47f3-be83-7bbdb471c0e1",
   "metadata": {},
   "source": [
    "## HF_TOKEN"
   ]
  },
  {
   "cell_type": "raw",
   "id": "99fe8748-6570-4002-889c-cd99fb2ea0de",
   "metadata": {
    "tags": []
   },
   "source": [
    "# HF_TOKEN method 2\n",
    "import os\n",
    "os.environ[\"HF_TOKEN\"] = \"\""
   ]
  },
  {
   "cell_type": "raw",
   "id": "196965fc-a6bb-4e4c-8bfd-c234848bace6",
   "metadata": {
    "tags": []
   },
   "source": [
    "%%bash\n",
    "# Download model\n",
    "mkdir -p chinese-llama-2-7b\n",
    "HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download hfl/chinese-llama-2-7b --local-dir chinese-llama-2-7b  --local-dir-use-symlinks False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b254d63b-67b1-46e9-9a88-8bf1cf3471cc",
   "metadata": {},
   "source": [
    "### LOAD LIBRARY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f68fe3d-8176-4015-9df7-b122a22b8c8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# LOAD LIBRARY\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline, BitsAndBytesConfig\n",
    "from transformers.generation.utils import GenerationConfig\n",
    "from langchain.llms.huggingface_pipeline import HuggingFacePipeline\n",
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e40f5aaa-5608-4b9f-acc7-b81d804cc343",
   "metadata": {},
   "source": [
    "## 輸入\n",
    "- inputs：輸入prompt。如果為空，則以batch size為1的bos_token_id初始化。對於只有decoder的模型（GPT系列），輸入需要是input_ids；對於encoder-decoder模型（BART、T5等），輸入更多樣化\n",
    "- max_length：產生序列的最大長度。\n",
    "- min_length：生成序列的最短長度，預設是10。\n",
    "- do_sample：是否開啟採樣，預設是False，即貪婪找最大條件機率的字。\n",
    "- early_stopping：是否在至少產生num_beams個句子後停止beam search，預設為False。\n",
    "- num_beams：預設是1，也就是不進行beam search。\n",
    "- temperature: 預設是1.0，溫度越低（小於1），softmax輸出的貧富差距越大；溫度越高，softmax差距越小。\n",
    "- top_k：top-k-filtering 演算法保留多少個最高機率的字作為候選，預設為50。詳見下文。\n",
    "- top_p：已知產生各字的總機率是1（即預設是1.0）如果top_p小於1，則由高到低累加直到top_p，取這前N個字為候選。\n",
    "- repetition_penalty：預設是1.0，重複詞懲罰。\n",
    "- pad_token_id (int, 可選) — 填充令牌的 id。\n",
    "- bos_token_id (int, 可選) — 序列開始標記的 id。\n",
    "- eos_token_id (int, 可選) — 序列結束標記的 id。\n",
    "- pad_token_id / bos_token_id / eos_token_id：填充詞<PAD>、起始附<s>、結束符</s> 的id。\n",
    "- length_penalty：長度懲罰，預設是1.0。\n",
    "- length_penalty=1.0：beam search分數會受到產生序列長度的懲罰\n",
    "- length_penalty=0.0：無懲罰\n",
    "- length_penalty<0.0：鼓勵模型生成長句子\n",
    "- length_penalty>0.0：鼓勵模型產生短句子\n",
    "- no_repeat_ngram_size：用來控制重複字生成，預設是0，如果大於0，則對應N-gram只出現一次\n",
    "- encoder_no_repeat_ngram_size：也是用來控制重複字生成，預設是0，如果大於0，則encoder_input_ids的N-gram不會出現在decoder_input_ids裡。\n",
    "- bad_words_ids：禁止產生的詞id列表，可用tokenizer(bad_words, add_prefix_space=True, add_special_tokens=False).input_ids方法取得ids。\n",
    "- force_words_ids：跟上面的bad_words_ids 相反，這個傳入必須產生的token id 清單。如果ids格式是[List[List[int]]]，例如[[1,2],[3,4]]，則觸發析取約束（Disjunctive Positive Constraint Decoding），大概意思是可以產生一個單字不同的形式，如「lonely」、「loneliness」等。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6602c22-3545-4ac1-acb7-faff9392a3c6",
   "metadata": {},
   "source": [
    "### 四、各解碼演算法原理簡述\n",
    "本小節主要介紹自迴歸文字產生的幾個最常用的解碼方法，包括\n",
    "- Greedy search\n",
    "- Beam search \n",
    "- Top-K sampling\n",
    "- Top-p sampling\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b24a24d2-cf1a-4c8f-9406-7bd8eddfb4dc",
   "metadata": {},
   "source": [
    "# MODEL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ba9365d9-0222-4e1c-be4b-94f1e6470621",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "73cfecaacff9497f8a9e8310fd661430",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import torch\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
    "\n",
    "MODEL_ID = \"/work/u00cjz00/slurm_jobs/github/models/Breeze-7B-Instruct-64k-v0.1\"\n",
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)\n",
    "#model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map=\"auto\")\n",
    "#model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map=\"auto\", torch_dtype=torch.float16).to(0)\n",
    "model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map=\"auto\", load_in_4bit=True)\n",
    "#model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map=\"auto\", torch_dtype=torch.float16, use_flash_attention_2=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba14ded1-89c7-4dac-b85b-3508d8c53f4b",
   "metadata": {},
   "source": [
    "##　1. 接龍"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "fcc70a3c-6260-414a-87df-af60c3746d1e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paris is the city of love\n",
      "Paris is the city of love\n",
      "Paris is the city of love\n",
      "Paris is the city of love\n",
      "Paris is the city of love\n",
      "Paris is the city of love\n",
      "Paris is the city of love\n",
      "Paris is the city of love\n",
      "Paris is the city of love\n",
      "Paris is the city of love\n"
     ]
    }
   ],
   "source": [
    "# Greedy search, top_k=1,do_sample=False\n",
    "text = \"Paris is the city\"\n",
    "inputs = tokenizer(text, return_tensors=\"pt\").to(0)\n",
    "for number in range(10):\n",
    "    outputs = model.generate(**inputs, max_new_tokens=2,top_k=1,do_sample=False,pad_token_id=tokenizer.eos_token_id)\n",
    "    print(tokenizer.decode(outputs[0], skip_special_tokens=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "2c658296-af70-4d01-8867-2109762ab196",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paris is the city of love\n",
      "Paris is the city of love\n",
      "Paris is the city of love\n",
      "Paris is the city of lights\n",
      "Paris is the city of love\n",
      "Paris is the city of lights\n",
      "Paris is the city of love\n",
      "Paris is the city of love\n",
      "Paris is the city of love\n",
      "Paris is the city of love\n"
     ]
    }
   ],
   "source": [
    "# Greedy search, top_k=2,do_sample=True\n",
    "text = \"Paris is the city\"\n",
    "inputs = tokenizer(text, return_tensors=\"pt\").to(0)\n",
    "for number in range(10):\n",
    "    outputs = model.generate(**inputs, max_new_tokens=2,top_k=2,do_sample=True,pad_token_id=tokenizer.eos_token_id)\n",
    "    print(tokenizer.decode(outputs[0], skip_special_tokens=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "168f20e5-0192-4a09-8e8c-041918ae0caf",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paris is the city of light\n",
      "Paris is the city of love\n",
      "Paris is the city of love\n",
      "Paris is the city of lights\n",
      "Paris is the city of light\n",
      "Paris is the city of love\n",
      "Paris is the city of lights\n",
      "Paris is the city of love\n",
      "Paris is the city of love\n",
      "Paris is the city of lights\n"
     ]
    }
   ],
   "source": [
    "# Greedy search, top_p,do_sample=True\n",
    "text = \"Paris is the city\"\n",
    "inputs = tokenizer(text, return_tensors=\"pt\").to(0)\n",
    "for number in range(10):\n",
    "    outputs = model.generate(**inputs, max_new_tokens=2,top_p=0.90,do_sample=True,pad_token_id=tokenizer.eos_token_id)\n",
    "    print(tokenizer.decode(outputs[0], skip_special_tokens=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "f6cfa84f-bf1f-4d92-a04a-cd86757fcc4e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paris is the city of love\n",
      "Paris is the city of love\n",
      "Paris is the city of lights\n",
      "Paris is the city of lights\n",
      "Paris is the city of romance\n",
      "Paris is the city of love\n",
      "Paris is the city of love\n",
      "Paris is the city of romance\n",
      "Paris is the city of light\n",
      "Paris is the city of lights\n"
     ]
    }
   ],
   "source": [
    "# Greedy search, top_p, Temperature\n",
    "text = \"Paris is the city\"\n",
    "inputs = tokenizer(text, return_tensors=\"pt\").to(0)\n",
    "for number in range(10):\n",
    "    outputs = model.generate(**inputs, max_new_tokens=2,top_p=0.90,do_sample=True,temperature=1.0,pad_token_id=tokenizer.eos_token_id)\n",
    "    print(tokenizer.decode(outputs[0], skip_special_tokens=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "affd7369-006d-475d-a51e-88d6493283e9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paris is the city of love, and it\n",
      "Paris is the city of love, and it’\n",
      "Paris is the city of love, and it’s\n",
      "Paris is the city of love, but it’s also\n",
      "Paris is the city of love, but it’s also a\n",
      "Paris is the city of love, but it’s also the city\n",
      "Paris is the city of love, but it’s also the city of\n",
      "Paris is the city of love, but it’s also the city of art\n",
      "Paris is the city of love, but it’s also the city of art,\n",
      "Paris is the city of love, but it’s also the city of art, culture\n"
     ]
    }
   ],
   "source": [
    "# Beams search, top_p, Temperature\n",
    "text = \"Paris is the city\"\n",
    "inputs = tokenizer(text, return_tensors=\"pt\").to(0)\n",
    "\n",
    "for number in range(10):\n",
    "    n=number+5\n",
    "    outputs = model.generate(**inputs, max_new_tokens=n,num_beams=2,do_sample=False,pad_token_id=tokenizer.eos_token_id)\n",
    "    print(tokenizer.decode(outputs[0], skip_special_tokens=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "264e5e62-9429-4f6c-bc2d-c960fed8fff4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/g00cjz00/.local/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:394: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Paris is the city of love, and it\n",
      "Paris is the city of love, and it’\n",
      "Paris is the city of love, and it’s\n",
      "Paris is the city of love, but it’s also\n",
      "Paris is the city of love, but it’s also a\n",
      "Paris is the city of love, but it’s also the city\n",
      "Paris is the city of love, but it’s also the city of\n",
      "Paris is the city of love, but it’s also the city of art\n",
      "Paris is the city of love, but it’s also the city of art,\n",
      "Paris is the city of love, but it’s also the city of art, culture\n"
     ]
    }
   ],
   "source": [
    "# Beams search\n",
    "text = \"Paris is the city\"\n",
    "inputs = tokenizer(text, return_tensors=\"pt\").to(0)\n",
    "\n",
    "for number in range(10):\n",
    "    n=number+5\n",
    "    outputs = model.generate(**inputs, max_new_tokens=n,num_beams=2,do_sample=False,pad_token_id=tokenizer.eos_token_id)\n",
    "    print(tokenizer.decode(outputs[0], skip_special_tokens=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34946669-521b-425e-908f-6b6fe2733b09",
   "metadata": {},
   "source": [
    "## 2. TOENIZER 樣板"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "470c9f5e-f7e7-4736-886d-5bd1b9e553e4",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<s>You are a helpful AI assistant built by MediaTek Research. The user you are helping speaks Traditional Chinese and comes from Taiwan.   [INST] Hello, how are you? [/INST] A01! [INST] Q02? [/INST] A02! [INST] Q03? [/INST] A03! \n"
     ]
    }
   ],
   "source": [
    "chat = [\n",
    "  {\"role\": \"user\", \"content\": \"Hello, how are you?\"},\n",
    "  {\"role\": \"assistant\", \"content\": \"A01!\"},\n",
    "  {\"role\": \"user\", \"content\": \"Q02?\"},\n",
    "  {\"role\": \"assistant\", \"content\": \"A02!\"},\n",
    "  {\"role\": \"user\", \"content\": \"Q03?\"},\n",
    "  {\"role\": \"assistant\", \"content\": \"A03!\"},\n",
    "    \n",
    "]\n",
    "\n",
    "print(tokenizer.apply_chat_template(chat, tokenize=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab2e4579-ae0a-4234-b7e1-8fb63ecac64f",
   "metadata": {},
   "source": [
    "## 3. Tokenizer 擴充詞語表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "0dedd7bd-57e2-45fd-982f-81bb1e9142e5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test text:\n",
      " The primary use of LLaMA is research on large language models, including\n",
      "['▁The', '▁primary', '▁use', '▁of', '▁L', 'La', 'MA', '▁is', '▁research', '▁on', '▁large', '▁language', '▁models', ',', '▁including'] -> Media\n",
      "['▁', '蔡', '英文', '，', '中華', '民國', '政治', '人物', '、', '法', '學家', '與', '律師', '，', '民主', '進步', '黨', '籍', '，', '現任', '中華', '民國', '總統', '。'] -> Media\n"
     ]
    }
   ],
   "source": [
    "text = '''The primary use of LLaMA is research on large language models, including'''\n",
    "print(\"Test text:\\n\", text)\n",
    "print(f\"{tokenizer.tokenize(text)} -> Media\")\n",
    "\n",
    "text = '''蔡英文，中華民國政治人物、法學家與律師，民主進步黨籍，現任中華民國總統。'''\n",
    "print(f\"{tokenizer.tokenize(text)} -> Media\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8658a45a-13ec-4c18-a3d4-4cf560a11ff4",
   "metadata": {},
   "source": [
    "## BK"
   ]
  },
  {
   "cell_type": "raw",
   "id": "cab62c5e-cd59-4f89-bd9c-79d9b8b41194",
   "metadata": {},
   "source": [
    "import numpy as np\n",
    "\n",
    "text = \"Paris is the city\"\n",
    "inputs = tokenizer(text, return_tensors=\"pt\")\n",
    "\n",
    "outputs = model.generate(**inputs, max_new_tokens=5, top_k=2, return_dict_in_generate=True, output_scores=True,do_sample=True,pad_token_id=tokenizer.eos_token_id)\n",
    "transition_scores = model.compute_transition_scores(\n",
    "    outputs.sequences, outputs.scores, normalize_logits=True\n",
    ")\n",
    "\n",
    "input_length = 1 if model.config.is_encoder_decoder else inputs.input_ids.shape[1]\n",
    "generated_tokens = outputs.sequences[:, input_length:]\n",
    "for tok, score in zip(generated_tokens[0], transition_scores[0]):\n",
    "    # | token | token string | logits | probability\n",
    "    print(f\"| {tok:5d} | {tokenizer.decode(tok):8s} | {score.numpy():.3f} | {np.exp(score.numpy()):.2%}\")\n",
    "\n",
    "    "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Image_S_work-genai01_pytorch_2.1.2-cuda11.8-cudnn8-devel_opencv",
   "language": "python",
   "name": "s_work-genai01_pytorch_2.1.2-cuda11.8-cudnn8-devel_opencv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}