B03_prompt_llama2_gptq.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "507788fa-cb65-4c26-a3b6-b933f8d82d4d",
   "metadata": {},
   "source": [
    "## Langchain PROMPT - Llama-2-7B-Chat-GPTQ"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b706cbb5-17e6-4a3f-aac6-d3a0903acf40",
   "metadata": {},
   "source": [
    "1. GPTQ\n",
    "- https://huggingface.co/TheBloke/Llama-2-7B-Chat-GPTQ\n",
    "\n",
    "2. LANGCHAIN 手冊\n",
    "https://python.langchain.com/docs/modules/model_io/prompts/prompt_templates/\n",
    "\n",
    "3. 學習LANGCHAIN -> llm +  prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d662e4d0-fed1-48c1-8c8e-a90a877f6db4",
   "metadata": {},
   "source": [
    "## 初始環境設定"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b17198d7-3e53-466f-ad3b-dd1fcb0a2daf",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "HOME = str(Path.home())\n",
    "Add_Binarry_Path=HOME+'/.local/bin'\n",
    "os.environ['PATH']=os.environ['PATH']+':'+Add_Binarry_Path\n",
    "current_foldr=!pwd\n",
    "current_foldr=current_foldr[0]\n",
    "current_foldr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c467a88-ed62-44ec-a045-15c7bfce3a59",
   "metadata": {},
   "source": [
    "## 確認CUDA版本, 以及否能使用GPU\n",
    "若無gpu 請點選右側->已連線->變更執行階段類型->T4 Gpu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4249d3d7-92f2-4981-9c6f-604f59df7de8",
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi\n",
    "import torch\n",
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e145ec44-0b71-41bd-92c0-fefd9affb806",
   "metadata": {},
   "source": [
    "## 安裝套件\n",
    "安裝完成後建議, 點選上方選單, 直接階段->重新啟動工作階段, 確保 library重置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f26741fe-c66f-407b-ac81-959529bb7b4e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install cohere gdown kaleido langchain openai pyngrok pypdf python-dotenv sentence-transformers tiktoken -q\n",
    "!pip install accelerate bitsandbytes hf_transfer huggingface_hub optimum transformers -q \n",
    "!pip install auto-gptq\n",
    "#!pip install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/  -q # Use cu117 if on CUDA 11.7"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "131aa253-8cb7-4171-9f32-7923fc3384ea",
   "metadata": {},
   "source": [
    "### LOAD LIBRARY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "261cc729-b6b3-43a8-9d69-2620c9d4016d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# LOAD LIBRARY\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline\n",
    "from langchain.llms.huggingface_pipeline import HuggingFacePipeline\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain.chains import LLMChain"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1561e895-1187-405a-92e9-6c2d0eb5d9a8",
   "metadata": {},
   "source": [
    "### Download model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5b9fdb4-569c-422c-8c04-bef242980493",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "# Download model\n",
    "mkdir -p /content/Llama-2-7B-Chat-GPTQ\n",
    "HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download TheBloke/Llama-2-7B-Chat-GPTQ --local-dir /content/Llama-2-7B-Chat-GPTQ --local-dir-use-symlinks False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b644634-295d-4baf-9d66-a1d11fdb3edd",
   "metadata": {},
   "source": [
    "### Load Model\n",
    "temperature 的參數值越小，模型就會回傳越確定的結果。如果調高該參數值，大語言模型可能會返回更隨機的結果，也就是說這可能會帶來更多樣化或更具創造性的產出\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0bb705af-4dca-4fbf-9afc-bc7d0a468606",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline\n",
    "\n",
    "MODEL_ID = \"/content/Llama-2-7B-Chat-GPTQ\"\n",
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)\n",
    "model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map=\"auto\")\n",
    "\n",
    "pipe = pipeline(\n",
    "    \"text-generation\",\n",
    "    model=model,\n",
    "    tokenizer=tokenizer,\n",
    "    max_new_tokens=512,\n",
    "    do_sample=True,\n",
    "    temperature=0.7,\n",
    "    top_p=0.95,\n",
    "    top_k=40,\n",
    "    repetition_penalty=1.1\n",
    ")\n",
    "\n",
    "llm = HuggingFacePipeline(pipeline=pipe, model_kwargs={\"temperature\": 0.0})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dab5eca6-b96f-4c1d-a7f7-89a99fecbb7e",
   "metadata": {},
   "source": [
    "### QUESTION to Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc2a2878-965f-4af9-becd-23674f33566d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "response=llm(\"什麼是聯邦式學習?\")\n",
    "\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5973eb23-daa1-48db-be83-aca73f6ce907",
   "metadata": {},
   "source": [
    "## LANGCHAIN PROMPT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b16efd0-c932-45c3-8967-9ba3086ec13c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# PROMPT01\n",
    "template01=\"\"\"[INST] <<SYS>>\n",
    "You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\n",
    "<</SYS>>\n",
    "{question}[/INST]\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "template02=\"\"\"<|im_start|>system\n",
    "You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.<|im_end|>\n",
    "<|im_start|>user\n",
    "{question}<|im_end|>\n",
    "<|im_start|>assistant\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "template03=\"\"\"You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe.  Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\n",
    "\n",
    "### Instruction:\n",
    "{question}\n",
    "\n",
    "### Response:\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "prompt01 = PromptTemplate(template=template01, input_variables=[\"question\"])\n",
    "prompt02 = PromptTemplate(template=template02, input_variables=[\"question\"])\n",
    "prompt03 = PromptTemplate(template=template03, input_variables=[\"question\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e47649e3-03af-47a9-8df2-080f23d42fe8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# PROMPT RESULT\n",
    "question = \"什麼是聯邦式學習?\"\n",
    "print(prompt01.format(question=question))\n",
    "\n",
    "print(prompt02.format(question=question))\n",
    "\n",
    "print(prompt03.format(question=question))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "578d3592-2d85-49a1-9318-ed6771c37786",
   "metadata": {},
   "source": [
    "## LANGCHAIN LLM+PROMPT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e88041b6-3cf2-45de-9de0-a2a1b675caba",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Create Chain (prompt + model)\n",
    "chain01 = LLMChain(llm=llm, prompt=prompt01)\n",
    "\n",
    "chain02 = LLMChain(llm=llm, prompt=prompt02)\n",
    "\n",
    "chain03 = LLMChain(llm=llm, prompt=prompt03)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "257c9d66-d409-4a3d-8477-ea35a1c5a8a2",
   "metadata": {},
   "source": [
    "### Lnagchain QUESTION"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0e382dd-bb7a-4cd9-9522-85924d2fc527",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "question = \"什麼是聯邦式學習?\"\n",
    "print(chain01.run(question=question))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82cdb1cc-9980-4984-9473-ec483a4a4525",
   "metadata": {},
   "outputs": [],
   "source": [
    "question = \"什麼是聯邦式學習?\"\n",
    "print(chain02.run(question=question))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59288253-f0b2-4f08-a61a-5fd84b8f3248",
   "metadata": {},
   "outputs": [],
   "source": [
    "question = \"什麼是聯邦式學習?\"\n",
    "print(chain03.run(question=question))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}