A03_model_llama2_gptq.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "507788fa-cb65-4c26-a3b6-b933f8d82d4d",
   "metadata": {},
   "source": [
    "## Langchain LLM - Llama-2-7B-Chat-GPTQ"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b706cbb5-17e6-4a3f-aac6-d3a0903acf40",
   "metadata": {},
   "source": [
    "1. GPTQ DEMO\n",
    "- https://huggingface.co/TheBloke/Llama-2-7B-Chat-GPTQ\n",
    "\n",
    "2. LANGCHAIN 手冊\n",
    "https://python.langchain.com/docs/integrations/llms/\n",
    "\n",
    "3. 學習llm 模組, 如 llm(\"什麼是聯邦式學習?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d662e4d0-fed1-48c1-8c8e-a90a877f6db4",
   "metadata": {},
   "source": [
    "## 初始環境設定"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b17198d7-3e53-466f-ad3b-dd1fcb0a2daf",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from pathlib import Path\n",
    "HOME = str(Path.home())\n",
    "Add_Binarry_Path=HOME+'/.local/bin'\n",
    "os.environ['PATH']=os.environ['PATH']+':'+Add_Binarry_Path\n",
    "current_foldr=!pwd\n",
    "current_foldr=current_foldr[0]\n",
    "current_foldr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12b9f514-0441-4d92-bfc2-b0c89eca55fd",
   "metadata": {},
   "source": [
    "## 確認CUDA版本, 以及否能使用GPU\n",
    "若無gpu 請點選右側->已連線->變更執行階段類型->T4 Gpu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5d9b03d-ba0a-433f-b1fc-572fe5250c67",
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi\n",
    "import torch\n",
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e145ec44-0b71-41bd-92c0-fefd9affb806",
   "metadata": {},
   "source": [
    "## 安裝套件\n",
    "安裝完成後建議, 點選上方選單, 直接階段->重新啟動工作階段, 確保 library重置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f26741fe-c66f-407b-ac81-959529bb7b4e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install cohere gdown kaleido langchain openai pyngrok pypdf python-dotenv sentence-transformers tiktoken -q\n",
    "!pip install accelerate bitsandbytes hf_transfer huggingface_hub optimum transformers -q \n",
    "!pip install auto-gptq\n",
    "#!pip install auto-gptq --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu118/  -q # Use cu117 if on CUDA 11.7"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "131aa253-8cb7-4171-9f32-7923fc3384ea",
   "metadata": {},
   "source": [
    "### LOAD LIBRARY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "261cc729-b6b3-43a8-9d69-2620c9d4016d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# LOAD LIBRARY\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline, BitsAndBytesConfig\n",
    "from langchain.llms.huggingface_pipeline import HuggingFacePipeline\n",
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1561e895-1187-405a-92e9-6c2d0eb5d9a8",
   "metadata": {},
   "source": [
    "### Download model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5b9fdb4-569c-422c-8c04-bef242980493",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "# Download model\n",
    "mkdir -p /content/Llama-2-7B-Chat-GPTQ\n",
    "HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download TheBloke/Llama-2-7B-Chat-GPTQ --local-dir /content/Llama-2-7B-Chat-GPTQ --local-dir-use-symlinks False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b644634-295d-4baf-9d66-a1d11fdb3edd",
   "metadata": {},
   "source": [
    "### Load Model\n",
    "temperature 的參數值越小，模型就會回傳越確定的結果。如果調高該參數值，大語言模型可能會返回更隨機的結果，也就是說這可能會帶來更多樣化或更具創造性的產出\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0bb705af-4dca-4fbf-9afc-bc7d0a468606",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline\n",
    "\n",
    "MODEL_ID = \"/content/Llama-2-7B-Chat-GPTQ\"\n",
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)\n",
    "model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map=\"auto\")\n",
    "\n",
    "pipe = pipeline(\n",
    "    \"text-generation\",\n",
    "    model=model,\n",
    "    tokenizer=tokenizer,\n",
    "    max_new_tokens=512,\n",
    "    do_sample=True,\n",
    "    temperature=0.7,\n",
    "    top_p=0.95,\n",
    "    top_k=40,\n",
    "    repetition_penalty=1.1\n",
    ")\n",
    "\n",
    "llm = HuggingFacePipeline(pipeline=pipe, model_kwargs={\"temperature\": 0.0})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dab5eca6-b96f-4c1d-a7f7-89a99fecbb7e",
   "metadata": {},
   "source": [
    "### QUESTION to Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc2a2878-965f-4af9-becd-23674f33566d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "response=llm(\"什麼是聯邦式學習?\")\n",
    "\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f747160-a5e2-4cb4-ae36-e17e48911446",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}