2024/07/FederatedLearning/Part02/L2/L2_Centralized_LLM_Fine-tuning_Executed.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "64c1a2de",
   "metadata": {},
   "source": [
    "# Lesson 2: Centralized LLM Fine-tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c16707d",
   "metadata": {},
   "source": [
    "Welcome to Lesson 2!\n",
    "\n",
    "To access the `requirements.txt` and `utils` files for this course, go to `File` and click `Open`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af1d0e65",
   "metadata": {},
   "source": [
    "#### 1. Import packages and utilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "000642f7",
   "metadata": {
    "height": 198
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.11/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32\n"
     ]
    }
   ],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=UserWarning)\n",
    "warnings.filterwarnings(\"ignore\", category=FutureWarning)\n",
    "import torch\n",
    "from datasets import load_dataset\n",
    "from trl import SFTTrainer\n",
    "from transformers import TrainingArguments\n",
    "from utils.utils import *\n",
    "from utils.LLM import LLM_pretrained, LLM_cen_partial\n",
    "from utils.LLM import get_fireworks_api_key,load_env"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e69d454",
   "metadata": {},
   "source": [
    "> Note: Throughout this course, we use Hydra, a framework for managing and configuring files. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b6db1fbe",
   "metadata": {
    "height": 98
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dataset:\n",
      "  name: medalpaca/medical_meadow_medical_flashcards\n",
      "model:\n",
      "  name: EleutherAI/pythia-70m\n",
      "  quantization: 4\n",
      "  gradient_checkpointing: true\n",
      "  use_fast_tokenizer: true\n",
      "  lora:\n",
      "    peft_lora_r: 16\n",
      "    peft_lora_alpha: 64\n",
      "    target_modules: null\n",
      "train:\n",
      "  save_every_round: 5\n",
      "  seq_length: 512\n",
      "  padding_side: left\n",
      "  training_arguments:\n",
      "    learning_rate: 0.0005\n",
      "    per_device_train_batch_size: 2\n",
      "    gradient_accumulation_steps: 1\n",
      "    logging_steps: 1\n",
      "    max_steps: 5\n",
      "    report_to: null\n",
      "    save_steps: 200\n",
      "    save_total_limit: 10\n",
      "    gradient_checkpointing: true\n",
      "    lr_scheduler_type: cosine\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Load config\n",
    "cfg = get_config(\"centralized\")\n",
    "\n",
    "# Inspect the config\n",
    "print_config(cfg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a917f369",
   "metadata": {},
   "source": [
    "#### 2. Load the dataset\n",
    "  \n",
    "**Note:** For more information about the dataset, check here [medAlpaca](https://arxiv.org/abs/2304.08247)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2f9fd8a3",
   "metadata": {
    "height": 147
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e16438ebde5f4be1b0dfc3dfbc8ffae8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading readme:   0%|          | 0.00/1.24k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e99c1d9318b64463bd5c7afb9f0a21e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/17.7M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "76741bad569143dc94e925ab6715d002",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset({\n",
      "    features: ['response', 'instruction'],\n",
      "    num_rows: 3395\n",
      "})\n"
     ]
    }
   ],
   "source": [
    "trainset_full = load_dataset(cfg.dataset.name, split='train')\n",
    "train_test = trainset_full.train_test_split(test_size=0.9, seed=1234)\n",
    "train_dataset = train_test[\"train\"]\n",
    "\n",
    "train_dataset = format_dataset(train_dataset)\n",
    "\n",
    "print(train_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7679e745",
   "metadata": {
    "height": 130
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'response': 'The IVC perforates the diaphragm at the level of T8.',\n",
       " 'instruction': 'At what level does the inferior vena cava (IVC) perforate the diaphragm?'}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print an example from the dataset\n",
    "example_index = 9\n",
    "\n",
    "data_point = train_dataset[example_index]\n",
    "\n",
    "data_point"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2315d3c4",
   "metadata": {},
   "source": [
    "#### 3. Asking the LLM \n",
    "\n",
    "\n",
    "* First attempt \n",
    "\n",
    "Ask a pre-trained LLM a question in a specific domain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c20902da",
   "metadata": {
    "height": 45
   },
   "outputs": [],
   "source": [
    "llm_pretrained = LLM_pretrained()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e6d9be1",
   "metadata": {},
   "source": [
    "* Write a prompt to ask the LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "cf56f92d",
   "metadata": {
    "height": 79
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "?  The weather is a very important factor in our\n",
      "\tlives. It can affect our mood, our health, and\n",
      "\teven our productivity. That’s why it’s important\n",
      "\tto know how to predict the weather.  There are a\n",
      "\tfew different ways to predict the weather. One\n",
      "\tway is to look at the sky. If the sky is clear,\n",
      "\tthen the weather is likely to be good\n"
     ]
    }
   ],
   "source": [
    "prompt = \"How to predict the weather\"\n",
    "llm_pretrained.eval(prompt, verbose = False)\n",
    "llm_pretrained.print_response(verbose = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38b13c69",
   "metadata": {},
   "source": [
    "* Evaluate pre-trained model on Medical Q&A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b44d1846",
   "metadata": {
    "height": 130
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt:\n",
      "\tAt what level does the inferior vena cava (IVC)\n",
      "\tperforate the diaphragm?\n",
      "Response:\n",
      "\tThe inferior vena cava (IVC) perforates the\n",
      "\tdiaphragm at the level of the 12th thoracic\n",
      "\tvertebra.  The IVC is a large vein that returns\n",
      "\tblood from the lower half of the body to the\n",
      "\theart. It begins at the confluence of the common\n",
      "\tiliac veins and ends at the\n",
      "Expected output:\n",
      "\tThe IVC perforates the diaphragm at the level of\n",
      "\tT8.\n"
     ]
    }
   ],
   "source": [
    "llm_pretrained.eval(data_point['instruction'], verbose=True)\n",
    "\n",
    "llm_pretrained.print_response()\n",
    "\n",
    "ex_response = format_string(data_point['response'])\n",
    "print(f\"Expected output:\\n\\t{ex_response}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7d2f9f1",
   "metadata": {},
   "source": [
    "* Second attempt:\n",
    "\n",
    "Ask a fine tuned LLM a question in a specific domain."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ceeeb1e",
   "metadata": {},
   "source": [
    "* Set the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "265fcae9",
   "metadata": {
    "height": 147
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4bf5fdfe29bd4eafb4744339576a7eb3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/567 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8fa997be51104f388c24dc820a8b9538",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/166M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trainable parameters: 196608\n",
      "All parameters: 70623232\n",
      "Trainable (%): 0.278\n"
     ]
    }
   ],
   "source": [
    "model = get_model(cfg.model)\n",
    "\n",
    "\n",
    "trainable, all_parameters = model.get_nb_trainable_parameters()\n",
    "print(f\"Trainable parameters: {trainable}\")\n",
    "print(f\"All parameters: {all_parameters}\")\n",
    "print(f\"Trainable (%): {100*trainable / all_parameters:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06e06af5",
   "metadata": {},
   "source": [
    "* Define the tokenizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3c22c284",
   "metadata": {
    "height": 147
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5bb203201d004822be31adb7fe6d5b9d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/396 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9b2a685491a34de792aa6ce544f9a1f5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/2.11M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "954a2be7478948909358bb7833936a14",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/99.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    }
   ],
   "source": [
    "(\n",
    "tokenizer, \n",
    "data_collator, \n",
    "format_prompts_fn\n",
    ") = get_tokenizer_and_data_collator_and_propt_formatting(\n",
    "        cfg.model.name, cfg.model.use_fast_tokenizer, cfg.train.padding_side\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78c5c437",
   "metadata": {},
   "source": [
    "* Define the finetune_centralised function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "14ab4a64",
   "metadata": {
    "height": 538
   },
   "outputs": [],
   "source": [
    "save_centralized = \"./my_centralized_model\"\n",
    "def finetune_centralised():\n",
    "\n",
    "    # The notebooks you are running in this course does not\n",
    "    # come with a GPU. However, we don't harcode \"cpu\" as the\n",
    "    # device to use in case you wish to download this notebook\n",
    "    # and run it on your own GPU.\n",
    "    use_cuda = torch.cuda.is_available()\n",
    "    training_arguments = TrainingArguments(\n",
    "                         **cfg.train.training_arguments,\n",
    "                         use_cpu=not(use_cuda),\n",
    "                         output_dir=save_centralized,\n",
    "    )\n",
    "\n",
    "    # Construct trainer\n",
    "    trainer = SFTTrainer(\n",
    "        tokenizer=tokenizer,\n",
    "        data_collator=data_collator,\n",
    "        formatting_func=format_prompts_fn,\n",
    "        max_seq_length=cfg.train.seq_length,\n",
    "        model=model,\n",
    "        args=training_arguments,\n",
    "        train_dataset=train_dataset,\n",
    "    )\n",
    "\n",
    "    # Do local training\n",
    "    trainer.train()\n",
    "\n",
    "    # Save the checkpoint\n",
    "    model.save_pretrained(save_centralized)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "218de263",
   "metadata": {},
   "source": [
    "* Let's do some fine-tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "dd8a0f40",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1fe3d76239554874bae576da279dcbd8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/3395 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Detected kernel version 4.18.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='5' max='5' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [5/5 00:05, Epoch 0/1]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.421900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4.437500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.687500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.921900</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "finetune_centralised()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3ca3dec",
   "metadata": {},
   "source": [
    "> Note: The previous fine-tuning was performed on a smaller model. To fine-tune a larger model (e.g., a 7 billion parameter model), you can use the same code with a different configuration. Refer to the code snippet below:\n",
    "\n",
    "\n",
    "```\n",
    " cfg = get_config(\"centralized_full\")\n",
    "```\n",
    "\n",
    "It is recommended to execute all prior cells with this configuration. However, please ensure that your machine has at least one GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef22e176",
   "metadata": {},
   "source": [
    "* Evaluate the Centrally fine-tuned LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6582cc5b",
   "metadata": {
    "height": 215
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prompt:\n",
      "\tAt what level does the inferior vena cava (IVC)\n",
      "\tperforate the diaphragm?\n",
      "Response:\n",
      "\tThe IVC perforates the diaphragm at the level of\n",
      "\tthe T8 vertebra. The IVC is a large blood vessel\n",
      "\tthat carries deoxygenated blood from the lower\n",
      "\tpart of the body back to the heart. It begins at\n",
      "\tthe junction of the right and left common iliac\n",
      "\tveins and runs upward along the right side of the\n",
      "\tabdomen, passing\n",
      "Expected output:\n",
      "\tThe IVC perforates the diaphragm at the level of\n",
      "\tT8.\n"
     ]
    }
   ],
   "source": [
    "llm_cen = LLM_cen_partial()\n",
    "example_index = 9\n",
    "\n",
    "data_point = train_dataset[example_index]\n",
    "\n",
    "llm_cen.eval(data_point['instruction'], verbose=True)\n",
    "\n",
    "llm_cen.print_response()\n",
    "\n",
    "ex_response = format_string(data_point['response'])\n",
    "print(f\"Expected output:\\n\\t{ex_response}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e32f9bf",
   "metadata": {},
   "source": [
    "#### 4. Visualize results of prompting with pretrained LLM and fine-tuned LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ec0ae775",
   "metadata": {
    "height": 45
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhgAAAFvCAYAAAAIS7L0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA6EklEQVR4nO3deVhV5f7//9cGZRJBSwX1oJg45oBzWGqdSBwybdLMDopDg8esg3bKb+WYokdzKtNOhpZWammfLKc8pGbq0RxwClFRQgtQM8ERdHP//vDnPu4AZdtC2vZ8XNe6Lve97rXWG2RtXqx9r3XbjDFGAAAAFvIo6QIAAMCth4ABAAAsR8AAAACWI2AAAADLETAAAIDlCBgAAMByBAwAAGC5UiVdwM2Wl5enn3/+WWXLlpXNZivpcgAAcBvGGJ0+fVpVqlSRh8e1r1H86QLGzz//rJCQkJIuAwAAt3XkyBH95S9/uWafP13AKFu2rKTL35yAgIASrgYAAPeRnZ2tkJAQx+/Sa/nTBYwrH4sEBAQQMAAAuAFFGWLAIE8AAGC5P0TAmDFjhkJDQ+Xj46NWrVppy5YthfadO3eubDab0+Lj43MTqwUAANdT4gFj4cKFio2N1YgRI7R9+3Y1btxYUVFROnbsWKHbBAQEKD093bH8+OOPN7FiAABwPSUeMCZPnqwBAwYoJiZG9evX16xZs+Tn56f4+PhCt7HZbAoODnYsQUFBN7FiAABwPSUaMHJzc7Vt2zZFRkY62jw8PBQZGalNmzYVut2ZM2dUvXp1hYSEqGvXrtq7d2+hfXNycpSdne20AACA4lWiAePEiROy2+35rkAEBQUpIyOjwG3q1Kmj+Ph4ffHFF5o/f77y8vLUunVrHT16tMD+cXFxCgwMdCw8AwMAgOJX4h+RuCoiIkLR0dEKDw9Xu3bttGTJElWsWFHvvvtugf2HDRumrKwsx3LkyJGbXDEAAH8+JfocjAoVKsjT01OZmZlO7ZmZmQoODi7SPkqXLq0mTZro4MGDBa739vaWt7f3764VAAAUXYlewfDy8lKzZs2UkJDgaMvLy1NCQoIiIiKKtA+73a7du3ercuXKxVUmAABwUYk/yTM2Nla9e/dW8+bN1bJlS02dOlVnz55VTEyMJCk6OlpVq1ZVXFycJGn06NG66667FBYWplOnTmnixIn68ccf1b9//5L8MgAAwFVKPGD06NFDx48f1/Dhw5WRkaHw8HCtXLnSMfAzLS3Naca2X3/9VQMGDFBGRobKly+vZs2aaePGjapfv35JfQkAAOA3bMYYU9JF3EzZ2dkKDAxUVlYWc5EAAOACV36HlvgVDAC4qUYGlnQFwM0zMqvEDu12t6kCAIA/PgIGAACwHAEDAABYjoABAAAsR8AAAACWI2AAAADLETAAAIDlCBgAAMByBAwAAGA5AgYAALAcAQMAAFiOgAEAACxHwAAAAJYjYAAAAMsRMAAAgOUIGAAAwHIEDAAAYDkCBgAAsBwBAwAAWI6AAQAALEfAAAAAliNgAAAAyxEwAACA5QgYAADAcgQMAABgOQIGAACwHAEDAABYjoABAAAsR8AAAACWI2AAAADLETAAAIDlCBgAAMByBAwAAGA5AgYAALAcAQMAAFiOgAEAACxHwAAAAJYjYAAAAMsRMAAAgOUIGAAAwHIEDAAAYDkCBgAAsBwBAwAAWI6AAQAALEfAAAAAliNgAAAAyxEwAACA5QgYAADAcn+IgDFjxgyFhobKx8dHrVq10pYtW4q03YIFC2Sz2dStW7fiLRAAALikxAPGwoULFRsbqxEjRmj79u1q3LixoqKidOzYsWtul5qaqqFDh6pNmzY3qVIAAFBUJR4wJk+erAEDBigmJkb169fXrFmz5Ofnp/j4+EK3sdvt6tWrl0aNGqU77rjjJlYLAACKokQDRm5urrZt26bIyEhHm4eHhyIjI7Vp06ZCtxs9erQqVaqkfv36XfcYOTk5ys7OdloAAEDxKtGAceLECdntdgUFBTm1BwUFKSMjo8BtvvvuO73//vt67733inSMuLg4BQYGOpaQkJDfXTcAALi2Ev+IxBWnT5/W3/72N7333nuqUKFCkbYZNmyYsrKyHMuRI0eKuUoAAFCqJA9eoUIFeXp6KjMz06k9MzNTwcHB+fqnpKQoNTVVXbp0cbTl5eVJkkqVKqXk5GTVrFnTaRtvb295e3sXQ/UAAKAwJXoFw8vLS82aNVNCQoKjLS8vTwkJCYqIiMjXv27dutq9e7cSExMdy0MPPaT77rtPiYmJfPwBAMAfRIlewZCk2NhY9e7dW82bN1fLli01depUnT17VjExMZKk6OhoVa1aVXFxcfLx8VGDBg2cti9Xrpwk5WsHAAAlp8QDRo8ePXT8+HENHz5cGRkZCg8P18qVKx0DP9PS0uTh4VZDRQAA+NOzGWNMSRdxM2VnZyswMFBZWVkKCAgo6XIA3GwjA0u6AuDmGZll6e5c+R3KpQEAAGA5AgYAALAcAQMAAFiOgAEAACxHwAAAAJYjYAAAAMsRMAAAgOUIGAAAwHIEDAAAYDkCBgAAsBwBAwAAWI6AAQAALOdywDh06FBx1AEAAG4hLgeMsLAw3XfffZo/f74uXLhQHDUBAAA353LA2L59uxo1aqTY2FgFBwfrmWee0ZYtW4qjNgAA4KZcDhjh4eGaNm2afv75Z8XHxys9PV333HOPGjRooMmTJ+v48ePFUScAAHAjNzzIs1SpUnrkkUf06aefasKECTp48KCGDh2qkJAQRUdHKz093co6AQCAG7nhgLF161YNHDhQlStX1uTJkzV06FClpKRo9erV+vnnn9W1a1cr6wQAAG6klKsbTJ48WXPmzFFycrI6deqkDz/8UJ06dZKHx+WsUqNGDc2dO1ehoaFW1woAANyEywFj5syZ6tu3r/r06aPKlSsX2KdSpUp6//33f3dxAADAPbkcMA4cOHDdPl5eXurdu/cNFQQAANyfy2Mw5syZo08//TRf+6effqoPPvjAkqIAAIB7czlgxMXFqUKFCvnaK1WqpHHjxllSFAAAcG8uB4y0tDTVqFEjX3v16tWVlpZmSVEAAMC9uRwwKlWqpF27duVr37lzp26//XZLigIAAO7N5YDRs2dPDR48WGvWrJHdbpfdbtc333yjF154QU888URx1AgAANyMy3eRjBkzRqmpqbr//vtVqtTlzfPy8hQdHc0YDAAAIOkGAoaXl5cWLlyoMWPGaOfOnfL19VXDhg1VvXr14qgPAAC4IZcDxhW1a9dW7dq1rawFAADcIm4oYBw9elRLly5VWlqacnNzndZNnjzZksIAAID7cjlgJCQk6KGHHtIdd9yhffv2qUGDBkpNTZUxRk2bNi2OGgEAgJtx+S6SYcOGaejQodq9e7d8fHy0ePFiHTlyRO3atdPjjz9eHDUCAAA343LASEpKUnR0tCSpVKlSOn/+vPz9/TV69GhNmDDB8gIBAID7cTlglClTxjHuonLlykpJSXGsO3HihHWVAQAAt+XyGIy77rpL3333nerVq6dOnTppyJAh2r17t5YsWaK77rqrOGoEAABuxuWAMXnyZJ05c0aSNGrUKJ05c0YLFy5UrVq1uIMEAABIcjFg2O12HT16VI0aNZJ0+eOSWbNmFUthAADAfbk0BsPT01Pt27fXr7/+Wlz1AACAW4DLgzwbNGigQ4cOFUctAADgFuFywHjjjTc0dOhQffXVV0pPT1d2drbTAgAA4PIgz06dOkmSHnroIdlsNke7MUY2m012u9266txI6CvLSroE4KZJHd+5pEsA8AfncsBYs2ZNcdQBAABuIS4HjHbt2hVHHQAA4BbicsD49ttvr7m+bdu2N1wMAAC4NbgcMO699958bVePxfizjsEAAAD/4/JdJL/++qvTcuzYMa1cuVItWrTQ119/XRw1AgAAN+PyFYzAwMB8bQ888IC8vLwUGxurbdu2WVIYAABwXy5fwShMUFCQkpOTrdodAABwYy5fwdi1a5fTa2OM0tPTNX78eIWHh1tVFwAAcGMuX8EIDw9XkyZNFB4e7vh3p06dlJubq9mzZ99QETNmzFBoaKh8fHzUqlUrbdmypdC+S5YsUfPmzVWuXDmVKVNG4eHhmjdv3g0dFwAAFA+Xr2AcPnzY6bWHh4cqVqwoHx+fGypg4cKFio2N1axZs9SqVStNnTpVUVFRSk5OVqVKlfL1v+222/Tqq6+qbt268vLy0ldffaWYmBhVqlRJUVFRN1QDAACwls0YY0qygFatWqlFixZ6++23JUl5eXkKCQnR888/r1deeaVI+2jatKk6d+6sMWPGXLdvdna2AgMDlZWVpYCAgN9V+9V4VDj+TNz6UeEj8w9UB25ZI7Ms3Z0rv0Nd/ohk8ODBmj59er72t99+Wy+++KJL+8rNzdW2bdsUGRn5v4I8PBQZGalNmzZdd3tjjBISEpScnFzoA75ycnKYkA0AgJvM5YCxePFi3X333fnaW7durc8++8ylfZ04cUJ2u11BQUFO7UFBQcrIyCh0u6ysLPn7+8vLy0udO3fWW2+9pQceeKDAvnFxcQoMDHQsISEhLtUIAABc53LA+OWXXwp8FkZAQIBOnDhhSVHXU7ZsWSUmJur777/X2LFjFRsbq7Vr1xbYd9iwYcrKynIsR44cuSk1AgDwZ+byIM+wsDCtXLlSgwYNcmpfsWKF7rjjDpf2VaFCBXl6eiozM9OpPTMzU8HBwYVu5+HhobCwMEmX72pJSkpSXFxcgY8x9/b2lre3t0t1AQCA38flgBEbG6tBgwbp+PHj+utf/ypJSkhI0JtvvqmpU6e6tC8vLy81a9ZMCQkJ6tatm6TLgzwTEhLyBZhrycvLU05OjkvHBgAAxcflgNG3b1/l5ORo7Nixjrs2QkNDNXPmTEVHR7tcQGxsrHr37q3mzZurZcuWmjp1qs6ePauYmBhJUnR0tKpWraq4uDhJl8dUNG/eXDVr1lROTo6WL1+uefPmaebMmS4fGwAAFA+XA4YkPffcc3ruued0/Phx+fr6yt/f/4YL6NGjh44fP67hw4crIyND4eHhWrlypWPgZ1pamjw8/jdU5OzZsxo4cKCOHj0qX19f1a1bV/Pnz1ePHj1uuAYAAGAtl5+DcfjwYV26dEm1atVyaj9w4IBKly6t0NBQK+uzHM/BAH4/noMBuAl3eg5Gnz59tHHjxnztmzdvVp8+fVzdHQAAuAW5HDB27NhR4HMw7rrrLiUmJlpREwAAcHMuBwybzabTp0/na8/KypLdbrekKAAA4N5cDhht27ZVXFycU5iw2+2Ki4vTPffcY2lxAADAPbl8F8mECRPUtm1b1alTR23atJEkrV+/XtnZ2frmm28sLxAAALgfl69g1K9fX7t27VL37t117NgxnT59WtHR0dq3b58aNGhQHDUCAAA3c0PPwahSpYrGjRvn1Hbq1Cm9/fbbLj2BEwAA3JpcvoLxWwkJCXryySdVuXJljRgxwoqaAACAm7uhgHHkyBGNHj1aNWrUUPv27SVJn3/++TWnWAcAAH8eRQ4YFy9e1KeffqqoqCjVqVNHiYmJmjhxojw8PPTaa6+pQ4cOKl26dHHWCgAA3ESRx2BUrVpVdevW1VNPPaUFCxaofPnykqSePXsWW3EAAMA9FfkKxqVLl2Sz2WSz2eTp6VmcNQEAADdX5IDx888/6+mnn9Ynn3yi4OBgPfroo/r8889ls9mKsz4AAOCGihwwfHx81KtXL33zzTfavXu36tWrp8GDB+vSpUsaO3asVq9ezaPCAQCApBu8i6RmzZp644039OOPP2rZsmXKycnRgw8+qKCgIKvrAwAAbuiGHrR1hYeHhzp27KiOHTvq+PHjmjdvnlV1AQAAN/a7H7R1RcWKFRUbG2vV7gAAgBuzLGAAAABcQcAAAACWI2AAAADLETAAAIDlXL6LxG63a+7cuUpISNCxY8eUl5fntP6bb76xrDgAAOCeXA4YL7zwgubOnavOnTurQYMGPMkTAADk43LAWLBggRYtWqROnToVRz0AAOAW4PIYDC8vL4WFhRVHLQAA4BbhcsAYMmSIpk2bJmNMcdQDAABuAS5/RPLdd99pzZo1WrFihe68806VLl3aaf2SJUssKw4AALgnlwNGuXLl9PDDDxdHLQAA4BbhcsCYM2dOcdQBAABuITc8m+rx48eVnJwsSapTp44qVqxoWVEAAMC9uTzI8+zZs+rbt68qV66stm3bqm3btqpSpYr69eunc+fOFUeNAADAzbgcMGJjY7Vu3Tp9+eWXOnXqlE6dOqUvvvhC69at05AhQ4qjRgAA4GZc/ohk8eLF+uyzz3Tvvfc62jp16iRfX191795dM2fOtLI+AADghly+gnHu3DkFBQXla69UqRIfkQAAAEk3EDAiIiI0YsQIXbhwwdF2/vx5jRo1ShEREZYWBwAA3JPLH5FMmzZNUVFR+stf/qLGjRtLknbu3CkfHx+tWrXK8gIBAID7cTlgNGjQQAcOHNBHH32kffv2SZJ69uypXr16ydfX1/ICAQCA+7mh52D4+flpwIABVtcCAABuEUUKGEuXLlXHjh1VunRpLV269Jp9H3roIUsKAwAA7qtIAaNbt27KyMhQpUqV1K1bt0L72Ww22e12q2oDAABuqkgBIy8vr8B/AwAAFMTl21Q//PBD5eTk5GvPzc3Vhx9+aElRAADAvbkcMGJiYpSVlZWv/fTp04qJibGkKAAA4N5cDhjGGNlstnztR48eVWBgoCVFAQAA91bk21SbNGkim80mm82m+++/X6VK/W9Tu92uw4cPq0OHDsVSJAAAcC9FDhhX7h5JTExUVFSU/P39Heu8vLwUGhqqRx991PICAQCA+ylywBgxYoQkKTQ0VD169JCPj0+xFQUAANyby0/y7N27d3HUAQAAbiEuBwy73a4pU6Zo0aJFSktLU25urtP6kydPWlYcAABwTy7fRTJq1ChNnjxZPXr0UFZWlmJjY/XII4/Iw8NDI0eOvKEiZsyYodDQUPn4+KhVq1basmVLoX3fe+89tWnTRuXLl1f58uUVGRl5zf4AAODmczlgfPTRR3rvvfc0ZMgQlSpVSj179tTs2bM1fPhw/fe//3W5gIULFyo2NlYjRozQ9u3b1bhxY0VFRenYsWMF9l+7dq169uypNWvWaNOmTQoJCVH79u31008/uXxsAABQPFwOGBkZGWrYsKEkyd/f3/HQrQcffFDLli1zuYDJkydrwIABiomJUf369TVr1iz5+fkpPj6+wP4fffSRBg4cqPDwcNWtW1ezZ89WXl6eEhISXD42AAAoHi4HjL/85S9KT0+XJNWsWVNff/21JOn777+Xt7e3S/vKzc3Vtm3bFBkZ+b+CPDwUGRmpTZs2FWkf586d08WLF3XbbbcVuD4nJ0fZ2dlOCwAAKF4uB4yHH37YcbXg+eef1+uvv65atWopOjpaffv2dWlfJ06ckN1uV1BQkFN7UFCQMjIyirSPl19+WVWqVHEKKVeLi4tTYGCgYwkJCXGpRgAA4DqX7yIZP3684989evRQtWrVtGnTJtWqVUtdunSxtLii1LJgwQKtXbu20OdyDBs2TLGxsY7X2dnZhAwAAIqZywHjtyIiIhQREXFD21aoUEGenp7KzMx0as/MzFRwcPA1t500aZLGjx+v//znP2rUqFGh/by9vV3+6AYAAPw+RQoYS5cuLfIOH3rooSL39fLyUrNmzZSQkOB4FPmVAZuDBg0qdLt//etfGjt2rFatWqXmzZsX+XgAAODmKFLAuPLL/wqbzSZjTL426fKDuFwRGxur3r17q3nz5mrZsqWmTp2qs2fPOqZ+j46OVtWqVRUXFydJmjBhgoYPH66PP/5YoaGhjrEa/v7+TvOjAACAklOkQZ55eXmO5euvv1Z4eLhWrFihU6dO6dSpU1qxYoWaNm2qlStXulxAjx49NGnSJA0fPlzh4eFKTEzUypUrHQM/09LSHHetSNLMmTOVm5urxx57TJUrV3YskyZNcvnYAACgeNjMby9FXEeDBg00a9Ys3XPPPU7t69ev19NPP62kpCRLC7Radna2AgMDlZWVpYCAAMv2G/qK688AAdxV6vjOJV3CjRsZWNIVADfPyCxLd+fK71CXb1NNSUlRuXLl8rUHBgYqNTXV1d0BAIBbkMsBo0WLFoqNjXW68yMzM1MvvfSSWrZsaWlxAADAPbkcMOLj45Wenq5q1aopLCxMYWFhqlatmn766Se9//77xVEjAABwMy4/ByMsLEy7du3S6tWrtW/fPklSvXr1FBkZ6biTBAAA/Lnd0IO2bDab2rdvr/bt21tdDwAAuAUUKWBMnz5dTz/9tHx8fDR9+vRr9h08eLAlhQEAAPdVpIAxZcoU9erVSz4+PpoyZUqh/Ww2GwEDAAAULWAcPny4wH8DAAAUxOW7SAAAAK6nSFcwrp7u/HomT558w8UAAIBbQ5ECxo4dO4q0M25TBQAAUhEDxpo1a4q7DgAAcAthDAYAALDcDT1oa+vWrVq0aJHS0tKUm5vrtG7JkiWWFAYAANyXy1cwFixYoNatWyspKUmff/65Ll68qL179+qbb75RYCDTIAMAgBsIGOPGjdOUKVP05ZdfysvLS9OmTdO+ffvUvXt3VatWrThqBAAAbsblgJGSkqLOnTtLkry8vHT27FnZbDb94x//0L///W/LCwQAAO7H5YBRvnx5nT59WpJUtWpV7dmzR5J06tQpnTt3ztrqAACAW3J5kGfbtm21evVqNWzYUI8//rheeOEFffPNN1q9erXuv//+4qgRAAC4mSIHjD179qhBgwZ6++23deHCBUnSq6++qtKlS2vjxo169NFH9dprrxVboQAAwH0UOWA0atRILVq0UP/+/fXEE09Ikjw8PPTKK68UW3EAAMA9FXkMxrp163TnnXdqyJAhqly5snr37q3169cXZ20AAMBNFTlgtGnTRvHx8UpPT9dbb72l1NRUtWvXTrVr19aECROUkZFRnHUCAAA34vJdJGXKlFFMTIzWrVun/fv36/HHH9eMGTNUrVo1PfTQQ8VRIwAAcDO/ay6SsLAw/b//9//02muvqWzZslq2bJlVdQEAADd2Q3ORSNK3336r+Ph4LV68WB4eHurevbv69etnZW0AAMBNuRQwfv75Z82dO1dz587VwYMH1bp1a02fPl3du3dXmTJliqtGAADgZoocMDp27Kj//Oc/qlChgqKjo9W3b1/VqVOnOGsDAABuqsgBo3Tp0vrss8/04IMPytPTszhrAgAAbq7IAWPp0qXFWQcAALiF/K67SAAAAApCwAAAAJYjYAAAAMsRMAAAgOUIGAAAwHIEDAAAYDkCBgAAsBwBAwAAWI6AAQAALEfAAAAAliNgAAAAyxEwAACA5QgYAADAcgQMAABgOQIGAACwHAEDAABYjoABAAAsR8AAAACWI2AAAADLlXjAmDFjhkJDQ+Xj46NWrVppy5Ythfbdu3evHn30UYWGhspms2nq1Kk3r1AAAFBkJRowFi5cqNjYWI0YMULbt29X48aNFRUVpWPHjhXY/9y5c7rjjjs0fvx4BQcH3+RqAQBAUZVowJg8ebIGDBigmJgY1a9fX7NmzZKfn5/i4+ML7N+iRQtNnDhRTzzxhLy9vW9ytQAAoKhKLGDk5uZq27ZtioyM/F8xHh6KjIzUpk2bLDtOTk6OsrOznRYAAFC8SixgnDhxQna7XUFBQU7tQUFBysjIsOw4cXFxCgwMdCwhISGW7RsAABSsxAd5Frdhw4YpKyvLsRw5cqSkSwIA4JZXqqQOXKFCBXl6eiozM9OpPTMz09IBnN7e3ozXAADgJiuxKxheXl5q1qyZEhISHG15eXlKSEhQRERESZUFAAAsUGJXMCQpNjZWvXv3VvPmzdWyZUtNnTpVZ8+eVUxMjCQpOjpaVatWVVxcnKTLA0N/+OEHx79/+uknJSYmyt/fX2FhYSX2dQAAAGclGjB69Oih48ePa/jw4crIyFB4eLhWrlzpGPiZlpYmD4//XWT5+eef1aRJE8frSZMmadKkSWrXrp3Wrl17s8sHAACFKNGAIUmDBg3SoEGDClz329AQGhoqY8xNqAoAAPwet/xdJAAA4OYjYAAAAMsRMAAAgOUIGAAAwHIEDAAAYDkCBgAAsBwBAwAAWI6AAQAALEfAAAAAliNgAAAAyxEwAACA5QgYAADAcgQMAABgOQIGAACwHAEDAABYjoABAAAsR8AAAACWI2AAAADLETAAAIDlCBgAAMByBAwAAGA5AgYAALAcAQMAAFiOgAEAACxHwAAAAJYjYAAAAMsRMAAAgOUIGAAAwHIEDAAAYDkCBgAAsBwBAwAAWI6AAQAALEfAAAAAliNgAAAAyxEwAACA5QgYAADAcgQMAABgOQIGAACwHAEDAABYjoABAAAsR8AAAACWI2AAAADLETAAAIDlCBgAAMByBAwAAGA5AgYAALAcAQMAAFiOgAEAACxHwAAAAJYjYAAAAMv9IQLGjBkzFBoaKh8fH7Vq1Upbtmy5Zv9PP/1UdevWlY+Pjxo2bKjly5ffpEoBAEBRlHjAWLhwoWJjYzVixAht375djRs3VlRUlI4dO1Zg/40bN6pnz57q16+fduzYoW7duqlbt27as2fPTa4cAAAUxmaMMSVZQKtWrdSiRQu9/fbbkqS8vDyFhITo+eef1yuvvJKvf48ePXT27Fl99dVXjra77rpL4eHhmjVrVr7+OTk5ysnJcbzOyspStWrVdOTIEQUEBFj2dTQYscqyfQF/dHtGRZV0CTcu7i8lXQFw8ww7aunusrOzFRISolOnTikwMPDanU0JysnJMZ6enubzzz93ao+OjjYPPfRQgduEhISYKVOmOLUNHz7cNGrUqMD+I0aMMJJYWFhYWFhYLFqOHDly3d/xpVSCTpw4IbvdrqCgIKf2oKAg7du3r8BtMjIyCuyfkZFRYP9hw4YpNjbW8TovL08nT57U7bffLpvN9ju/ApSkK0na6qtRAKzFuXrrMMbo9OnTqlKlynX7lmjAuBm8vb3l7e3t1FauXLmSKQbFIiAggDctwA1wrt4arvvRyP+vRAd5VqhQQZ6ensrMzHRqz8zMVHBwcIHbBAcHu9QfAADcfCUaMLy8vNSsWTMlJCQ42vLy8pSQkKCIiIgCt4mIiHDqL0mrV68utD8AALj5SvwjktjYWPXu3VvNmzdXy5YtNXXqVJ09e1YxMTGSpOjoaFWtWlVxcXGSpBdeeEHt2rXTm2++qc6dO2vBggXaunWr/v3vf5fkl4ES4O3trREjRuT7CAzAHwvn6p9Tid+mKklvv/22Jk6cqIyMDIWHh2v69Olq1aqVJOnee+9VaGio5s6d6+j/6aef6rXXXlNqaqpq1aqlf/3rX+rUqVMJVQ8AAH7rDxEwAADAraXEn+QJAABuPQQMAABgOQIGAACwHAEDfwihoaGaOnVqsR/n3nvv1YsvvljsxymKkSNHKjw8vMj9U1NTZbPZlJiYWGw14Y/nj/QzW1z+aD/brr4fuXou/1kQMG4hffr0kc1mk81mk5eXl8LCwjR69GhdunTJ8mNZ/ab3/fff6+mnn7Zsf4A7ufrcvXo5ePCglixZojFjxlh+vG7dulm6T+C3Svw5GLBWhw4dNGfOHOXk5Gj58uX6+9//rtKlS2vYsGH5+ubm5srLy6vYajHGyG63q1Sp6/+YVaxYsdjqANzBlXP3ahUrVpSnp2cJVQT8PlzBuMV4e3srODhY1atX13PPPafIyEgtXbpU0v/+ahk7dqyqVKmiOnXqSJKOHDmi7t27q1y5crrtttvUtWtXpaamFnqMPn36aN26dZo2bZrjL63U1FStXbtWNptNK1asULNmzeTt7a3vvvtOKSkp6tq1q4KCguTv768WLVroP//5j9M+f3tJ0mazafbs2Xr44Yfl5+enWrVqOb6OK/bs2aOOHTvK399fQUFB+tvf/qYTJ0441p89e1bR0dHy9/dX5cqV9eabb173+3flUmd8fLyqVasmf39/DRw4UHa7Xf/6178UHBysSpUqaezYsU7bpaWlqWvXrvL391dAQIC6d++e75H248ePV1BQkMqWLat+/frpwoUL+Y4/e/Zs1atXTz4+Pqpbt67eeeed69aMW8OVc/fqxdPTM9/VwtDQUI0bN059+/ZV2bJlVa1atXwPGrzWOT1y5Eh98MEH+uKLLxzn79q1ax3n76lTpxz7SUxMdJzfkjR37lyVK1dOq1atUr169eTv768OHTooPT3d6fjX+znesmWLmjRpIh8fHzVv3lw7duy47vcnNDRUb7zxhuOcrl69upYuXarjx487zr1GjRpp69atTtstXrxYd955p7y9vRUaGprvfeDYsWPq0qWLfH19VaNGDX300Uf5jn3q1Cn1799fFStWVEBAgP76179q586d1635z46AcYvz9fVVbm6u43VCQoKSk5O1evVqffXVV7p48aKioqJUtmxZrV+/Xhs2bHC8aVy93dWmTZumiIgIDRgwQOnp6UpPT1dISIhj/SuvvKLx48crKSlJjRo10pkzZ9SpUyclJCRox44d6tChg7p06aK0tLRr1j5q1Ch1795du3btUqdOndSrVy+dPHlS0uUT/q9//auaNGmirVu3auXKlcrMzFT37t0d27/00ktat26dvvjiC3399ddau3attm/fft3vWUpKilasWKGVK1fqk08+0fvvv6/OnTvr6NGjWrdunSZMmKDXXntNmzdvlnT58fZdu3bVyZMntW7dOq1evVqHDh1Sjx49HPtctGiRRo4cqXHjxmnr1q2qXLlyvjfdjz76SMOHD9fYsWOVlJSkcePG6fXXX9cHH3xw3Zrx5/Lmm286fjEPHDhQzz33nJKTkyXpuuf00KFD1b17d0cwSE9PV+vWrYt87HPnzmnSpEmaN2+evv32W6WlpWno0KGO9df7OT5z5owefPBB1a9fX9u2bdPIkSOdtr+WKVOm6O6779aOHTvUuXNn/e1vf1N0dLSeeuopbd++XTVr1lR0dLSuPN5p27Zt6t69u5544gnt3r1bI0eO1Ouvv+704MY+ffroyJEjWrNmjT777DO98847OnbsmNNxH3/8cR07dkwrVqzQtm3b1LRpU91///2O9yMU4roTusNt9O7d23Tt2tUYY0xeXp5ZvXq18fb2NkOHDnWsDwoKMjk5OY5t5s2bZ+rUqWPy8vIcbTk5OcbX19esWrWq0GO1a9fOvPDCC05ta9asMZLM//3f/1231jvvvNO89dZbjtfVq1c3U6ZMcbyWZF577TXH6zNnzhhJZsWKFcYYY8aMGWPat2/vtM8jR44YSSY5OdmcPn3aeHl5mUWLFjnW//LLL8bX1zdf3VcbMWKE8fPzM9nZ2Y62qKgoExoaaux2u6OtTp06Ji4uzhhjzNdff208PT1NWlqaY/3evXuNJLNlyxZjjDERERFm4MCBTsdq1aqVady4seN1zZo1zccff+zUZ8yYMSYiIsIYY8zhw4eNJLNjx45C64d76t27t/H09DRlypRxLI899pgxJv+5Vr16dfPUU085Xufl5ZlKlSqZmTNnGmOKdk5f/V5xxZXz99dff3W07dixw0gyhw8fNsYYM2fOHCPJHDx40NFnxowZJigoyPH6ej/H7777rrn99tvN+fPnHetnzpx53Z/t337d6enpRpJ5/fXXHW2bNm0ykkx6eroxxpgnn3zSPPDAA077eemll0z9+vWNMcYkJyc7nafGGJOUlGQkOd6P1q9fbwICAsyFCxec9lOzZk3z7rvvGmMuv29cfS7jMsZg3GK++uor+fv76+LFi8rLy9OTTz6pkSNHOtY3bNjQadzFzp07dfDgQZUtW9ZpPxcuXFBKSorWr1+vjh07Otrfffdd9erV65o1NG/e3On1mTNnNHLkSC1btkzp6em6dOmSzp8/f90rGI0aNXL8u0yZMgoICHD8ZbFz506tWbNG/v7++bZLSUnR+fPnlZub63jkvCTddtttjo+FriU0NNTp+xEUFCRPT095eHg4tV2pJSkpSSEhIU5XcerXr69y5copKSlJLVq0UFJSkp599lmn40RERGjNmjWSLn+ck5KSon79+mnAgAGOPpcuXSry1Mhwb/fdd59mzpzpeF2mTJlC+159bthsNgUHBzudG9c6p38vPz8/1axZ0/G6cuXKjmMX5ef4ypVNHx8fx/qiTlZ59dcdFBQk6fJ72m/bjh07puDgYCUlJalr165O+7j77rs1depU2e12JSUlqVSpUmrWrJljfd26dVWuXDnH6507d+rMmTO6/fbbnfZz/vx5S76ftzICxi3mypuUl5eXqlSpkm+A5W/ftM6cOaNmzZoV+LljxYoV5eXl5XTr2JUT+Fp+e4yhQ4dq9erVmjRpksLCwuTr66vHHnus0I9grihdurTTa5vNpry8PEfdXbp00YQJE/JtV7lyZR08ePC6dbpy3GvVYoUzZ85Ikt577z2nUCSJQX5/EmXKlFFYWFiR+l7v3LjWOV2YKwHaXDV7xMWLF4t07CvbFPfP8dXHttlshbZZfW5WrlxZa9euzbfu6iCC/AgYtxhX3qQkqWnTplq4cKEqVaqkgICAAvsUtD8vLy/Z7fYiHWPDhg3q06ePHn74YUmXT9hrDSItiqZNm2rx4sUKDQ0t8C6VmjVrqnTp0tq8ebOqVasmSfr111+1f/9+tWvX7ncd+7fq1aunI0eO6MiRI46rGD/88INOnTql+vXrO/ps3rxZ0dHRju3++9//Ov4dFBSkKlWq6NChQ9e9QgRcS1HO6YLO3yvhIz09XeXLl5ckl59LUZSf43r16mnevHm6cOGC4yrG1eeClerVq6cNGzY4tW3YsEG1a9eWp6en6tatq0uXLmnbtm1q0aKFJCk5OdlpoGvTpk2VkZGhUqVKKTQ0tFjqvFUxyPNPrlevXqpQoYK6du2q9evX6/Dhw1q7dq0GDx6so0ePFrpdaGioNm/erNTUVJ04ceKafzHUqlVLS5YsUWJionbu3Kknn3zyd/+F8fe//10nT55Uz5499f333yslJUWrVq1STEyM7Ha7/P391a9fP7300kv65ptvtGfPHvXp08fpYw6rREZGqmHDhurVq5e2b9+uLVu2KDo6Wu3atXN8XPTCCy8oPj5ec+bM0f79+zVixAjt3bvXaT+jRo1SXFycpk+frv3792v37t2aM2eOJk+ebHnNuHUV5ZwODQ3Vrl27lJycrBMnTujixYsKCwtTSEiIRo4cqQMHDmjZsmVFuvPqt673c/zkk0/KZrNpwIAB+uGHH7R8+XJNmjTJ0u/BFUOGDFFCQoLGjBmj/fv364MPPtDbb7/tGFRap04ddejQQc8884w2b96sbdu2qX///vL19XXsIzIyUhEREerWrZu+/vprpaamauPGjXr11Vfz3bECZwSMPzk/Pz99++23qlatmh555BHVq1fPcQtlYX/9SJc/9vD09FT9+vVVsWLFa46nmDx5ssqXL6/WrVurS5cuioqKUtOmTX9X3VWqVNGGDRtkt9vVvn17NWzYUC+++KLKlSvnCBETJ05UmzZt1KVLF0VGRuqee+5x+qzVKjabTV988YXKly+vtm3bKjIyUnfccYcWLlzo6NOjRw+9/vrr+uc//6lmzZrpxx9/1HPPPee0n/79+2v27NmaM2eOGjZsqHbt2mnu3LmqUaOG5TXj1lWUc3rAgAGqU6eOmjdvrooVK2rDhg0qXbq0PvnkE+3bt0+NGjXShAkT9MYbb7h8/Ov9HPv7++vLL7/U7t271aRJE7366qsFftRphaZNm2rRokVasGCBGjRooOHDh2v06NHq06ePo8+cOXNUpUoVtWvXTo888oiefvppVapUybHeZrNp+fLlatu2rWJiYlS7dm098cQT+vHHH4v0kfGfGdO1AwAAy3EFAwAAWI6AAQAALEfAAAAAliNgAAAAyxEwAOAafvnlF1WqVOl3P7ulJF2ZxO+KmzFd+28nMLwVFTRBXHF55ZVX9Pzzzxf7caxEwACAaxg7dqy6du16Ux+ydGXW0uIybdo0pwm//qguXLigPn36qGHDhipVqlShoWjt2rVq2rSpvL29FRYWlu9r++ijjxQSEqLy5csrNjbWaV1qaqpq166t7OzsYvoqnP12dtyiGjp0qD744AMdOnTI+qKKCQEDAApx7tw5vf/+++rXr19Jl1Kg6z1uvzCBgYFu8Zhru90uX19fDR48WJGRkQX2OXz4sDp37qz77rtPiYmJevHFF9W/f3+tWrVKknTixAn1799fkyZN0tdff6358+frq6++cmw/cOBAjR8//prP/fkjqFChgqKiopzmq/mjI2AAQCGWL18ub29v3XXXXU7te/fu1YMPPqiAgACVLVtWbdq0cZr4avbs2apXr558fHxUt25dvfPOO451qampstlsWrJkie677z75+fmpcePG2rRpk6TLf43HxMQoKytLNptNNpvNMWFhaGioxowZo+joaAUEBOjpp5+WJL388suqXbu2/Pz8dMcdd+j1118vcB6RK67+iORKPb9d7r33Xkf/7777Tm3atJGvr69CQkI0ePBgnT171rH+2LFj6tKli3x9fVWjRo0C50G5EWXKlNHMmTM1YMAABQcHF9hn1qxZqlGjht58803Vq1dPgwYN0mOPPaYpU6ZIkg4dOqTAwED16NFDLVq00H333aekpCRJ0ieffKLSpUvrkUceKVI9y5cvV+3ateXr66v77rsv38dmv/zyi3r27KmqVavKz89PDRs21CeffOJY36dPH61bt07Tpk1zfJ9TU1Nlt9vVr18/1ahRQ76+vqpTp46mTZuW7/hdunTRggULilTrH0LJTuYKAH9cgwcPNh06dHBqO3r0qLntttvMI488Yr7//nuTnJxs4uPjzb59+4wxxsyfP99UrlzZLF682Bw6dMgsXrzY3HbbbWbu3LnGGGMOHz5sJJm6deuar776yiQnJ5vHHnvMVK9e3Vy8eNHk5OSYqVOnmoCAAJOenm7S09PN6dOnjTGXpywPCAgwkyZNMgcPHnRMmz5mzBizYcMGc/jwYbN06VITFBRkJkyY4Kj5t9OJXz1d+6VLlxzHSU9PNzt27DC33367Yxr0gwcPmjJlypgpU6aY/fv3mw0bNpgmTZqYPn36OPbXsWNH07hxY7Np0yazdetW07p1a+Pr6+uY8twKBU0xb4wxbdq0cZrO3hhj4uPjTUBAgDHGmJMnT5qyZcua7du3m19++cXUqFHDrFy50pw8edLUrFnTpKWlFen4aWlpxtvb28TGxpp9+/aZ+fPnm6CgIKcp7o8ePWomTpxoduzYYVJSUsz06dONp6en2bx5szHGmFOnTpmIiAgzYMAAx/f70qVLJjc31wwfPtx8//335tChQ2b+/PnGz8/PLFy40KmGK1PJHz582KXvXUkhYABAIbp27Wr69u3r1DZs2DBTo0YNk5ubW+A2NWvWNB9//LFT25gxY0xERIQx5n8BY/bs2Y71e/fuNZJMUlKSMcaYOXPmmMDAwHz7rl69uunWrdt16544caJp1qyZ4/W1AsbVzp8/b1q1amUefPBBY7fbjTHG9OvXzzz99NNO/davX288PDzM+fPnTXJyspFktmzZ4lh/5RfhzQgYtWrVMuPGjXNqW7ZsmZFkzp07Z4wxZsmSJaZBgwamZs2aZsSIEcYYY/r27WumTJli1q1bZ8LDw82dd95pPv3000KPP2zYMFO/fn2ntpdfftkpYBSkc+fOZsiQIY7X7dq1yxeICvL3v//dPProo05tWVlZRpJZu3btdbf/I2A2VQAoxPnz5x0zfl6RmJioNm3a5Ju2XJLOnj2rlJQU9evXTwMGDHC0X7p0SYGBgU59GzVq5Ph35cqVJV3+qKFu3brXrOnKBHpXW7hwoaZPn66UlBSdOXNGly5duqExBX379tXp06e1evVqx5w+O3fu1K5du5w+9jDGKC8vT4cPH9b+/ftVqlQpp3l+6tate80xHuvXr1fHjh0dr999991inUX44YcfdszmLEnr1q3Trl279NZbbyksLEyffPKJgoOD1bJlS7Vt29ZpLpIrkpKS8k1BHxER4fTabrdr3LhxWrRokX766Sfl5uYqJydHfn5+161xxowZio+PV1pams6fP6/c3FynO38kOSZhO3fuXFG/9BJFwACAQlSoUEG//vqrU9vVM23+1pkzZyRJ7733Xr5fRp6enk6vrw4oNptNkoo0y3CZMmWcXm/atEm9evXSqFGjFBUVpcDAQC1YsMDlmVDfeOMNrVq1Slu2bFHZsmWdvqZnnnlGgwcPzrdNtWrVtH//fpeOI10OSVdPBf97Jg0LDg5WZmamU1tmZqYCAgIK/L/KycnRwIEDNW/ePB08eFCXLl1Su3btJEm1a9fW5s2b1aVLlxuqZeLEiZo2bZqmTp2qhg0bqkyZMnrxxRevOxh3wYIFGjp0qN58801FRESobNmymjhxojZv3uzU7+TJk5KkihUr3lB9NxsBAwAK0aRJE82fP9+prVGjRvrggw908eLFfFcxgoKCVKVKFR06dOh3/UXu5eUlu91epL4bN25U9erV9eqrrzrafvzxR5eOt3jxYo0ePVorVqxQzZo1ndY1bdpUP/zwg8LCwgrctm7durp06ZK2bdumFi1aSJKSk5Ov+WwIX1/fQvfnqoiICC1fvtypbfXq1fmuLlzxxhtvqEOHDmratKl27NihS5cuOdZdvHix0O97vXr1tHTpUqe2//73v06vN2zYoK5du+qpp56SdDkw7t+/X/Xr13f0Kej/dsOGDWrdurUGDhzoaLt60PAVe/bsUenSpXXnnXcWWOMfDXeRAEAhoqKitHfvXqerGIMGDVJ2draeeOIJbd26VQcOHNC8efOUnJwsSRo1apTi4uI0ffp07d+/X7t379acOXM0efLkIh83NDRUZ86cUUJCgk6cOHHNS+K1atVSWlqaFixYoJSUFE2fPl2ff/55kY+1Z88eRUdH6+WXX9add96pjIwMZWRkOP5afvnll7Vx40YNGjRIiYmJOnDggL744gsNGjRIklSnTh116NBBzzzzjDZv3qxt27apf//+17zS44offvhBiYmJOnnypLKyspSYmOh09ePZZ5/VoUOH9M9//lP79u3TO++8o0WLFukf//hHgftauHChRo8eLelyOPLw8ND777+vZcuWad++fY6Q9FvPPvusDhw4oJdeeknJycn6+OOP8z1vo1atWlq9erU2btyopKQkPfPMM/muroSGhmrz5s1KTU3ViRMnlJeXp1q1amnr1q1atWqV9u/fr9dff13ff/99vhrWr1/vuJvHLZT0IBAA+CNr2bKlmTVrllPbzp07Tfv27Y2fn58pW7asadOmjUlJSXGs/+ijj0x4eLjx8vIy5cuXN23btjVLliwxxvxvkOeOHTsc/X/99VcjyaxZs8bR9uyzz5rbb7/dSHIMTKxevXqBAydfeuklc/vttxt/f3/To0cPM2XKFKdBotca5DlnzhwjKd/Srl07R/8tW7aYBx54wPj7+5syZcqYRo0ambFjxzrWp6enm86dOxtvb29TrVo18+GHHxZaq6uqV69eYH1XW7NmjeP7fccdd5g5c+bk209eXp65++67zZdffunU/uWXX5pq1aqZoKAg8957712zli+//NKEhYUZb29v06ZNGxMfH+80yPOXX34xXbt2Nf7+/qZSpUrmtddeM9HR0U6DU5OTk81dd91lfH19HXeEXLhwwfTp08cEBgaacuXKmeeee8688sorTv9nxhhTp04d88knnxT5e1fSbMYYUwK5BgDcwrJly/TSSy9pz549joGPwM22YsUKDRkyRLt27VKpUu4xusE9qgSAEtK5c2cdOHBAP/30k0JCQkq6HPxJnT17VnPmzHGbcCFJXMEAAACW43ofAACwHAEDAABYjoABAAAsR8AAAACWI2AAAADLETAAAIDlCBgAAMByBAwAAGA5AgYAALDc/wew7oxNdLpLOQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 600x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_results(results=['7b/pretrained', '7b/cen_10'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2568a9f",
   "metadata": {},
   "source": [
    "#### Extra! \n",
    "\n",
    "#### Generate the data structure for systematic evaluation\n",
    "\n",
    "We used this code to evaluate the performance of the finetuned 7B LLM you tested in the previous code cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8128e02b",
   "metadata": {
    "height": 334
   },
   "outputs": [],
   "source": [
    "# Lauch evalution code\n",
    "# from utils.utils import inference, evaluate\n",
    "\n",
    "# Step 1: generate answers\n",
    "# To run inference on the pre-trained model:\n",
    "# inference(base_model_name_path=cfg.model.name, run_name=\"pretrained\")\n",
    "\n",
    "# To run inference on the centralised finetuned model:\n",
    "# inference(\n",
    "#           base_model_name_path=cfg.model.name, \n",
    "#           peft_path=path/to/your/checkpoint/directory, \n",
    "#           run_name=\"centralised_finetuned\",\n",
    "# )\n",
    "\n",
    "\n",
    "# Step 2: evaluation --- accuracy value will be printed\n",
    "# evaluate(run_name=\"pretrained\")\n",
    "# evaluate(run_name=\"centralised_finetuned\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}