Lesson_1_Semantic_Search.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "77448887-ed71-48e1-bf0d-2ff499d0c7ca",
   "metadata": {},
   "source": [
    "# Lesson 1 - Semantic Search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c3ea165",
   "metadata": {},
   "source": [
    "Welcome to Lesson 1. \n",
    "\n",
    "To access the `requirement.txt` file, go to `File` and click on `Open`.\n",
    " \n",
    "I hope you enjoy this course!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "750cd7dd",
   "metadata": {},
   "source": [
    "### Import the Needed Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "809aa032-d737-450d-aafa-e32bfba9d8f8",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "30698fb9-4709-4088-9905-9ccb4efd5e09",
   "metadata": {
    "height": 166
   },
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "from sentence_transformers import SentenceTransformer\n",
    "from pinecone import Pinecone, ServerlessSpec\n",
    "from DLAIUtils import Utils\n",
    "import DLAIUtils\n",
    "\n",
    "import os\n",
    "import time\n",
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8ab484bb-3bfb-4c52-a5bd-bcbe4a7a63d2",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "from tqdm.auto import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba92fc2d",
   "metadata": {},
   "source": [
    "### Load the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ce859e4b-9b50-4f53-b357-28d3e3872c87",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5a88be3d59bd4ff8810f23a1a06b1d6c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading builder script:   0%|          | 0.00/2.38k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1cb95c01882e491198e3e36159dea90b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading readme:   0%|          | 0.00/5.69k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c59489b7f2b24a8aaf0d2b19a7c02b01",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8db912278c0543aaa0f7784e49f93e99",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/58.2M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "516e19f1ab884bcb9ccd8d6c83137bb6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Extracting data files:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9437ce60172c45508708e14d0e9ccc09",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train split:   0%|          | 0/404290 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dataset = load_dataset('quora', split='train[240000:290000]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "356d4112-fa51-4092-9841-8b266e3b6a2c",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'questions': [{'id': [207550, 351729],\n",
       "   'text': ['What is the truth of life?', \"What's the evil truth of life?\"]},\n",
       "  {'id': [33183, 351730],\n",
       "   'text': ['Which is the best smartphone under 20K in India?',\n",
       "    'Which is the best smartphone with in 20k in India?']},\n",
       "  {'id': [351731, 351732],\n",
       "   'text': ['Steps taken by Canadian government to improve literacy rate?',\n",
       "    'Can I send homemade herbal hair oil from India to US via postal or private courier services?']},\n",
       "  {'id': [37799, 94186],\n",
       "   'text': ['What is a good way to lose 30 pounds in 2 months?',\n",
       "    'What can I do to lose 30 pounds in 2 months?']},\n",
       "  {'id': [351733, 351734],\n",
       "   'text': ['Which of the following most accurately describes the translation of the graph y = (x+3)^2 -2 to the graph of y = (x -2)^2 +2?',\n",
       "    'How do you graph x + 2y = -2?']}],\n",
       " 'is_duplicate': [False, True, False, True, False]}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "332d1241-61ae-4d09-bf46-52081c133c0c",
   "metadata": {
    "height": 132
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What is the truth of life?\n",
      "What's the evil truth of life?\n",
      "Which is the best smartphone under 20K in India?\n",
      "Which is the best smartphone with in 20k in India?\n",
      "Steps taken by Canadian government to improve literacy rate?\n",
      "Can I send homemade herbal hair oil from India to US via postal or private courier services?\n",
      "What is a good way to lose 30 pounds in 2 months?\n",
      "What can I do to lose 30 pounds in 2 months?\n",
      "Which of the following most accurately describes the translation of the graph y = (x+3)^2 -2 to the graph of y = (x -2)^2 +2?\n",
      "How do you graph x + 2y = -2?\n",
      "--------------------------------------------------\n",
      "Number of questions: 100000\n"
     ]
    }
   ],
   "source": [
    "questions = []\n",
    "for record in dataset['questions']:\n",
    "    questions.extend(record['text'])\n",
    "question = list(set(questions))\n",
    "print('\\n'.join(questions[:10]))\n",
    "print('-' * 50)\n",
    "print(f'Number of questions: {len(questions)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab3c0402",
   "metadata": {},
   "source": [
    "### Check cuda and Setup the model\n",
    "\n",
    "**Note**: \"Checking cuda\" refers to checking if you have access to GPUs (faster compute). In this course, we are using CPUs. So, you might notice some code cells taking a little longer to run.\n",
    "\n",
    "We are using *all-MiniLM-L6-v2* sentence-transformers model that maps sentences to a 384 dimensional dense vector space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5fb67759-ab38-4472-bfb0-4a56d1c05955",
   "metadata": {
    "height": 81
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sorry no cuda.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e5fa1c72e05344539d0c25fed8ac0166",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       ".gitattributes:   0%|          | 0.00/1.23k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "380698c88b4c406f9dfd2212544543bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "1_Pooling/config.json:   0%|          | 0.00/190 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c165e7c55a3e4af7b577d8407704f01f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "README.md:   0%|          | 0.00/10.7k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4137c66273a94a11aac61ef656a36ca4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/612 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0476927f219f4688bfb47bb2c66d8da5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config_sentence_transformers.json:   0%|          | 0.00/116 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f6b22b5649144b79a8072b9d12bfda2e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "data_config.json:   0%|          | 0.00/39.3k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fff59ea8d6724b5f95350468d5550625",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/90.9M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6f510c16d90c4ff696099ac3393c385c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.onnx:   0%|          | 0.00/90.4M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "345a2497a2714b67ae945b0d802f0505",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "pytorch_model.bin:   0%|          | 0.00/90.9M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5a9d727b6e514c588116eebe1a4a2e31",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "sentence_bert_config.json:   0%|          | 0.00/53.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2310ac748474471a8b33f6ca284e2075",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/112 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e4158403227641abadf446cef097e161",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "100466c475e94881957d6197815ac37f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/350 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "72800cde35e54301ac2247302c1598f0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "train_script.py:   0%|          | 0.00/13.2k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1bb336c7e0e24852a2b0406711fc5725",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "946df34dc0524ae69bcfb73c523f3d54",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "modules.json:   0%|          | 0.00/349 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "if device != 'cuda':\n",
    "    print('Sorry no cuda.')\n",
    "model = SentenceTransformer('all-MiniLM-L6-v2', device=device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d90ec5ec-5397-4ed5-8163-7a901b6ecb0c",
   "metadata": {
    "height": 64
   },
   "outputs": [],
   "source": [
    "query = 'which city is the most populated in the world?'\n",
    "xq = model.encode(query)\n",
    "xq.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1780a189",
   "metadata": {},
   "source": [
    "### Setup Pinecone"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b3e3a94e-127f-4667-a9ae-7a17d7304ee6",
   "metadata": {
    "height": 43
   },
   "outputs": [],
   "source": [
    "utils = Utils()\n",
    "PINECONE_API_KEY = utils.get_pinecone_api_key()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "17a75eca-60f0-478b-bdcf-b68732c1545d",
   "metadata": {
    "height": 200
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dl-ai-g6gvsyaw-zgrtmdqzxwfxtqbjl5h7h1i7r8c\n",
      "<pinecone.data.index.Index object at 0x7f7e3a9a81f0>\n"
     ]
    }
   ],
   "source": [
    "pinecone = Pinecone(api_key=PINECONE_API_KEY)\n",
    "INDEX_NAME = utils.create_dlai_index_name('dl-ai')\n",
    "\n",
    "if INDEX_NAME in [index.name for index in pinecone.list_indexes()]:\n",
    "    pinecone.delete_index(INDEX_NAME)\n",
    "print(INDEX_NAME)\n",
    "pinecone.create_index(name=INDEX_NAME, \n",
    "    dimension=model.get_sentence_embedding_dimension(), \n",
    "    metric='cosine',\n",
    "    spec=ServerlessSpec(cloud='aws', region='us-west-2'))\n",
    "\n",
    "index = pinecone.Index(INDEX_NAME)\n",
    "print(index)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d88d9424",
   "metadata": {},
   "source": [
    "### Create Embeddings and Upsert to Pinecone"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ea551303-aa81-41cd-adc5-dc9ea8072397",
   "metadata": {
    "height": 352
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b522fcde226d490e9fbe34775ee876f8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/50 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "batch_size=200\n",
    "vector_limit=10000\n",
    "\n",
    "questions = question[:vector_limit]\n",
    "\n",
    "import json\n",
    "\n",
    "for i in tqdm(range(0, len(questions), batch_size)):\n",
    "    # find end of batch\n",
    "    i_end = min(i+batch_size, len(questions))\n",
    "    # create IDs batch\n",
    "    ids = [str(x) for x in range(i, i_end)]\n",
    "    # create metadata batch\n",
    "    metadatas = [{'text': text} for text in questions[i:i_end]]\n",
    "    # create embeddings\n",
    "    xc = model.encode(questions[i:i_end])\n",
    "    # create records list for upsert\n",
    "    records = zip(ids, xc, metadatas)\n",
    "    # upsert to Pinecone\n",
    "    index.upsert(vectors=records)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6153920a-f4c4-420e-9790-262dd2299fc6",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'dimension': 384,\n",
       " 'index_fullness': 0.0,\n",
       " 'namespaces': {'': {'vector_count': 10000}},\n",
       " 'total_vector_count': 10000}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index.describe_index_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71135418",
   "metadata": {},
   "source": [
    "### Run Your Query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "28b20b81-4782-4ce2-aec3-9576c7779f2e",
   "metadata": {
    "height": 115
   },
   "outputs": [],
   "source": [
    "# small helper function so we can repeat queries later\n",
    "def run_query(query):\n",
    "  embedding = model.encode(query).tolist()\n",
    "  results = index.query(top_k=10, vector=embedding, include_metadata=True, include_values=False)\n",
    "  for result in results['matches']:\n",
    "    print(f\"{round(result['score'], 2)}: {result['metadata']['text']}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4c4244d6-7be0-4ee1-a36a-0d586f0555f7",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7: What is the most isolated city in the world, with over a million metro area inhabitants?\n",
      "0.64: Which is the largest country in the world?\n",
      "0.64: Which city has the most museums per capita?\n",
      "0.62: How many cities are on Earth?\n",
      "0.61: What is the most modern city in China?\n",
      "0.58: What is the biggest country?\n",
      "0.56: What is best city in India?\n",
      "0.55: Which is the highest mountain in the world?\n",
      "0.55: About 50% of the world population is concentrated between the latitudes of?\n",
      "0.55: Which country is the largest democracy in the world?\n"
     ]
    }
   ],
   "source": [
    "run_query('which city has the highest population in the world?')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f85b20fe-2328-47eb-84cd-bf3b27c6d4aa",
   "metadata": {
    "height": 47
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.87: How do I make cake?\n",
      "0.77: What are some ways to add chocolate chips to a cake mix?\n",
      "0.58: What's a good recipe for cake featuring Ciroc?\n",
      "0.55: How do you make a pumpkin pie?\n",
      "0.53: What is the red velvet cake frosting made of?\n",
      "0.49: How do I make art?\n",
      "0.49: Are You Looking For Tasty Chocolates in Bangalore?\n",
      "0.45: How do you make scrambled eggs without milk?\n",
      "0.44: What are some favorite recipes?\n",
      "0.43: Where can I get highest quality, tastiest cupcakes across the Gold Coast?\n"
     ]
    }
   ],
   "source": [
    "query = 'how do i make chocolate cake?'\n",
    "run_query(query)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}