Ingest data into Pinecone.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **How to ingest data to Pinecone using LangChain**\n",
    "\n",
    "- This notebook shows you how you can ingest data from a document into a Pinecone Database.\n",
    "- It uses Langchain to ease the process of split and ingest the data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **1. Import dependencies**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "e:\\Coding\\ai\\building-qa-app-with-openai-pinecone-and-streamlit\\.venv\\lib\\site-packages\\pinecone\\index.py:4: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from tqdm.autonotebook import tqdm\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import pinecone\n",
    "from langchain.document_loaders import UnstructuredPDFLoader\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
    "from langchain.vectorstores import Pinecone\n",
    "from langchain.chat_models import AzureChatOpenAI\n",
    "from langchain.chains.question_answering import load_qa_chain"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **2. Read PineCone and Azure OpenAI environment variables**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "PINECONE_API_KEY = os.getenv('PINECONE_API_KEY')\n",
    "PINECONE_API_ENV = os.getenv('PINECONE_ENVIRONMENT')\n",
    "PINECONE_INDEX_NAME =  os.getenv('PINECONE_INDEX_NAME')\n",
    "AZURE_OPENAI_API_KEY = os.getenv('AZURE_OPENAI_APIKEY')\n",
    "AZURE_OPENAI_API_BASE = os.getenv('AZURE_OPENAI_BASE_URI')\n",
    "AZURE_OPENAI_API_TYPE = 'azure'\n",
    "AZURE_OPENAI_API_VERSION = '2023-03-15-preview'\n",
    "AZURE_OPENAI_EMBEDDINGS_MODEL_NAME= os.getenv('AZURE_OPENAI_EMBEDDINGS_MODEL_NAME')\n",
    "AZURE_OPENAI_GPT4_MODEL_NAME= os.getenv('AZURE_OPENAI_GPT4_MODEL_NAME')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **3. Load your data files and split it into chunks**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "detectron2 is not installed. Cannot use the hi_res partitioning strategy. Falling back to partitioning with the fast strategy.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You have a total of 917 chunks\n"
     ]
    }
   ],
   "source": [
    "loader = UnstructuredPDFLoader(\"./docs/NET-Microservices-Architecture-for-Containerized-NET-Applications.pdf\")\n",
    "data = loader.load()\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "chunks = text_splitter.split_documents(data)\n",
    "\n",
    "print (f'You have a total of {len(chunks)} chunks')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **4. Init Azure OpenAI and Pinecone clients**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = OpenAIEmbeddings(\n",
    "    openai_api_base=AZURE_OPENAI_API_BASE, \n",
    "    openai_api_key=AZURE_OPENAI_API_KEY, \n",
    "    openai_api_type=AZURE_OPENAI_API_TYPE,\n",
    "    model=AZURE_OPENAI_EMBEDDINGS_MODEL_NAME, \n",
    "    chunk_size=1)\n",
    "\n",
    "pinecone.init(\n",
    "    api_key=PINECONE_API_KEY,\n",
    "    environment=PINECONE_API_ENV  \n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **5. Create and store embeddings into Pinecone**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "docsearch = Pinecone.from_texts([t.page_content for t in chunks], embeddings, index_name=PINECONE_INDEX_NAME)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **6. Ask a question**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer: \n",
      "\n",
      " 1. Use asynchronous communication: Implement message-based communication across internal microservices to minimize ripple effects and enforce a higher level of microservice autonomy.\n",
      "\n",
      "2. Implement retries with exponential backoff: In case of transient faults, use the \"Retry pattern\" to retry the operation with increasing time intervals between attempts, allowing the system to recover.\n",
      "\n",
      "3. Use the Circuit Breaker pattern: Track the number of failed requests, and if the error rate exceeds a configured limit, trip the circuit breaker to prevent further attempts. After a timeout period, try again and close the circuit breaker if new requests are successful.\n"
     ]
    }
   ],
   "source": [
    "llm = AzureChatOpenAI(\n",
    "    temperature=0, \n",
    "    openai_api_base=AZURE_OPENAI_API_BASE, \n",
    "    openai_api_key=AZURE_OPENAI_API_KEY, \n",
    "    openai_api_version=AZURE_OPENAI_API_VERSION, \n",
    "    deployment_name=AZURE_OPENAI_GPT4_MODEL_NAME)\n",
    "\n",
    "chain = load_qa_chain(llm, chain_type=\"stuff\")\n",
    "\n",
    "query = \"Enumerate 3 strategies to handle partial failures?\"\n",
    "docs = docsearch.similarity_search(query, include_metadata=True)\n",
    "\n",
    "result = chain.run(input_documents=docs, question=query)\n",
    "\n",
    "print(f\"Answer: \\n\\n {result}\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}