02-LoadCSVOneToMany-ACogSearch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4077b4ee-73ea-4155-afbd-1de66dd6b650",
   "metadata": {},
   "source": [
    "# Load CSVs (one-to-many) to Azure Cognitive Search\n",
    "\n",
    "In this Jupyter Notebook, we create and run steps to index a CSV file in which each row is an individual and independent record/document. Each row then becomes searchable in Azure Cognitive Search. \n",
    "The reference documentation can be found at [Indexing blobs and files to produce multiple search documents](https://learn.microsoft.com/en-us/azure/search/search-howto-index-one-to-many-blobs).\n",
    "\n",
    "By default, an indexer will treat the contents of a blob or file as a single search document. If you want a more granular representation in a search index, you can set parsingMode values to create multiple search documents from one blob or file.\n",
    "\n",
    "We are going to be using the same private Blob Storage account but a different container that has abstracts of 90k Medical publications about COVID-19 published in 2020-2022. This file is a subset of a much bigger dataset (770k articles) called CORD19 [HERE](https://github.com/allenai/cord19)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c088c844-1e71-4279-a8fe-a77a007c15c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import requests\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"credentials.env\")\n",
    "\n",
    "# Name of the container in your Blob Storage Datasource ( in credentials.env)\n",
    "BLOB_CONTAINER_NAME = \"cord19\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c4908539-1d17-46a3-b9e0-dcc46a210c4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the names for the data source, index and indexer\n",
    "datasource_name = \"cogsrch-datasource-csv\"\n",
    "skillset_name = \"cogsrch-skillset-csv\"\n",
    "index_name = \"cogsrch-index-csv\"\n",
    "indexer_name = \"cogsrch-indexer-csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f2434379-070e-4110-8f5a-7d5bda9a0b7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup the Payloads header\n",
    "headers = {'Content-Type': 'application/json','api-key': os.environ['AZURE_SEARCH_KEY']}\n",
    "params = {'api-version': os.environ['AZURE_SEARCH_API_VERSION']}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecad0e75-e3c8-4147-b8c6-b938435bc8f5",
   "metadata": {},
   "source": [
    "## Create Data Source (Blob container with the Litcovid CSV data file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a9fa6c09-a489-4b6d-8c93-5fc26bae63a0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "204\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Create a data source\n",
    "\n",
    "datasource_payload = {\n",
    "    \"name\": datasource_name,\n",
    "    \"description\": \"Demo files to demonstrate cognitive search capabilities of one-to-many.\",\n",
    "    \"type\": \"azureblob\",\n",
    "    \"credentials\": {\n",
    "        \"connectionString\": os.environ['BLOB_CONNECTION_STRING']\n",
    "    },\n",
    "    \"container\": {\n",
    "        \"name\": BLOB_CONTAINER_NAME\n",
    "    }\n",
    "}\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/datasources/\" + datasource_name,\n",
    "                 data=json.dumps(datasource_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86b7ff86-19fc-48d3-88d1-b098e8d01302",
   "metadata": {},
   "source": [
    "## Inspect CSV file so we can understand the column types before creating the Index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cf6879a-a3da-4e54-97ed-f4122325abb1",
   "metadata": {},
   "source": [
    "In our private dataset we have place a smaller version of the original the metadata.csv file in the cord19 dataset. \n",
    "Let's see what the file looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2fbbbd0d-3015-4601-9ef1-7008ad168167",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Download the csv files to disk and inspect using pandas\n",
    "import pandas as pd\n",
    "remote_file_path = \"https://demodatasetsp.blob.core.windows.net/cord19/metadata.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "aaac918a-8859-45f5-9519-2cf56bfded88",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No. of lines: 90000\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_5ea2e\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_5ea2e_level0_col0\" class=\"col_heading level0 col0\" >cord_uid</th>\n",
       "      <th id=\"T_5ea2e_level0_col1\" class=\"col_heading level0 col1\" >source_x</th>\n",
       "      <th id=\"T_5ea2e_level0_col2\" class=\"col_heading level0 col2\" >title</th>\n",
       "      <th id=\"T_5ea2e_level0_col3\" class=\"col_heading level0 col3\" >abstract</th>\n",
       "      <th id=\"T_5ea2e_level0_col4\" class=\"col_heading level0 col4\" >authors</th>\n",
       "      <th id=\"T_5ea2e_level0_col5\" class=\"col_heading level0 col5\" >url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_5ea2e_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_5ea2e_row0_col0\" class=\"data row0 col0\" >ug7v899j</td>\n",
       "      <td id=\"T_5ea2e_row0_col1\" class=\"data row0 col1\" >PMC</td>\n",
       "      <td id=\"T_5ea2e_row0_col2\" class=\"data row0 col2\" >Clinical features of culture-p...</td>\n",
       "      <td id=\"T_5ea2e_row0_col3\" class=\"data row0 col3\" >OBJECTIVE: This retrospective ...</td>\n",
       "      <td id=\"T_5ea2e_row0_col4\" class=\"data row0 col4\" >Madani, Tariq A; Al-Ghamdi, Ai...</td>\n",
       "      <td id=\"T_5ea2e_row0_col5\" class=\"data row0 col5\" ><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC35282/\">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC35282/</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5ea2e_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_5ea2e_row1_col0\" class=\"data row1 col0\" >02tnwd4m</td>\n",
       "      <td id=\"T_5ea2e_row1_col1\" class=\"data row1 col1\" >PMC</td>\n",
       "      <td id=\"T_5ea2e_row1_col2\" class=\"data row1 col2\" >Nitric oxide: a pro-inflammato...</td>\n",
       "      <td id=\"T_5ea2e_row1_col3\" class=\"data row1 col3\" >Inflammatory diseases of the r...</td>\n",
       "      <td id=\"T_5ea2e_row1_col4\" class=\"data row1 col4\" >Vliet, Albert van der; Eiseric...</td>\n",
       "      <td id=\"T_5ea2e_row1_col5\" class=\"data row1 col5\" ><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC59543/\">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC59543/</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5ea2e_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_5ea2e_row2_col0\" class=\"data row2 col0\" >ejv2xln0</td>\n",
       "      <td id=\"T_5ea2e_row2_col1\" class=\"data row2 col1\" >PMC</td>\n",
       "      <td id=\"T_5ea2e_row2_col2\" class=\"data row2 col2\" >Surfactant protein-D and pulmo...</td>\n",
       "      <td id=\"T_5ea2e_row2_col3\" class=\"data row2 col3\" >Surfactant protein-D (SP-D) pa...</td>\n",
       "      <td id=\"T_5ea2e_row2_col4\" class=\"data row2 col4\" >Crouch, Erika C...</td>\n",
       "      <td id=\"T_5ea2e_row2_col5\" class=\"data row2 col5\" ><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC59549/\">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC59549/</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5ea2e_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_5ea2e_row3_col0\" class=\"data row3 col0\" >2b73a28n</td>\n",
       "      <td id=\"T_5ea2e_row3_col1\" class=\"data row3 col1\" >PMC</td>\n",
       "      <td id=\"T_5ea2e_row3_col2\" class=\"data row3 col2\" >Role of endothelin-1 in lung d...</td>\n",
       "      <td id=\"T_5ea2e_row3_col3\" class=\"data row3 col3\" >Endothelin-1 (ET-1) is a 21 am...</td>\n",
       "      <td id=\"T_5ea2e_row3_col4\" class=\"data row3 col4\" >Fagan, Karen A; McMurtry, Ivan...</td>\n",
       "      <td id=\"T_5ea2e_row3_col5\" class=\"data row3 col5\" ><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC59574/\">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC59574/</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5ea2e_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_5ea2e_row4_col0\" class=\"data row4 col0\" >9785vg6d</td>\n",
       "      <td id=\"T_5ea2e_row4_col1\" class=\"data row4 col1\" >PMC</td>\n",
       "      <td id=\"T_5ea2e_row4_col2\" class=\"data row4 col2\" >Gene expression in epithelial ...</td>\n",
       "      <td id=\"T_5ea2e_row4_col3\" class=\"data row4 col3\" >Respiratory syncytial virus (R...</td>\n",
       "      <td id=\"T_5ea2e_row4_col4\" class=\"data row4 col4\" >Domachowske, Joseph B; Bonvill...</td>\n",
       "      <td id=\"T_5ea2e_row4_col5\" class=\"data row4 col5\" ><a href=\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC59580/\">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC59580/</a></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f899eab9d50>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metadata = pd.read_csv(remote_file_path + os.environ['BLOB_SAS_TOKEN'])\n",
    "print(\"No. of lines:\",metadata.shape[0])\n",
    "\n",
    "simple_schema = ['cord_uid', 'source_x', 'title', 'abstract', 'authors', 'url']\n",
    "\n",
    "def make_clickable(address):\n",
    "    '''Make the url clickable'''\n",
    "    return '<a href=\"{0}\">{0}</a>'.format(address)\n",
    "\n",
    "def preview(text):\n",
    "    '''Show only a preview of the text data.'''\n",
    "    return text[:30] + '...'\n",
    "\n",
    "format_ = {'title': preview, 'abstract': preview, 'authors': preview, 'url': make_clickable}\n",
    "\n",
    "metadata[simple_schema].head().style.format(format_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0b3935d-8546-4756-95cd-7f4fcecb9836",
   "metadata": {},
   "source": [
    "## Create Skillset - Text Splitter, Language Detection\n",
    "We will use cognitive services enrichment for spliting the text of each content field into chunks (pages) and for language detection. We should always split the text since we don't know how big the content of each row might be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b46cfa90-28b4-4602-b6ff-743a3407fd72",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "201\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Create a skillset\n",
    "skillset_payload = {\n",
    "    \"name\": skillset_name,\n",
    "    \"description\": \"Splits Text and detect language\",\n",
    "    \"skills\":\n",
    "    [\n",
    "        {\n",
    "            \"@odata.type\": \"#Microsoft.Skills.Text.LanguageDetectionSkill\",\n",
    "            \"description\": \"If you have multilingual content, adding a language code is useful for filtering\",\n",
    "            \"context\": \"/document\",\n",
    "            \"inputs\": [\n",
    "                {\n",
    "                  \"name\": \"text\",\n",
    "                  \"source\": \"/document/abstract\"\n",
    "                }\n",
    "            ],\n",
    "            \"outputs\": [\n",
    "                {\n",
    "                  \"name\": \"languageCode\",\n",
    "                  \"targetName\": \"language\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"@odata.type\": \"#Microsoft.Skills.Text.SplitSkill\",\n",
    "            \"context\": \"/document\",\n",
    "            \"textSplitMode\": \"pages\",\n",
    "            \"maximumPageLength\": 5000, # 5000 is default\n",
    "            \"defaultLanguageCode\": \"en\",\n",
    "            \"inputs\": [\n",
    "                {\n",
    "                    \"name\": \"text\",\n",
    "                    \"source\": \"/document/abstract\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"languageCode\",\n",
    "                    \"source\": \"/document/language\"\n",
    "                }\n",
    "            ],\n",
    "            \"outputs\": [\n",
    "                {\n",
    "                    \"name\": \"textItems\",\n",
    "                    \"targetName\": \"pages\"\n",
    "                }\n",
    "            ]\n",
    "        }\n",
    "    ],\n",
    "    \"cognitiveServices\": {\n",
    "        \"@odata.type\": \"#Microsoft.Azure.Search.CognitiveServicesByKey\",\n",
    "        \"description\": os.environ['COG_SERVICES_NAME'],\n",
    "        \"key\": os.environ['COG_SERVICES_KEY']\n",
    "    }\n",
    "}\n",
    "\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/skillsets/\" + skillset_name,\n",
    "                 data=json.dumps(skillset_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a321916-cd14-4d34-837d-1d153edb1221",
   "metadata": {},
   "source": [
    "## Create the Index\n",
    "In Azure Cognitive Search, both blob indexers and file indexers support a delimitedText parsing mode for CSV files that treats each line in the CSV as a separate search document.\n",
    "\n",
    "### **Important**:\n",
    "As you can see below and from the prior Notebook, there are 7 mandatory fields in the schema: `id, title, content, chunks, language, name, location`. These fields must exist in any index that you create regardles of the datasource. Any additional fields are good to add so the engine can search relevant documents, however the mandatory fields must exist for all the code downstream work with no issues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5284b80c-9ba6-49d6-8109-5bfdbaa6ddc5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "201\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "index_payload = {\n",
    "    \"name\": index_name,  \n",
    "    \"fields\": [\n",
    "        {\"name\": \"id\", \"type\": \"Edm.String\", \"key\": \"true\", \"searchable\": \"false\", \"retrievable\": \"true\", \"facetable\": \"false\", \"filterable\": \"false\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"title\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"facetable\": \"false\", \"filterable\": \"true\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"content\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"facetable\": \"false\", \"filterable\": \"false\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"chunks\",\"type\": \"Collection(Edm.String)\", \"searchable\": \"false\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"language\", \"type\": \"Edm.String\", \"searchable\": \"false\", \"retrievable\": \"true\", \"sortable\": \"true\", \"filterable\": \"true\", \"facetable\": \"true\"},\n",
    "        {\"name\": \"name\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"location\", \"type\": \"Edm.String\", \"searchable\": \"false\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"vectorized\", \"type\": \"Edm.Boolean\", \"searchable\": \"false\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"authors\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"facetable\": \"false\", \"filterable\": \"false\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"metadata_storage_name\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"metadata_storage_path\", \"type\":\"Edm.String\", \"searchable\": \"false\", \"retrievable\": \"true\", \"filterable\": \"false\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"metadata_storage_last_modified\", \"type\":\"Edm.DateTimeOffset\", \"searchable\": \"false\", \"retrievable\": \"false\", \"filterable\": \"false\", \"sortable\": \"false\"}\n",
    "    ],\n",
    "    \"semantic\": {\n",
    "        \"configurations\": [\n",
    "            {\n",
    "                \"name\": \"my-semantic-config\",\n",
    "                \"prioritizedFields\": {\n",
    "                    \"titleField\": \n",
    "                        {\n",
    "                            \"fieldName\": \"title\"\n",
    "                        },\n",
    "                    \"prioritizedContentFields\": [\n",
    "                        { \n",
    "                            \"fieldName\":\"content\" \n",
    "                        }\n",
    "                    ]\n",
    "                }\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "}\n",
    "\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexes/\" + index_name,\n",
    "                 data=json.dumps(index_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51849738-6f66-452a-b7df-d34afd11f943",
   "metadata": {},
   "source": [
    "## Create and Run the Indexer - (runs the pipeline)\n",
    "To create one-to-many indexers with CSV blobs, create or update an indexer definition with the delimitedText parsing mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b87b8ebd-8091-43b6-9124-cc17021cfb78",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "201\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "indexer_payload = {\n",
    "    \"name\": indexer_name,\n",
    "    \"dataSourceName\": datasource_name,\n",
    "    \"targetIndexName\": index_name,\n",
    "    \"skillsetName\": skillset_name,\n",
    "    \"schedule\" : { \"interval\" : \"PT2H\"},\n",
    "    \"fieldMappings\": [\n",
    "        {\n",
    "          \"sourceFieldName\" : \"cord_uid\",\n",
    "          \"targetFieldName\" : \"id\"\n",
    "        },\n",
    "        {\n",
    "          \"sourceFieldName\" : \"abstract\",\n",
    "          \"targetFieldName\" : \"content\"\n",
    "        },\n",
    "        {\n",
    "          \"sourceFieldName\" : \"metadata_storage_name\",\n",
    "          \"targetFieldName\" : \"name\"\n",
    "        },\n",
    "        {\n",
    "          \"sourceFieldName\" : \"url\",\n",
    "          \"targetFieldName\" : \"location\"\n",
    "        }\n",
    "    ],\n",
    "    \"outputFieldMappings\":\n",
    "    [\n",
    "        {\n",
    "            \"sourceFieldName\": \"/document/language\",\n",
    "            \"targetFieldName\": \"language\"\n",
    "        },\n",
    "        {\n",
    "            \"sourceFieldName\": \"/document/pages/*\",\n",
    "            \"targetFieldName\": \"chunks\"\n",
    "        }\n",
    "    ],\n",
    "    \"parameters\" : { \n",
    "        \"configuration\" : { \n",
    "            \"dataToExtract\": \"contentAndMetadata\",\n",
    "            \"parsingMode\" : \"delimitedText\", \n",
    "            \"firstLineContainsHeaders\" : True,\n",
    "            \"delimitedTextDelimiter\": \",\"\n",
    "        } \n",
    "    }\n",
    "}\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexers/\" + indexer_name,\n",
    "                 data=json.dumps(indexer_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6132c041-7213-410e-a206-1a8c7385128e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\n",
      "Status: inProgress\n",
      "Items Processed: 20000\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Optionally, get indexer status to confirm that it's running\n",
    "r = requests.get(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexers/\" + indexer_name +\n",
    "                 \"/status\", headers=headers, params=params)\n",
    "# pprint(json.dumps(r.json(), indent=1))\n",
    "print(r.status_code)\n",
    "print(\"Status:\",r.json().get('lastResult').get('status'))\n",
    "print(\"Items Processed:\",r.json().get('lastResult').get('itemsProcessed'))\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2152806f-245c-45db-93c6-c19c0569d73a",
   "metadata": {},
   "source": [
    "**When the indexer finishes running we will have all 90,000 rows indexed properly as separate documents in our Search Engine!.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9d67bce-61be-47e4-bd1c-fdfda862f399",
   "metadata": {},
   "source": [
    "## Creation of its corresponding vector-based index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ec359823-3b9f-4b7f-af38-c3f2f916d5fa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "201\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "index_payload = {\n",
    "    \"name\": index_name + \"-vector\",\n",
    "    \"fields\": [\n",
    "        {\"name\": \"id\", \"type\": \"Edm.String\", \"key\": \"true\", \"filterable\": \"true\" },\n",
    "        {\"name\": \"title\",\"type\": \"Edm.String\",\"searchable\": \"true\",\"retrievable\": \"true\"},\n",
    "        {\"name\": \"chunk\",\"type\": \"Edm.String\",\"searchable\": \"true\",\"retrievable\": \"true\"},\n",
    "        {\"name\": \"chunkVector\",\"type\": \"Collection(Edm.Single)\",\"searchable\": \"true\",\"retrievable\": \"true\",\"dimensions\": 1536,\"vectorSearchConfiguration\": \"vectorConfig\"},\n",
    "        {\"name\": \"name\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"location\", \"type\": \"Edm.String\", \"searchable\": \"false\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "\n",
    "    ],\n",
    "    \"vectorSearch\": {\n",
    "        \"algorithmConfigurations\": [\n",
    "            {\n",
    "                \"name\": \"vectorConfig\",\n",
    "                \"kind\": \"hnsw\"\n",
    "            }\n",
    "        ]\n",
    "    },\n",
    "    \"semantic\": {\n",
    "        \"configurations\": [\n",
    "            {\n",
    "                \"name\": \"my-semantic-config\",\n",
    "                \"prioritizedFields\": {\n",
    "                    \"titleField\": {\n",
    "                        \"fieldName\": \"title\"\n",
    "                    },\n",
    "                    \"prioritizedContentFields\": [\n",
    "                        {\n",
    "                            \"fieldName\": \"chunk\"\n",
    "                        }\n",
    "                    ],\n",
    "                    \"prioritizedKeywordsFields\": []\n",
    "                }\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "}\n",
    "\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexes/\" + index_name + \"-vector\",\n",
    "                 data=json.dumps(index_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0eed6f22-437f-4a49-9b67-5fa2e7d066bf",
   "metadata": {},
   "source": [
    "# Reference\n",
    "\n",
    "- https://learn.microsoft.com/en-us/azure/search/search-howto-index-csv-blobs\n",
    "- https://learn.microsoft.com/en-us/azure/search/knowledge-store-create-rest\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d9f82a9-cb4c-44b9-b125-bc124ea23aa8",
   "metadata": {},
   "source": [
    "# NEXT\n",
    "Now that we have two separate text-based indexes loaded with two different types of information and its correspongind vector-based indexes, In the next notebook 3, we will do a Multi-Index query, sort the results based on the reranker semantic score of Azure Search, and then use OpenAI to understand this results and give the best answer possible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7505d8f9-39c7-4b87-a85f-283b6fea3de0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK v2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}