01-Load-Data-ACogSearch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load and Enrich multiple file types Azure Cognitive Search\n",
    "\n",
    "In this Jupyter Notebook, we create and run enrichment steps to unlock searchable content in the specified Azure blob. It performs operations over mixed content in Azure Storage, such as images and application files, using a skillset that analyzes and extracts text information that becomes searchable in Azure Cognitive Search. \n",
    "The reference sample can be found at [Tutorial: Use Python and AI to generate searchable content from Azure blobs](https://docs.microsoft.com/azure/search/cognitive-search-tutorial-blob-python).\n",
    "\n",
    "In this demo we are going to be using a private (so we can mimic a private data lake scenario) Blob Storage container that has ~9.8k Computer Science publication PDFs from the Arxiv dataset.\n",
    "https://www.kaggle.com/datasets/Cornell-University/arxiv\n",
    "\n",
    "If you want to explore the dataset, go [HERE](https://console.cloud.google.com/storage/browser/arxiv-dataset/arxiv/cs/pdf?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&prefix=&forceOnObjectsSortingFiltering=false)<br>\n",
    "Note: This dataset has been copy to a public azure blob container for this demo\n",
    "\n",
    "Although only  PDF files are used here, this can be done at a much larger scale and Azure Cognitive Search supports a range of other file formats including: Microsoft Office (DOCX/DOC, XSLX/XLS, PPTX/PPT, MSG), HTML, XML, ZIP, and plain text files (including JSON).\n",
    "Azure Search support the following sources: [Data Sources Gallery](https://learn.microsoft.com/EN-US/AZURE/search/search-data-sources-gallery)\n",
    "\n",
    "This notebook creates the following objects on your search service:\n",
    "\n",
    "+ data source\n",
    "+ skillset\n",
    "+ search index\n",
    "+ indexer\n",
    "\n",
    "This notebook calls the [Search REST APIs](https://docs.microsoft.com/rest/api/searchservice/), but you can also use the Azure.Search.Documents client library in the Azure SDK for Python to perform the same steps. See this [Python quickstart](https://docs.microsoft.com/azure/search/search-get-started-python) for details.\n",
    "\n",
    "To run this notebook, you should have already created the Azure services on README. Once you've done this, you can run all cells, but the query won't return results until the indexer is finished and the search index is loaded. \n",
    "\n",
    "We recommend running each step and making sure it completes before moving on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![cog-search](./images/Cog-Search-Enrich.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import requests\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv(\"credentials.env\")\n",
    "\n",
    "# Name of the container in your Blob Storage Datasource ( in credentials.env)\n",
    "BLOB_CONTAINER_NAME = \"arxivcs\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the names for the data source, skillset, index and indexer\n",
    "datasource_name = \"cogsrch-datasource-files\"\n",
    "skillset_name = \"cogsrch-skillset-files\"\n",
    "index_name = \"cogsrch-index-files\"\n",
    "indexer_name = \"cogsrch-indexer-files\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup the Payloads header\n",
    "headers = {'Content-Type': 'application/json','api-key': os.environ['AZURE_SEARCH_KEY']}\n",
    "params = {'api-version': os.environ['AZURE_SEARCH_API_VERSION']}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Data Source (Blob container with the Arxiv CS pdfs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "204\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# The following code sends the json paylod to Azure Search engine to create the Datasource\n",
    "\n",
    "datasource_payload = {\n",
    "    \"name\": datasource_name,\n",
    "    \"description\": \"Demo files to demonstrate cognitive search capabilities.\",\n",
    "    \"type\": \"azureblob\",\n",
    "    \"credentials\": {\n",
    "        \"connectionString\": os.environ['BLOB_CONNECTION_STRING']\n",
    "    },\n",
    "    \"dataDeletionDetectionPolicy\" : {\n",
    "        \"@odata.type\" :\"#Microsoft.Azure.Search.NativeBlobSoftDeleteDeletionDetectionPolicy\" # this makes sure that if the item is deleted from the source, it will be deleted from the index\n",
    "    },\n",
    "    \"container\": {\n",
    "        \"name\": BLOB_CONTAINER_NAME\n",
    "    }\n",
    "}\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/datasources/\" + datasource_name,\n",
    "                 data=json.dumps(datasource_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 201 - Successfully created\n",
    "- 204 - Succesfully overwritten\n",
    "- 40X - Authentication Error\n",
    "\n",
    "For information on Change and Delete file detection please see [HERE](https://learn.microsoft.com/en-us/azure/search/search-howto-index-changed-deleted-blobs?tabs=rest-api)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you have a 403 code, probably you have a wrong endpoint or key, you can debug by uncomment this\n",
    "# r.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Skillset - OCR, Text Splitter, Language Detection, KeyPhrase extraction, Entity Recognition\n",
    "\n",
    "We need to create now the skillset. This is a set of steps in which we use many Cognitive Services to enrich the documents by extracting information, applying OCR, translating, etc.\n",
    "\n",
    "https://learn.microsoft.com/en-us/azure/search/cognitive-search-working-with-skillsets\n",
    "\n",
    "https://learn.microsoft.com/en-us/azure/search/cognitive-search-predefined-skills\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "204\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Create a skillset\n",
    "skillset_payload = {\n",
    "    \"name\": skillset_name,\n",
    "    \"description\": \"Extract entities, detect language and extract key-phrases\",\n",
    "    \"skills\":\n",
    "    [\n",
    "        {\n",
    "            \"@odata.type\": \"#Microsoft.Skills.Vision.OcrSkill\",\n",
    "            \"description\": \"Extract text (plain and structured) from image.\",\n",
    "            \"context\": \"/document/normalized_images/*\",\n",
    "            \"defaultLanguageCode\": \"en\",\n",
    "            \"detectOrientation\": True,\n",
    "            \"inputs\": [\n",
    "                {\n",
    "                  \"name\": \"image\",\n",
    "                  \"source\": \"/document/normalized_images/*\"\n",
    "                }\n",
    "            ],\n",
    "                \"outputs\": [\n",
    "                {\n",
    "                  \"name\": \"text\",\n",
    "                  \"targetName\" : \"images_text\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"@odata.type\": \"#Microsoft.Skills.Text.MergeSkill\",\n",
    "            \"description\": \"Create merged_text, which includes all the textual representation of each image inserted at the right location in the content field. This is useful for PDF and other file formats that supported embedded images.\",\n",
    "            \"context\": \"/document\",\n",
    "            \"insertPreTag\": \" \",\n",
    "            \"insertPostTag\": \" \",\n",
    "            \"inputs\": [\n",
    "                {\n",
    "                  \"name\":\"text\", \"source\": \"/document/content\"\n",
    "                },\n",
    "                {\n",
    "                  \"name\": \"itemsToInsert\", \"source\": \"/document/normalized_images/*/images_text\"\n",
    "                },\n",
    "                {\n",
    "                  \"name\":\"offsets\", \"source\": \"/document/normalized_images/*/contentOffset\"\n",
    "                }\n",
    "            ],\n",
    "            \"outputs\": [\n",
    "                {\n",
    "                  \"name\": \"mergedText\", \n",
    "                  \"targetName\" : \"merged_text\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"@odata.type\": \"#Microsoft.Skills.Text.LanguageDetectionSkill\",\n",
    "            \"context\": \"/document\",\n",
    "            \"description\": \"If you have multilingual content, adding a language code is useful for filtering\",\n",
    "            \"inputs\": [\n",
    "                {\n",
    "                  \"name\": \"text\",\n",
    "                  \"source\": \"/document/content\"\n",
    "                }\n",
    "            ],\n",
    "            \"outputs\": [\n",
    "                {\n",
    "                  \"name\": \"languageCode\",\n",
    "                  \"targetName\": \"language\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"@odata.type\": \"#Microsoft.Skills.Text.SplitSkill\",\n",
    "            \"context\": \"/document\",\n",
    "            \"textSplitMode\": \"pages\",\n",
    "            \"maximumPageLength\": 5000, # 5000 is default\n",
    "            \"defaultLanguageCode\": \"en\",\n",
    "            \"inputs\": [\n",
    "                {\n",
    "                    \"name\": \"text\",\n",
    "                    \"source\": \"/document/merged_text\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"languageCode\",\n",
    "                    \"source\": \"/document/language\"\n",
    "                }\n",
    "            ],\n",
    "            \"outputs\": [\n",
    "                {\n",
    "                    \"name\": \"textItems\",\n",
    "                    \"targetName\": \"pages\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"@odata.type\": \"#Microsoft.Skills.Text.KeyPhraseExtractionSkill\",\n",
    "            \"context\": \"/document/pages/*\",\n",
    "            \"maxKeyPhraseCount\": 2,\n",
    "            \"defaultLanguageCode\": \"en\",\n",
    "            \"inputs\": [\n",
    "                {\n",
    "                    \"name\": \"text\", \n",
    "                    \"source\": \"/document/pages/*\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"languageCode\",\n",
    "                    \"source\": \"/document/language\"\n",
    "                }\n",
    "            ],\n",
    "            \"outputs\": [\n",
    "                {\n",
    "                    \"name\": \"keyPhrases\",\n",
    "                    \"targetName\": \"keyPhrases\"\n",
    "                }\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"@odata.type\": \"#Microsoft.Skills.Text.V3.EntityRecognitionSkill\",\n",
    "            \"context\": \"/document/pages/*\",\n",
    "            \"categories\": [\"Person\", \"URL\", \"Email\"],\n",
    "            \"minimumPrecision\": 0.5, \n",
    "            \"defaultLanguageCode\": \"en\",\n",
    "            \"inputs\": [\n",
    "                {\n",
    "                    \"name\": \"text\", \n",
    "                    \"source\":\"/document/pages/*\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"languageCode\",\n",
    "                    \"source\": \"/document/language\"\n",
    "                }\n",
    "            ],\n",
    "            \"outputs\": [\n",
    "                {\n",
    "                    \"name\": \"persons\", \n",
    "                    \"targetName\": \"persons\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"urls\", \n",
    "                    \"targetName\": \"urls\"\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"emails\", \n",
    "                    \"targetName\": \"emails\"\n",
    "                }\n",
    "            ]\n",
    "        }\n",
    "    ],\n",
    "    \"cognitiveServices\": {\n",
    "        \"@odata.type\": \"#Microsoft.Azure.Search.CognitiveServicesByKey\",\n",
    "        \"description\": os.environ['COG_SERVICES_NAME'],\n",
    "        \"key\": os.environ['COG_SERVICES_KEY']\n",
    "    }\n",
    "}\n",
    "\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/skillsets/\" + skillset_name,\n",
    "                 data=json.dumps(skillset_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Azure Cognitive Search, a search index is your searchable content, available to the search engine for indexing, full text search, and filtered queries. An index is defined by a schema and saved to the search service. This content exists within your search service, apart from your primary data stores, which is necessary for the millisecond response times expected in modern applications. Except for specific indexing scenarios, the search service will never connect to or query your local data.\n",
    "\n",
    "The body of the request defines the schema of the search index. A fields collection requires one field to be designated as the key. For blob type, this field is often the \"metadata_storage_path\" that uniquely identifies each file in the container.\n",
    "\n",
    "Reference:\n",
    "\n",
    "https://learn.microsoft.com/en-us/azure/search/search-what-is-an-index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "204\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Create an index\n",
    "# Queries operate over the searchable fields and filterable fields in the index\n",
    "index_payload = {\n",
    "    \"name\": index_name,\n",
    "    \"fields\": [\n",
    "        {\"name\": \"id\", \"type\": \"Edm.String\", \"key\": \"true\", \"searchable\": \"false\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\",\"facetable\": \"false\"},\n",
    "        {\"name\": \"title\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"facetable\": \"false\", \"filterable\": \"true\", \"sortable\": \"false\"},\n",
    "        {\"name\": \"content\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\",\"facetable\": \"false\"},\n",
    "        {\"name\": \"chunks\",\"type\": \"Collection(Edm.String)\", \"searchable\": \"false\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"language\", \"type\": \"Edm.String\", \"searchable\": \"false\", \"retrievable\": \"true\", \"sortable\": \"true\", \"filterable\": \"true\", \"facetable\": \"true\"},\n",
    "        {\"name\": \"name\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"location\", \"type\": \"Edm.String\", \"searchable\": \"false\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"vectorized\", \"type\": \"Edm.Boolean\", \"searchable\": \"false\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"images_text\", \"type\": \"Collection(Edm.String)\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"keyPhrases\", \"type\": \"Collection(Edm.String)\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"true\", \"facetable\": \"true\"},\n",
    "        {\"name\": \"persons\", \"type\": \"Collection(Edm.String)\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"urls\", \"type\": \"Collection(Edm.String)\", \"searchable\": \"false\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"emails\", \"type\": \"Collection(Edm.String)\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"true\", \"facetable\": \"false\"}\n",
    "        \n",
    "    ],\n",
    "    \"semantic\": {\n",
    "      \"configurations\": [\n",
    "        {\n",
    "          \"name\": \"my-semantic-config\",\n",
    "          \"prioritizedFields\": {\n",
    "            \"titleField\": \n",
    "                {\n",
    "                    \"fieldName\": \"title\"\n",
    "                },\n",
    "            \"prioritizedContentFields\": [\n",
    "                {\n",
    "                    \"fieldName\": \"content\"\n",
    "                }\n",
    "                ]\n",
    "          }\n",
    "        }\n",
    "      ]\n",
    "    }\n",
    "}\n",
    "\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexes/\" + index_name,\n",
    "                 data=json.dumps(index_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### Semantic Search capabilities\n",
    "As you can see above in the index payload, there is a `semantic configuration`. What is that?\n",
    "\n",
    "Azure Search has a feature called: Semantic Search. This is a Deep Neural Network that lives on the engine that tries to find results based on the semantic meaning of the query and the content, not keyword mathching/counting. \n",
    "From the [official documentation](https://learn.microsoft.com/en-us/azure/search/semantic-search-overview):\n",
    "\n",
    "Semantic search is a collection of features that improve the quality of initial search results for text-based queries. When you enable it on your search service, semantic search extends the query execution pipeline in two ways:\n",
    "\n",
    "- First, it adds secondary ranking over an initial result set, promoting the most semantically relevant results to the top of the list.\n",
    "\n",
    "- Second, it extracts and returns captions and answers in the response, which you can render on a search page to improve the user's search experience.\n",
    "\n",
    "For deeper explanation and limitations see [HERE](https://learn.microsoft.com/en-us/azure/search/semantic-ranking)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create and Run the Indexer - (runs the pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The three components you have created thus far (data source, skillset, index) are inputs to an indexer. Creating the indexer on Azure Cognitive Search is the event that puts the entire pipeline into motion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "201\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Create an indexer\n",
    "indexer_payload = {\n",
    "    \"name\": indexer_name,\n",
    "    \"dataSourceName\": datasource_name,\n",
    "    \"targetIndexName\": index_name,\n",
    "    \"skillsetName\": skillset_name,\n",
    "    \"schedule\" : { \"interval\" : \"PT2H\"}, # How often do you want to check for new content in the data source\n",
    "    \"fieldMappings\": [\n",
    "        {\n",
    "          \"sourceFieldName\" : \"metadata_storage_path\",\n",
    "          \"targetFieldName\" : \"id\",\n",
    "          \"mappingFunction\" : { \"name\" : \"base64Encode\" }\n",
    "        },\n",
    "        {\n",
    "          \"sourceFieldName\" : \"metadata_title\",\n",
    "          \"targetFieldName\" : \"title\"\n",
    "        },\n",
    "        {\n",
    "          \"sourceFieldName\" : \"metadata_storage_name\",\n",
    "          \"targetFieldName\" : \"name\"\n",
    "        },\n",
    "        {\n",
    "          \"sourceFieldName\" : \"metadata_storage_path\",\n",
    "          \"targetFieldName\" : \"location\"\n",
    "        }\n",
    "    ],\n",
    "    \"outputFieldMappings\":\n",
    "    [\n",
    "        {\n",
    "            \"sourceFieldName\": \"/document/merged_text\",\n",
    "            \"targetFieldName\": \"content\"\n",
    "        },\n",
    "        {\n",
    "            \"sourceFieldName\": \"/document/pages/*\",\n",
    "            \"targetFieldName\": \"chunks\"\n",
    "        },\n",
    "        {\n",
    "            \"sourceFieldName\" : \"/document/normalized_images/*/images_text\",\n",
    "            \"targetFieldName\" : \"images_text\"\n",
    "        },\n",
    "        {\n",
    "            \"sourceFieldName\": \"/document/language\",\n",
    "            \"targetFieldName\": \"language\"\n",
    "        },\n",
    "        {\n",
    "            \"sourceFieldName\": \"/document/pages/*/keyPhrases/*\",\n",
    "            \"targetFieldName\": \"keyPhrases\"\n",
    "        },\n",
    "        {\n",
    "          \"sourceFieldName\" : \"/document/pages/*/persons/*\", \n",
    "          \"targetFieldName\" : \"persons\"\n",
    "        },\n",
    "        {\n",
    "            \"sourceFieldName\": \"/document/pages/*/urls/*\",\n",
    "            \"targetFieldName\": \"urls\"\n",
    "        },\n",
    "        {\n",
    "            \"sourceFieldName\": \"/document/pages/*/emails/*\",\n",
    "            \"targetFieldName\": \"emails\"\n",
    "        }\n",
    "    ],\n",
    "    \"parameters\":\n",
    "    {\n",
    "        \"maxFailedItems\": -1,\n",
    "        \"maxFailedItemsPerBatch\": -1,\n",
    "        \"configuration\":\n",
    "        {\n",
    "            \"dataToExtract\": \"contentAndMetadata\",\n",
    "            \"imageAction\": \"generateNormalizedImages\"\n",
    "        }\n",
    "    }\n",
    "}\n",
    "\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexers/\" + indexer_name,\n",
    "                 data=json.dumps(indexer_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment if you find an error\n",
    "# r.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: If you get a 400 unauthorize error, make sure that you are using the Azure Search MANAGEMENT KEY, not the QUERY key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\n",
      "Status: inProgress\n",
      "Items Processed: 990\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Optionally, get indexer status to confirm that it's running\n",
    "r = requests.get(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexers/\" + indexer_name +\n",
    "                 \"/status\", headers=headers, params=params)\n",
    "# pprint(json.dumps(r.json(), indent=1))\n",
    "print(r.status_code)\n",
    "print(\"Status:\",r.json().get('lastResult').get('status'))\n",
    "print(\"Items Processed:\",r.json().get('lastResult').get('itemsProcessed'))\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**When the indexer finishes running we will have all 9.8k documents indexed in your Search Engine!.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creation of its corresponding vector-based index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Azure Cognitive Search has now vector search capabilities** ([Watch this video](https://aka.ms/Vector_SearchSnackableVideo)). The advantages of vector search in Azure Cognitive Search include its integration with other capabilities of Azure Cognitive Search, the ability to use any type of data (text, image, audio, video, etc) from diverse Azure datastores to inform a single generative AI-powered application, and the support of vector fields in the search indexes. It also offers pure vector search, hybrid retrieval, and a sophisticated re-ranking system powered by Bing in a single integrated solution (check the release [blog site](https://techcommunity.microsoft.com/t5/azure-ai-services-blog/announcing-vector-search-in-azure-cognitive-search-public/ba-p/3872868)).\n",
    "\n",
    "\n",
    "![vector-search](https://techcommunity.microsoft.com/t5/image/serverpage/image-id/489211i001E2B9B34F483C2/image-dimensions/876x416?v=v2)\n",
    "\n",
    "\n",
    "**The main limitations (for now) of vector search in Azure Cognitive Search are:**\n",
    "\n",
    "- It does not generate vector embeddings for the content. Users need to provide the embeddings themselves by using a service such as Azure OpenAI.\n",
    "- There is not field type for Collection of vectors, meaning that each document in the vector-based index must be either a small document or a chunk of a bigger document.\n",
    "\n",
    "We are going to come back to these limitations and solve them in the next notebooks, but for now let's just create our corresponding vector-based index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "201\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "index_payload = {\n",
    "    \"name\": index_name + \"-vector\",\n",
    "    \"fields\": [\n",
    "        {\"name\": \"id\", \"type\": \"Edm.String\", \"key\": \"true\", \"filterable\": \"true\" },\n",
    "        {\"name\": \"title\",\"type\": \"Edm.String\",\"searchable\": \"true\",\"retrievable\": \"true\"},\n",
    "        {\"name\": \"chunk\",\"type\": \"Edm.String\",\"searchable\": \"true\",\"retrievable\": \"true\"},\n",
    "        {\"name\": \"chunkVector\",\"type\": \"Collection(Edm.Single)\",\"searchable\": \"true\",\"retrievable\": \"true\",\"dimensions\": 1536,\"vectorSearchConfiguration\": \"vectorConfig\"},\n",
    "        {\"name\": \"name\", \"type\": \"Edm.String\", \"searchable\": \"true\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "        {\"name\": \"location\", \"type\": \"Edm.String\", \"searchable\": \"false\", \"retrievable\": \"true\", \"sortable\": \"false\", \"filterable\": \"false\", \"facetable\": \"false\"},\n",
    "\n",
    "    ],\n",
    "    \"vectorSearch\": {\n",
    "        \"algorithmConfigurations\": [\n",
    "            {\n",
    "                \"name\": \"vectorConfig\",\n",
    "                \"kind\": \"hnsw\"\n",
    "            }\n",
    "        ]\n",
    "    },\n",
    "    \"semantic\": {\n",
    "        \"configurations\": [\n",
    "            {\n",
    "                \"name\": \"my-semantic-config\",\n",
    "                \"prioritizedFields\": {\n",
    "                    \"titleField\": {\n",
    "                        \"fieldName\": \"title\"\n",
    "                    },\n",
    "                    \"prioritizedContentFields\": [\n",
    "                        {\n",
    "                            \"fieldName\": \"chunk\"\n",
    "                        }\n",
    "                    ],\n",
    "                    \"prioritizedKeywordsFields\": []\n",
    "                }\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "}\n",
    "\n",
    "r = requests.put(os.environ['AZURE_SEARCH_ENDPOINT'] + \"/indexes/\" + index_name + \"-vector\",\n",
    "                 data=json.dumps(index_payload), headers=headers, params=params)\n",
    "print(r.status_code)\n",
    "print(r.ok)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "- https://learn.microsoft.com/en-us/azure/search/cognitive-search-tutorial-blob\n",
    "- https://github.com/Azure-Samples/azure-search-python-samples/blob/main/Tutorial-AI-Enrichment/PythonTutorial-AzureSearch-AIEnrichment.ipynb\n",
    "- https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/search/azure-search-documents/samples\n",
    "- https://learn.microsoft.com/en-us/azure/search/search-get-started-python\n",
    "- https://github.com/Azure-Samples/azure-search-python-samples/blob/main/Tutorial-AI-Enrichment/PythonTutorial-AzureSearch-AIEnrichment.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NEXT\n",
    "In the next notebook 02, we will implement another type of indexing call One-to-Many, in which a single CSV or JSON file can be converted into multiple individual searchable documents in Azure Search. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10 - SDK v2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  },
  "vscode": {
   "interpreter": {
    "hash": "9ff083f0c83558f9261023d47a77b9b3eb892c62cdbe066d046abcad1a5edb5c"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}