tests/notebooks/TF4ces_Search_Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "97f9ea35-c5af-46fc-bc15-287fbc872494",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# TF4ces Search Engine\n",
    "## User Queries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2eac753c-7e23-4936-972c-932c493e7bf0",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Setup system paths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5ea39c34-c4d7-4584-b6aa-476766f5b23b",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import sys\n",
    "\n",
    "__WORKSPACE__ = Path.cwd().parent.parent\n",
    "sys.path.append(str(__WORKSPACE__))\n",
    "\n",
    "__WORKSPACE__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ef6f53d-ec8a-4374-afdd-29fd570d636c",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Native imports\n",
    "from copy import deepcopy\n",
    "from IPython.core.display import HTML\n",
    "\n",
    "# Third-party imports\n",
    "import pandas as pd\n",
    "\n",
    "# User imports\n",
    "from config.conf import __ALL_MODELS__\n",
    "from src.TF4ces_search_engine.data.data_gathering import DataGathering\n",
    "from src.main_ensemble import TF4cesSearchEnsemble\n",
    "from src.utils.ensemble_strategy import EnsembleStrategy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf5bd57f-75b0-47eb-a0e8-a33b8b6ffdcc",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Configs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a9c7873b-691f-4e35-bf4f-346d1cd340aa",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Model Names\n",
    "TFIDF_MODEL = __ALL_MODELS__[0]\n",
    "BM25_MODEL = __ALL_MODELS__[1]\n",
    "MPNET_MODEL = __ALL_MODELS__[2]\n",
    "ROBERTA_MODEL = __ALL_MODELS__[3]\n",
    "\n",
    "# Dataset Configs\n",
    "DATASET_NAME = 'lotte'\n",
    "DATASET_CATEGORY = 'lifestyle'\n",
    "SPLIT = 'test'\n",
    "\n",
    "# Path Configs\n",
    "TFIDF_MODEL_PATH = __WORKSPACE__ / \"models/tfidf/lotte/lifestyle/tfidf.v0.0.1.pkl\"\n",
    "MPNET_EMB_PATH = __WORKSPACE__ / \"dataset/embeddings_test/test_v0.0.1/all-mpnet-base-v2/lotte/lifestyle\"\n",
    "ROBERTA_EMB_PATH = __WORKSPACE__ / \"dataset/embeddings_test/test_v0.0.1/all-roberta-large-v1/lotte/lifestyle\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "601740f3-30aa-46e5-9180-19cacd7d0c2a",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "FILTER_TOP_N = 3_000\n",
    "VOTING_TOP_N = 100\n",
    "\n",
    "FILTER_MODEL = {\n",
    "    TFIDF_MODEL: {'model_path': TFIDF_MODEL_PATH},\n",
    "    # BM25_MODEL: {'model_path': None},\n",
    "}\n",
    "VOTER_MODELS = {\n",
    "    MPNET_MODEL: {'emb_path': MPNET_EMB_PATH},\n",
    "    ROBERTA_MODEL: {'emb_path': ROBERTA_EMB_PATH},\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ea426fa-5a39-4a3b-b586-c488f39944ea",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "#### Path Checks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6a304bdd-e81c-4402-9bdd-14cdf880590a",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "drive_link = \"https://drive.google.com/file/d/1x-lOZMwBK5Ea9eCDE18gQCTnukZ16xQT/view?usp=share_link\"\n",
    "\n",
    "if not MPNET_EMB_PATH.exists() or not ROBERTA_EMB_PATH.exists():\n",
    "    raise Exception(f\"Embeddings for MPNET model is missing at this path: '{MPNET_EMB_PATH}'.\\n\\n Please download embeddings from here '{drive_link}'.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bfc0153-a906-4ab4-856f-6bc3a46553b0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Step 1 : Data Gathering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ae7af74d-8458-407e-81a1-03814f481dc2",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "data_gathering = DataGathering(dataset_name=DATASET_NAME, )\n",
    "docs_obj = data_gathering.get_documents(dataset_category=DATASET_CATEGORY, dataset_split=SPLIT)\n",
    "del data_gathering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8f5a1342-4cf5-4190-a2aa-31fb42ec6f7f",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'document': 'Normal double-acting baking powder makes CO2 (thus giving a rising effect) in two ways: when it gets wet, and when it is heated. Baking soda only makes CO2 when it gets wet. From Wikipedia: The acid in a baking powder can be either fast-acting or slow-acting.[6] A fast-acting acid reacts in a wet mixture with baking soda at room temperature, and a slow-acting acid will not react until heated in an oven. Baking powders that contain both fast- and slow-acting acids are double acting; those that contain only one acid are single acting. By providing a second rise in the oven, double-acting baking powders increase the reliability of baked goods by rendering the time elapsed between mixing and baking less critical, and this is the type most widely available to consumers today.'}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#sample doc\n",
    "docs_obj[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6653a20d-fe9c-41d5-b9b2-f544143ed788",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Step 2 : Init the Ensemble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e8f39852-a260-4e5c-bd96-9ed073f2b903",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------------\n",
      "\t\tTF4ces Search Engine\n",
      "----------------------------------------------------------------------\n",
      "Filter Model\t: ['tfidf']\n",
      "Voter Models\t: ['all-mpnet-base-v2', 'all-roberta-large-v1']\n",
      "Num of Docs\t: 119461\n",
      "Filter Top N\t: 3000\n",
      "Voting Top N\t: 100\n",
      "----------------------------------------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ensemble = TF4cesSearchEnsemble(\n",
    "    filter_model_dict=FILTER_MODEL,\n",
    "    voter_models_dict=VOTER_MODELS,\n",
    "    docs_obj=deepcopy(docs_obj),\n",
    "    filter_top_n=FILTER_TOP_N,\n",
    "    voting_top_n=VOTING_TOP_N,\n",
    ")\n",
    "ensemble.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f423b6ab-3cfa-438d-9e89-53fb130db5b9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Step 3 : Load models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d0459b6d-ced0-4987-8bbc-1cbc727c675a",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model [TF-IDF] : Loaded with vocab_size (166387) from : '/Users/ataago/Documents/git/TF4ces/TF4ces-search-engine/models/tfidf/lotte/lifestyle/tfidf.v0.0.1.pkl'\n",
      "Model [all-mpnet-base-v2] : Loaded on 'mps'\n",
      "Model [all-roberta-large-v1] : Loaded on 'mps'\n"
     ]
    }
   ],
   "source": [
    "ensemble.load_filter_model(bl_train=False)\n",
    "ensemble.load_voter_models()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce3e110f-cd08-4c35-926d-4b6f524e6d85",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Step 4: Query Helper Methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "31f67fdb-0495-4423-bd38-8b8cd406d046",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_prediction_dfs(preds, user_queries_obj):\n",
    "    pred_dfs = dict()\n",
    "    for (q_id, q_data), pred_doc_ids in zip(user_queries_obj.items(), preds):\n",
    "        pred_dfs[q_id] = pd.DataFrame(\n",
    "            map(lambda doc_id: docs_obj[doc_id]['document'], pred_doc_ids[:5]),\n",
    "            columns=[q_data['query']],\n",
    "            index=pred_doc_ids[:5]\n",
    "        )\n",
    "    return pred_dfs\n",
    "\n",
    "def query_it(query):\n",
    "    \n",
    "    # Generate query obj\n",
    "    user_queries = [query]\n",
    "    user_queries_obj = {q_id: {'query': query, 'rel_doc_ids': list()} for q_id, query in enumerate(user_queries)}\n",
    "    \n",
    "    # Filter docs using filter model\n",
    "    q_ids, filtered_rel_doc_ids = ensemble.filter_docs(queries_obj=deepcopy(user_queries_obj), bl_eval=False)\n",
    "    \n",
    "    # Ensemble with Voter predictions\n",
    "    #ensemble.load_embeddings(filtered_rel_doc_ids=filtered_rel_doc_ids)\n",
    "    ensemble.load_embeddings_if_not_present(filtered_rel_doc_ids=filtered_rel_doc_ids)\n",
    "    ensemble.find_most_relevant_docs(q_ids=q_ids, filtered_rel_doc_ids=filtered_rel_doc_ids, queries_obj=user_queries_obj)\n",
    "    preds = ensemble.ensemble_voting(ensemble_strategy=EnsembleStrategy.TXIOI)\n",
    "    \n",
    "    # Generate data frame of docs\n",
    "    pred_dfs = get_prediction_dfs(preds=preds, user_queries_obj=user_queries_obj)\n",
    "    display(HTML(pred_dfs[0].to_html()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b482accb",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Test Query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cefcaa36-010f-47ca-bce2-0face24d7bbb",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preprocessed data loaded from : /Users/ataago/Documents/git/TF4ces/TF4ces-search-engine/dataset/preprocessed/ensemble/tfidf\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "70fa4c819184434b8351dd64bebb4762",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Pre-Processing Queries:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filter Model [tfidf] : Retrieving top 3000 Docs for queries(1) documents(119461).\n",
      "Model [TF-IDF] : Vector embeddings generated for queries(1) and docs (119461)\n",
      "Model [TF-IDF] : Finding cosine similarities between Queries & Docs...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bde5b07c64eb47249587cc2e7b6b1d63",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Voting Model [all-mpnet-base-v2] : Loading Embeddings:   0%|          | 0/3000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9b385e4a9fe24ccfb890e1a190dfcd7d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Voting Model [all-roberta-large-v1] : Loading Embeddings:   0%|          | 0/3000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5c773fd0946b4dbdb1904aeee0b69f1d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Retrieve Top 100 docs using Voter Model [all-mpnet-base-v2]:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cf1d6428e0c14cb1a23c02c27d0f2945",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Retrieve Top 100 docs using Voter Model [all-roberta-large-v1]:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>are clear pomegranate seeds good to eat?</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>32117</th>\n",
       "      <td>I just opened one up from the tree in my yard. I was sad when I found white seeds on the inside. I was under the impression the seeds were red. The tree was here when I moved here so this fruit is a new experience for me. I tasted the white seeds and was surprised at the sweet flavor. Not tart in taste at all, that's what I was expecting from the google search I did. Only one of the pomegranates on the tree was split open so I am thinking I will watch carefully and give them a smidge longer, it's September and from what I read October is a good month for the fruit.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16020</th>\n",
       "      <td>Apparently it's a different variety of pomegranate. They are good, but taste sweeter than the deep red seeds, which have a great tart flavor.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14700</th>\n",
       "      <td>Some pomegranate varieties produce pink or white seeds so yours sounds perfectly normal. I have had white seeds and they are as delicious as the red ones. Go for it.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9538</th>\n",
       "      <td>There are a few seeds which are good to eat. These generally get sold in the supermarket (pumpkin seeds, apricot kernels). If you buy the fruit containing them, you can keep the seeds. If you want to store them, you should dry them first. Spread the cleaned seeds in a single layer on paper and put it in a warm dry place (not in direct sunlight), and wait a few days. You should deshell them right before consuming, that way they keep longer. If you want to eat them right away, don't dry them. They taste better when fresh. Some should be roasted in the shell before eaten (pumpkin seeds, sunflower seeds). There is little sense in eating seeds not commonly sold in supermarkets, like melon seeds or apple pips. They don't have any nutritional value for humans, or taste bad (e.g. very bitter), or both. It is even dangerous to experiment too much. For example, peach kernels contain poisonous cyanide compounds. So stick to what is commonly eaten, there is a reason the other seeds aren't used.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24746</th>\n",
       "      <td>Got pomegranate with pale pink/white seed. I was surprised when i first opened it because i have been always getting the dark red seed varity. I thought i got a bad fruit. After I taste it, those pale pink/white seed taste sweeter and it is less tart than the drak red varity. It is also more juicy. I LOVED it!!</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "query_it(\"are clear pomegranate seeds good to eat?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e98d9ddd-0bb2-4e5f-bba6-5eb5487e2714",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preprocessed data loaded from : /Users/ataago/Documents/git/TF4ces/TF4ces-search-engine/dataset/preprocessed/ensemble/tfidf\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "da69b1555b2544d1b9c64431b0492534",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Pre-Processing Queries:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filter Model [tfidf] : Retrieving top 3000 Docs for queries(1) documents(119461).\n",
      "Model [TF-IDF] : Vector embeddings generated for queries(1) and docs (119461)\n",
      "Model [TF-IDF] : Finding cosine similarities between Queries & Docs...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "46244d6d77ec4f4aac13bb77c3452609",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Voting Model [all-mpnet-base-v2] : Loading Embeddings:   0%|          | 0/2885 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "693e1e780f7045549ebd5f3b1628da54",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Voting Model [all-roberta-large-v1] : Loading Embeddings:   0%|          | 0/2885 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dab1d3eca537419781283aad17009bad",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Retrieve Top 100 docs using Voter Model [all-mpnet-base-v2]:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "486f12a42ad64467ae45e26564fc0029",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Retrieve Top 100 docs using Voter Model [all-roberta-large-v1]:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>do nutmeg and cinnamon go together?</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>33083</th>\n",
       "      <td>Powdered sugar doesn't seem like a great thickener to me. It takes a lot to thicken a small amount of liquid, and as you've noticed, that means it'll end up pretty sweet. As an example, you can make a glaze (e.g. for cinnamon rolls) with a cup of powdered sugar and only a few tablespoons of liquid. Generally, people thicken things using starch, most commonly flour and cornstarch, but also things like potato starch, arrowroot powder, tapioca/cassava starch. They're stronger thickeners, and don't really have much flavor of their own. So I'd use as much sugar as you want to balance the pomegranate's sourness, and then use starch to thicken. You probably won't need extra sour at that point, but if you want it, you could certainly add something like lemon juice or vinegar.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39748</th>\n",
       "      <td>Probably harmless. I'd go back to the store and tell them about it. If unsure, do not eat them and find another store.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33164</th>\n",
       "      <td>I just used them in my chicken noodle soup - some cooked, some at the very last minute - and it was delicious! So much better than spending money on fresh parsley, which usually goes bad in my fridge before I can use all of it.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41814</th>\n",
       "      <td>I'm pretty sure it's just crushing the husks a bit so they crack open - that's how I do it when I see \"lightly crushed\" for cardamom pods. It gives access to the seeds inside so flavor can infuse out of the pod and into the dish. The whole pod should be visible in the recipe, and removed before eating (would be a woody bite, else). If you crush the husk in smaller bits, which would be needed to get to &amp; crush the seeds, it would be harder to find and fish out, and more likely some huskish bit would find its way into and leave its texture in a bite. Its similar to how cinnamon would be used in big chunks of a stick that can be fished out, or else ground really finely into dust, but not left in a dish in in-between-sized little fragments - the texture is just not desirable If the recipe wanted crushed or ground seeds to release the flavor, it would have asked for seeds instead of (or as well as) a whole pod.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41685</th>\n",
       "      <td>Yes, they're called unstabilized oats, and they can go rancid, so must be refrigerated.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "query_it(\"do nutmeg and cinnamon go together?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "4a102ed0-3fa7-4180-a3ff-b9a076622ba7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preprocessed data loaded from : /Users/ataago/Documents/git/TF4ces/TF4ces-search-engine/dataset/preprocessed/ensemble/tfidf\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ba1ae3ed92bf401aa02bbcda2f9f8485",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Pre-Processing Queries:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filter Model [tfidf] : Retrieving top 3000 Docs for queries(1) documents(119461).\n",
      "Model [TF-IDF] : Vector embeddings generated for queries(1) and docs (119461)\n",
      "Model [TF-IDF] : Finding cosine similarities between Queries & Docs...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9efe70d4a0444685ae3336223ad93e20",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Voting Model [all-mpnet-base-v2] : Loading Embeddings:   0%|          | 0/2707 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "81dac1ed231f4327ba24fa1c9b9f0535",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Voting Model [all-roberta-large-v1] : Loading Embeddings:   0%|          | 0/2707 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "030189d5b82b4c2c8e83c7f920422d8e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Retrieve Top 100 docs using Voter Model [all-mpnet-base-v2]:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5f3d4e4ca7d641f684e75a5e30a8b3c9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Retrieve Top 100 docs using Voter Model [all-roberta-large-v1]:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>How do I flip an egg when preparing it over easy?</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>32334</th>\n",
       "      <td>Pumpkin seeds roasted with a little of salty water is a great (and commonly sold and home made in México) snack. They are also used along with sunflower seeds in preparing some hot sauces and Mole (the hard way). Seeds of Cantaloupe (melon) and Watermelon can be blendend along with their edible parts to strengthen flavor and keep freshness when preparing fruit flavored/fresh water (\"Agua fresca\"). When watermelon seeds are really small, you can eat and chew them along with the fruit: they are usually sweet.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3710</th>\n",
       "      <td>I think it depends on the dish. If it is a biryani or other rice dish, I'd be ok leaving them in because they are easy for the eater to see and remove. But if it is a wet curry for example, I'd probably do what Manako says - extract the seeds at the beginning rather than try to fish around for the pod later.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5414</th>\n",
       "      <td>These are the seeds: These are the leaves of the more common variety (there are many others): While dried seeds are full of aroma and flavour, dried leaves are not. BTW, it is VERY easy to grow cilantro (as parsley) in a pot, just use the seeds ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>927</th>\n",
       "      <td>When they're still young, I grill them -- trim ends cut into planks about 1/4\" to 3/8\" thick (~1cm) toss in olive oil sprinkle with salt grill over direct heat flip when you develop good char marks. pull from the grill once the other side is slightly charred. As they get older, the seed cavity starts developing -- you can cut the sides off, leaving the middle, but it's just not as sweet.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10991</th>\n",
       "      <td>I wouldn't trust any store brand to omit seeds entirely; it's just not going to be a priority for them. I'd suggest making your own instead. It's fairly easy and fast to seed tomatoes using a chinois, and to seed peppers using a knife. And it'll taste better, too!</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "query_it(\"How do I flip an egg when preparing it over easy?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8977b27a-c212-4f3b-a634-ea35c943b878",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Thank you"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}