2-Data_Embedding_and_Storage.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "source_text = \"llm.txt\" will be used by the function that will\n",
    "add the data to our vector store. We then briefly check the document just to\n",
    "be sure.\n",
    "\n",
    "\n",
    "We will now chunk the data. We will determine a chunk size defined by the\n",
    "number of characters. In this case, it is CHUNK_SIZE = 1000, but we can select\n",
    "chunk sizes using different strategies.\n",
    "\n",
    "\n",
    "Chunking is necessary to optimize data processing: selecting portions of\n",
    "text, embedding, and loading the data. It also makes the embedded dataset\n",
    "easier to query. The following code chunks a document to complete the\n",
    "preparation process:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_text = \"llm.txt\"\n",
    "with open(source_text, 'r') as f:\n",
    "    text = f.read()\n",
    "\n",
    "CHUNK_SIZE = 1000\n",
    "chunked_text = [text[i:i+CHUNK_SIZE] for i in range(0,len(text), CHUNK_SIZE)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Exploration of space, planets, and moons \"Space Exploration\" redirects here. For the company, see SpaceX . For broader coverage of this topic, see Exploration . Buzz Aldrin taking a core sample of the Moon during the Apollo 11 mission Self-portrait of Curiosity rover on Mars 's surface Part of a series on Spaceflight History History of spaceflight Space Race Timeline of spaceflight Space probes Lunar missions Mars missions Applications Communications Earth observation Exploration Espionage Military Navigation Colonization Habitation Exploration Telescopes Tourism Spacecraft Robotic spacecraft Satellite Space probe Cargo spacecraft Crewed spacecraft Apollo Lunar Module Space capsules Space Shuttle Space stations Spaceplanes Vostok Space launch Spaceport Launch pad Expendable and reusable launch vehicles Escape velocity Non-rocket spacelaunch Spaceflight types Sub-orbital Orbital Interplanetary Interstellar Intergalactic List of space organizations Space agencies Space forces Companies S\n"
     ]
    }
   ],
   "source": [
    "print(chunked_text[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ### Verifying if the vector store exists and creating it if not\n",
    "\n",
    "First, we need to define the path of our Activeloop vector store path, whether\n",
    "our dataset exists or not:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "vector_store_path = \"./space_exploration_v1\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %pip install deeplake==3.9.18\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deep Lake Dataset in ./space_exploration_v1 already exists, loading from the storage\n",
      "Vector store exists\n"
     ]
    }
   ],
   "source": [
    "# Create a vector store with default tensors\n",
    "from deeplake.core.vectorstore.deeplake_vectorstore import VectorStore\n",
    "\n",
    "# data = VectorStore(\n",
    "#       \"./my_vector_store\"\n",
    "# )\n",
    "\n",
    "try:\n",
    "    # Attempt to load the vector store\n",
    "    vector_store = VectorStore(path=vector_store_path)\n",
    "    print(\"Vector store exists\")\n",
    "except FileNotFoundError:\n",
    "    print(\"Vector store does not exist. You can create it.\")\n",
    "    # Code to create the vector store goes here\n",
    "    create_vector_store=True\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Embedding Function\n",
    "\n",
    "The embedding function will transform the chunks of data we created into\n",
    "vectors to enable vector-based search. In this program, we will use \"text-embedding-3-small\" to embed the documents.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-0.019143932, -0.025292054, -0.0017211713, 0.018834507, -0.033821393, -0.01968206, -0.021027382, 0.05160655, -0.0321801, -0.030431183, -0.0021508336, -0.028924422, -0.002487164, -0.031480536, 0.010291713, 0.018565442, -0.046144545, 0.041409012, 0.00043050305, 0.041166853, 0.05365144, 0.0018481361, 0.004564005, 0.009955383, 0.04781274, 0.0021642868, -0.0098477565, 0.038422395, 0.0009131373, -0.05209087, 0.051122233, -0.032529887, -0.014085521, -0.012605667, 0.013271601, 0.018565442, 0.0016320437, -0.0008479733, -0.012773832, -0.029677803, -0.0045101917, -0.015309764, 0.025668744, 0.00904729, -0.036834914, 0.020287456, -0.040709443, -0.0026771908, 0.03554341, 0.048485406, -0.033659957, -0.0024417595, 0.017273935, 0.0760376, 0.00095433777, -0.042700518, 0.00837463, 0.075983785, -0.047274616, 0.015107966, 0.014260413, 0.024753924, 0.010163908, -0.0010005832, 0.013729011, -0.0100428285, -0.020691052, -0.0014664009, -0.011704301, 0.049373318, 0.0011166172, 0.03785736, -0.01938609, 0.0058252444, -0.04633289, -0.004564005, -0.022359252, 0.0087916795, -0.02652975, -0.054431725, -0.044234186, 0.01930537, -0.030942405, -0.060270425, -0.018901775, 0.0044092927, -0.023892919, -0.011959912, -0.016601274, 0.0022147363, -0.007977759, 0.015296311, -0.014973434, 0.012471135, 0.014542931, -0.02774054, -0.022762848, 0.056664962, 0.06113143, -0.049319502, 0.014610196, -0.02732349, 0.013311961, -0.020543067, 0.029893054, 0.022587957, 0.00819301, -0.04065563, -0.026597016, 0.004553915, -0.12473825, 0.00944416, 0.03180341, -0.031050032, 0.0058050645, 0.030888593, 0.05365144, -0.03960628, 0.023422055, -0.053759065, 0.008657147, -0.0114957765, 0.019534076, -0.041812606, -0.043131024, -0.04299649, 0.0058286074, 0.057472154, -0.031453628, 0.019924218, 0.039417934, 0.039848436, 0.0112737985, -0.0002980729, -0.045579508, 0.0018565443, -0.040628724, -0.01339268, -0.04910425, 0.047328427, 0.056503523, -0.047274616, -0.010890381, -0.009975563, -0.00592278, -0.041139945, -0.008885852, 0.026058886, -0.009955383, -0.006201934, 0.03037737, 0.060593303, -0.05131058, 0.025749462, 0.0037803547, -0.0026687824, 0.024498314, -0.016776165, -0.027821258, 0.015067606, 0.055077482, 0.039579373, -0.044368718, -0.00789704, -0.008105565, 0.008542795, -0.011778294, 0.0024081264, -0.043588433, -0.0022550959, 0.023919825, 0.020906303, -0.021161914, -0.0451221, -0.019197745, 0.026314497, 0.023448963, -0.04956166, -0.01629185, -0.025224786, 0.028493918, 0.04541807, 0.0022904107, 0.019143932, -0.026556656, -0.011145992, -0.0044933753, -0.0068140556, -0.011872467, 0.0008525978, -0.032987297, 0.056126833, 0.023489323, 0.0019826682, 0.0044900123, -0.03656585, 0.06694322, 0.005361108, -0.021552058, 0.024269609, -0.00037858202, 0.012666207, -0.051364392, -0.031937942, -0.023422055, -0.041839514, 1.4793287e-05, 0.070817746, -0.00835445, -0.05246756, 0.058656037, -0.014664009, -0.05418957, -0.002118882, -0.049292598, -0.017892782, 0.041301385, 0.022291984, -0.011206532, -0.031615067, -0.07937399, 0.07157113, 0.08518579, -0.041247573, 0.030942405, -0.0133657735, -0.008455348, -0.008172831, 0.04996526, 0.014583291, 0.020879397, 0.018242566, 0.024256155, -0.021444432, -0.010453152, -0.0043487535, 0.036619663, -0.021377167, -0.0056570787, -0.0071302066, -0.038745273, -0.0011334338, -0.006080855, 0.0342519, 0.04563332, -0.016063144, -0.042673614, -0.022359252, 0.011065274, 0.04073635, 0.01791969, 0.02724277, 0.0054182843, -0.020368176, 0.017233575, -0.0102513535, 0.0066458904, -0.027471475, 0.042673614, -0.021054288, -0.04883519, -0.062100064, -0.03535506, -0.027794352, 0.012646027, -0.027377302, 0.0040628724, -0.013910629, -0.01658782, -0.013197608, 0.043884404, -0.007553983, 0.032691322, -0.004379023, -0.02577637, 0.0033078105, -0.0051660365, -0.017327746, -0.019856952, 0.036996353, -0.025749462, -0.0015033974, 0.010506964, 0.03400974, -0.051364392, 0.008784953, 0.027108237, -0.011415057, 0.0077692345, -0.02486155, 0.03277204, 0.02774054, 0.01625149, -0.020368176, 0.025937807, -0.002830221, -0.0013713876, -0.041678075, 0.03815333, 0.0025376135, 0.07700623, 0.00074791495, 0.042942677, -0.005845424, -0.0017136039, 0.04148973, 0.015888253, -0.016977964, 0.059517045, -0.0013436403, -0.008973298, 0.009585419, 0.056664962, -0.027821258, 0.013500306, -0.045821667, 0.019830046, -0.019359183, 0.0009459295, -0.040036783, 0.023502775, -0.061992437, 0.009430707, -0.015726814, -0.015161779, -0.008603334, 0.0455526, 0.021538604, 0.0019910766, 0.0137693705, 0.014664009, -0.027094785, -0.0046413606, 0.010863475, 0.0072311056, -0.01692415, -0.04383059, 0.054216474, 0.012787285, 0.045767855, -0.026502842, -0.027901977, 0.021807669, -0.036458224, -0.011986819, -0.0042310376, 0.033902112, -0.027121691, 0.00434539, 0.00437566, -0.017368106, -0.0052366657, 0.01642638, -0.024646299, -0.017569905, -0.009854483, 0.07162494, -0.024673205, -0.019574435, 0.021202274, -0.006659344, -0.006928408, 0.0041536815, -0.012094445, 0.0338483, 0.030404277, -0.007204199, -0.04073635, -0.0073320046, 0.04383059, 0.013332141, 0.0148119945, 0.03212629, -0.0036559124, -0.034790028, -0.015121419, 0.011778294, -0.00018161847, 0.05962467, -0.009242362, 0.04035966, 0.03368686, -0.019749327, -0.026583562, 0.010311893, -0.0058723306, 0.020704506, 0.011401604, -0.016991416, 0.008549521, 0.03349852, -0.011529409, 0.0044631055, 0.00952488, 0.007796141, -0.017569905, -0.021848029, -0.015202139, -0.022668675, -0.057687405, 0.04716699, -0.023677668, 0.00750017, 0.011374697, -0.020220188, -0.03347161, 0.012228977, 0.05109533, -0.0036592758, -0.023516228, 0.004066236, -0.052198492, 0.033982832, 0.031911038, -0.025063349, 0.0019473536, -0.023220258, -0.039256494, -0.000755062, -0.004106595, 0.006097672, 0.01904976, -0.014152788, 0.025883995, -0.03255679, 0.040063687, 0.030619528, -0.028063416, 0.018847961, -0.08260277, -0.0015815941, -0.03575866, -0.0036794555, 0.03621607, 0.026301045, -0.04617145, 0.002162605, -0.0063566463, 0.058656037, 0.023220258, -0.002535932, 0.008986751, 0.008832038, 0.0022920924, -0.009363441, -0.053839784, -0.003139645, -0.013856816, -0.040601816, 0.017825516, 0.003518017, -0.043588433, 0.013204335, 0.013964443, 0.03129219, 0.015188685, -0.00027284812, 0.01684343, 0.019897312, 0.02636831, -0.0039014337, 0.02315299, -0.047732025, 0.05214468, 0.0064138225, -0.003585283, -0.022803208, 0.02331443, -0.016305303, -0.022278532, -0.0125249475, 0.025063349, 0.008226644, -0.023785293, -0.020274002, -0.01821566, -0.05448554, 0.015457749, -0.04412656, 0.00018151336, 0.027498381, 0.006968768, 0.041409012, 0.029166581, -0.003294357, 0.009874663, -0.072055444, 0.026771907, -0.0040225126, -0.01025808, -0.034332618, 0.061938625, 0.06403732, -0.015525016, -0.009713225, -0.00829391, 0.0056368993, -0.013258148, -0.0029597085, -0.028870609, -0.0071503865, -0.041543543, -0.0066795237, 0.019897312, 0.061992437, -0.0069822213, -0.046871018, -0.014246959, 0.0074059973, -0.03376758, -0.043480806, -0.008139198, 0.0076817884, -0.005549453, 0.012148257, -0.0031530983, 0.009531606, 0.028278667, -0.012020452, -0.03083478, 0.026664281, 0.032529887, -0.006541628, -0.008011392, -0.0033767582, 0.019412996, -0.05160655, -0.022090187, 0.008515888, 0.027848165, 0.019655153, 0.0050012344, 0.028682264, 0.013621385, -0.007775961, -0.011085453, -0.00059152127, 0.011980092, -0.0047153537, -0.022870474, 0.007089847, 0.029489458, -0.0064676353, -0.0007983645, -0.0114957765, -0.0078365, 0.046897925, 0.0064340024, -0.0024501677, 0.025211334, 0.008105565, -0.016318755, -0.013924083, -0.012094445, 0.03387521, 0.0027125054, -0.021525152, -0.006649254, 0.015148326, 0.06586696, 0.0048027993, -0.010426246, 0.0137693705, 0.027928885, -5.260419e-05, 0.03183032, 0.007190746, -0.021282993, -0.037749734, -0.038530022, -0.03613535, -0.024000544, -0.026058886, -0.063875884, 0.021175368, -0.0075808894, 0.033902112, 0.026395217, 0.0012873049, -0.0048700655, -0.00472208, -0.01674926, -0.0016303621, 0.029758522, -0.041193757, 0.04875447, 0.021848029, 0.014394945, 0.02470011, -0.032233916, 0.01220207, 0.0039922427, 0.009228908, 0.0426198, 0.0012847825, -0.03527434, -0.018027315, -0.020381628, -0.029408738, 0.036996353, 0.0118052, 0.0063734627, -0.022251625, -0.048458498, 0.027552195, -0.014179694, -0.01105182, -0.0018952223, 0.0025577934, -0.049615473, -0.02498263, 0.037749734, -0.021552058, -0.054135755, -0.004167135, 0.030484997, -0.011367971, 0.011778294, -0.015202139, -0.021901842, -0.008260277, -0.042350736, 0.0027259586, 0.028197948, -0.0024989357, 0.016789619, -0.018538537, 0.002322362, -0.02373148, 0.019816592, 0.012874731, 0.014973434, 0.0059732297, -0.0016715626, -0.0065752612, -0.004994508, -0.00869078, -0.016560914, 0.0039754263, -0.023368243, 0.04956166, 0.020865943, 0.0028369478, -0.017596811, 0.05192943, 0.014287319, 0.013103436, 0.031399816, 0.009531606, -0.013338868, 0.026677735, 0.00397879, 0.00334817, 0.001732102, 0.012921818, -0.020543067, -0.028171042, -0.018430911, -0.021040836, 0.022332344, -0.0046480875, 0.07496134, -0.01625149, -0.0061952076, -0.009094376, 0.010009196, 0.019480262, 0.0142066, -0.008092112, -0.019614795, 0.008677327, -0.014233506, 0.042485267, 0.0033952564, 0.03659276, 0.0018817692, -0.030081399, -0.007278192, 0.0018548627, -0.0244445, 0.013345594, 0.0063061966, -0.041086134, -0.036323693, -0.0020717958, 0.028951328, 0.029812334, 0.010695309, -0.003595373, -0.00052509597, -0.029193487, -0.021767309, -0.003629006, 0.042458363, -0.010480058, -0.031534348, 0.022695582, 0.004476559, -0.012868005, -0.03823405, 0.024659751, -0.010984554, -0.0063936426, 0.021175368, 0.026892986, -0.023502775, -0.02414853, 0.034413338, 0.02803651, 0.014327679, -0.04700555, 0.0053442917, -0.029166581, 0.009289448, -0.058817476, -0.0029328018, -0.029704709, -0.06075474, -0.0067568794, 0.0022954557, -0.03430571, 0.000416209, -0.02365076, 0.010735669, -0.021323353, -0.035032183, 0.0027949065, -0.015982425, 0.03126528, 0.01825602, 0.028520826, -0.018417457, 0.013655018, 0.025843635, 0.01674926, -0.0009980607, -0.023852559, -0.014018255, 0.0150137935, -0.014986887, -0.022924287, 0.024928816, 0.044072747, 0.02293774, -0.016641634, 0.012834372, 0.0010602819, 0.0017506002, -0.01638602, -7.026154e-05, 0.011724481, 0.010419519, -0.028655358, -0.0035583766, 0.0044731954, -0.012417322, 0.01767753, -0.0015992515, -0.017139401, 0.021552058, -0.01888832, 0.03083478, 0.0043285736, 0.051498923, -0.020381628, 0.006460909, 0.011953185, 0.0012973949, -0.0017943232, 0.0033448068, 0.046063825, -0.0039754263, 0.011374697, 0.0022096913, 0.0026839173, 0.006397006, -0.041570447, 0.022345798, -0.042270016, -0.018430911, -0.00631965, 0.0094239805, -0.024135076, -0.0015832759, 0.014058615, -0.01779861, 0.0015605735, 0.011515956, 0.022224719, -0.042296924, -0.02418889, -0.04630598, -0.011677395, 0.0053678346, 0.005263572, 0.04487994, -0.018229112, -0.032287728, 0.010634771, 0.0038341677, 0.004587548, -0.04754368, 0.020744866, 0.040036783, 0.004775893, -0.026623921, -0.014381492, 0.008838765, 0.006407096, -0.008246824, 0.021552058, 0.034117367, -0.012101172, -0.012363509, 0.0056671686, -0.039848436, 0.006410459, 0.0035045638, 0.023045365, -0.00014283536, 0.020771772, 0.008347723, -0.01851163, -0.014233506, 0.046871018, 0.021834577, -0.02402745, 0.014300773, 0.033740677, -0.0035314702, -0.018578896, 0.0023038639, -0.0068207826, -0.0065752612, 0.006541628, 0.029220393, 0.008569701, -0.01688379, -0.03522053, -0.0106684035, 0.00035882258, -0.0030992855, 0.0010779392, -0.016157316, 0.026045434, -0.008058479, 0.021565512, -0.024538673, -0.0028873973, 0.01295545, -0.0434539, -0.012841098, 0.026220325, -0.00668625, -0.032072477, 0.016063144, -0.026005074, -0.012168437, -0.00297148, -0.019534076, -0.005391378, 0.01968206, 0.018498177, 0.017152855, 0.03322945, -0.014744729, -0.0006112807, -0.012424048, 0.04956166, 0.05009979, 0.0015740267, -0.00031173634, 0.0013478445, -0.029462552, 0.020771772, 0.044745408, -0.05470079, 0.027848165, -0.006618984, -0.0076817884, 0.022803208, 0.042646706, 0.010352253, 0.0067905127, -0.036027722, -0.026058886, 0.014959981, 0.023260618, 0.013500306, 0.00819301, -0.014341132, 0.0031530983, -0.0016875382, -0.009679591, 0.0062658372, -0.021269541, -0.0106751295, 0.027188957, 0.014919621, 0.009255815, 0.0106751295, -0.007950853, 0.021242633, 0.027794352, 0.018121487, -0.0075607095, -0.0042747604, -0.0020650693, -0.006208661, 0.0018548627, -0.014664009, -0.052117772, 0.029220393, 0.02448486, 0.04797418, 0.009074197, 0.015565375, 0.022211267, -0.0061279414, 0.0061649377, -0.018942133, -0.043534618, -0.016951056, 0.005720982, -0.004076326, -0.009141463, -0.03285276, 0.055642515, 0.006935135, -0.017435372, 0.029139673, 0.029623989, -0.016063144, 0.01809458, 0.007910493, -0.0134734, -0.05160655, 0.030538809, 0.002762955, -0.031157658, -0.030996218, 0.038799085, 0.019359183, 0.005411558, 0.062207688, 0.0055427267, 0.017933141, -0.004056146, -0.0152155915, -0.019157385, -0.0236104, -0.002017983, 0.00078365003, 0.0075405296, -0.010829842, 0.008186284, 0.044099655, -0.008926211, -0.0054855505, 0.012383689, -0.01658782, -0.01646674, -0.024794284, -0.012195344, -0.0016606319, -0.004644724, -0.01871343, -0.013803003, 0.009773764, 0.01563264, -0.027350396, 0.0006596282, 0.03656585, -0.01142851, 0.0023677666, -0.01726048, -0.035328154, 0.025843635, 0.040548004, -0.0063162865, 0.02365076, 0.022655223, 0.025870541, -0.0051256767, -0.0052871155, 0.001331028, 0.014664009, 0.0060169525, -0.01621113, -0.014462211, 0.007258012, -0.0067568794, -0.020933209, 0.03454787, 0.039794624, -0.003048836, -0.025937807, -0.011253619, -0.021296447, -0.016695445, 0.0023189988, 0.016856885, 0.024135076, -0.0024047631, 0.01625149, -0.0012948724, -0.006548355, 0.0061144885, 0.0129419975, -0.015726814, 0.030727154, -0.0035112903, -0.014085521, 0.025843635, 0.026502842, 0.005182853, -0.029085862, 0.041409012, -0.0046514506, 0.014421851, -0.025964715, -0.005384651, 0.0054351008, -0.009699771, -0.008011392, -0.018955586, 0.010695309, -0.011166172, -0.029166581, 0.027632913, -0.018282926, 0.021121554, -0.0039855163, 0.036081534, -0.007224379, 0.03137291, -0.025883995, -0.012033905, -0.017771704, -0.009430707, -0.0047590765, -0.018350191, -0.0133590475, 0.023247164, 0.003618916, -0.004745623, -0.030269744, 0.029032048, 0.0035550133, -0.010607864, -0.015928613, 0.029032048, -0.00627929, 0.024215795, 0.022090187, 0.009787218, 0.010163908, -0.014098974, 0.005462007, 0.02281666, -0.054377913, -0.029058954, -0.029866148, 0.0037669016, 0.016829979, -0.02803651, 0.02335479, 0.00395861, 0.0081795575, 0.036538944, -0.019022852, 0.014502571, -0.012773832, 0.033283267, -0.0090069305, -0.018081127, 0.029946867, -0.025937807, 0.016534006, -0.010998007, 0.021780763, -0.011024914, 0.023300977, -0.03121147, -0.039767716, -0.03121147, 0.0022365977, 0.0056436257, 0.029032048, -0.014704369, 0.0029933413, 0.0048162527, -0.026301045, 0.016453288, -0.044449437, 0.018000407, 0.027108237, 0.010063008, 0.053167123, -0.02644903, -0.00041641918, 0.020274002, 0.023327883, -0.018942133, -0.010029376, 0.0079844855, -0.013042897, -0.013883723, 0.0035348334, 0.03922959, 0.021578964, -0.0039754263, 0.003652549, -0.0096190525, 0.024081264, 0.00085974485, -0.022534143, 0.0002566622, 0.00042734994, -0.0078028673, 0.020287456, -0.036915634, 0.020045297, -0.006857779, -0.008636967, -0.009511426, 0.053032592, 0.021619324, -0.03349852, 0.007379091, 0.02414853, 0.0051357667, 0.022399612, -0.043722965, 0.015430843, 0.050745543, -0.017273935, -0.017193215, -0.027215863, 0.022009468, 0.048323967, -0.0072849183, -0.00833427, -0.0009972199, -0.037964985, -0.0056167194, -0.006851052, 0.100899145, -0.004964238, -0.022655223, 0.004301667, 0.03815333, -0.028709171, 0.007823047, -0.0008513366, -0.009491246, -0.015323217, -0.012296243, -0.012646027, -0.045364257, -0.008058479, -0.008246824, 0.029516364, -0.006131305, -0.032072477, -0.007870134, 0.0016564276, 0.008125745, -0.05047648, -0.009188549, 0.0058656037, 0.0018279562, -0.024269609, -0.040117502, -0.01617077, 0.019951126, 0.038368583, -0.014314226, 0.00033612028, 0.046198357, -0.018726882, -0.013910629, 0.020220188, -0.04487994, -0.021713497, -0.03694254, 0.024969175, -0.03626988, 0.0027545467, 0.028009603, -0.025292054, 0.0016345662, 0.005260209, 0.0071974727, -0.0076010693, 0.0009089332, -0.014462211, -0.005105497, 0.041220665, -0.004944058, 0.01299581, 0.014220053, -0.012060812, 0.033740677, 0.055077482, 0.055965394, -0.012888185, 0.00073572295, 0.012101172, 0.014273866, 0.010392612, 0.0057949745, -0.015820986, -0.039633185, 0.038879804, -0.010345526, 0.04700555, -0.008462075, 0.019184291, 0.014771636, 0.0338483, 0.009262542, 0.012874731, -0.015538469, 0.022978099, -0.0019877132, 0.03922959, -0.0075607095, -0.023206804, 0.008011392, 0.041974045, -0.014327679, 0.03242226, -0.0063667363, -0.005462007, -0.015134872, -0.008448622, 0.002835266, -0.005680622, 0.023085725, 0.0068645054, -0.01255858, 0.02515752, -0.054001223, 0.020892851, -0.02377184, 0.020327816, 0.020421987, 0.019211197, 0.03355233, 0.019372636, -0.028951328, -0.011018187, -0.008105565, -0.058333162, -0.039391026, 0.031076938, -0.00023501093, -0.0098343035, 0.015094512, -0.00473217, 0.05106842, -0.07065631, -0.036242973, -0.031615067, 0.0004296622, 0.013540666, 0.045149006, 0.033121828, 0.014287319, -0.022453424, 0.03659276, -0.00835445, 0.05001907, 0.013406133, 0.02519788, -0.021888388, 0.0034171178, -0.014125881, -0.0015210547, 0.014529477, 0.020583427, -0.00060161116, 0.013661745, -0.01295545, -0.0027427753, 0.0043353, 0.0013999757, -0.03648513, 0.0133590475, -0.009235635, -0.017395014, -0.0028234946, 0.025964715, -0.0059295064, -0.03559722, -0.039929155, -0.0032304544, -0.021753857, 0.009255815, 0.0029546635, -0.0046783574, -0.004853249, -0.029704709, 0.003474294, 0.017031776, -0.03573175, -0.047355335, -0.013089983, -0.01343304, -0.003497837, -0.0158748, 0.023395149, -0.023408603, 0.021673137, -0.009336535, -0.002889079, 0.022951193, -0.031076938, -0.019278463, 0.014462211, 0.0058958735, -0.016722351, 0.005593176, 0.014341132, 0.042135485, 0.008764773, -0.06425258, -0.03702326, 0.022036374, -0.018377097, -0.0028823523, 0.024121623, -0.014973434, -0.01930537, -0.01295545, -0.032691322, 0.0018380461, -0.018982492, -0.0056638056, -0.0052198493, -0.013520486, -0.01775825, -0.034790028, -0.040843975, 0.021027382, -0.010089915, -0.047624398, -0.026623921, 0.017664077, 0.012773832, 0.031615067, 0.01642638, -0.017825516, 0.022964645, -0.013843363, -0.018363645, -0.0043521165, 0.024135076, -0.012397142, 0.009181823, 0.051498923, 0.054593164, -0.00085007533, 0.03220701, -0.003474294, 0.017166309, -0.0062927436, -0.00091229647, 0.014018255, -0.01345322, -0.013729011, 0.019103572, -0.005438464, -0.020085657, -0.020368176, -0.0152290445, 0.02494227, 0.011838834, 0.025009535, 0.041624263, -0.013170702, 0.00835445, -0.0046514506, 0.00013526792, -0.014098974, -0.008441895, -0.00819301, -0.021552058, 0.0061918446, -0.017852422, -0.0016152272, 0.037749734, 0.033740677, -0.0008248505, 0.018161846, -0.0015126464, 0.0018313195, 0.02803651, -0.023072273, -0.01188592, 0.008428442, -0.0053308383, 0.0045707314, -0.0069015017, 0.021511698, -0.0069889477, -0.02690644, -0.006528175, 0.008488982, -0.03250298, -0.00047002186, -0.029919961, 0.022224719, 0.019708967, -0.0013007582, 0.027767446, -0.010728943, 0.0069216816, 0.008132472, -0.028224856, -0.02436378, 0.01976278, 0.03285276, -0.01813494, 0.022224719, -0.0135742985, -0.009713225, -0.020300908, -0.011771567, 0.007654882, -0.014018255, 0.015646094, -0.00821319, 0.0011519319, -0.031157658, -0.011105633, 0.06775042, -0.008832038, 0.008569701, 0.020973569, -0.014919621, 0.0050113243, -0.012006999, 0.0052904785, 0.020987023, -0.003682819, -0.02486155, 0.00029954434, 0.004079689, -0.013856816, -0.0043420265, 0.023462415, 0.018780695, 0.00875132, -0.012060812, 0.0008391446, 0.012504768, 0.020960117, -0.008845492, -0.0057747946, -0.017099041, 0.020892851, -0.009975563, -0.011946459, -0.0045505515, 0.0359201, 0.026691187, 0.014946527, -0.007358911, -0.010473331, -0.027875071, -0.02595126, 0.0056436257, -0.0041536815, 0.027875071, -0.039417934, 0.0032001846, 0.004597638, 0.040386565, -0.028332481, 0.03613535, -0.027875071, -0.021565512, 0.011145992, 0.022386158, -0.015767174, 0.0252786, -0.005071864, 0.0022550959, 0.013069803, 0.011690848, 0.023206804, 0.028816797, -0.016157316, 0.009397074, 0.008603334, -0.023825653, 0.0087984055, 0.02013947, 0.0010586003, -0.056934025, 0.029220393, 0.003294357, -0.031911038, 0.022910833, -0.021915294, -0.0018649526, -0.041220665, -0.018646162, 0.009134736, -0.029623989, 0.060324237, -0.025049895, -0.017731344, 0.03376758, -0.020274002, -0.017193215, -0.025372772, 0.030969312, -0.034036648, 0.019655153, -0.0063566463, -0.010002469, -0.03113075, 0.014664009, 0.013244695, -0.020529613, -0.022130547, 0.011583222, -0.005522547, 0.043776777, 0.0045101917, -0.022668675, 0.0056436257, -0.010628044], [0.027567066, -0.0265443, -0.027091362, 0.057798076, -0.0034756148, -0.0028126019, 0.031848405, 0.01012061, -0.034108, -0.04464485, -0.034488562, -0.009365429, -0.022322426, -0.0004307354, -0.013248365, 0.02133534, -0.02854226, 0.0146992635, 0.00797994, 0.0040732184, 0.025735604, -0.019123314, 0.0015237405, 0.02275056, 0.05056737, -0.0004678998, 0.006879874, 0.04669038, -0.010881737, -0.02376143, 0.04267068, -0.039697528, 0.025378825, 0.035368618, -0.0070642093, 0.04041108, -0.05708452, 0.020514749, -0.004209983, -0.06336382, -0.002717461, -0.047475293, 0.010519012, 0.019539556, 0.006160371, 0.01842165, -0.0347502, 0.0009558684, 0.032752242, 0.023309512, -0.0328236, 0.015507962, 0.043027457, 0.08947998, 0.023440331, -0.013010513, 0.023559257, -0.006196049, 0.033989076, -0.0020410689, -0.01942063, -0.032181397, 0.006653914, -0.018385973, -0.011190944, -0.022358103, -0.005280318, 0.018933032, -0.010685508, 0.0052565327, 0.008782691, 0.025307471, 0.0061306395, -0.004611359, -0.009198933, 0.019111421, -0.00013964524, -0.053469166, -0.0042159297, 0.04271825, -0.041600343, -0.0071415114, -0.036462735, 0.011369334, -0.020919098, -0.02635402, -0.00087707984, 0.0033953397, -0.0025985348, -0.029565023, -0.0055835797, 0.038008776, -0.06678889, 0.0001334202, 0.022262963, -0.039126683, -0.0042129564, 0.017303744, 0.023392761, 0.0019593071, 0.062269695, -0.012041266, 0.0021540488, 0.015567424, 0.06602776, 0.014021385, 0.003380474, -0.017672416, -0.0072425986, -0.01660208, -0.12691791, -0.0538973, -0.016982645, 0.03769957, 0.015555532, 0.017934054, 0.06160371, -0.015496069, 0.0071831355, -0.067312166, 0.01942063, -0.010578475, 0.045524903, -0.032871168, -0.08229685, 0.034345854, -0.030944567, 0.023749538, -0.043146383, -0.018980604, 0.037414145, 0.037128724, 0.010935253, 0.0008919456, -0.005533036, 0.037556857, -0.01892114, 0.011506098, -0.042646892, 0.031253774, 0.051090647, -0.030088298, -0.011517991, -0.007254491, 0.018397866, -0.018076764, -0.02690108, 0.07482829, 0.0021941862, -0.050852794, 0.03142027, -0.010132502, -0.082439564, 0.022262963, 0.01645937, -0.034702633, 0.016958859, -0.009228664, 0.0073199, 0.024284706, 0.0011870309, 0.045382194, -0.027329214, 0.00014670647, -0.02566425, -0.010804434, -0.03639138, 0.016126378, 0.0061306395, -0.010804434, -0.04669038, 0.029898018, -0.022358103, -0.014092741, -0.009817348, -0.050139237, 0.04509677, 0.00257029, -0.047213655, -0.01235642, -0.0063090287, 0.01628098, 0.02115695, -0.010138448, 0.008289148, 0.0140094925, -0.053421598, 0.026258878, -0.05480114, -0.017268067, 0.014972794, 0.059653323, 0.01103634, 0.015127398, -0.0047213654, -0.02804277, 0.04126735, 0.011012555, -0.0154009275, -0.01951577, 0.008990812, 0.012065051, -0.027186502, -0.05175663, 0.03320416, -0.027923845, -0.010037362, 0.02476041, -0.0041713323, 0.011958018, 0.060224168, -0.008883778, 0.012927265, 0.034131784, -0.001623341, 0.004828399, 0.027971415, 0.032799814, 0.011827199, 0.010768757, -0.073210895, 0.01142285, 0.035178334, 0.008824315, 0.012053158, -0.011333656, 0.015067935, -0.003992943, -0.012332635, -0.005928465, -0.009175147, 0.016269088, 0.008628087, 0.015186861, -0.029469883, 0.018659502, 0.021454265, -0.011303924, 0.028803898, 0.009698422, 0.023404652, 0.0076826247, -0.006820411, 0.028566046, 0.017470242, 0.010840112, 0.030349936, -0.04276582, -0.016566403, -0.0034993999, 0.014473304, -2.2821265e-05, 0.023190586, -0.031253774, -0.01153583, -0.017125355, -0.02923203, 0.038246628, 0.029850448, -0.03972131, -0.014663586, -0.036034603, -0.014401949, -0.052089624, -0.023131123, -0.01586474, 0.013438648, 0.018124335, -0.02968395, -0.014116526, 0.024189565, -0.01377164, 0.03634381, -0.029255817, 0.022334319, 0.027043791, 0.006062257, 0.0034280443, -0.0082356315, -0.0022328373, 0.0025152867, -0.018457329, 0.0133435065, 0.0073555782, -0.050234377, 0.029422313, -0.040006734, -0.023749538, 0.0070463703, 0.060985297, 0.036034603, -0.008943241, -0.019694159, -0.008949188, -0.011874769, 0.00062770676, 0.002955313, -0.04690445, -0.018338403, -0.007236652, 0.07007125, 0.016768578, 0.018873569, 0.021347232, -0.004694607, 0.056656387, 0.0023086525, -0.0071890815, -0.021121273, -0.0334658, -0.04152899, 0.020764494, 0.016388014, -0.031134848, -0.05413515, -0.012296957, 0.026282664, 5.263408e-05, 0.0593679, -0.021715902, 0.02630645, -0.04126735, -0.035582684, -0.034678847, 0.0015727974, -0.0075993766, 0.012522916, -0.02471284, -0.008259417, -0.013664607, 0.0064636325, -0.062269695, -0.007736142, -0.018778428, 0.02954124, 0.014473304, -0.038270414, 0.025521537, 0.047070943, 0.030349936, -0.02575939, -0.047166083, -0.040863004, -0.05951061, -0.0064398474, -0.021846721, -0.018718965, 0.04152899, -0.065361775, 0.004596493, 0.031753264, 0.014413841, 0.0039810506, -0.020942884, -0.014449519, -0.024689054, 0.03972131, 0.0005463167, 0.0019221428, 0.005761969, -0.009395161, -0.0015623914, -0.0040405137, 0.01016818, 0.043622088, 0.038460694, -0.05480114, -0.028280623, -0.012606164, 0.035677824, 0.0059403577, 0.044383213, 0.0014308294, 0.01377164, -0.05075765, 0.0006589248, 0.043241523, 0.0018938979, -0.006802572, 0.017803235, 0.05185177, 0.04257554, -0.057988357, -0.033727437, 0.039126683, -0.012261279, -0.00066784426, -0.0050335466, 0.019622803, 0.020360146, 0.04431186, -0.025164759, 0.025545323, -0.016150162, 0.023202479, -0.024094425, 0.06426766, 0.042290114, -0.05722723, -0.04171927, 0.040648934, -0.012136406, 0.006588505, -0.0039572655, 0.05289832, 0.035772964, -0.041576557, -0.0139975995, 0.020443393, 0.013569466, 0.021394802, -0.032799814, 0.0326571, 0.056180682, -0.009097845, -0.019539556, 0.021941863, 0.00025885005, 0.01827894, 0.01595988, -0.019266026, 0.0038056346, 0.030992137, 0.0024037934, -0.0119639635, -0.030445077, 0.013938137, -0.06488607, 0.031444058, -0.028946608, 0.0078847995, 0.012844017, 0.03639138, -0.0070404243, -0.011416904, 0.0020410689, 0.014627908, 0.004510272, 0.02566425, -0.0026044813, -0.021775365, -0.03788985, 0.021240199, -0.033846363, -0.041410062, -0.027138932, 0.030873211, -0.021121273, 0.004281339, -0.04864077, -0.038270414, -0.006475525, -0.019765515, 0.046071965, -0.040863004, 0.03734279, 0.03272846, -0.013926244, 0.013569466, -0.030587789, 0.031158634, -0.037152506, -0.024784196, 0.048688337, 0.024213351, 0.01285591, 0.020752601, 3.3378277e-05, -0.004962191, -0.01951577, -0.0069155516, 0.023380868, -0.01285591, 0.022833807, -0.005200043, -0.016721006, 0.0006202739, 0.02275056, 0.016946968, -0.00551817, 0.035630256, 0.039745096, -0.008324826, 0.034226928, -0.029469883, -0.02020554, 0.0026877294, 0.037176292, -0.06479093, 0.03213383, 0.008776745, 0.023773324, 0.053374026, -0.009377321, -0.007528021, 0.021644548, -0.040458653, -0.033537157, 0.019456306, -0.021299662, -0.01377164, 0.04057758, -0.01180936, 0.032110043, 0.034226928, -0.0053992444, 0.02471284, -0.009276235, 0.017993517, -0.024130102, -0.0012621031, 0.030468863, -0.008081027, -0.021073703, -0.014211667, 0.015531747, 0.030564003, 0.035820536, -0.019801192, -0.038532052, 0.021073703, 0.0102573745, 0.035463758, -0.042456612, 0.0010889169, 0.0076172156, -0.017910268, 0.015305786, -0.044240505, -5.9788228e-05, 0.023047876, -0.03501184, -0.048450485, 0.004985976, -0.012725091, 0.025355041, 0.030397506, -0.038555834, -0.019884441, -0.040696505, 0.008402128, 0.046809305, -0.012273172, -0.010798489, -0.0102930525, 0.009704368, 0.020586105, -0.018814106, 0.014520874, 0.015008472, 0.025069619, -0.004284312, 0.020669352, 0.017494027, 0.001398868, 0.009989791, 0.016292874, -0.019123314, -0.0010777676, 0.027424354, -0.020253113, -0.025592893, -0.0059552235, 0.018457329, -0.0146992635, 0.04445457, 0.004575681, 0.023261942, 0.04091057, -0.0028586858, -0.040197015, 0.006529042, 0.02115695, -0.019539556, 0.020253113, 0.017922161, 0.0073436857, 0.012927265, -0.04355073, -0.01285591, -0.03356094, -0.040791646, 0.0058957604, -0.0043318826, -0.012344527, 0.004438916, 0.004843265, 0.008081027, 0.01377164, 0.035083193, -0.0015623914, -0.014461412, 0.008057242, 0.01322458, 0.010417925, -0.029802876, -0.015341464, -0.025236115, -0.038056348, 0.021775365, -0.005179231, 0.013890566, -0.0020024178, -0.045834113, -0.035582684, 0.011327709, 0.0055003315, 0.031253774, -0.03232411, -0.05142364, -0.003793742, 0.0045400034, 0.0033477691, -0.088385865, 0.01942063, 0.02001526, -0.014449519, 0.0036094065, -0.016031235, -0.009644905, -0.019646589, -0.005592499, 0.00645174, 0.0055389823, 0.03337066, 0.0022239177, -0.024296599, -0.00015423226, -0.010661723, -0.024237135, -0.0023160854, 0.022310533, 0.023749538, -0.006570666, -0.0011862877, 0.020550428, -0.017660523, -0.016530726, 0.037747137, -0.015519854, 0.016340444, -0.0024364982, 0.03351337, 0.008473484, 0.009472462, -0.012760769, -0.047499076, 0.0019711999, -0.014937116, -0.0246177, 0.008194007, -0.055562265, -0.017006429, -0.024189565, -0.015496069, 0.020134186, -0.01327215, -0.013914351, -0.014211667, 0.015924202, 0.033893935, 0.025592893, -0.00552709, 0.014354378, -0.023678184, 0.009781671, -0.0016605054, -0.023511687, -0.016709115, 0.0054676267, -0.003868071, -0.027329214, 0.050424658, -0.055134133, 0.05237505, -0.0034548028, -0.0039423998, -0.03739036, -0.020550428, -0.016388014, -0.029731521, 0.016542617, -0.028470904, 0.008931349, 0.032538176, 0.034036644, 0.046333604, -0.030873211, -0.007843175, 0.053183746, -0.015210646, 0.0013067003, 0.040791646, -0.005803593, 0.026425375, 0.0020693138, 0.008384289, 0.021501835, 0.012974836, -0.101420164, 0.0028349005, -0.03953103, -0.03127756, -0.010328731, -0.022691097, -0.020621782, -0.032395463, -0.00012784555, 0.034179356, 0.057512652, -0.036510307, 0.011541776, -0.0053100497, -0.0053814054, 0.03077807, 0.004105923, -0.020859635, 0.00262678, 0.018540576, -0.013688392, -0.0042337687, -0.014794405, -0.015971772, -0.053374026, 0.038722333, -0.017482134, -0.041148424, -0.016673436, 0.009858972, -0.01285591, 0.003187219, -0.018588146, -0.020871527, 0.004183225, 0.034393422, 0.008735121, -0.01942063, -0.011720166, -0.05166149, -0.0032020847, -0.00035956557, 0.011732058, 0.023523578, -0.027210288, 0.010400086, 0.035915677, 0.014306808, 0.03903154, -0.048902404, -0.0076588397, 0.012178031, 0.008527, -0.047475293, 0.033346873, -0.007177189, -0.02690108, 0.045738973, 0.019170884, 0.0119045, 0.026330234, -0.0038205003, 0.032704674, 0.002959773, 0.009781671, -0.030207224, 0.030373722, 0.004849211, -0.0012598732, -0.018064871, 0.002745706, 0.0076350546, 0.006534988, -0.01668533, -0.0062495656, 0.050900362, -0.002201619, -0.03232411, 0.032752242, -0.030587789, -0.018457329, 0.019432522, -0.0034666953, 0.007486397, 0.029065534, -0.018029194, -0.03377501, 0.005886841, 0.009020544, 0.033727437, -0.015924202, -0.003148568, -0.003752118, 0.03543997, -0.0056549353, -0.018635716, 0.027900059, 0.001829975, -0.036129743, 0.018267047, 0.02215593, -0.027138932, 0.009805456, -0.0023517632, -0.02740057, 0.027852489, -0.0197893, -0.044882704, 0.0116369175, 0.0058808946, -0.028708756, -0.019682266, -0.016126378, -0.020895313, 0.018718965, 0.011179051, -0.016982645, -0.02197754, 0.0010012089, 0.0011654756, -0.0089372955, 0.04818885, 0.027519496, -0.011595293, -0.0017749718, 0.034274496, 0.0035083194, -0.011220676, -0.020003367, -0.0055984454, -0.019765515, 0.023416545, -0.029350957, -0.026211308, -0.007391256, 0.002475149, 0.015270109, -0.047998566, -0.008830261, 0.014283023, -0.016292874, 0.005063278, -0.03988781, 0.017743772, 0.023261942, 0.009448677, 0.047641788, -0.0020009314, -0.008729175, 0.026044812, 0.02083585, 0.01832651, 0.004025648, 0.041885767, 0.017815128, -0.024249028, -0.0016887503, -0.0019384952, -0.0024216324, -0.003868071, -0.029993158, -0.024474988, 0.02749571, 0.017339423, -0.024546344, 0.020467179, -0.013295936, 0.025140975, -0.007195028, -0.022429459, 0.0035856215, -0.00088302616, -0.014520874, -0.0026178604, -0.030016944, -0.0049057007, -0.0013862322, -0.01481819, -0.0012569, -0.022488922, -0.0126775205, 0.026163738, -0.0133316135, 0.015496069, 0.031800836, 0.0066003976, 0.0066658068, 0.041505203, -0.009442731, -0.010025469, 0.040648934, 0.0051881503, 0.011601239, 0.0065409346, 0.010406032, 0.005226801, -0.0017303745, -0.022465138, 0.0037045474, -0.013985707, -0.016875612, 0.008610249, -0.0059403577, 0.005925492, 0.014330593, 0.013605144, 0.025236115, 0.021525621, 0.046952017, -0.0067133773, 0.011006609, 0.0060652304, -0.0362011, 0.00711178, 0.029755305, -0.015531747, 0.044716205, -0.022132143, 0.0074328803, 0.008836208, -0.046619024, -0.0077480343, 0.0042307954, 0.014925224, 0.010108717, -0.0035469704, 0.01827894, 0.008015618, 0.03679573, -0.041552775, 0.013926244, 0.005851163, -0.0076885712, -0.04709473, 0.010067093, 0.024379848, 0.027162718, -0.0029612596, 0.0009841133, -0.030992137, -0.030587789, -0.0017467268, -0.009395161, 0.019277917, 0.01750592, 0.038555834, 0.006784733, -0.039840236, 0.02083585, 0.016352337, -0.0126656275, -0.012820232, 0.028518474, 0.0040197014, 0.021478051, 0.016292874, -0.004774882, 0.022762451, -0.029255817, 0.010804434, 0.023499794, -0.023250049, -0.0146398, -0.005690613, -0.0032942526, -0.030349936, 0.025450181, 0.0063803843, 0.006475525, 0.004299178, 0.018956818, -0.012148299, 0.049996525, 0.013533788, 0.024427418, 0.020467179, -0.0048194793, 0.02115695, -0.0016173947, 0.02083585, -0.018183798, 0.055419553, 0.050852794, 0.015781492, 0.007426934, -0.057893217, -0.0054230294, 0.025307471, 0.013307828, -0.023785217, -0.00525356, 0.042290114, 0.010953092, 0.03720008, 0.003481561, -0.0031545144, -0.021525621, 0.017886482, -0.06169885, 0.029469883, -0.02083585, 0.008972973, -0.04942568, 0.041148424, -0.010935253, 0.010519012, -0.011250407, 0.009401107, 0.019943904, 0.014889546, 0.02485555, 0.025117189, -0.03903154, 0.015829062, -0.0025747498, 0.017529704, -0.028233051, 0.0049413787, 0.023107339, 0.0025970484, -0.0016010423, 0.018611932, 0.021739688, 0.042385254, 0.008485376, -0.014627908, -0.0133553995, 0.00809292, 0.02125209, 0.009133523, 0.017482134, 0.009127577, 0.046880662, 0.02110938, 0.046119533, -0.006653914, -0.002689216, -0.0147230495, -0.06393466, -0.015008472, 0.026211308, -0.0061128004, -0.01044171, -0.005571687, -0.0042872853, -0.016566403, 0.00891351, -0.03465506, 0.022477029, 0.0081999535, 0.00020328927, -0.006303082, -0.010477388, -0.029565023, -0.0021153977, 0.033917718, -0.03215761, -0.0076885712, -0.010340623, -0.0022239177, 0.020407716, -0.01240399, 0.0052981568, -0.0147230495, -0.020276897, -0.030635359, -0.013831103, 0.006636075, -0.0017690255, -0.042242546, -0.0078729065, 0.02393982, 0.042646892, 0.017803235, 0.024879336, 0.0021020186, -0.010976877, 0.05846406, -0.0313727, -0.026687013, 0.098946504, 0.016863719, 0.005417083, 0.005182204, -0.008312933, -0.0389364, 0.0029984238, -0.0246177, -0.02101424, 0.019266026, -0.013236473, 0.018409757, -0.0016634786, -0.006201995, 0.010078985, -0.006546881, -0.005753049, 0.0078491215, 0.0218824, 0.04509677, 0.004421077, -0.01742267, -0.02740057, 0.024427418, -0.0028601722, 0.019503878, 0.0096032815, 0.002784357, -0.020229327, -0.011791521, -0.007991833, 0.023000304, -0.007034478, 0.010245482, -0.019753622, -0.017945945, -0.018742751, -0.037770923, -0.016340444, 0.00018981718, -0.011072018, -0.0027739508, 0.0037461717, 0.010810381, 0.0063209212, -0.024926906, 0.028613616, 0.052279904, 0.009157308, -0.013486218, 0.0492354, 0.046404958, -0.024296599, 0.009020544, -0.013854888, 0.023238156, 0.015650673, 0.0061246934, 0.002815575, -0.028090341, 0.014830083, 0.03762821, 0.0062674046, -0.024344169, 0.011161213, 0.0071831355, -0.020586105, -0.025212329, 0.0942846, -0.03191976, -0.02963638, -0.03798499, 0.0028973366, -0.0133435065, -0.017684309, -0.0030653197, 0.0077242493, 0.02839955, 0.008871886, 0.0054735728, 0.008312933, 0.0020172836, -0.017934054, 0.03534483, 0.014116526, 0.005185177, 0.012962943, -0.018849785, 0.007831283, -0.033537157, -0.004673795, -0.004792721, -0.022619741, -0.029731521, 0.017244281, 0.0014390056, 0.008419966, 0.0032704673, -0.008045349, 0.017838912, -0.0050365194, 0.028375763, -0.009305966, 0.04007809, -0.020693138, -0.023749538, 0.0077658733, 0.033037666, -0.0020782333, -0.0049740835, 0.029113105, 0.004985976, 0.023321405, -0.007135565, 0.029041748, 0.002562857, -0.0031426218, -0.01741078, -0.002779897, 0.027138932, 0.036105957, 0.011749897, 0.046309818, -0.009050275, 0.007212867, 0.02754328, 0.018136228, 0.02038393, 0.0037848225, -0.02175158, 0.0019875523, -0.010417925, -0.009751939, 0.0477845, 0.011107696, 0.008164275, -0.013105654, 0.016078806, 0.022393782, 0.0004508042, -0.01819569, 0.014283023, -0.0013401483, 0.014794405, -0.028351977, 0.009995737, -0.0009053248, 0.03251439, -0.029160675, -0.023273835, -0.02165644, -0.0022135116, -0.021668332, 0.0040821377, 0.005443841, 0.0035826482, 0.002539072, -0.0021168843, 0.0022001325, 0.010964985, 0.009924382, -0.02571182, 0.02361872, 0.014175989, -0.010114663, 0.036986012, -0.020597998, 0.015769599, -0.031705692, -0.00596117, -0.022179715, 0.014021385, 0.041029498, -0.04171927, 0.032871168, -0.024546344, -0.008312933, -0.04723744, 0.06412494, -0.008473484, -0.012320742, 0.029350957, 0.010400086, -0.035582684, 0.03739036, -0.035463758, -0.008961081, -0.006980961, 0.035820536, 0.040434867, -0.016709115, -0.03206247, 0.034369636, 0.011958018, 0.029612595, 0.015698243, -0.012047213, -0.02102613, 0.0049562445, 0.0031039708, 0.01235642, -0.0032734405, 0.02165644, -0.0058184587, -0.02844712, 0.024926906, 0.004340802, -0.0030593735, -0.059463043, -0.034536134, 0.010084932, -0.011678541, 0.0073020617, -0.011066072, 0.0133435065, -0.023214372, -0.01563878, -0.02968395, -0.008378343, 0.010917414, 0.0023963605, 0.019860655, 0.025521537, 0.0027947628, 0.028946608, -0.031848405, -0.011684488, -0.016304767, 0.0030445077, 0.004427023, 0.0013460947, -0.010263321, 0.017006429, 0.042456612, 0.03370365, 0.02685351, -0.03579675, -0.017482134, -0.009638959, -0.00094471907, -0.03146784, -0.012784554, 0.036724374, 0.007070156, 0.002657998, 0.018397866, -0.045025416, 0.003871044, -0.017363207, 0.017517813, -0.0070820483, -0.035273477, -0.030873211, -0.0048640766, 0.012772662, -0.020752601, 0.0034577758, -0.0029969374, -0.015234431, -0.012867802, 0.0068144645, 0.026639443, 0.019384952, 0.06393466, -0.06883442, -0.042432826, 0.03291874, 0.019896334, -0.01655451, 0.0064993105, -0.017684309, 0.024082532, -0.016304767, 0.007855068, 0.008972973, 0.022393782, -0.00039914568, 0.0036094065, -0.012499131, -0.00617821, -0.01750592, 0.023178693, 0.024308491, -0.0014137338, -0.002342844, 0.0402208, 0.01659019, 0.023273835, -0.013284043, -0.0026698906, 0.017684309, -0.016055021, 0.0033983127, 0.002162968, -0.0072663836, -0.017006429, -0.010352516, -0.014104634, 0.047760714, 0.0075577525, 0.018825999, 0.016709115, 0.0006072663, 0.048260204, -0.03256196, -0.0077064103, -0.008919456, -0.027781133, 0.0033150646, -0.024641484, -0.015210646, -0.014342486, -0.009448677, 0.01180936, -0.006653914, 0.0036331918, 0.0016515859, 0.021941863, 0.022643527, -0.021347232, 0.0068144645, -0.010126556, 0.03013587, -0.03634381, 0.018409757, -0.019801192, 0.018718965, -0.005845217, 0.013081869, -0.0016917235, 0.03337066, -0.036510307, -0.0016278008, 0.012998621, 0.017577276, 0.017862698, -0.004400265, -0.022881377, 0.0010071553, -0.010406032, -0.011345549, -0.0004162413, 0.010132502, 0.04007809, -0.018992495, 0.003986997, 0.026139952, -0.020122293, 0.004034567, 0.03762821, 2.9685065e-05, 0.02575939, 0.0100552, 0.0052684257, 0.0046827146, -0.0013944083, -0.027257858, -0.0037907688, 0.038413126, 0.006303082, -0.0033983127, 0.017054, -0.020942884, 0.038865045, -0.009056221, -0.05384973, 0.004718392, 0.025188545, 0.013022406, -0.024903122, -0.011749897, 0.021727795, -0.023059767, 0.048712123, -0.03013587, -0.012047213, -0.0015237405, 0.01928981, 0.007212867, -0.008069134, -0.018267047, -0.012332635, -0.03206247, -0.008039403, -0.017981624, 0.0073555782, -0.0062674046, 0.01827894, 0.016399907, 0.035582684, 0.020919098, -0.021275876, 0.0016590188, -0.0105725285, -0.026330234, 0.0063565993, 0.007956155, -0.024784196, 0.027424354, -0.0062138876, -0.004343775, 0.024641484, 0.02028879, -0.001383259, -0.015460391, 0.0070998874, 0.019503878, -0.020348253, -0.031087277, -0.06260269, 0.010673616, -0.005610338, 0.0008993785, 0.03077807, -0.02001526, -0.049806245, 0.0265443, 0.028566046, 0.0032318162, -0.0048402916, 0.03387015, -0.0043140436, -0.034726415, 0.012606164, 0.018433543, -0.0011015529, 0.015757706, -0.026829725, -0.0053487006, -0.050710082, 0.031134848, 0.022667311, -0.020990454, 0.020942884, -0.018754642, 0.02954124, 0.00019418399, 0.01829083, -0.0133791845, -0.00012626605, -0.005702506, -0.06450551, -0.034488562, -0.03624867, 0.041291136, -0.0011781115, 0.018445436, 0.006647968, 0.020229327, -0.040696505, -0.010025469, 0.017113464, -0.011428797, 0.0112860855, -0.018088657, -0.040268373, 0.002521233]]\n"
     ]
    }
   ],
   "source": [
    "from openai import OpenAI\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv()\n",
    "\n",
    "# Initialize OpenAI client\n",
    "client = OpenAI()  # Assumes API key is set in env variables or ~/.openai_key\n",
    "\n",
    "def embedding_function(texts, model=\"text-embedding-3-small\"):\n",
    "    \"\"\"Generates embeddings for input texts using OpenAI API.\"\"\"\n",
    "    \n",
    "    if isinstance(texts, str):\n",
    "        texts = [texts]  # Convert single string to list\n",
    "    \n",
    "    texts = [t.replace(\"\\n\", \" \") for t in texts]  # Remove newlines\n",
    "    \n",
    "    try:\n",
    "        response = client.embeddings.create(\n",
    "            model=model,\n",
    "            input=texts,\n",
    "            encoding_format=\"float\"  # Ensure correct format\n",
    "        )\n",
    "        \n",
    "        embeddings = [data.embedding for data in response.data]  # Correct attribute access\n",
    "        return embeddings\n",
    "\n",
    "    except Exception as e:\n",
    "        print(f\"Error generating embeddings: {e}\")\n",
    "        return None\n",
    "\n",
    "# Example Usage\n",
    "texts = [\"Hello, world!\", \"How are you?\"]\n",
    "embeddings = embedding_function(texts)\n",
    "print(embeddings)  # Prints list of embeddings\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting python-dotenv\n",
      "  Using cached python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB)\n",
      "Using cached python_dotenv-1.0.1-py3-none-any.whl (19 kB)\n",
      "Installing collected packages: python-dotenv\n",
      "Successfully installed python-dotenv-1.0.1\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding data to the vector store\n",
    "\n",
    "We set the adding data flag to True:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "add_to_vector_store=True\n",
    "source_text = \"llm.txt\"\n",
    "\n",
    "if add_to_vector_store == True:\n",
    "    with open(source_text, 'r') as f:\n",
    "        text = f.read()\n",
    "        CHUNK_SIZE = 1000\n",
    "        chunked_text = [text[i:i+1000] for i in range(0, len(text), CHUNK_SIZE)]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Creating 1651 embeddings in 4 batches of size 500:: 100%|██████████| 4/4 [00:49<00:00, 12.43s/it]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset(path='./space_exploration_v1', tensors=['embedding', 'id', 'metadata', 'text'])\n",
      "\n",
      "  tensor      htype       shape       dtype  compression\n",
      "  -------    -------     -------     -------  ------- \n",
      " embedding  embedding  (1651, 1536)  float32   None   \n",
      "    id        text      (1651, 1)      str     None   \n",
      " metadata     json      (1651, 1)      str     None   \n",
      "   text       text      (1651, 1)      str     None   \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "vector_store.add(text = chunked_text,\n",
    "              embedding_function = embedding_function,\n",
    "              embedding_data = chunked_text,\n",
    "              metadata = [{\"source\": source_text}]*len(chunked_text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset(path='./space_exploration_v1', tensors=['embedding', 'id', 'metadata', 'text'])\n",
      "\n",
      "  tensor      htype       shape       dtype  compression\n",
      "  -------    -------     -------     -------  ------- \n",
      " embedding  embedding  (1651, 1536)  float32   None   \n",
      "    id        text      (1651, 1)      str     None   \n",
      " metadata     json      (1651, 1)      str     None   \n",
      "   text       text      (1651, 1)      str     None   \n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(vector_store.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./space_exploration_v1 loaded successfully.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": []
    }
   ],
   "source": [
    "import deeplake\n",
    "ds = deeplake.load(vector_store_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Estimates the size in bytes of the dataset.\n",
    "ds_size=ds.size_approx()\n",
    "     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset size in megabytes: 55.31311 MB\n",
      "Dataset size in gigabytes: 0.05402 GB\n"
     ]
    }
   ],
   "source": [
    "# Convert bytes to megabytes and limit to 5 decimal places\n",
    "ds_size_mb = ds_size / 1048576\n",
    "print(f\"Dataset size in megabytes: {ds_size_mb:.5f} MB\")\n",
    "\n",
    "# Convert bytes to gigabytes and limit to 5 decimal places\n",
    "ds_size_gb = ds_size / 1073741824\n",
    "print(f\"Dataset size in gigabytes: {ds_size_gb:.5f} GB\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "py3_11",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}