09_simple_rag.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyMt5uFF5RXwe1alQ5OVY+l0",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/badlogic/genai-workshop/blob/main/09_simple_rag.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Building a simple RAG\n",
        "Let's put what we've learned to practice. We want to build a chatbot that can answer questions related to a subset of the documentation of WinCC OA, namely the [Getting Started](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GETTINGSTARTED_DE.html) documentation. It should also output links to the original documentation where appropriate.\n",
        "\n",
        "> **Note:** Execute the code cells below, as the remainder of this section requires those helper functions."
      ],
      "metadata": {
        "id": "0QzdCZGj27gN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip -q install openai tiktoken umap-learn"
      ],
      "metadata": {
        "id": "BZ4Huheo3Oo1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ebca9fd0-d854-48d8-a056-8e8d53fd5465"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m226.7/226.7 kB\u001b[0m \u001b[31m3.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.8/1.8 MB\u001b[0m \u001b[31m11.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m90.9/90.9 kB\u001b[0m \u001b[31m3.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m55.8/55.8 kB\u001b[0m \u001b[31m5.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.8/77.8 kB\u001b[0m \u001b[31m9.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m6.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Building wheel for umap-learn (setup.py) ... \u001b[?25l\u001b[?25hdone\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "> **Note:** Enter your own OpenAI API key below."
      ],
      "metadata": {
        "id": "NNiwJ71mJC-M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from openai import OpenAI\n",
        "import tiktoken\n",
        "\n",
        "# Use your own OpenAI API key here.\n",
        "client = OpenAI(api_key = \"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\")\n",
        "\n",
        "messages = []\n",
        "model_name=\"gpt-3.5-turbo\"\n",
        "max_tokens = 12000\n",
        "temperature=0\n",
        "\n",
        "# Uncomment to use a model served locally via `ollama serve`\n",
        "# client = OpenAI(\n",
        "#    base_url = 'http://localhost:11434/v1',\n",
        "#    api_key='ollama', # required, but unused\n",
        "#)\n",
        "#model_name=\"mixtral:latest\"\n",
        "\n",
        "enc = tiktoken.get_encoding(\"cl100k_base\")\n",
        "def num_tokens(message):\n",
        "    return len(enc.encode(message))\n",
        "\n",
        "def truncate_messages(messages, max_tokens):\n",
        "    total_tokens = sum(num_tokens(message[\"content\"]) for message in messages)\n",
        "    if total_tokens <= max_tokens:\n",
        "        return messages\n",
        "\n",
        "    truncated_messages = messages[:1]\n",
        "    remaining_tokens = max_tokens - num_tokens(truncated_messages[0][\"content\"])\n",
        "    for message in reversed(messages[1:]):\n",
        "        tokens = num_tokens(message[\"content\"])\n",
        "        if remaining_tokens >= tokens:\n",
        "            truncated_messages.insert(1, message)\n",
        "            remaining_tokens -= tokens\n",
        "        else:\n",
        "            break\n",
        "    return truncated_messages\n",
        "\n",
        "def complete(message, max_response_tokens=2048, silent=False):\n",
        "    global messages\n",
        "    messages.append({\"role\": \"user\", \"content\": message})\n",
        "    truncated_messages = truncate_messages(messages, max_tokens=max_tokens)\n",
        "    stream = client.chat.completions.create(\n",
        "        model=model_name,\n",
        "        messages=truncated_messages,\n",
        "        stream=True,\n",
        "        temperature=temperature,\n",
        "        max_tokens=max_response_tokens\n",
        "    )\n",
        "    reply = \"\"\n",
        "    for response in stream:\n",
        "        token = response.choices[0].delta.content\n",
        "        if (token is None):\n",
        "            break\n",
        "        reply += token\n",
        "        if not silent:\n",
        "          print(token, end='')\n",
        "\n",
        "    reply = {\"role\": \"assistant\", \"content\": reply}\n",
        "    messages.append(reply)\n",
        "    total_tokens = sum(num_tokens(message[\"content\"]) for message in truncated_messages)\n",
        "    if not silent:\n",
        "      print(f'\\nTokens: {total_tokens}')\n",
        "\n",
        "def clear_history():\n",
        "  global messages\n",
        "  messages = [];\n",
        "\n",
        "def print_history():\n",
        "  global messages\n",
        "  for message in messages:\n",
        "    print(\"<\" + message[\"role\"] + \">\")\n",
        "    print(message[\"content\"])\n",
        "    print()\n",
        "\n",
        "def system_prompt(message):\n",
        "  global messages\n",
        "  prompt = { \"role\": \"system\", \"content\": message }\n",
        "  if (len(messages) == 0):\n",
        "    messages.append(prompt)\n",
        "  else:\n",
        "    messages[0] = prompt"
      ],
      "metadata": {
        "id": "lbo7v9pt3Pmf"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Data\n",
        "I've taken each raw HTML file from the Getting Started guide, converted it to corresponding Markdown offline, zipped the result and uploaded it to [https://marioslab.io/uploads/genai/wincc-oa-getting-started.zip](https://marioslab.io/uploads/genai/wincc-oa-getting-started.zip).\n",
        "\n",
        "Each file starts with a line containing the URL it came from.\n",
        "\n",
        "Let's fetch and load the data.\n"
      ],
      "metadata": {
        "id": "ivEekdfY27DG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!wget https://marioslab.io/uploads/genai/wincc-oa-getting-started.zip\n",
        "!unzip -o wincc-oa-getting-started.zip"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rUv0LfcS6Tjw",
        "outputId": "a92c1e73-9838-4680-fb8c-ec059a77c411"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-02-26 20:40:25--  https://marioslab.io/uploads/genai/wincc-oa-getting-started.zip\n",
            "Resolving marioslab.io (marioslab.io)... 95.216.8.184\n",
            "Connecting to marioslab.io (marioslab.io)|95.216.8.184|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 97486 (95K) [application/zip]\n",
            "Saving to: ‘wincc-oa-getting-started.zip’\n",
            "\n",
            "wincc-oa-getting-st 100%[===================>]  95.20K   293KB/s    in 0.3s    \n",
            "\n",
            "2024-02-26 20:40:26 (293 KB/s) - ‘wincc-oa-getting-started.zip’ saved [97486/97486]\n",
            "\n",
            "Archive:  wincc-oa-getting-started.zip\n",
            "  inflating: GettingStarted-01.md    \n",
            "  inflating: GettingStarted-02.md    \n",
            "  inflating: GettingStarted-06.md    \n",
            "  inflating: GettingStarted-09.md    \n",
            "  inflating: GettingStarted-12.md    \n",
            "  inflating: GettingStarted-13.md    \n",
            "  inflating: GettingStarted-14.md    \n",
            "  inflating: GettingStarted-20.md    \n",
            "  inflating: GettingStarted-21.md    \n",
            "  inflating: GettingStarted-22.md    \n",
            "  inflating: GettingStarted-23.md    \n",
            "  inflating: GettingStarted-24.md    \n",
            "  inflating: GettingStarted-25.md    \n",
            "  inflating: GettingStarted-26.md    \n",
            "  inflating: GettingStarted-27.md    \n",
            "  inflating: GettingStarted-28.md    \n",
            "  inflating: GettingStarted-30.md    \n",
            "  inflating: GettingStarted-31.md    \n",
            "  inflating: GettingStarted-32.md    \n",
            "  inflating: GettingStarted-33.md    \n",
            "  inflating: GettingStarted-34.md    \n",
            "  inflating: GettingStarted-35.md    \n",
            "  inflating: GettingStarted-36.md    \n",
            "  inflating: GettingStarted-37.md    \n",
            "  inflating: GettingStarted-39.md    \n",
            "  inflating: GettingStarted-40.md    \n",
            "  inflating: GettingStarted-41.md    \n",
            "  inflating: GettingStarted-42.md    \n",
            "  inflating: GettingStarted-44.md    \n",
            "  inflating: GettingStarted-45.md    \n",
            "  inflating: GettingStarted-46.md    \n",
            "  inflating: GettingStarted-47.md    \n",
            "  inflating: GettingStarted-50.md    \n",
            "  inflating: GettingStarted-52.md    \n",
            "  inflating: GettingStarted-53.md    \n",
            "  inflating: GettingStarted-54.md    \n",
            "  inflating: GettingStarted-56.md    \n",
            "  inflating: GettingStarted-57.md    \n",
            "  inflating: GettingStarted-59.md    \n",
            "  inflating: GettingStarted-62.md    \n",
            "  inflating: GettingStarted-63.md    \n",
            "  inflating: GettingStarted-64.md    \n",
            "  inflating: GettingStarted-66.md    \n",
            "  inflating: GettingStarted-68.md    \n",
            "  inflating: GettingStarted-73.md    \n",
            "  inflating: GettingStarted-75.md    \n",
            "  inflating: GettingStarted-76.md    \n",
            "  inflating: GettingStarted-78.md    \n",
            "  inflating: GettingStarted-79.md    \n",
            "  inflating: GettingStarted-83.md    \n",
            "  inflating: GettingStarted-84.md    \n",
            "  inflating: GettingStarted-87.md    \n",
            "  inflating: GettingStarted-89.md    \n",
            "  inflating: GettingStarted-91.md    \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, let's load the contents of all .md files. Each file is transformed into an object with fields `url`, `content`, `num_tokens`"
      ],
      "metadata": {
        "id": "YKbkz2Wk6nqj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import glob\n",
        "import os\n",
        "import pandas as pd\n",
        "\n",
        "documents = []\n",
        "for md_file in glob.glob(\"*.md\"):\n",
        "    with open(md_file, 'r', encoding='utf-8') as file:\n",
        "        lines = file.readlines()\n",
        "        url = lines[0].strip()\n",
        "        content = ''.join(lines[1:])\n",
        "        documents.append({\"url\": url, \"content\": content, \"num_tokens\": num_tokens(content)})\n",
        "\n",
        "df = pd.DataFrame(documents, columns=[\"url\", \"content\", \"num_tokens\"])\n",
        "print(documents[10][\"content\"])\n",
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "8IP_EHrU6nZW",
        "outputId": "ae90c367-60cd-4e0c-f31b-7303302f4cd0"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "# Data point Concept, Process Image\n",
            "\n",
            "The **variables** of the **process** that should be handled and monitored must also be found in the software in the control room. There has to be a variable that represents the value of every **logic state**, every **measured value** or **set value** within the system.\n",
            "\n",
            "Figure 1. Mapping of States, Guidelines and Measurements to Data points (Process Variables)\n",
            "\n",
            "![](image065.gif)\n",
            "\n",
            "![](image065.gif)\n",
            "\n",
            "|                  |                                         |      |       |\n",
            "| ---------------- | --------------------------------------- | ---- | ----- |\n",
            "| Data point       | Description                             | Unit | Value |\n",
            "| V02.state.closed | Valve V02 response, End position closed | -    | TRUE  |\n",
            "| V02.cmd.open     | Valve V02 command open                  | -    | FALSE |\n",
            "| ...              |                                         |      |       |\n",
            "| PI09.value       | Pressure measurement PI09 actual value  | bar  | 2,74  |\n",
            "| ...              |                                         |      |       |\n",
            "\n",
            "In WinCC OA these **variables of the process image** are called **\"data points\"**. Other names are often used, depending on the product or the region (tag, process variable, PV, item, Point, I/O point, ...)\n",
            "\n",
            "Whereas common SCADA systems assign one data point to each single process variable, WinCC OA chooses a more modern way: almost all information of the process belong logically to a more or less complex ensemble: **a device**.\n",
            "\n",
            "Figure 2. Signals of a real Device\n",
            "\n",
            "![](dp_concept_1.jpg)\n",
            "\n",
            "The practice shows that the number of **information belonging together (of such a device)** is typically between 4 and 30. Intelligent units like digital controllers, function modules or robots can easily exceed this number.\n",
            "\n",
            "Instead of making this logically close combined values to independent variables isolated of each other, WinCC OA defines **structured, device orientated data points**. The data points are defined in a **tree structure** with arbitrary branching levels.\n",
            "\n",
            "Figure 3. Structured Data point as a Mapping of a real Device (device orientation)\n",
            "\n",
            "![](dp_concept_2.jpg)\n",
            "\n",
            "The **values of the actual process values** are saved on **data point elements** (the outer leaves of this tree structure). Each process variable corresponds to a data point **element** within a data point. In addition, the tree structure can have as many nodes as necessary for a clear organization of the data.\n",
            "\n",
            "Each data point element is **addressed** individually by the **description chain within the structure**. The operating notification \"**Open**\" in the example above could for example be addressed through:\n",
            "\n",
            "Valve.Response.OperatingNotif.Open\n",
            "\n",
            "In practice you certainly want to save desk work and use short descriptions. WinCC OA, however, allows almost unlimited character strings.\n",
            "\n",
            "In addition to the **naming convention** and a **storage****location** for the actual value, specific **control functionalities** can be defined for a data point. This can be, for example, a value range check, the definition of an alert handling or a statistic calculation rule. **These functionalities defined for a data point element** are called \"**configs**\" in WinCC OA. Only configs that are also needed for a certain data point, are defined.\n",
            "\n",
            "## ****Data point Type and Data point****\n",
            "\n",
            "The user may configure **an appropriate data point type** for **each real device type** (drive, valve, stirring unit, controller, intrusion sensor, ...). A data point for **each real device** is derived from this data point type (kind of **template)**. In the object-oriented software engineering you would call the data point type a \"class\" and the representation of an individual device (the data point) an instance.\n",
            "\n",
            "Figure 4. Data points slider 1 to 3 as instances of the data point type \"valve\"\n",
            "\n",
            "![](dp_concept_inheritance.jpg)\n",
            "\n",
            "The **creation** and also the **configuration** of **numerousness process factors** that represent a device is therefore limited to a **single operation**. Already defined data point types that represent a module (for example service hour counters) can be set to be **a part of a new data point type**. This kind of hierarchically built data points (type-in-type) offer new, efficient engineering possibilities.\n",
            "\n",
            "Changes of the data point type also change the data points (instances).\n",
            "\n",
            "In conjunction with the structured data points and their inheritance concept between type and instance we often also talk about object orientation. This is not fully comparable with the object orientation of higher languages like C++, C# or Java. It is rather a question of the general approach to define data and methods (functions) as capsuled units and therefore guarantee their easy and efficient reusability. Structured data points correspond to a form of \"object\".\n",
            "\n",
            "![](image065.gif)\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                  url  \\\n",
              "0   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "1   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "2   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "3   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "4   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "5   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "6   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "7   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "8   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "9   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "10  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "11  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "12  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "13  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "14  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "15  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "16  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "17  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "18  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "19  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "20  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "21  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "22  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "23  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "24  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "25  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "26  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "27  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "28  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "29  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "30  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "31  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "32  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "33  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "34  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "35  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "36  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "37  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "38  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "39  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "40  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "41  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "42  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "43  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "44  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "45  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "46  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "47  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "48  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "49  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "50  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "51  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "52  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "53  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "\n",
              "                                              content  num_tokens  \n",
              "0   # Detect System Data\\n\\nIf you are not familia...         297  \n",
              "1   # Display of an Alert for an Object\\n\\n**Alert...        1995  \n",
              "2   # Functionality of the Device Oriented Data Ob...        1011  \n",
              "3   # Dialogs for Graphic Object - Test Panel\\n\\nI...         797  \n",
              "4   # Master Data Points\\n\\nIt is recommended to u...        1826  \n",
              "5   # Drivers\\n\\nThe **communication between contr...         806  \n",
              "6   # What is WinCC OA?\\n\\n\"WinCC OA\" is the abbre...         391  \n",
              "7   # After the Start, first Steps\\n\\nWhen a proje...         257  \n",
              "8   # Preview in the Graphic Editor\\n\\nYou can now...         275  \n",
              "9   # Addressing of Data Point Elements / Configs ...         899  \n",
              "10  # Data point Concept, Process Image\\n\\nThe **v...         995  \n",
              "11  # General Information\\n\\nWinCC OA possesses a ...         290  \n",
              "12  # Import the Current Project State\\n\\nYou shou...         418  \n",
              "13  # Emergency Mode\\n\\nWinCC OA automatically mon...         136  \n",
              "14  # Defining an Alert Handling for a Data Point\\...         697  \n",
              "15  # The Control Editor\\n\\n**Specific input help*...         734  \n",
              "16  # Naming Convention/ Notation\\n\\nAll **names**...        1164  \n",
              "17  # Dynamic Colors - Blinking\\n\\nWinCC OA allows...         286  \n",
              "18  # Dynamism of Graphic Elements\\n\\nIn the previ...        2094  \n",
              "19  # Data points as bearer of information\\n\\nIn W...        1696  \n",
              "20  # Modeling of Data Point Types\\n\\nIn the previ...         490  \n",
              "21  # Enable Historic Recording for a Data Point E...         774  \n",
              "22  # Scripting\\n\\nIn WinCC OA you can **create** ...         278  \n",
              "23  # Creating Data Points with the Database Edito...         787  \n",
              "24  # WinCC OA Getting Started - Tutorial\\n\\nThis ...         880  \n",
              "25  # Setting the Simulation\\n\\nHow to integrate t...         881  \n",
              "26  # Addressing\\n\\nIt doesn't matter where you wa...         399  \n",
              "27  # Panel Topology\\n\\nThe panel topology in WinC...        2415  \n",
              "28  # Creating Process Images - The Graphic Editor...         333  \n",
              "29  # Modules, Panels, Child Panels\\n\\nSo far only...         975  \n",
              "30  # Simple Draw Operations\\n\\nBy selecting a **g...        1027  \n",
              "31  # Input Dialog for Reference\\n\\n**The valves**...        1455  \n",
              "32  # Start, Stop\\n\\nAs soon as the desired projec...         639  \n",
              "33  # Configuration of the PowerConfig Settings fo...         603  \n",
              "34  # Installation - Activate TCP/IP Network\\n\\nOn...         414  \n",
              "35  # Add the Simulation\\n\\nIn order to configure ...         717  \n",
              "36  # Settings for a Data Point Element\\n\\nThe new...         206  \n",
              "37  # Import Current Project State\\n\\nNow all data...         105  \n",
              "38  # Handling of Modules, Panels and Child Panels...         229  \n",
              "39  # Installation Process\\n\\nThe installation des...         605  \n",
              "40  # Handling of Graphic Properties\\n\\nGraphic el...         834  \n",
              "41  # System, Distribution, Configuration\\n\\nA str...         512  \n",
              "42  # Import From ASCII File (Mass Engineering)\\n\\...         783  \n",
              "43  # Directory Structure\\n\\nEach project is saved...         386  \n",
              "44  # Control Manager (Runtime Scripts)\\n\\nIn addi...        1205  \n",
              "45  # Configuration of a Project\\n\\nExcept for the...         673  \n",
              "46  # Copy and Back up Project\\n\\nA project can be...         388  \n",
              "47  # Create Project\\n\\nIn WinCC OA, a **\"project\"...         886  \n",
              "48  # Task - Exercise project\\n\\nAll work steps wi...         950  \n",
              "49  # System Requirements\\n\\nYou can find the syst...         212  \n",
              "50  # Paths\\n\\nConsider that **neither** the insta...          79  \n",
              "51  # Creating a Device Symbol (Reference)\\n\\nIn a...        3803  \n",
              "52  # Architecture\\n\\nWinCC OA is a **modularly bu...        1024  \n",
              "53  # Standard Symbols\\n\\nThe easiest way to creat...         790  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-b7e06975-0e36-46b1-894e-522682ac583e\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>url</th>\n",
              "      <th>content</th>\n",
              "      <th>num_tokens</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Detect System Data\\n\\nIf you are not familia...</td>\n",
              "      <td>297</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Display of an Alert for an Object\\n\\n**Alert...</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Functionality of the Device Oriented Data Ob...</td>\n",
              "      <td>1011</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Dialogs for Graphic Object - Test Panel\\n\\nI...</td>\n",
              "      <td>797</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Master Data Points\\n\\nIt is recommended to u...</td>\n",
              "      <td>1826</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Drivers\\n\\nThe **communication between contr...</td>\n",
              "      <td>806</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># What is WinCC OA?\\n\\n\"WinCC OA\" is the abbre...</td>\n",
              "      <td>391</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># After the Start, first Steps\\n\\nWhen a proje...</td>\n",
              "      <td>257</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Preview in the Graphic Editor\\n\\nYou can now...</td>\n",
              "      <td>275</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Addressing of Data Point Elements / Configs ...</td>\n",
              "      <td>899</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Data point Concept, Process Image\\n\\nThe **v...</td>\n",
              "      <td>995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># General Information\\n\\nWinCC OA possesses a ...</td>\n",
              "      <td>290</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Import the Current Project State\\n\\nYou shou...</td>\n",
              "      <td>418</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Emergency Mode\\n\\nWinCC OA automatically mon...</td>\n",
              "      <td>136</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Defining an Alert Handling for a Data Point\\...</td>\n",
              "      <td>697</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># The Control Editor\\n\\n**Specific input help*...</td>\n",
              "      <td>734</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Naming Convention/ Notation\\n\\nAll **names**...</td>\n",
              "      <td>1164</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Dynamic Colors - Blinking\\n\\nWinCC OA allows...</td>\n",
              "      <td>286</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Dynamism of Graphic Elements\\n\\nIn the previ...</td>\n",
              "      <td>2094</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Data points as bearer of information\\n\\nIn W...</td>\n",
              "      <td>1696</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Modeling of Data Point Types\\n\\nIn the previ...</td>\n",
              "      <td>490</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Enable Historic Recording for a Data Point E...</td>\n",
              "      <td>774</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Scripting\\n\\nIn WinCC OA you can **create** ...</td>\n",
              "      <td>278</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Creating Data Points with the Database Edito...</td>\n",
              "      <td>787</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># WinCC OA Getting Started - Tutorial\\n\\nThis ...</td>\n",
              "      <td>880</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Setting the Simulation\\n\\nHow to integrate t...</td>\n",
              "      <td>881</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Addressing\\n\\nIt doesn't matter where you wa...</td>\n",
              "      <td>399</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Panel Topology\\n\\nThe panel topology in WinC...</td>\n",
              "      <td>2415</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Creating Process Images - The Graphic Editor...</td>\n",
              "      <td>333</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Modules, Panels, Child Panels\\n\\nSo far only...</td>\n",
              "      <td>975</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Simple Draw Operations\\n\\nBy selecting a **g...</td>\n",
              "      <td>1027</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Input Dialog for Reference\\n\\n**The valves**...</td>\n",
              "      <td>1455</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Start, Stop\\n\\nAs soon as the desired projec...</td>\n",
              "      <td>639</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>33</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Configuration of the PowerConfig Settings fo...</td>\n",
              "      <td>603</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Installation - Activate TCP/IP Network\\n\\nOn...</td>\n",
              "      <td>414</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Add the Simulation\\n\\nIn order to configure ...</td>\n",
              "      <td>717</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Settings for a Data Point Element\\n\\nThe new...</td>\n",
              "      <td>206</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Import Current Project State\\n\\nNow all data...</td>\n",
              "      <td>105</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Handling of Modules, Panels and Child Panels...</td>\n",
              "      <td>229</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>39</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Installation Process\\n\\nThe installation des...</td>\n",
              "      <td>605</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>40</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Handling of Graphic Properties\\n\\nGraphic el...</td>\n",
              "      <td>834</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>41</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># System, Distribution, Configuration\\n\\nA str...</td>\n",
              "      <td>512</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>42</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Import From ASCII File (Mass Engineering)\\n\\...</td>\n",
              "      <td>783</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>43</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Directory Structure\\n\\nEach project is saved...</td>\n",
              "      <td>386</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>44</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Control Manager (Runtime Scripts)\\n\\nIn addi...</td>\n",
              "      <td>1205</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>45</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Configuration of a Project\\n\\nExcept for the...</td>\n",
              "      <td>673</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>46</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Copy and Back up Project\\n\\nA project can be...</td>\n",
              "      <td>388</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>47</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Create Project\\n\\nIn WinCC OA, a **\"project\"...</td>\n",
              "      <td>886</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>48</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Task - Exercise project\\n\\nAll work steps wi...</td>\n",
              "      <td>950</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># System Requirements\\n\\nYou can find the syst...</td>\n",
              "      <td>212</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Paths\\n\\nConsider that **neither** the insta...</td>\n",
              "      <td>79</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>51</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Creating a Device Symbol (Reference)\\n\\nIn a...</td>\n",
              "      <td>3803</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>52</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Architecture\\n\\nWinCC OA is a **modularly bu...</td>\n",
              "      <td>1024</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>53</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Standard Symbols\\n\\nThe easiest way to creat...</td>\n",
              "      <td>790</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-b7e06975-0e36-46b1-894e-522682ac583e')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-b7e06975-0e36-46b1-894e-522682ac583e button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-b7e06975-0e36-46b1-894e-522682ac583e');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-6d9108e5-3d94-41d9-83e3-bb75b8c93f20\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-6d9108e5-3d94-41d9-83e3-bb75b8c93f20')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-6d9108e5-3d94-41d9-83e3-bb75b8c93f20 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 54,\n  \"fields\": [\n    {\n      \"column\": \"url\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 54,\n        \"samples\": [\n          \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-27.html\",\n          \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-13.html\",\n          \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-26.html\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"content\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 54,\n        \"samples\": [\n          \"# Data points as bearer of information\\n\\nIn WinCC OA it's possible to individually create device orientated data points. The basic concept of such a structured data point was introduced in the section [Data point Concept, Process Image](GettingStarted-12.html). In the following sections the practical use of data points combined with detailed background information will be provided.\\n\\nIn order to work with WinCC OA and to be able to create user interfaces for the use, the **carrier of the presented information** must be defined. These carriers are variables whose values present the current process information. Such **process variables** are called **data point elements** in WinCC OA.\\n\\nIn order to display existing data points or to create new ones, you usually use the **database editor PARA**. It can be opened with the already opened graphic editor GEDI using the ![](image152.gif) symbol.\\n\\nFigure 1. Database Editor PARA\\n\\n![](GettingStarted-37.png)\\n\\nAlternatively, we could also define the database editor as its own manager in the console and start it from the console. Therefore, open the manager selection dialog (see figure on the page [Configuration of a Project](GettingStarted-23.html)) through the ![](image123.gif) button, select the entry WCCOAui and enter the parameter \\\"**-m para**\\\" in the \\\"**Options**\\\" field. Consider that the restart option is set to \\\"**manual**\\\".\\n\\n## Data point types\\n\\nThe list in the field on the left side (in the figure above) shows all **data point types** available in the current project. Each of these data point types represents a whole class of **devices or logical units**. If you click on the \\\"+\\\" sign in front of the name of the data point type, a list of **data points** (instances/devices) of this type will be shown.\\n\\nA **data point type is a form of template** for structured data points. **Structure**, **naming** and partly also configurations are already configured when defining the type.\\n\\nFigure 2. Data point Type \\\"PUMP1\\\" as a Template of a whole Class of Devices\\n\\n![](para_dpType_PUMP1.gif)\\n\\nBefore it's possible to create a structured data point as a representative of a device, a corresponding data point type has to be created as a template.\\n\\n## Data points\\n\\nEach **data point** can represent a **real device** or a **logical combination of information**. It is built up of one or several, almost arbitrary structured **data point elements**. Each data point belongs to its data point type and is accordingly shown in the module PARA. A double click on the name of the data point type or the \\\"+\\\" symbol in the front of the name opens a list of all data points of one type.\\n\\nThe following image shows **two types****available by default** and their data points (instances). The data point type **ExampleDP\\\\_Float has 4 data points (instances) in this case. This data point type consists of**only one data point element**and can therefore only represent a single process variable. This corresponds to the common data model of most SCADA systems but is an exception in WinCC OA. This type (and also the other** ExampleDP\\\\* types) are merely available for **test or demonstration purposes.**\\n\\nFigure 3. Existing Instances of the Data point Types ExampleDP\\\\_Float and PUMP1\\n\\n![](GettingStarted-64.png)\\n\\nThe **data point type** PUMP1 (displayed at the right) corresponds more to the common structure in WinCC OA. Numerous process variables are **part of the device-oriented data point**. At the start, PUMP1 has just a single data point instance. Even this is an exception: it is a **master data point (MP)** that can display certain configuration information for the type. The [master data points](GettingStarted-33.html) will be described later. There are no real data point instances available for the type PUMP1 since this is a new, initially empty project.\\n\\nFigure 4. Database Editor PARA - Data point Type ExampleDP\\\\_Float with data point ExampleDP\\\\_Arg1\\n\\n![](GettingStarted-65.png)\\n\\nData point types and data points only exist within a project. Each of the projects on a computer can have other data point types and data points even with the same names. In order to transfer existing data point types and data points from a project to another one, the [Import/Export](GettingStarted-37.html) function of the ASCII manager is used. For more information, see [ASCII Manager Panel, basics](../Manager_ASCII/Manager_ASCII.html).\\n\\n## Information at the data point element\\n\\nEach data point element primarily represents **the value** of a process variable, however there is some important additional information about this value. It's a **complete process****image** since it maps all **the essential attributes** of a recorded value.\\n\\nFigure 5. Information on the data point Element - Value with Attributes (Time, Quality, Origin)\\n\\n![](vtqs.gif)\\n\\nThese additional attributes of the value could also be seen in the database editor PARA. Additionally, you can see that there are **two different fields for value display**.\\n\\n![](image065.gif)\\n\\n|                |                                                                                                                                                                                                                                                                                             |\\n| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\\n| Original value | The value that was originally recorded (e.g. received from a controller). This original value always remains and can be queried also after automatic corrections (substitute value enabled). Active write operations are always performed on the original value.                            |\\n| Online value   | All function modules within WinCC OA work with this value. Normally, it is an identic copy of the original value. Only if a value is noted as invalid, a substitute value might be displayed. When reading a data point element (display, processing), the online value is always accessed. |\\n\\nFor more information on the **original value**, see the chapter [\\\\_original (Original attributes)](../Referenz_PARA/Referenz_PARA-19.html)\\n\\n## Data types\\n\\nData point **elements** can represent quite **different information**. In addition to the common binary and analog variables also **higher and structured data types** are available. Therefore, the data points can not only be used for the pure representation of process states of a control system but also for **administration tasks**, **configurations**, **recipes** and the like.\\n\\n![](image065.gif)\\n\\n|                       |               |                                                                                                                               |\\n| --------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------- |\\n| Data type             | Data Width    | Meaning                                                                                                                       |\\n| bool                  | 1 Bit         | Binary variable                                                                                                               |\\n| int                   | 32 Bit        | Integer                                                                                                                       |\\n| unsigned              | 32 Bit        | Unsigned integer                                                                                                              |\\n| float                 | 64 Bit        | Float                                                                                                                         |\\n| bit32                 | 32 Bit        | Bit pattern                                                                                                                   |\\n| char                  | 8 Bit         | Character                                                                                                                     |\\n| string                | n x 8 Bit     | String (n characters long)                                                                                                    |\\n| langString            | m x n x 8 Bit | Multilingual string (n languages)                                                                                             |\\n| time                  | 48 Bit        | Time / length                                                                                                                 |\\n| dpid                  | -             | Data point identifier                                                                                                         |\\n| typeref               | -             | Type reference                                                                                                                |\\n| blob                  | arbitrary     | Binary structure (**b**inary **l**arge **o****b**ject)                                                                        |\\n| dyn\\\\_\\u2026(dyn\\\\_string,\\u2026) |               | Dynamic fields of above mentioned basic types (the used size and required space can be changed for each instance dynamically) |\\n| array\\u2026(char array,\\u2026)  |               | Static fields of above mentioned basic types (the number of field elements is already defined when configuring).              |\\n\\nThe data type of a data point element is already defined **when creating the data point type** (together with the identification schema). This data type can **be changed later** which results in the change of all data points of this type.\\n\\nWhen using floating point numbers, a point \\\".\\\" is used as the decimal separator. This is conform to the worldwide programming standard C/C++.\\n\\n![](image065.gif)\\n\",\n          \"# System Requirements\\n\\nYou can find the system requirements [Hardware and software requirements for WinCC OA](../WCCOA_Voraussetzungen/Voraussetzungen-01.html) .\\n\\n**Note before installing the software** that the **system requirements** are met. The hardware requirements strongly depend on the size of the project and the dynamic of the process factors. Use quality hardware with corresponding functionalities such as redundant power supplies or RAID hard disks. WinCC OA supports dual and multi processor operation and benefits from it. As in case of RAM, CPU and HDD: the more the better.\\n\\nIf you need a tutorial that shows how to check this system data, see section [determine system data](GettingStarted-83.html).\\n\\nIn case of modern Windows models, the configuration and activation of the **TCP/IP network protocol** is already completed with the installation of the operating system. If this option is not available on your computer, you have to activate it manually (see also [Installation - activate TCP/IP network](GettingStarted-84.html).\\n\",\n          \"# Task - Exercise project\\n\\nAll work steps will be explained via small examples in order to keep the projection sequences of WinCC OA as practical as possible. The **end result** looks like this:\\n\\nFigure 1. Process Image \\\"Process tanks\\\" (process.pnl) embedded in a general User Interface\\n\\n![](GettingStarted-34.png)\\n\\nThe core of our application is a **small tank system**. The first **tank** (T1) is filled through a **feed valve** (V1). The liquid medium is transferred to an open **reservoir** (T2) by a second **valve** (V2) and a **pump** (P1). The outflow of the **reservoir** (T2) is controlled by another valve (V3). The procedural sense of the configuration is not considered. The primary goal of this example is to learn the **modeling of devices** in form of **structured data points** and to prepare these devices graphically for the use.\\n\\nAt first, we have the following devices:\\n\\n|                                                |        |                    |\\n| ---------------------------------------------- | ------ | ------------------ |\\n| Device type                                    | Number | Device (instances) |\\n| Pump (rotation speed variable / FU)            | 1      | P1                 |\\n| Fill level sensors (level measurement)         | 2      | T1, T2             |\\n| Engine operated shut-off valve (valve, slider) | 3      | V1, V2, V3         |\\n\\n## ****Pump (**** GS\\\\_PUMP****)****\\n\\nIt should be possible to switch between manual operation and automatic operation for the pump P1. In the manual operation it's possible to specify the rotation speed between 0...100 (in %) and to switch the pump on and off. The operating state (running, stopped, rotation speed, manual operation) as well as a possible alarm should be displayed for the user.\\n\\n## ****Fill level sensor (**** GS\\\\_LEVEL****)****\\n\\nThe fill level sensor provides an absolute level in meters. In addition to the automatic value range monitoring, the user should also be alarmed in case of a too low or too high fill level. Although the tank T1 and the reservoir T2 are designed differently, the level can be measured identically.\\n\\n## ****Shut off valve (**** GS\\\\_VALVE****)****\\n\\nIt should be possible to switch between manual and automatic operation for the valves (slider). The commands **On**/ **Stop**/ **Off** as well as a specification of the opening level should be possible in the manual mode. The valves should show the operating status (**end position****open**, **end position closed**), the current opening level in %, manual operation and a possible alarm.\\n\\nAll **commands** should be entered using **own dialogs**, starting from the device symbol (two-stage operation). The properties of these devices correspond approximate to the following simplified **measuring/message list**. The necessary information is repeated later by numerous step by step instructions.\\n\\nFigure 2. Simplified Device List for the Exercise Example\\n\\n![](simple_device_list.gif)\\n\\nThe figures of the devices as well as their **operation dialogs** are modeled as **reusable objects** in WinCC OA. The created process image should be included in a superior, flexible configurable **operation image**. In addition to the process image, a detailed image should show the chronological sequence of the levels in T1 and T2 also graphically (**trend**). An **overview image** should facilitate the navigation to the process image (tanks) as well as to other further parts of the plant.\\n\\nFigure 3. Chronological Sequence of the Levels in a Trend\\n\\n![](GettingStarted-35.png)\\n\\nFigure 4. Overview Image with combined Navigation and Sum Alert Elements\\n\\n![](GettingStarted-36.png)\\n\\nIdentifier in WinCC OA depend on the exact [notation](GettingStarted-91.html) including upper and lower case (case sensitivity). This applies to data point (element) names as well as variable identifiers or the names of graphic objects (buttons, lines, ....). Consider that all names correspond exactly to the notation specified here, since prepared application parts are used later and these have to suitable for your objects/names.\\n\\nPrepared [Simulation programs](GettingStarted-62.html) that can be simply included in the self created application, enable a simple use with confirmations even without connection to a real process.\\n\\nThe complete configuration exercise project is also available as the application GettingStarted\\\\_3.19. If necessary, you can reconstruct some steps using the ready application or also adopt parts of it.\\n\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"num_tokens\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 661,\n        \"min\": 79,\n        \"max\": 3803,\n        \"num_unique_values\": 54,\n        \"samples\": [\n          1696,\n          212,\n          950\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next we split all documents which are larger than 1024 tokens into smaller chunks of maximally 1024 tokens. Looking at the data frame table above, there are only a handful of documents bigger than 1024 tokens.\n",
        "\n",
        "Let's write a function that does the splitting into chunks for us.\n",
        "\n"
      ],
      "metadata": {
        "id": "uB6UwYe59ltl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def chunk(documents, max_tokens):\n",
        "    chunked_documents = []\n",
        "\n",
        "    for doc in documents:\n",
        "        content = doc['content']\n",
        "        if num_tokens(content) <= max_tokens:\n",
        "            chunked_documents.append(doc)\n",
        "        else:\n",
        "            content_parts = []\n",
        "            words = content.split()\n",
        "            current_part = []\n",
        "            for word in words:\n",
        "                current_part.append(word)\n",
        "                if num_tokens(' '.join(current_part)) > max_tokens:\n",
        "                    current_part.pop()\n",
        "                    content_parts.append(' '.join(current_part))\n",
        "                    current_part = [word]\n",
        "            if current_part:\n",
        "                content_parts.append(' '.join(current_part))\n",
        "\n",
        "            for part in content_parts:\n",
        "                chunked_documents.append({'url': doc['url'], 'content': part, 'num_tokens': num_tokens(part)})\n",
        "\n",
        "    return chunked_documents"
      ],
      "metadata": {
        "id": "8UPry-FU-n_B"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "chunked_documents = chunk(documents, 1024)"
      ],
      "metadata": {
        "id": "sEk6e_zP-vBo"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pd.DataFrame(chunked_documents, columns=[\"url\", \"content\", \"num_tokens\"])"
      ],
      "metadata": {
        "id": "wOsUT2mD-26y",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 423
        },
        "outputId": "bcf5cd2d-dae3-47b0-fe93-55be13521940"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                  url  \\\n",
              "0   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "1   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "2   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "3   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "4   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "..                                                ...   \n",
              "63  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "64  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "65  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "66  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "67  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "\n",
              "                                              content  num_tokens  \n",
              "0   # Detect System Data\\n\\nIf you are not familia...         297  \n",
              "1   # Display of an Alert for an Object **Alerts**...        1024  \n",
              "2   However, there is one new characteristic: the ...         988  \n",
              "3   # Functionality of the Device Oriented Data Ob...        1011  \n",
              "4   # Dialogs for Graphic Object - Test Panel\\n\\nI...         797  \n",
              "..                                                ...         ...  \n",
              "63  `.../panels/objects/GS_VALVE_ref.pnl`*.*The lo...        1024  \n",
              "64  data point element is: Figure 10. Structure of...        1024  \n",
              "65  objects** can be created **for universal use**...         756  \n",
              "66  # Architecture\\n\\nWinCC OA is a **modularly bu...        1024  \n",
              "67  # Standard Symbols\\n\\nThe easiest way to creat...         790  \n",
              "\n",
              "[68 rows x 3 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-c61fecbf-c924-48eb-a0c2-83594971ed72\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>url</th>\n",
              "      <th>content</th>\n",
              "      <th>num_tokens</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Detect System Data\\n\\nIf you are not familia...</td>\n",
              "      <td>297</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Display of an Alert for an Object **Alerts**...</td>\n",
              "      <td>1024</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td>However, there is one new characteristic: the ...</td>\n",
              "      <td>988</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Functionality of the Device Oriented Data Ob...</td>\n",
              "      <td>1011</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Dialogs for Graphic Object - Test Panel\\n\\nI...</td>\n",
              "      <td>797</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>63</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td>`.../panels/objects/GS_VALVE_ref.pnl`*.*The lo...</td>\n",
              "      <td>1024</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>64</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td>data point element is: Figure 10. Structure of...</td>\n",
              "      <td>1024</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td>objects** can be created **for universal use**...</td>\n",
              "      <td>756</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Architecture\\n\\nWinCC OA is a **modularly bu...</td>\n",
              "      <td>1024</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>67</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Standard Symbols\\n\\nThe easiest way to creat...</td>\n",
              "      <td>790</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>68 rows × 3 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c61fecbf-c924-48eb-a0c2-83594971ed72')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-c61fecbf-c924-48eb-a0c2-83594971ed72 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-c61fecbf-c924-48eb-a0c2-83594971ed72');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-5fea5888-3b84-47bc-8321-6320e059d06a\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-5fea5888-3b84-47bc-8321-6320e059d06a')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-5fea5888-3b84-47bc-8321-6320e059d06a button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"pd\",\n  \"rows\": 68,\n  \"fields\": [\n    {\n      \"column\": \"url\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 54,\n        \"samples\": [\n          \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-27.html\",\n          \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-13.html\",\n          \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-26.html\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"content\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 68,\n        \"samples\": [\n          \"# Settings for a Data Point Element\\n\\nThe new data point P2 is ready for use in our application. However, it is often desirable to make some further settings for the elements. For example, every element can carry the following additional information:\\n\\n* **Description**\\n\\n* **Alias address** (second internal notation system in addition to the data point description)\\n\\n* **Number format** (number of digits before/after the decimal point)\\n\\n* **Unit**\\n\\n**Figure: Enter the Description Text, Number Format and Unit for the Data point Element (config: \\\\_common)**\\n\\nThe settings can be displayed by a \\\\[leftMouseClick] on the \\\"\\\\_common\\\" sub element (config) for each data point element.\\n\\nEnter the settings for the data point P2 as shown in the figure above.\\n\\nIn a multilingual project you can enter the description multilingual. For more information on multilingual projects, see [Multilingual Projects, Basics](../Mehrsprachigkeit/Mehrsprachigkeit.html).\\n\",\n          \"# Defining an Alert Handling for a Data Point\\n\\nAn **alert handling** is a **property** that can be provided **per data point element**. A **certain alert state** (continuous or discrete) can be assigned to a range of values of the element. While binary data point elements can accept two states, the value range of a analog value can be separated to endless **monitoring ranges**. Accordingly, for binary alert handlings, an alert can be assigned exactly to one of the two possible states. For an analog value, there are generally 2,3 or 5 ranges in which the whole value range of the data point element will be divided (maximum 255).\\n\\nIn order to **configure an alert handling**, select the associated element in the database editor PARA and add an alert handling as a **config**. This was already described in the chapter [Functionality of the Device Oriented Data Object - Configs](GettingStarted-32.html).\\n\\nFigure 1. Configuration of an analog Alert Handling with 3 Ranges\\n\\n![](GettingStarted-51.png)\\n\\nIf you work with **master data points**, the alert handling will be added as a **PowerConfig** directly to the respective element of the master data point. You can then set alert limits and other settings using \\\"**Configure PowerConfig**\\\" on the particular element of a data point instance.\\n\\nIf you work **without master data point**, add the alert handling directly to the data point element as a normal config \\\"**alarm handling**\\\" (\\\\_alert\\\\_hdl).\\n\\nIn case of analog alert handlings you can specify **hysteresis** for each alert range so that when thresholds are exceeded or underrun continuously, new alerts will not be triggered permanently. A number of properties is additionally assigned to each **alert range**. These properties are, for example:\\n\\n* Priority of the alert (Priority of the alert range)\\n\\n* State model of the alert\\n\\n* Required **authorization for acknowledging** the alert\\n\\n* Standard color that is used to signalize the color\\n\\n* Historic recording of the alert history (**storage**)\\n\\n* Message actions\\n\\n* ...\\n\\nSo that you do not have to define these properties for each individual alert range again, **alert classes** exist in WinCC OA. An alert class comprises such properties so that you only have to **refer** to the associated alert class in case of an individual alert range of a data point element. In the previous figure, you can see such references to alert classes on the right in the \\\"alert class\\\" column next to the high and low alerts. The middle row shows the **OK range** of the value so that you do **not have to define an alert class** here.\\n\\nThe indication of the alert class is in the example \\\"060\\\\_alert\\\". This means priority = \\\"60\\\" and class = \\\"alert\\\". **Alert****classes** are specified in WinCC OA by using a **config with the same name**. You can find all alert classes existing by default in the module PARA through the internal data point type \\\"\\\\_AlertClass\\\" and data points of the same name. In order to view the existing alert classes, open the module PARA, activate the check box \\\"**Internal data points**\\\" above the tree view and navigate to the type \\\\_AlertClass.\\n\\nFigure 2. Summary of Alert Properties through Alert Classes (the DP presents an alert class)\\n\\n![](GettingStarted-52.png)\\n\",\n          \"# Dialogs for Graphic Object - Test Panel\\n\\nIn order to test the created symbol view of the pump regarding its dynamic behavior, add a **test panel** to the symbol. If you \\\\[right click] on the pump symbol P1 at run time, a universal test panel should open which can be used to **monitor** and **change** the values of all data point elements of P1. This test panel is a part of WinCC OA (scope of delivery) and you only have to call it.\\n\\nFigure 1. Open a Child Panel through \\\\[right mouse click] - Configuration in Wizards\\n\\n![](image569.png)\\n\\n1. Select the circle symbol of the pump P1 in the process.pnl. panel\\n\\n2. Select the standard tab in the properties panel and click on the \\\"**Open attribute wizards**\\\" symbol next to \\\"**RightMousePressed**\\\".\\n\\n3. Select \\\"**Panel functions**\\\" in the first step of the wizards and click on **Next**.\\n\\n4. Select the desired functionality \\\"**Open panel**\\\" (in a new module) in the next step and click on **Next**.\\n\\n5. Open the file selection dialog ![](image295.gif) and select the version directory (for example. C:/Siemens/Automation/WinCC\\\\_OA/\\\\<version>/panels/). You must switch between the current project directory and version directory since the desired test panel is a part of WinCC OA.\\n\\n6. Select the file dp\\\\_monitor.pnl from the directory \\\\<wincc\\\\_oa\\\\_path>/panels/vision/ and confirm the dialog with **Open**.\\n\\n7. In the field \\\"**Panel name:**\\\" enter the text \\\"**Monitor** P1\\\" - this will be the title of the panel that will be opened later.\\n\\n8. When you select the panel to be opened, also the **parameters** for this panel will be shown. This means that the panel will receive further information when it will be opened. The first parameter $b\\\\_editable specifies whether the user may input data into the panel, the second parameter $dp specifies that you should enter the data point to be shown.\\n\\n9. Click on $b\\\\_editable in the selection list so that the entry will be **applied** to the fields \\\"**Parameter name**\\\" and \\\"**Parameter**\\\" in the new panel.\\n\\n10. Enter the value 1 or TRUE to the field \\\"**Parameter**\\\" to be able to input values later. Confirm the input by clicking on the Apply button with the green checkmark ![](image297.gif). Your input will be applied **to the list**. Note: If you would enter \\\"0\\\" or \\\"**FALSE**\\\", the test panel would be opened \\\"**ReadOnly**\\\".\\n\\n11. Click on the $dp in the selection list and enter the text \\\"P1\\\" in the field Parameter. You should be able to monitor and change all data point elements of the pump P1 in the test panel after opening.\\n\\n12. Confirm this entry by clicking on ![](image297.gif) and proceed with **Next**.\\n\\n13. Select that the window should be opened as \\\"**Child panel in the calling panel\\\", \\\"relative to the calling object**\\\" with an offset of x = 60 and y = -70 (to the right of the object). Close the configuration by finishing it.\\n\\nFigure 2. Configuration Process for opening a Panel including Parameter Passing\\n\\n![](299.png)\\n\\n![](GettingStarted-105.png)\\n\\nThe parameters passed to the new panel are also called \\\"**Dollar parameters**\\\" in WinCC OA. For more information about the $ parameters and their general meaning, see chapters [Creating a device symbol (reference)](GettingStarted-52.html) and [Input Dialog for Reference](GettingStarted-53.html)\\n\\nFigure 3. Specifying the Window Properties and the Result in the Runtime View (Preview GEDI)\\n\\n![](image309.png)\\n\\n![](image310.png)\\n\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"num_tokens\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 329,\n        \"min\": 20,\n        \"max\": 1024,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          297,\n          803,\n          371\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Index & retrieval\n",
        "We have a total of 68 chunks, which is ridiculously small. We do not really need a full-blown vector store like [Chroma](https://www.trychroma.com/) or [Pinecone](https://www.pinecone.io/).\n",
        "\n",
        "Instead, we'll build a simple in-memory vector store. To embed our chunks, We'll use [OpenAI's embedding API](https://platform.openai.com/docs/guides/embeddings/what-are-embeddings), which is cheap, fast, and produces great embeddings.\n",
        "\n",
        "Let's write a function that takes our list of `chunked_documents`, and embeds each chunk's text using OpenAI's `text-embedding-3-small` embedder. Each returned vector will have a lenght of 1536 dimensions. We batch submission to the OpenAI API."
      ],
      "metadata": {
        "id": "ELMiJxm0_Ee-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def embed_chunks(chunks):\n",
        "    batch = []\n",
        "    batch_tokens = 0\n",
        "    for chunk in chunks:\n",
        "        if batch_tokens + chunk['num_tokens'] <= 7000:\n",
        "            batch.append(chunk)\n",
        "            batch_tokens += chunk['num_tokens']\n",
        "        else:\n",
        "            if batch:\n",
        "                response = client.embeddings.create(\n",
        "                    input=[chunk['content'] for chunk in batch],\n",
        "                    model=\"text-embedding-3-small\"\n",
        "                )\n",
        "                for chunk, data in zip(batch, response.data):\n",
        "                    chunk['vector'] = data.embedding\n",
        "\n",
        "                batch = [chunk]\n",
        "                batch_tokens = chunk['num_tokens']\n",
        "\n",
        "    if batch:\n",
        "        response = client.embeddings.create(\n",
        "            input=[chunk['content'] for chunk in batch],\n",
        "            model=\"text-embedding-3-small\"\n",
        "        )\n",
        "        for chunk, data in zip(batch, response.data):\n",
        "            chunk['vector'] = data.embedding"
      ],
      "metadata": {
        "id": "zkBN1BM2AceF"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "embed_chunks(chunked_documents)"
      ],
      "metadata": {
        "id": "8RVP2sSzA5qO"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pd.DataFrame(chunked_documents, columns=[\"url\", \"content\", \"vector\", \"num_tokens\"])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 597
        },
        "id": "Yfb89vxWCEQz",
        "outputId": "9394617f-f830-45db-9d1d-27cdf466e817"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                  url  \\\n",
              "0   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "1   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "2   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "3   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "4   https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "..                                                ...   \n",
              "63  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "64  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "65  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "66  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "67  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "\n",
              "                                              content  \\\n",
              "0   # Addressing of Data Point Elements / Configs ...   \n",
              "1   # Simple Draw Operations By selecting a **grap...   \n",
              "2   Basics](../Native_GEDI/Referenz_Native_GEDI.ht...   \n",
              "3   # Control Manager (Runtime Scripts) In additio...   \n",
              "4   scripts for the simulation run in a single CON...   \n",
              "..                                                ...   \n",
              "63  # Panel Topology The panel topology in WinCC O...   \n",
              "64  on \"**OK**\". 15. A dialog box pop-ups: \"Apply ...   \n",
              "65  ![](pt_configur_directAccess_collage_de.png) D...   \n",
              "66  # System Requirements\\n\\nYou can find the syst...   \n",
              "67  # Installation - Activate TCP/IP Network\\n\\nOn...   \n",
              "\n",
              "                                               vector  num_tokens  \n",
              "0   [0.004041814710944891, 0.03711238130927086, 0....         899  \n",
              "1   [-0.042427342385053635, 0.05359514430165291, 0...        1013  \n",
              "2   [-0.05739807337522507, 0.043246570974588394, 0...          20  \n",
              "3   [-0.017868608236312866, 0.07958424091339111, 0...        1024  \n",
              "4   [0.016829390078783035, 0.04174959659576416, 0....         175  \n",
              "..                                                ...         ...  \n",
              "63  [-0.052065569907426834, 0.02563834935426712, 0...        1024  \n",
              "64  [-0.04392346367239952, 0.02678963914513588, 0....        1018  \n",
              "65  [-0.01521751843392849, 0.013226008042693138, 0...         371  \n",
              "66  [-0.057610463351011276, 0.03982636332511902, 0...         212  \n",
              "67  [-0.040496695786714554, -0.000200380411115475,...         414  \n",
              "\n",
              "[68 rows x 4 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-f37cc32c-d0d9-413f-bc6a-378289494e38\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>url</th>\n",
              "      <th>content</th>\n",
              "      <th>vector</th>\n",
              "      <th>num_tokens</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Addressing of Data Point Elements / Configs ...</td>\n",
              "      <td>[0.004041814710944891, 0.03711238130927086, 0....</td>\n",
              "      <td>899</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Simple Draw Operations By selecting a **grap...</td>\n",
              "      <td>[-0.042427342385053635, 0.05359514430165291, 0...</td>\n",
              "      <td>1013</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td>Basics](../Native_GEDI/Referenz_Native_GEDI.ht...</td>\n",
              "      <td>[-0.05739807337522507, 0.043246570974588394, 0...</td>\n",
              "      <td>20</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Control Manager (Runtime Scripts) In additio...</td>\n",
              "      <td>[-0.017868608236312866, 0.07958424091339111, 0...</td>\n",
              "      <td>1024</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td>scripts for the simulation run in a single CON...</td>\n",
              "      <td>[0.016829390078783035, 0.04174959659576416, 0....</td>\n",
              "      <td>175</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>63</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Panel Topology The panel topology in WinCC O...</td>\n",
              "      <td>[-0.052065569907426834, 0.02563834935426712, 0...</td>\n",
              "      <td>1024</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>64</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td>on \"**OK**\". 15. A dialog box pop-ups: \"Apply ...</td>\n",
              "      <td>[-0.04392346367239952, 0.02678963914513588, 0....</td>\n",
              "      <td>1018</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>65</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td>![](pt_configur_directAccess_collage_de.png) D...</td>\n",
              "      <td>[-0.01521751843392849, 0.013226008042693138, 0...</td>\n",
              "      <td>371</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># System Requirements\\n\\nYou can find the syst...</td>\n",
              "      <td>[-0.057610463351011276, 0.03982636332511902, 0...</td>\n",
              "      <td>212</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>67</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Installation - Activate TCP/IP Network\\n\\nOn...</td>\n",
              "      <td>[-0.040496695786714554, -0.000200380411115475,...</td>\n",
              "      <td>414</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>68 rows × 4 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f37cc32c-d0d9-413f-bc6a-378289494e38')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-f37cc32c-d0d9-413f-bc6a-378289494e38 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-f37cc32c-d0d9-413f-bc6a-378289494e38');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-53eb2b61-f825-43fc-8564-84a597d496a4\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-53eb2b61-f825-43fc-8564-84a597d496a4')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-53eb2b61-f825-43fc-8564-84a597d496a4 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"pd\",\n  \"rows\": 68,\n  \"fields\": [\n    {\n      \"column\": \"url\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 54,\n        \"samples\": [\n          \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-34.html\",\n          \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-76.html\",\n          \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-46.html\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"content\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 68,\n        \"samples\": [\n          \"# Start, Stop\\n\\nAs soon as the desired project is created, the project administration is not required anymore.\\n\\nSelect the desired project using the mouse and switch to the console using the ![](openConsole.png) symbol. Alternatively, you can open the console through the context menu (\\\\[right mouse click] on the project row) and start the project immediately.\\n\\nFigure 1. Start a project using the context menu of the project administration\\n\\n![](GettingStarted-07.png)\\n\\nYou can also open a project that should be started individually directly in the console without using the project administration (Combo box \\\"**Project**\\\").\\n\\nFigure 2. Console as control center of a project - manager list with operating status\\n\\n![](GettingStarted-06.png)\\n\\nThe console is the **control center of a project**. Here you specify **which managers** must run for the plant operation. Additionally, you define their **start order** and the necessary measures for the **process monitoring**.\\n\\nThe selected project is **started** by clicking the green **traffic light symbol**![](image118.gif) . The **sequential** proceeding start process can be observed in the console.\\n\\nA project can also be stopped via the console by clicking the **red traffic light** symbol. The project is stopped in the inverse order as it was started. The colored first row always shows the **current operating statuses** of the current manager.\\n\\nAlternatively to the complete start of a project, **each of the listed managers could also be started or stopped individually**. Use the context menu \\\\[right mouse click] in the manager list or one of the vertical arranged buttons on the right. Consider that some managers can only be started manually if data and event managers are already running. See the chapter [Console panel](../Pmon_Consolepanel/Pmon_Consolepanel-11.html) for detailed information on the individual buttons.\\n\\nThe console works independent from the **actual WinCC OA system**. Even if the console is closed, the managers will be running. As soon as the console is opened again, it shows the operating states of the managers again. In normal operation it may remain closed like the project administration.\\n\\nThe following part of the console as well as the sections [directory structure](GettingStarted-22.html) to [copy and backup project](GettingStarted-24.html) can be considered as **detailed information**. They are not mandatory at the start but help you to understand WinCC OA better.\\n\\nThe console does not communicate with different managers. The PMON, a monitoring process, runs in the background. It starts and stops managers and monitors constantly their function integrity. Furthermore, the PMON can run as service so that WinCC OA can also be used without a login user.\\n\\nThe console represents the user interface of the PMON. Without having an effect on the functionality, it shows the information of the background process and passes the user inputs to the process. The PMON possesses comprehensive interfaces so that it can be included in the existing IT infrastructure (SNMP, HTTP). For further information see chapter [Process monitor, basics](../Pmon_Consolepanel/Pmon_Consolepanel-01.html).\\n\",\n          \"# Settings for a Data Point Element\\n\\nThe new data point P2 is ready for use in our application. However, it is often desirable to make some further settings for the elements. For example, every element can carry the following additional information:\\n\\n* **Description**\\n\\n* **Alias address** (second internal notation system in addition to the data point description)\\n\\n* **Number format** (number of digits before/after the decimal point)\\n\\n* **Unit**\\n\\n**Figure: Enter the Description Text, Number Format and Unit for the Data point Element (config: \\\\_common)**\\n\\nThe settings can be displayed by a \\\\[leftMouseClick] on the \\\"\\\\_common\\\" sub element (config) for each data point element.\\n\\nEnter the settings for the data point P2 as shown in the figure above.\\n\\nIn a multilingual project you can enter the description multilingual. For more information on multilingual projects, see [Multilingual Projects, Basics](../Mehrsprachigkeit/Mehrsprachigkeit.html).\\n\",\n          \"scripts for the simulation run in a single CONTROL manager. You can define what scripts should be run in a CONTROL manager as call parameters in the console - in the available project through script lists (-f sim.lst) - see also [figure](GettingStarted-62.html#GettingStarted-62___Ref79212257) on the page \\\"Creating the simulation\\\". ## Notes and Restrictions * The file extension for a script does not have to be specified. Per default scripts are browsed in the following order: 1. given name 2\\\\. \\\\<Name>.ctl 3\\\\. \\\\<Name>.ctc * A script can be called with additional parameters. These parameters must not include spaces. Any spaces will be interpreted as splitters and result in multiple parameters. e.g.: \\\"Te st\\\" will be passed as two parameters, \\\"Te\\\" and \\\"st\\\" respectively.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"vector\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"num_tokens\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 329,\n        \"min\": 20,\n        \"max\": 1024,\n        \"num_unique_values\": 57,\n        \"samples\": [\n          899,\n          803,\n          673\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "To complete our in-memory vector store, we need a way to find the top-k\n",
        "most similar chunks given a user query. We do so by:\n",
        "1. Embedding the user query query using the same embedding model.\n",
        "2. Iterate through all chunks and calculate the cosine similarity\n",
        "3. Return the top-k most similar chunks.\n",
        "\n",
        "For the cosine similarity, we reuse the `cosine_similarity` function from an earlier section on embeddings.\n",
        "\n",
        "Let's wrap this up in a function:"
      ],
      "metadata": {
        "id": "C8Bl-lSHCp3J"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from numpy.linalg import norm\n",
        "import numpy as np\n",
        "\n",
        "cosine_similarity = lambda a,b: (a @ b.T) / (norm(a)*norm(b))"
      ],
      "metadata": {
        "id": "W1PjGLZODMhn"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def query_chunks(query, chunks, top_k):\n",
        "  response = client.embeddings.create(\n",
        "    input=query,\n",
        "    model=\"text-embedding-3-small\"\n",
        "  )\n",
        "  query_vector = response.data[0].embedding\n",
        "\n",
        "  cosine_similarity = lambda a, b: np.dot(a, b) / (norm(a) * norm(b))\n",
        "  similarities = []\n",
        "  for chunk in chunks:\n",
        "    if \"vector\" not in chunk:\n",
        "      continue\n",
        "    similarity = cosine_similarity(np.array(query_vector), np.array(chunk[\"vector\"]))\n",
        "    similarities.append((chunk, similarity))\n",
        "  sorted_chunks = sorted(similarities, key=lambda x: x[1], reverse=True)[:top_k]\n",
        "\n",
        "  return [chunk[0] for chunk in sorted_chunks]"
      ],
      "metadata": {
        "id": "sIubajG4CurT"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's try it out"
      ],
      "metadata": {
        "id": "yaBsIjxND50Q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "top_k = query_chunks(\"What is WinCC OA?\", chunked_documents, 5)\n",
        "pd.DataFrame(top_k, columns=[\"url\", \"content\", \"vector\", \"num_tokens\"])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "zgyKyPlGD8Aa",
        "outputId": "577a2e48-49ea-453d-c416-208530e0e858"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                 url  \\\n",
              "0  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "1  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "2  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "3  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "4  https://www.winccoa.com/documentation/WinCCOA/...   \n",
              "\n",
              "                                             content  \\\n",
              "0  # System Requirements\\n\\nYou can find the syst...   \n",
              "1  # WinCC OA Getting Started - Tutorial\\n\\nThis ...   \n",
              "2  # General Information\\n\\nWinCC OA possesses a ...   \n",
              "3  # Architecture\\n\\nWinCC OA is a **modularly bu...   \n",
              "4  # Drivers\\n\\nThe **communication between contr...   \n",
              "\n",
              "                                              vector  num_tokens  \n",
              "0  [-0.057610463351011276, 0.03982636332511902, 0...         212  \n",
              "1  [-0.05473048612475395, 0.022419877350330353, 0...         880  \n",
              "2  [-0.013674133457243443, 0.041188377887010574, ...         290  \n",
              "3  [-0.03359668701887131, 0.02406434901058674, 0....        1024  \n",
              "4  [-0.014540722593665123, 0.009579542092978954, ...         806  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-6aaa4d16-62cb-4638-a51a-9f549d7f60d6\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>url</th>\n",
              "      <th>content</th>\n",
              "      <th>vector</th>\n",
              "      <th>num_tokens</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># System Requirements\\n\\nYou can find the syst...</td>\n",
              "      <td>[-0.057610463351011276, 0.03982636332511902, 0...</td>\n",
              "      <td>212</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># WinCC OA Getting Started - Tutorial\\n\\nThis ...</td>\n",
              "      <td>[-0.05473048612475395, 0.022419877350330353, 0...</td>\n",
              "      <td>880</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># General Information\\n\\nWinCC OA possesses a ...</td>\n",
              "      <td>[-0.013674133457243443, 0.041188377887010574, ...</td>\n",
              "      <td>290</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Architecture\\n\\nWinCC OA is a **modularly bu...</td>\n",
              "      <td>[-0.03359668701887131, 0.02406434901058674, 0....</td>\n",
              "      <td>1024</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>https://www.winccoa.com/documentation/WinCCOA/...</td>\n",
              "      <td># Drivers\\n\\nThe **communication between contr...</td>\n",
              "      <td>[-0.014540722593665123, 0.009579542092978954, ...</td>\n",
              "      <td>806</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-6aaa4d16-62cb-4638-a51a-9f549d7f60d6')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-6aaa4d16-62cb-4638-a51a-9f549d7f60d6 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-6aaa4d16-62cb-4638-a51a-9f549d7f60d6');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-3ff99e5f-fde6-4f7e-b1f6-e77c2fd75bfa\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-3ff99e5f-fde6-4f7e-b1f6-e77c2fd75bfa')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-3ff99e5f-fde6-4f7e-b1f6-e77c2fd75bfa button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"pd\",\n  \"rows\": 5,\n  \"fields\": [\n    {\n      \"column\": \"url\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-01.html\",\n          \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-79.html\",\n          \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-66.html\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"content\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 5,\n        \"samples\": [\n          \"# WinCC OA Getting Started - Tutorial\\n\\nThis GettingStarted tutorial is addressed to automation technicians, project engineers and developers who perform tasks in the fields of process visualization, control technology and the Collaborative Manufacturing Management (CMM).\\n\\nThrough its scalable and modular system concept, WinCC OA provides a powerful and future-proof basis for each form of control center application. The consistently implemented client-server architecture allows multi-user access with simultaneous multilingualism. The possible configurations range from autonomous single systems with minimum configuration to multi-computer applications for load sharing as well as multi-server clusters (distributed systems). Hot-standby redundancy of the servers as well as the full network redundancy provide high availability.\\n\\nThis tutorial describes the basics of technics and use of the automation software WinCC OA on the basis of practical examples. The fields \\\"structured data model\\\" and \\\"Object orientation\\\" build the main focus. The purpose is to get to know WinCC OA quickly and independently though it's not possible to go into details of the whole range of functionality. In addition we recommend to participate in a professional training.\\n\\nThe content of this tutorial was up to date at the time of creation. Due to the constant development of the product to further improve it, certain information is no longer valid and needs to be replaced by new data. Please refer to the relevant information media for customers and contact the ETM support in case of doubt. Possible errors concerning the content and form may exist and can not form the basis of any warranty claims. Numerous screenshots relate to one of several display modes of the operating systems Windows XP or Windows 2000 - slight differences to other display modes are possible. For didactic purposes Some modified screen views are used (collages etc.) for didactic purposes. The product does not include the views in the displayed form.\\n\\nBrand names, logos and product descriptions of third parties, particularly of Microsoft Corporation, are subject to separate copyright and trademark rights.\\n\\n|                                                                                |                                                                                                                                                |\\n| ------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- |\\n| Chapter                                                                        | Content                                                                                                                                        |\\n| [Introduction](GettingStarted-02.html)                                         | What is WinCC OA? Target groups and requirements for working with this tutorial.                                                               |\\n| [Basic Concepts](GettingStarted-06.html)                                       | Basic terms and presentation of the concepts in WinCC OA.                                                                                      |\\n| [Installation](GettingStarted-13.html)                                         | Installation of WinCC OA under Windows.                                                                                                        |\\n| [Project Administration](GettingStarted-20.html)                               | Description of the console and project administration in WinCC OA. It allows to administer individual projects easily under Windows and Linux. |\\n| [Exercise Project](GettingStarted-26.html)                                     | Presentation of the exercise project. The project is the result of this tutorial.                                                              |\\n| [Data Model](GettingStarted-27.html)                                           | Basic terms for the data point concept in WinCC OA.                                                                                            |\\n| [Creating Data points](GettingStarted-35.html)                                 | Introduction to database editor - PARA.                                                                                                        |\\n| [Graphical Interfaces - \\\"Panels\\\"](GettingStarted-41.html)                      | Introduction to graphic editor - GEDI.                                                                                                         |\\n| [Alarms](GettingStarted-54.html)                                               | Colored presentation of malfunction information according to standards, display in object and presentation of the alarm screen in WinCC OA.    |\\n| [Archiving](GettingStarted-59.html)                                            | Save historic data, presentation and evaluation of historic values in trends.                                                                  |\\n| [Navigation & Image Administration - \\\"Panel-Topology\\\"](GettingStarted-63.html) | Administration of the process panels in WinCC OA. Basics of modules, panels and child panels.                                                  |\\n| [Scripting in the Graphics](GettingStarted-66.html)                            | General description of the scripting language CONTROL in WinCC OA. Example for Control scripts with graphic objects.                           |\\n| [Scripting for general Tasks](GettingStarted-78.html)                          | Control language at runtime for processing scripts in an own manager.                                                                          |\\n| [Periphery Connection](GettingStarted-79.html)                                 | General information on periphery connection in WinCC OA and description of how to set an OPC client-server connection.                         |\\n| [Hints and Trouble Shooting](GettingStarted-83.html)                           | Important information concerning WinCC OA (path statements, notation, ...)                                                                     |\\n\",\n          \"# Drivers\\n\\nThe **communication between control station and basic automation** (PLC, DDC, RTU) is executed in WinCC OA through own managers, the so-called drivers. Depending on the communication protocol and the used bus physics, specific drivers are used.\\n\\n* Seriel protocols: RK512, 3964R, Muehbus,...\\n\\n* Field bus: Profibus DP, Profibus FMS, ...\\n\\n* Ethernet: ModbusTCP (OpenModbus), Industrial Ethernet (S7), Ethernet IP (AB)...\\n\\n* **Telecontrol system**: SSI (Ethernet), IEC 60870-5-101, IEC 60870-5-104,....\\n\\n* ****Multivendor interfaces**** : OPC DataAccess, OPC Alarms\\\\&Events\\n\\nAn arbitrary, vendor-specific communication form (protocol, bus physics) is adapted to the general **WinCC OA communication form** by a driver manager. This a standardization where the driver process executes a number of tasks already automatically.\\n\\n* Adjustment to the **bus physics** (PCI card, low-level driver, TCP stack, ...)\\n\\n* Decoding of the **communication protocol** (evaluation of telegrams / frames)\\n\\n* Connection monitoring\\n\\n* Automatic old-new **comparison** (Low-Level smoothing)\\n\\n* Value and time-dependent smoothing (more significant smoothing)\\n\\n* Data type-**Transformations**\\n\\n* Transition of arbitrary communication form to **event-oriented data exchange**\\n\\nAdd a driver in the same way as each other manager to **the start list** of the console. Subsequently, you have to **configure the drivers individually**. These necessary processes are described in the online help in the chapter [driver](../Treiber_ComDrv/driver_overview.html).\\n\\nFigure 1. Manager properties of a driver\\n\\n![](GettingStarted-28.png)\\n\\nWithout taking further steps, all existing data point elements are **internal**: merely the different managers of WinCC OA can read or write their values. The data is not exchanged automatically with the base automation (PLC, DDC, RTU).\\n\\nIn order to receive values for **a data point element** from an external device, for example PLC, assign a **peripheral address in the form of a config** to this element. You can assign a peripheral address as follows in the database editor PARA:\\n\\n* Add a corresponding **config** (\\\\_address) or **PowerConfig** to a data point element\\n\\n* **Select** the driver to be used individually (for example WCCOApr for Profibus DP or Profibus FMS)\\n\\n* Enter the hardware or bus specific **address** (for example '%MW1021' for Modbus TCP in conjunction with Schneider Premium or Quantum PLCs)\\n\\n* **Enter the data direction** and for some protocols also the type of access (input, output, input/output)\\n\\nIn this example, the driver that exists by default \\\"**WCCILsim**\\\" was used. It serves as a form of placeholder as long as you do not configure any real drivers. In order to configure driver-specific settings, **at least this simulation driver** has to exist. **Consider that only one driver with the same number** (for example -num 2) is active at the same time. In order to run a driver with the number 1, you have to deactivate the by default existing SIM driver (if you don't specify the driver number, it does automatically mean -1 internal).\\n\\nHow to connect to a programmable controller of the basis automation, see chapter [driver](../Treiber_ComDrv/driver_overview.html) of the online help.\\n\\nYou can run **several drivers simultaneously** in WinCC OA. These can be of the same but also of a **different type**. For example, it's possible to establish connections to a Simatic controller via the S7 protocol, to a telecontrol system via IEC 60870-5-104 and to an arbitrary OPC server via OPC DA .\\n\",\n          \"# General Information\\n\\nWinCC OA possesses a **powerful script language**. You can use the language to implement own **logics**, design **symbols** and **configuration dialogs** as well as reports or **permanent calculation rules**. The language is available in the user interface as well as within an explicit CONTROL manager. Furthermore, you can define arithmetic data point functions in CONTROL and run them in the Event Manager.\\n\\nFigure 1. Use of the Script Language CONTROL within the managers of WinCC OA\\n\\n![](manager_using_control.jpg)\\n\\nIt is a **procedural high level language** with all typical control structures. The syntax corresponds **as far as possible to the ANSI-C standard**. Some simplifications facilitate the use. A number of **functions** adjusted to the needs in the automation technology are delivered with WinCC OA.\\n\\n* Access to the **current process image** - data point elements\\n\\n* Access to the **history** of values and alerts\\n\\n* Access to the **configuration** of data point elements or configs\\n\\n* Access to the properties and methods of **graphic objects**\\n\\n* Access to the operating system level, files, serial interface, TCP connections, external databases. email, SMS etc.\\n\\nThe code will be processed through a quick **intermediate code interpreter** so that no compilation or linking is necessary. The program instructions are normally executed **event-driven** but they can also be implemented time-controlled or cyclical.\\n\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"vector\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"num_tokens\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 366,\n        \"min\": 212,\n        \"max\": 1024,\n        \"num_unique_values\": 5,\n        \"samples\": [\n          880,\n          806,\n          290\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Looks like this returns pretty good results. We also want query expansion to resolve references in a user query."
      ],
      "metadata": {
        "id": "pcMcWYpRFLJl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Putting it all together\n",
        "\n",
        "We can now build our RAG system on-top of these primitives. We create an API that works similarly to the API for normal completion above.\n",
        "\n",
        "Our RAG system needs to keep track of the converstion history, which we store in `rag_history`. We also provide a function to clear the history called `rag_clear_history`, analogous to our `clear_history` function.\n",
        "\n"
      ],
      "metadata": {
        "id": "FAMt9yzTFd7w"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "rag_history = []\n",
        "\n",
        "def rag_clear_history():\n",
        "  global rag_history\n",
        "  rag_history = []\n"
      ],
      "metadata": {
        "id": "TE2MjsxCMjxy"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, we need a few help functions to format the conversation history and chunks so we can inject them into our prompt."
      ],
      "metadata": {
        "id": "dKO7h0X6Mkfj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def format_messages(messages):\n",
        "  formatted_str = \"\"\n",
        "  for message in messages:\n",
        "      formatted_str += f\"<{message['role']}>:\\n{message['content']}\\n\\n\"\n",
        "  return formatted_str\n",
        "\n",
        "def format_chunks(chunks):\n",
        "    formatted_str = \"[\\n\"\n",
        "    chunk_strings = []\n",
        "    for chunk in chunks:\n",
        "        content = chunk['content'].replace(\"\\n\", \"\\\\n\")\n",
        "        formatted_chunk = f\"   {{ url: \\\"{chunk['url']}\\\", content: \\\"{content}\\\"}}\"\n",
        "        chunk_strings.append(formatted_chunk)\n",
        "    formatted_str += \",\\n\".join(chunk_strings)\n",
        "    formatted_str += \"\\n]\"\n",
        "    return formatted_str"
      ],
      "metadata": {
        "id": "Lgrfxz--Mwbr"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Since this is a turn-by-turn system, we also need query expansion. We reuse what we had in the previous section and wrap it in a function. Note how we use our `clear_history` and `complete` primitive functions from above."
      ],
      "metadata": {
        "id": "uP2eYLTCMxpZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def expand_query(query, history):\n",
        "  clear_history()\n",
        "  complete(f\"\"\"You are given a conversation and new message, both delimited by triple backticks.\n",
        "  Expand the new message by resolving and references to persons, entities or locations, in the\n",
        "  conversation with their full name.\n",
        "\n",
        "  Cconversation:\n",
        "  ```\n",
        "  {format_messages(history)}\n",
        "  ```\n",
        "\n",
        "  New Message:\n",
        "  ```\n",
        "  {query}\n",
        "  ```\n",
        "  \"\"\", 2024, True)\n",
        "  expanded = messages[-1][\"content\"]\n",
        "  clear_history()\n",
        "  return expanded"
      ],
      "metadata": {
        "id": "QYLMCYJAM4Rv"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Finally, we create a function called `rag_complete`, which works analogously to `complete`, except it will augment the prompt with relevant information for the LLM to use for answering the user query.\n",
        "\n",
        "The function first truncates the RAG history to 4000 tokens. Next we expand the query using the truncated history to not explode the LLMs token window.\n",
        "\n",
        "We then query the top-k relevant chunks based on the expanded query, which will result in an additional 5120 tokens to be injected into the prompt.\n",
        "\n",
        "When then construct the final prompt based on all the pieces, with instructions for the LLM on how to use the information to answer the query.\n",
        "\n",
        "We send of the completion request and record the response in the RAG history."
      ],
      "metadata": {
        "id": "cnzUrhBhM49Q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def rag_complete(query, debug=False):\n",
        "  global rag_history\n",
        "  truncated_history = truncate_messages(rag_history, 4000) # 4000 tokens\n",
        "  expanded_query = expand_query(query, truncated_history)\n",
        "  if debug:\n",
        "    print(\"Expanded query: \" + expanded_query)\n",
        "  top_k = query_chunks(expanded_query, chunked_documents, 5) # 5 * 1024 tokens = 5120 tokens\n",
        "  prompt = f\"\"\"\n",
        "You are provided with a conversation history, a set of relevant information, and a user query.\n",
        "\n",
        "The conversation history:\n",
        "```\n",
        "{format_messages(truncated_history)}\n",
        "```\n",
        "\n",
        "The relevant information:\n",
        "```\n",
        "{format_chunks(top_k)}\n",
        "```\n",
        "\n",
        "The query:\n",
        "```\n",
        "{query}\n",
        "```\n",
        "\n",
        "- Answer the query in your own words based on the relevant information and conversation history.\n",
        "- Output your answer in Markdown\n",
        "- Include links to the relevant information you used to answer the query inline.\n",
        "- Use a descriptive label for each link, based on the contents of the relevant information you cite\n",
        "  \"\"\"\n",
        "  if debug:\n",
        "    print(prompt)\n",
        "  clear_history()\n",
        "  complete(prompt, 2048, False)\n",
        "  rag_history.append({\"role\": \"user\", \"content\": query})\n",
        "  rag_history.append(messages[-1])\n",
        "  if debug:\n",
        "    print(rag_history)"
      ],
      "metadata": {
        "id": "TpLz7agBFy2d"
      },
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We ask GPT-3.5 to generate Markdown output. Let's use the ipython Markdown display output to convert the raw Markdown to nicely rendered HTML output in the notebook, alongside the raw, streamed answer from the model."
      ],
      "metadata": {
        "id": "jYQziiumLKO2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from IPython.display import display, Markdown\n",
        "\n",
        "def print_markdown_message(message):\n",
        "  display(Markdown(message[\"content\"]))"
      ],
      "metadata": {
        "id": "TKQCDZYFKf_Q"
      },
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Alright, let's give it a try."
      ],
      "metadata": {
        "id": "VqEWoUTUKtBK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "rag_clear_history()\n",
        "rag_complete(\"Can you tell me what WinCC OA is?\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "79ozs3wJGIPK",
        "outputId": "d442b369-b23f-48d4-8a6a-18f8ba10a15b"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "WinCC OA, which stands for WinCC Open Architecture, is a modularly built system designed for process visualization, control technology, and Collaborative Manufacturing Management (CMM). It provides a powerful and future-proof basis for control center applications, supporting multi-user access with simultaneous multilingualism. The system architecture allows for various configurations, from autonomous single systems to multi-computer applications and multi-server clusters with high availability features like hot-standby redundancy and full network redundancy. WinCC OA is equipped with managers like drivers, Event Manager, Data Manager, Control, and UI Manager, each serving specific functions within the system. These managers handle tasks such as communication with control devices, maintaining process variables, executing alerts, archiving data, implementing algorithms, and providing user interfaces for interaction. The system requirements for WinCC OA include hardware and software specifications that ensure optimal performance, scalability, and reliability. For more detailed information, you can refer to the [WinCC OA Getting Started Tutorial](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-01.html) and the [Architecture section](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-06.html) of the documentation.\n",
            "Tokens: 3893\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print_markdown_message(rag_history[-1])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 168
        },
        "id": "0m2KVogTK0o-",
        "outputId": "65ffcdf4-0cb2-44fd-efca-f90b77d1fb31"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "WinCC OA, which stands for WinCC Open Architecture, is a modularly built system designed for process visualization, control technology, and Collaborative Manufacturing Management (CMM). It provides a powerful and future-proof basis for control center applications, supporting multi-user access with simultaneous multilingualism. The system architecture allows for various configurations, from autonomous single systems to multi-computer applications and multi-server clusters with high availability features like hot-standby redundancy and full network redundancy. WinCC OA is equipped with managers like drivers, Event Manager, Data Manager, Control, and UI Manager, each serving specific functions within the system. These managers handle tasks such as communication with control devices, maintaining process variables, executing alerts, archiving data, implementing algorithms, and providing user interfaces for interaction. The system requirements for WinCC OA include hardware and software specifications that ensure optimal performance, scalability, and reliability. For more detailed information, you can refer to the [WinCC OA Getting Started Tutorial](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-01.html) and the [Architecture section](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-06.html) of the documentation."
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's see if our query expansion correctly resolves \"it\" to WinCC OA based on the conversation history:"
      ],
      "metadata": {
        "id": "iUWglTDXKA_-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "rag_complete(\"What does it cost?\", True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nQjTEGr-PNaz",
        "outputId": "02aa556c-39be-48c6-94af-063d34cd7ed5"
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Expanded query: What does WinCC Open Architecture (WinCC OA) cost?\n",
            "\n",
            "You are provided with a conversation history, a set of relevant information, and a user query.\n",
            "\n",
            "The conversation history:\n",
            "```\n",
            "<user>:\n",
            "Can you tell me what WinCC OA is?\n",
            "\n",
            "<assistant>:\n",
            "WinCC OA, which stands for WinCC Open Architecture, is a modularly built system designed for process visualization, control technology, and Collaborative Manufacturing Management (CMM). It provides a powerful and future-proof basis for control center applications, supporting multi-user access with simultaneous multilingualism. The system architecture allows for various configurations, from autonomous single systems to multi-computer applications and multi-server clusters with high availability features like hot-standby redundancy and full network redundancy. WinCC OA is equipped with managers like drivers, Event Manager, Data Manager, Control, and UI Manager, each serving specific functions within the system. These managers handle tasks such as communication with control devices, maintaining process variables, executing alerts, archiving data, implementing algorithms, and providing user interfaces for interaction. The system requirements for WinCC OA include hardware and software specifications that ensure optimal performance, scalability, and reliability. For more detailed information, you can refer to the [WinCC OA Getting Started Tutorial](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-01.html) and the [Architecture section](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-06.html) of the documentation.\n",
            "\n",
            "\n",
            "```\n",
            "\n",
            "The relevant information:\n",
            "```\n",
            "[\n",
            "   { url: \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-13.html\", content: \"# System Requirements\\n\\nYou can find the system requirements [Hardware and software requirements for WinCC OA](../WCCOA_Voraussetzungen/Voraussetzungen-01.html) .\\n\\n**Note before installing the software** that the **system requirements** are met. The hardware requirements strongly depend on the size of the project and the dynamic of the process factors. Use quality hardware with corresponding functionalities such as redundant power supplies or RAID hard disks. WinCC OA supports dual and multi processor operation and benefits from it. As in case of RAM, CPU and HDD: the more the better.\\n\\nIf you need a tutorial that shows how to check this system data, see section [determine system data](GettingStarted-83.html).\\n\\nIn case of modern Windows models, the configuration and activation of the **TCP/IP network protocol** is already completed with the installation of the operating system. If this option is not available on your computer, you have to activate it manually (see also [Installation - activate TCP/IP network](GettingStarted-84.html).\\n\"},\n",
            "   { url: \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-66.html\", content: \"# General Information\\n\\nWinCC OA possesses a **powerful script language**. You can use the language to implement own **logics**, design **symbols** and **configuration dialogs** as well as reports or **permanent calculation rules**. The language is available in the user interface as well as within an explicit CONTROL manager. Furthermore, you can define arithmetic data point functions in CONTROL and run them in the Event Manager.\\n\\nFigure 1. Use of the Script Language CONTROL within the managers of WinCC OA\\n\\n![](manager_using_control.jpg)\\n\\nIt is a **procedural high level language** with all typical control structures. The syntax corresponds **as far as possible to the ANSI-C standard**. Some simplifications facilitate the use. A number of **functions** adjusted to the needs in the automation technology are delivered with WinCC OA.\\n\\n* Access to the **current process image** - data point elements\\n\\n* Access to the **history** of values and alerts\\n\\n* Access to the **configuration** of data point elements or configs\\n\\n* Access to the properties and methods of **graphic objects**\\n\\n* Access to the operating system level, files, serial interface, TCP connections, external databases. email, SMS etc.\\n\\nThe code will be processed through a quick **intermediate code interpreter** so that no compilation or linking is necessary. The program instructions are normally executed **event-driven** but they can also be implemented time-controlled or cyclical.\\n\"},\n",
            "   { url: \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-06.html\", content: \"# Architecture\\n\\nWinCC OA is a **modularly built system.** The required functionalities are handled by **specific****units** that were created for different tasks. In WinCC OA these units are called **managers** - the managers are own software processes.\\n\\nFigure 1. A WinCC OA system is composed of function-specific units, managers\\n\\n![](manager_concept.png)\\n\\nIn the following, the most important managers of the figure above are briefly described. Thereby, it is merely **a simple configuration** and can be much more comprehensive in practice. In fact, there are a number of managers that are not dealt with in detail here.\\n\\n## ****Process interface****\\n\\nThe **process interface modules**, which are called **drivers****(D)** in WinCC OA, form the lowest level in a WinCC OA system. The drivers are special programs that handle the communication with the **control and field level**. Since there are numerous different communication forms with **PLCs** or remote control nodes, there are **several different****drivers** available.\\n\\nTherefore, the used **PLC** or the associated communication bus defines which WinCC OA driver should be used. To put it simply, the driver is a module that converts a certain protocol into the internal communication form of WinCC OA. The driver reads current states, measured values or counter values from the field and passes commands and set values to the subordinate controls (= the word \"control\" is used in this tutorial for all possible equipment of the basic automation (PLC, DDC, telecontrol system,...)\\n\\n## **Process image, history**\\n\\nThe **central processing center** in WinCC OA is called Event Manager (EV). This unit always keeps a current **image of all process variables** in the memory. Each other function unit (manager) that wants to access data, receives the data from the process image of the event manager and does not have to communicate directly with a control. A command from an operator station is only set as a value change in the process image of the Event manager at first. The forwarding to the appropriate target device (for example PLC) will be executed by the responsible manager automatically.\\n\\nThe Event manager is a kind of **central data distributor**, the communication center for WinCC OA. Furthermore, this manager also executes the **alert handling** and is able to execute different **calculating functions** stand-alone.\\n\\nThe data manager supports the Event manager. The Data manager is the link to a database. The Data manager handles the **configuration data of an application** that is saved in the database. Furthermore, the **historical data - value changes** and **alerts** are saved in a database. If you would like to query historic data later, the query is also executed by the Data manager and not the database itself.\\n\\nThe concept for archiving process data handles the saving and readout of information that occurs during process control or visualization. This involves values and messages that are generated by value changes. The process data is saved in Value Archives (VA). Each archive is managed by a separate archiving process. Each archive consists of a series of chronologically ordered archive files.\\n\\n## ****Processing, Control****\\n\\nIn WinCC OA there are numerous possibilities to implement own **algorithms** and **processing**. The internal language control (CTRL) and the common programming interface API (Application Programming Interface) are the most important.\\n\\nControl is a powerful scripting language. The code is processed interpretatively so compiling (= conversion process which interprets a source code into executable machine code for the processor) is required. The syntax corresponds to the ANSI-C (= \"C\"... international standardized and exquisitely widespread, higher-level programming language.) with some simplifying modifications. It is an advanced **procedural high level language** with multithreading (parallel processing of single programs, the processing control is executed by the system itself). The language provides a widespread function library for tasks of control and visualization technics. Control can be used as**a stand-alone process**(CONTROL manager) for animation and**user interface design**(UI manager) or for standardized,**data object based processing** (Event manager).\\n\\nThe **API** (WinCC OA API) presents the most powerful form of function extensions. It is a C++ class library and allows the software developer to implement individual functions as an independent additional manager (prognosis system, simulation, planning tools, proprietary databases), ....\\n\\n## ****Present, handle****\\n\\nThe User Interface Manager (UI) is the interface to the **user**. Thereby, there is the **graphics editor** (GEDI), a **database editor** (PARA) or the general **user interface** of the application (Module VISION). In the user interface, values are shown, commands are executed or alarms are tracked . Furthermore, **trends** or **reports** are normally part of the UI. In WinCC OA the user interaction is technically completely separated from the background processing. It is merely a view of the data of the current process image or the history.\\n\"},\n",
            "   { url: \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-01.html\", content: \"# WinCC OA Getting Started - Tutorial\\n\\nThis GettingStarted tutorial is addressed to automation technicians, project engineers and developers who perform tasks in the fields of process visualization, control technology and the Collaborative Manufacturing Management (CMM).\\n\\nThrough its scalable and modular system concept, WinCC OA provides a powerful and future-proof basis for each form of control center application. The consistently implemented client-server architecture allows multi-user access with simultaneous multilingualism. The possible configurations range from autonomous single systems with minimum configuration to multi-computer applications for load sharing as well as multi-server clusters (distributed systems). Hot-standby redundancy of the servers as well as the full network redundancy provide high availability.\\n\\nThis tutorial describes the basics of technics and use of the automation software WinCC OA on the basis of practical examples. The fields \"structured data model\" and \"Object orientation\" build the main focus. The purpose is to get to know WinCC OA quickly and independently though it's not possible to go into details of the whole range of functionality. In addition we recommend to participate in a professional training.\\n\\nThe content of this tutorial was up to date at the time of creation. Due to the constant development of the product to further improve it, certain information is no longer valid and needs to be replaced by new data. Please refer to the relevant information media for customers and contact the ETM support in case of doubt. Possible errors concerning the content and form may exist and can not form the basis of any warranty claims. Numerous screenshots relate to one of several display modes of the operating systems Windows XP or Windows 2000 - slight differences to other display modes are possible. For didactic purposes Some modified screen views are used (collages etc.) for didactic purposes. The product does not include the views in the displayed form.\\n\\nBrand names, logos and product descriptions of third parties, particularly of Microsoft Corporation, are subject to separate copyright and trademark rights.\\n\\n|                                                                                |                                                                                                                                                |\\n| ------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- |\\n| Chapter                                                                        | Content                                                                                                                                        |\\n| [Introduction](GettingStarted-02.html)                                         | What is WinCC OA? Target groups and requirements for working with this tutorial.                                                               |\\n| [Basic Concepts](GettingStarted-06.html)                                       | Basic terms and presentation of the concepts in WinCC OA.                                                                                      |\\n| [Installation](GettingStarted-13.html)                                         | Installation of WinCC OA under Windows.                                                                                                        |\\n| [Project Administration](GettingStarted-20.html)                               | Description of the console and project administration in WinCC OA. It allows to administer individual projects easily under Windows and Linux. |\\n| [Exercise Project](GettingStarted-26.html)                                     | Presentation of the exercise project. The project is the result of this tutorial.                                                              |\\n| [Data Model](GettingStarted-27.html)                                           | Basic terms for the data point concept in WinCC OA.                                                                                            |\\n| [Creating Data points](GettingStarted-35.html)                                 | Introduction to database editor - PARA.                                                                                                        |\\n| [Graphical Interfaces - \"Panels\"](GettingStarted-41.html)                      | Introduction to graphic editor - GEDI.                                                                                                         |\\n| [Alarms](GettingStarted-54.html)                                               | Colored presentation of malfunction information according to standards, display in object and presentation of the alarm screen in WinCC OA.    |\\n| [Archiving](GettingStarted-59.html)                                            | Save historic data, presentation and evaluation of historic values in trends.                                                                  |\\n| [Navigation & Image Administration - \"Panel-Topology\"](GettingStarted-63.html) | Administration of the process panels in WinCC OA. Basics of modules, panels and child panels.                                                  |\\n| [Scripting in the Graphics](GettingStarted-66.html)                            | General description of the scripting language CONTROL in WinCC OA. Example for Control scripts with graphic objects.                           |\\n| [Scripting for general Tasks](GettingStarted-78.html)                          | Control language at runtime for processing scripts in an own manager.                                                                          |\\n| [Periphery Connection](GettingStarted-79.html)                                 | General information on periphery connection in WinCC OA and description of how to set an OPC client-server connection.                         |\\n| [Hints and Trouble Shooting](GettingStarted-83.html)                           | Important information concerning WinCC OA (path statements, notation, ...)                                                                     |\\n\"},\n",
            "   { url: \"https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-20.html\", content: \"# Create Project\\n\\nIn WinCC OA, a **\"project\"** is **an executable unit** for solving a task in the field of control technology. Normally, only **one project runs on a computer**. **The project contains all specific configurations of a task,** in addition to the programs and basic components that were provided with the installation. These comprise primarily:\\n\\n* Data point types (Definition of devices)\\n\\n* Data points (Variables of the process image /representation of devices)\\n\\n* **Panels** (Process images, Dialogs, Symbols)\\n\\n* **System-dependent processing** (scripts, clock timers, recipes)\\n\\n* Configurations (Archive settings, Alerting, Trends,...)\\n\\nIn order to work with WinCC OA, **at least one project** has to be available on the computer. The previous installation has already created and configured 2 projects ready for use (see the figure below). For the following configuration steps we create **a new project** - proceed as described below:\\n\\nFigure 1. Create a project using the wizards of the project administration\\n\\n![](pa_create_proj_1_de.png)\\n\\n1. Open the project administration (**Start -> Programs -> Siemens Automation -> Runtime Systems -> WinCC OA 3.19 ->****WinCC OA Project Administration**)\\n\\n2. Click the![](image108.gif) (**New**)symbol or use the shortcut \\[ctrl] + \\[n].\\n\\n3. Select the installation option \"**Standard project**\" from the wizard and click on **Next**.\\n\\n4. Enter the name \"**myGettingStarted**\" into the project name field.\\n\\n5. **Select the project languages \"English -US\" and \"German - Austria\".**\\n\\n6. Select the target directory for the project using the directory selector or enter the path manually C:/WinCC\\_OA\\_Proj/. Click on**Next**.\\n\\n7. Confirm the following dialog with **OK** in order to create the project. Wait during the creation of the database (this could take a while depending on the language selection).\\n\\nEach from of string in WinCC OA **differentiates strictly the case sensitivity**. Consider always that the names that you have assigned correspond exactly to the spelling here.\\n\\nFigure 2. Selection of the project language(s) and the target directory\\n\\n![](../Pmon_Consolepanel/Pmon_Consolepanel-118.png)\\n\\nIf you move the mouse over the toolbars and buttons, context sensitive text descriptions (tooltips) of the functionality are shown.\\n\\nAfter creating the project, the new project is shown in the project administration. It is now ready for the first **start**. (see section [Start, Stop](GettingStarted-21.html)).\\n\\n|                       |           |             |\\n| --------------------- | --------- | ----------- |\\n| Project name          | Version   | Pmon Status |\\n| DemoApplication\\_3.19 | e.g. 3.19 | not running |\\n| GettingStarted\\_3.19  | e.g. 3.19 | not running |\\n| myGettingStarted      | e.g. 3.19 | not running |\\n\\nIn order to create a project for **multilingual user interfaces**, you have to press the \\[CTRL] key to select **further languages**. Later you can define **individual texts and fonts** for the user interfaces of the workstations (multilingualism). You can also set a password for the a project user. A password for the user \"root\" can be set when creating the project (after clicking OK to create a project), see the second figure below. If you do not specify a password, the user interface is started without user.\\n\\nFigure 3. Enter Password Window\\n\\n![](GettingStarted-104.png)\\n\\nFigure 4. Set \"root\" password\\n\\n![](GettingStarted-08.png)\\n\\nThe database generation process sometimes takes several minutes in case of other languages than \"German-Austria\".\\n\\nFor additional information regarding the creation of a new project see: [Project administration > Create project](../Pmon_Consolepanel/create_project/topics/pmon_create_proj.html)\\n\\n## Licensing\\n\\nIf you start WinCC OA without a valid license, a message will be shown. The system stops without further warning after a 30 minutes demo period. For more information, see chapter [Licensing](../Installation/WinCCOA_licensing/licensing_furtherInfo.html).\\n\"}\n",
            "]\n",
            "```\n",
            "\n",
            "The query:\n",
            "```\n",
            "What does it cost?\n",
            "```\n",
            "\n",
            "- Answer the query in your own words based on the relevant information and conversation history.\n",
            "- Output your answer in Markdown\n",
            "- Include links to the relevant information you used to answer the query inline.\n",
            "- Use a descriptive label for each link, based on the contents of the relevant information you cite\n",
            "  \n",
            "Based on the provided conversation history and relevant information, the cost of WinCC OA is not explicitly mentioned. The focus of the information is on explaining the system requirements, architecture, scripting language, and project creation process of WinCC OA. For detailed information on licensing and potential costs associated with WinCC OA, you can refer to the [Licensing section](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-20.html) of the documentation.\n",
            "Tokens: 4098\n",
            "[{'role': 'user', 'content': 'Can you tell me what WinCC OA is?'}, {'role': 'assistant', 'content': 'WinCC OA, which stands for WinCC Open Architecture, is a modularly built system designed for process visualization, control technology, and Collaborative Manufacturing Management (CMM). It provides a powerful and future-proof basis for control center applications, supporting multi-user access with simultaneous multilingualism. The system architecture allows for various configurations, from autonomous single systems to multi-computer applications and multi-server clusters with high availability features like hot-standby redundancy and full network redundancy. WinCC OA is equipped with managers like drivers, Event Manager, Data Manager, Control, and UI Manager, each serving specific functions within the system. These managers handle tasks such as communication with control devices, maintaining process variables, executing alerts, archiving data, implementing algorithms, and providing user interfaces for interaction. The system requirements for WinCC OA include hardware and software specifications that ensure optimal performance, scalability, and reliability. For more detailed information, you can refer to the [WinCC OA Getting Started Tutorial](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-01.html) and the [Architecture section](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-06.html) of the documentation.'}, {'role': 'user', 'content': 'What does it cost?'}, {'role': 'assistant', 'content': 'Based on the provided conversation history and relevant information, the cost of WinCC OA is not explicitly mentioned. The focus of the information is on explaining the system requirements, architecture, scripting language, and project creation process of WinCC OA. For detailed information on licensing and potential costs associated with WinCC OA, you can refer to the [Licensing section](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-20.html) of the documentation.'}]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print_markdown_message(rag_history[-1])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        },
        "id": "zrvOXTwmLBMT",
        "outputId": "67f91396-437d-44a1-cd42-b281a5828b8c"
      },
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "Based on the provided conversation history and relevant information, the cost of WinCC OA is not explicitly mentioned. The focus of the information is on explaining the system requirements, architecture, scripting language, and project creation process of WinCC OA. For detailed information on licensing and potential costs associated with WinCC OA, you can refer to the [Licensing section](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-20.html) of the documentation."
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "rag_complete(\"Describe what the process interface is\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Md2ZsgSwPTkz",
        "outputId": "cf61b079-21d9-427f-a4c3-b983b0ca15e2"
      },
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The **process interface** in WinCC OA, also known as **drivers**, serves as the communication link between the system and the control and field level devices. These drivers are specialized programs that handle various communication protocols with PLCs or remote control nodes. Depending on the communication bus or PLC used, different drivers are available to convert the specific protocol into the internal communication format of WinCC OA. The drivers read current states, measured values, or counter values from the field and send commands and set values to the control devices. This ensures seamless communication between the system and the external devices, enabling data exchange and control operations.\n",
            "\n",
            "For more detailed information on the process interface in WinCC OA, you can refer to the [Architecture section](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-06.html) of the documentation.\n",
            "Tokens: 5188\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print_markdown_message(rag_history[-1])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 139
        },
        "id": "DJxo4CygLFOp",
        "outputId": "8175370d-6591-4514-a435-36f42fe10a98"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "The **process interface** in WinCC OA, also known as **drivers**, serves as the communication link between the system and the control and field level devices. These drivers are specialized programs that handle various communication protocols with PLCs or remote control nodes. Depending on the communication bus or PLC used, different drivers are available to convert the specific protocol into the internal communication format of WinCC OA. The drivers read current states, measured values, or counter values from the field and send commands and set values to the control devices. This ensures seamless communication between the system and the external devices, enabling data exchange and control operations.\n\nFor more detailed information on the process interface in WinCC OA, you can refer to the [Architecture section](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-06.html) of the documentation."
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "rag_complete(\"What programming languages can I use?\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gptib3BRPinE",
        "outputId": "e50748e8-2934-42a2-bd95-11764c412308"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "To implement custom logics, design symbols, configuration dialogs, reports, and permanent calculation rules in WinCC OA, you can use the **script language CONTROL**. This language is available within the user interface and in the explicit CONTROL manager. It is a procedural high-level language with control structures similar to the ANSI-C standard. The language provides functions for accessing process variables, historical data, configurations, graphic objects, operating system resources, external databases, and more. The code is processed interpretatively without the need for compilation or linking, allowing for event-driven, time-controlled, or cyclical execution ([General Information](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-66.html)).\n",
            "\n",
            "In addition to the script language CONTROL, WinCC OA also supports the **API (Application Programming Interface)**, which is a C++ class library enabling software developers to implement custom functions as independent additional managers. This API offers the most powerful form of function extensions in WinCC OA, allowing for the creation of specialized functionalities like prognosis systems, simulations, planning tools, and proprietary databases ([Architecture](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-06.html)).\n",
            "Tokens: 4524\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print_markdown_message(rag_history[-1])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 174
        },
        "id": "xfYAgArjLIjw",
        "outputId": "6b313eea-1fa2-4175-f453-651a9be202a5"
      },
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "To implement custom logics, design symbols, configuration dialogs, reports, and permanent calculation rules in WinCC OA, you can use the **script language CONTROL**. This language is available within the user interface and in the explicit CONTROL manager. It is a procedural high-level language with control structures similar to the ANSI-C standard. The language provides functions for accessing process variables, historical data, configurations, graphic objects, operating system resources, external databases, and more. The code is processed interpretatively without the need for compilation or linking, allowing for event-driven, time-controlled, or cyclical execution ([General Information](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-66.html)).\n\nIn addition to the script language CONTROL, WinCC OA also supports the **API (Application Programming Interface)**, which is a C++ class library enabling software developers to implement custom functions as independent additional managers. This API offers the most powerful form of function extensions in WinCC OA, allowing for the creation of specialized functionalities like prognosis systems, simulations, planning tools, and proprietary databases ([Architecture](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-06.html))."
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "rag_complete(\"How do i create a new project?\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P_Ufh63LQs1W",
        "outputId": "2b15eff3-ea55-4a1a-eb20-0846d4d4fd95"
      },
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "To create a new project in WinCC OA, you need to follow these steps:\n",
            "\n",
            "1. Open the project administration tool by navigating to **Start -> Programs -> Siemens Automation -> Runtime Systems -> WinCC OA 3.19 -> WinCC OA Project Administration**. You can also use the shortcut `[ctrl] + [n]`.\n",
            "   - For more detailed instructions, refer to the [Create Project section](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-20.html) of the documentation.\n",
            "\n",
            "2. Select the option for creating a \"Standard project\" and proceed by entering a name for your project (e.g., \"myGettingStarted\").\n",
            "   - Choose the project languages (e.g., \"English - US\" and \"German - Austria\") and specify the target directory for the project.\n",
            "   - Confirm the dialog to create the project and wait for the database creation process to complete.\n",
            "\n",
            "3. After creating the project, it will be visible in the project administration tool and ready for use.\n",
            "   - You can find additional information on creating multilingual user interfaces and setting passwords for project users in the [relevant documentation](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-20.html).\n",
            "\n",
            "By following these steps, you can successfully create a new project in WinCC OA.\n",
            "Tokens: 4826\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print_markdown_message(rag_history[-1])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 261
        },
        "id": "rXPlvi9pLQKg",
        "outputId": "f96a1570-fa00-49ca-d78c-f1e5a6917a81"
      },
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "To create a new project in WinCC OA, you need to follow these steps:\n\n1. Open the project administration tool by navigating to **Start -> Programs -> Siemens Automation -> Runtime Systems -> WinCC OA 3.19 -> WinCC OA Project Administration**. You can also use the shortcut `[ctrl] + [n]`.\n   - For more detailed instructions, refer to the [Create Project section](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-20.html) of the documentation.\n\n2. Select the option for creating a \"Standard project\" and proceed by entering a name for your project (e.g., \"myGettingStarted\").\n   - Choose the project languages (e.g., \"English - US\" and \"German - Austria\") and specify the target directory for the project.\n   - Confirm the dialog to create the project and wait for the database creation process to complete.\n\n3. After creating the project, it will be visible in the project administration tool and ready for use.\n   - You can find additional information on creating multilingual user interfaces and setting passwords for project users in the [relevant documentation](https://www.winccoa.com/documentation/WinCCOA/latest/en_US/GettingStarted/GettingStarted-20.html).\n\nBy following these steps, you can successfully create a new project in WinCC OA."
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "A pretty decent system for the amount of code. We can compare its outputs with the outputs of the current system.\n",
        "\n",
        "<center><img src=\"https://marioslab.io/uploads/genai/brag.png\" width=480></center>\n",
        "<center><img src=\"https://marioslab.io/uploads/genai/brag-2.png\" width=480></center>\n",
        "<center><img src=\"https://marioslab.io/uploads/genai/brag-3.png\" width=480></center>\n",
        "<center><img src=\"https://marioslab.io/uploads/genai/brag-5.png\" width=480></center>\n",
        "<center><img src=\"https://marioslab.io/uploads/genai/brag-4.png\" width=480></center>"
      ],
      "metadata": {
        "id": "b8xTJs-qQyNe"
      }
    }
  ]
}