13_3_LangChain_text_splitter__M.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "587cb857-5ec7-4f1a-8fd9-53bee22d879b"
      },
      "source": [
        "<p style=\"text-align:center\">\n",
        "    <a href=\"https://skills.network\" target=\"_blank\">\n",
        "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n",
        "    </a>\n",
        "</p>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "947f20ed-9c87-4571-80e2-cf0eb53c4112"
      },
      "source": [
        "# **Apply Text Splitting Techniques to Enhance Model Responsiveness**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "52a2b4a3-b3d1-49f9-b4b6-f7f5177bfd75"
      },
      "source": [
        "Estimated time needed: **30** minutes\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9b75e8a1-4f12-418f-abe4-1e7fa9a7b474"
      },
      "source": [
        "## Overview\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6f2c48a9-3e72-4f9c-b874-3dd3a9ef622c"
      },
      "source": [
        "In many data processing tasks, especially those involving large documents, breaking down text into smaller, more manageable chunks is essential. Text splitters are tools specifically designed to accomplish this, ensuring that lengthy texts are divided into coherent segments. This division is crucial for maintaining the integrity and readability of the information, making it easier to handle and process. Effective text splitting helps prevent overwhelming systems with large, unwieldy blocks of text and ensures that each segment remains meaningful and contextually relevant.\n",
        "\n",
        "The significance of text splitters becomes even more apparent in the context of retrieval-augmented generation (RAG). RAG involves fetching relevant pieces of information from a large dataset and using them to generate accurate and context-aware responses. Without properly split text, the retrieval process can become inefficient, potentially missing critical pieces of information or returning irrelevant data. By using text splitters to create well-defined chunks, the retrieval process can be streamlined, ensuring that the most relevant information is easily accessible. This not only enhances the efficiency of data retrieval but also improves the quality and relevance of the generated responses, making text splitters an important tool in the RAG workflow.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "abe93483-f924-4239-9e51-0f678d9dfa73"
      },
      "source": [
        "<figure>\n",
        "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/3Y4eB0v7LXU5cSVbeN1b8A/text-splitter.png\" width=\"50%\" alt=\"langchain\">\n",
        "    <figcaption><a>source: DALL-E</a></figcaption>\n",
        "</figure>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f8327090-046e-466d-8a15-e05aa9cf2cfc"
      },
      "source": [
        "This lab will guide you about how to use some commonly used text splitters from LangChain to split your source document.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c9a4a7cd-158f-4acc-b378-1740eddafd1c"
      },
      "source": [
        "## __Table of Contents__\n",
        "\n",
        "<ol>\n",
        "    <li><a href=\"#Objectives\">Objectives</a></li>\n",
        "    <li>\n",
        "        <a href=\"#Setup\">Setup</a>\n",
        "        <ol>\n",
        "            <li><a href=\"#Installing-required-libraries\">Installing required libraries</a></li>\n",
        "        </ol>\n",
        "    </li>\n",
        "    <li>\n",
        "        <a href=\"#Text-splitters\">Text splitters</a>\n",
        "        <ol>\n",
        "            <li><a href=\"#Key-parameters\">Key parameters</a></li>\n",
        "            <li><a href=\"#Prepare-the-document\">Prepare the document</a></li>\n",
        "            <li><a href=\"#Split-by-Character\">Split by Character</a></li>\n",
        "            <li><a href=\"#Recursively-Split-by-Character\">Recursively Split by Character</a></li>\n",
        "            <li><a href=\"#Split-Code\">Split Code</a></li>\n",
        "            <li><a href=\"#Markdown-Header-Text-Splitter\">Markdown Header Text Splitter</a></li>\n",
        "            <li><a href=\"#Split-by-HTML\">Split by HTML</a></li>\n",
        "        </ol>\n",
        "    </li>\n",
        "</ol>\n",
        "\n",
        "<a href=\"#Exercises\">Exercises</a>\n",
        "<ol>\n",
        "    <li><a href=\"#Exercise-1---Changing-separator-for-CharacterTextSplitter\">Exercise 1. Changing separator for CharacterTextSplitter</a></li>\n",
        "    <li><a href=\"#Exercise-2---Splitting-Latex-code\">Exercise 2. Splitting Latex code</a></li>\n",
        "</ol>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f1cb2a85-02c5-40e9-b8df-e1a6f79be5a0"
      },
      "source": [
        "## Objectives\n",
        "\n",
        "After completing this lab you will be able to:\n",
        "\n",
        " - Use commonly used text splitters from LangChain.\n",
        " - Split source documents into chunks for downstream use in RAG\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3e2ed166-447e-4f83-83bd-4bddb6e4b938"
      },
      "source": [
        "----\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6012e0b4-7395-4aeb-b117-be347c71bba6"
      },
      "source": [
        "## Setup\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d4245b51-c2f3-48b7-b59b-dab9776b28b0"
      },
      "source": [
        "For this lab, you will use the following libraries:\n",
        "\n",
        "*   [`langchain`, `langchain-text-splitters`](https://www.langchain.com/) for using relevant features and text splitters from Langchain.\n",
        "*   [`lxml`](https://pypi.org/project/lxml/) for libxml2 and libxslt libraries, which is used for splitting html text.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8373a5ed-f599-4792-8492-3f52d685cbcc"
      },
      "source": [
        "### Installing required libraries\n",
        "\n",
        "The following required libraries are __not__ preinstalled in the Skills Network Labs environment. __You must run the following cell__ to install them:\n",
        "\n",
        "**Note:** The version is being pinned here to specify the version. It's recommended that you do this as well. Even if the library is updated in the future, the installed library could still support this lab work.\n",
        "\n",
        "This might take approximately 1 minute.\n",
        "\n",
        "As `%%capture` is used to capture the installation, you won't see the output process. But after the installation completes, you will see a number beside the cell.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "79b4b5ca-451f-4aa5-9738-007530d9bc6d"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "!pip install \"langchain==0.2.7\"\n",
        "!pip install \"langchain-core==0.2.20\"\n",
        "!pip install \"langchain-text-splitters==0.2.2\"\n",
        "!pip install \"lxml==5.2.2\""
      ],
      "execution_count": 1
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b6439f44-5184-4e6b-8e9d-d0655c9cddb8"
      },
      "source": [
        "After you install the libraries, restart your kernel. You can do that by clicking the **Restart the kernel** icon.\n",
        "\n",
        "<p style=\"text-align:left\">\n",
        "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/1_Bd_EvpEzLH9BbxRXXUGQ/screenshot-to-replace.png\" width=\"50%\"/>\n",
        "    </a>\n",
        "</p>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3e3f33f2-fdfe-4dab-b241-cbf71ca19777"
      },
      "source": [
        "## Text splitters\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0a3a20b7-d2bc-4a99-9004-fc9704203617"
      },
      "source": [
        "### Key parameters\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2f0085b1-bcef-4166-89ec-b09d4f34d0f1"
      },
      "source": [
        "When using the splitter, you can customize several key parameters to fit your needs:\n",
        "- **separator**: Define the characters that will be used for splitting the text.\n",
        "- **chunk_size**: Specify the maximum size of your chunks to ensure they are as granular or broad as needed.\n",
        "- **chunk_overlap**: Maintain context between chunks by setting the `chunk_overlap` parameter, which determines the number of characters that overlap between consecutive chunks. This helps ensure that information isn't lost at the chunk boundaries.\n",
        "- **length_function**: Define how the length of chunks is calculated.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b9a294c3-1d04-47bd-8031-f4121a8c3c57"
      },
      "source": [
        "### Prepare the document\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "692a53fb-d2ca-4622-8f51-52686c62d156"
      },
      "source": [
        "A long document has been prepared for this project to demonstrate the performance of each splitter. Run the following code to download it.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ff0198c9-105b-4ef5-90ce-7dc9c0daec97",
        "outputId": "594ac44b-6950-4c7b-f1f2-10eb2b79e6b3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-12-03 15:05:58--  https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/YRYau14UJyh0DdiLDdzFcA/companypolicies.txt\n",
            "Resolving cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)... 198.23.119.245\n",
            "Connecting to cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)|198.23.119.245|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 15660 (15K) [text/plain]\n",
            "Saving to: ‘companypolicies.txt’\n",
            "\n",
            "companypolicies.txt 100%[===================>]  15.29K  --.-KB/s    in 0s      \n",
            "\n",
            "2024-12-03 15:05:59 (265 MB/s) - ‘companypolicies.txt’ saved [15660/15660]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/YRYau14UJyh0DdiLDdzFcA/companypolicies.txt\""
      ],
      "execution_count": 2
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "564b28fe-284f-46b1-8fef-31f8649876c6"
      },
      "source": [
        "Let's take a look at what the document looks like.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7cb4a550-7db2-4ae1-ac2b-140336649591"
      },
      "outputs": [],
      "source": [
        "# This is a long document you can split up.\n",
        "with open(\"companypolicies.txt\") as f:\n",
        "    companypolicies = f.read()"
      ],
      "execution_count": 3
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aba38804-c6e2-43b1-a0fc-5a65e62de25c",
        "outputId": "e4dd1300-9a23-42a6-d3a1-3fe205d15aa1"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1.\tCode of Conduct\n",
            "\n",
            "Our Code of Conduct outlines the fundamental principles and ethical standards that guide every member of our organization. We are committed to maintaining a workplace that is built on integrity, respect, and accountability.\n",
            "Integrity: We hold ourselves to the highest ethical standards. This means acting honestly and transparently in all our interactions, whether with colleagues, clients, or the broader community. We respect and protect sensitive information, and we avoid conflicts of interest.\n",
            "Respect: We embrace diversity and value each individual's contributions. Discrimination, harassment, or any form of disrespectful behavior is unacceptable. We create an inclusive environment where differences are celebrated and everyone is treated with dignity and courtesy.\n",
            "Accountability: We take responsibility for our actions and decisions. We follow all relevant laws and regulations, and we strive to continuously improve our practices. We report any potential violations of this code and support the investigation of such matters.\n",
            "Safety: We prioritize the safety of our employees, clients, and the communities we serve. We maintain a culture of safety, including reporting any unsafe conditions or practices.\n",
            "Environmental Responsibility: We are committed to minimizing our environmental footprint and promoting sustainable practices.\n",
            "Our Code of Conduct is not just a set of rules; it is the foundation of our organization's culture. We expect all employees to uphold these principles and serve as role models for others, ensuring we maintain our reputation for ethical conduct, integrity, and social responsibility.\n",
            "\n",
            "2.\tRecruitment Policy\n",
            "\n",
            "Our Recruitment Policy reflects our commitment to attracting, selecting, and onboarding the most qualified and diverse candidates to join our organization. We believe that the success of our company relies on the talents, skills, and dedication of our employees.\n",
            "Equal Opportunity: We are an equal opportunity employer and do not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, or any other protected status. We actively promote diversity and inclusion.\n",
            "Transparency: We maintain transparency in our recruitment processes. All job vacancies are advertised internally and externally when appropriate. Job descriptions and requirements are clear and accurately represent the role.\n",
            "Selection Criteria: Our selection process is based on the qualifications, experience, and skills necessary for the position. Interviews and assessments are conducted objectively, and decisions are made without bias.\n",
            "Data Privacy: We are committed to protecting the privacy of candidates' personal information and adhere to all relevant data protection laws and regulations.\n",
            "Feedback: Candidates will receive timely and constructive feedback on their application and interview performance.\n",
            "Onboarding: New employees receive comprehensive onboarding to help them integrate into the organization effectively. This includes information on our culture, policies, and expectations.\n",
            "Employee Referrals: We encourage and appreciate employee referrals as they contribute to building a strong and engaged team.\n",
            "Our Recruitment Policy is a foundation for creating a diverse, inclusive, and talented workforce. It ensures that we attract and hire the best candidates who align with our company values and contribute to our continued success. We continuously review and update this policy to reflect evolving best practices in recruitment.\n",
            "\n",
            "3.\tInternet and Email Policy\n",
            "\n",
            "Our Internet and Email Policy is established to guide the responsible and secure use of these essential tools within our organization. We recognize their significance in daily business operations and the importance of adhering to principles that maintain security, productivity, and legal compliance.\n",
            "Acceptable Use: Company-provided internet and email services are primarily meant for job-related tasks. Limited personal use is allowed during non-work hours, provided it doesn't interfere with work responsibilities.\n",
            "Security: Safeguard your login credentials, avoiding the sharing of passwords. Exercise caution with email attachments and links from unknown sources. Promptly report any unusual online activity or potential security breaches.\n",
            "Confidentiality: Reserve email for the transmission of confidential information, trade secrets, and sensitive customer data only when encryption is applied. Exercise discretion when discussing company matters on public forums or social media.\n",
            "Harassment and Inappropriate Content: Internet and email usage must not involve harassment, discrimination, or the distribution of offensive or inappropriate content. Show respect and sensitivity to others in all online communications.\n",
            "Compliance: Ensure compliance with all relevant laws and regulations regarding internet and email usage, including those related to copyright and data protection.\n",
            "Monitoring: The company retains the right to monitor internet and email usage for security and compliance purposes.\n",
            "Consequences: Policy violations may lead to disciplinary measures, including potential termination.\n",
            "Our Internet and Email Policy aims to promote safe, responsible usage of digital communication tools that align with our values and legal obligations. Each employee is expected to understand and follow this policy. Regular reviews ensure its alignment with evolving technology and security standards.\n",
            "\n",
            "4.\tMobile Phone Policy\n",
            "\n",
            "The Mobile Phone Policy sets forth the standards and expectations governing the appropriate and responsible usage of mobile devices in the organization. The purpose of this policy is to ensure that employees utilize mobile phones in a manner consistent with company values and legal compliance.\n",
            "Acceptable Use: Mobile devices are primarily intended for work-related tasks. Limited personal usage is allowed, provided it does not disrupt work obligations.\n",
            "Security: Safeguard your mobile device and access credentials. Exercise caution when downloading apps or clicking links from unfamiliar sources. Promptly report security concerns or suspicious activities related to your mobile device.\n",
            "Confidentiality: Avoid transmitting sensitive company information via unsecured messaging apps or emails. Be discreet when discussing company matters in public spaces.\n",
            "Cost Management: Keep personal phone usage separate from company accounts and reimburse the company for any personal charges on company-issued phones.\n",
            "Compliance: Adhere to all pertinent laws and regulations concerning mobile phone usage, including those related to data protection and privacy.\n",
            "Lost or Stolen Devices: Immediately report any lost or stolen mobile devices to the IT department or your supervisor.\n",
            "Consequences: Non-compliance with this policy may lead to disciplinary actions, including the potential loss of mobile phone privileges.\n",
            "The Mobile Phone Policy is aimed at promoting the responsible and secure use of mobile devices in line with legal and ethical standards. Every employee is expected to comprehend and abide by these guidelines. Regular reviews of the policy ensure its ongoing alignment with evolving technology and security best practices.\n",
            "\n",
            "5.\tSmoking Policy\n",
            "\n",
            "Policy Purpose: The Smoking Policy has been established to provide clear guidance and expectations concerning smoking on company premises. This policy is in place to ensure a safe and healthy environment for all employees, visitors, and the general public.\n",
            "Designated Smoking Areas: Smoking is only permitted in designated smoking areas, as marked by appropriate signage. These areas have been chosen to minimize exposure to secondhand smoke and to maintain the overall cleanliness of the premises.\n",
            "Smoking Restrictions: Smoking inside company buildings, offices, meeting rooms, and other enclosed spaces is strictly prohibited. This includes electronic cigarettes and vaping devices.\n",
            "Compliance with Applicable Laws: All employees and visitors must adhere to relevant federal, state, and local smoking laws and regulations.\n",
            "Disposal of Smoking Materials: Properly dispose of cigarette butts and related materials in designated receptacles. Littering on company premises is prohibited.\n",
            "No Smoking in Company Vehicles: Smoking is not permitted in company vehicles, whether they are owned or leased, to maintain the condition and cleanliness of these vehicles.\n",
            "Enforcement and Consequences: All employees and visitors are expected to adhere to this policy. Non-compliance may lead to appropriate disciplinary action, which could include fines, or, in the case of employees, possible termination of employment.\n",
            "Review of Policy: This policy will be reviewed periodically to ensure its alignment with evolving legal requirements and best practices for maintaining a healthy and safe workplace.\n",
            "We appreciate your cooperation in maintaining a smoke-free and safe environment for all.\n",
            "\n",
            "6.\tDrug and Alcohol Policy\n",
            "\n",
            "Policy Objective: The Drug and Alcohol Policy is established to establish clear expectations and guidelines for the responsible use of drugs and alcohol within the organization. This policy aims to maintain a safe, healthy, and productive workplace.\n",
            "Prohibited Substances: The use, possession, distribution, or sale of illegal drugs or unauthorized controlled substances is strictly prohibited on company premises or during work-related activities. This includes the misuse of prescription drugs.\n",
            "Alcohol Consumption: The consumption of alcoholic beverages is not allowed during work hours, on company property, or while performing company-related duties. Exception may be made for company-sanctioned events.\n",
            "Impairment: Employees are expected to perform their job duties without impairment from drugs or alcohol. The use of substances that could impair job performance or pose a safety risk is prohibited.\n",
            "Testing and Searches: The organization reserves the right to conduct drug and alcohol testing as per applicable laws and regulations. Employees may be subject to testing in cases of reasonable suspicion, post-accident, or as part of routine workplace safety measures.\n",
            "Reporting: Employees should report any concerns related to drug or alcohol misuse by themselves or their colleagues, as well as safety concerns arising from such misuse.\n",
            "Treatment and Assistance: Employees with substance abuse issues are encouraged to seek help. The organization is committed to providing support, resources, and information to assist those seeking treatment.\n",
            "Consequences: Violation of this policy may result in disciplinary actions, up to and including termination of employment. Legal action may also be pursued when necessary.\n",
            "Policy Review: This policy will undergo periodic review to ensure its continued relevance and compliance with evolving legal requirements and best practices for a safe and productive work environment.\n",
            "Your adherence to this policy is appreciated as it helps to maintain a safe and drug-free workplace for all.\n",
            "\n",
            "7.\tHealth and Safety Policy\n",
            "\n",
            "Our commitment to health and safety is paramount. We prioritize the well-being of our employees, customers, and the public. We diligently comply with all relevant health and safety laws and regulations. Our objective is to maintain a workplace free from hazards, preventing accidents, injuries, and illnesses. Every individual within our organization is responsible for upholding these standards. We regularly assess and improve our safety measures, provide adequate training, and encourage open communication regarding safety concerns. Through collective dedication, we aim to ensure a safe, healthy, and secure environment for all. Your cooperation is essential in achieving this common goal.\n",
            "\n",
            "8.\tAnti-discrimination and Harassment Policy\n",
            "\n",
            "The Anti-Discrimination and Harassment Policy is a testament to the commitment of this organization in fostering a workplace that is free from discrimination, harassment, and any form of unlawful bias. This policy applies to every individual within the organization, including employees, contractors, visitors, and clients.\n",
            "Non-Discrimination: This organization strictly prohibits discrimination based on race, color, religion, gender, national origin, age, disability, sexual orientation, or any other legally protected characteristic in all aspects of employment, including recruitment, hiring, compensation, benefits, promotions, and terminations.\n",
            "Harassment: Harassment in any form, whether based on the aforementioned characteristics or any other protected status, is unacceptable. This encompasses unwelcome advances, offensive jokes, slurs, and other verbal or physical conduct that creates a hostile or intimidating work environment.\n",
            "Reporting: Individuals who experience or witness any form of discrimination or harassment are encouraged to promptly report the incident to their supervisor, manager, or the designated HR representative. The organization is committed to a timely and confidential investigation of such complaints.\n",
            "Consequences: Violation of this policy may result in disciplinary action, including termination of employment. The organization is committed to taking appropriate action against any individual found to be in violation of this policy.\n",
            "Review and Update: This policy is subject to regular review and update to remain aligned with evolving legal requirements and best practices in preventing discrimination and harassment. This organization considers it a collective responsibility to ensure a workplace free from discrimination and harassment, and it is essential that every individual within the organization plays their part in upholding these principles.\n",
            "\n",
            "9.\tDiscipline and Termination Policy\n",
            "\n",
            "The Discipline and Termination Policy underscores the organization's commitment to maintaining a productive, ethical, and respectful work environment. This policy applies to all personnel, including employees, contractors, and temporary staff.\n",
            "Performance and Conduct Expectations: Employees are expected to meet performance standards and adhere to conduct guidelines. The organization will provide clear expectations, feedback, and opportunities for improvement when performance or conduct issues arise.\n",
            "Disciplinary Actions: When necessary, disciplinary actions will be taken, which may include verbal warnings, written warnings, suspension, or other appropriate measures. Disciplinary actions are designed to address issues constructively and maintain performance standards.\n",
            "Termination: In situations where an employee's performance or conduct issues persist, the organization may resort to termination. Termination may also occur for reasons such as redundancy, violation of policies, or restructuring.\n",
            "Termination Procedure: The organization will follow appropriate procedures, ensuring fairness and adherence to legal requirements during the termination process. Employees may be eligible for notice periods, severance pay, or other benefits as per employment agreements and applicable laws.\n",
            "Exit Process: The organization will conduct an exit process to ensure a smooth transition for departing employees, including the return of company property, final pay, and cancellation of access and benefits.\n",
            "This policy serves as a framework for handling discipline and termination. The organization recognizes the importance of fairness and consistency in these processes, and decisions will be made after careful consideration. Every employee is expected to understand and adhere to this policy, contributing to a respectful and productive workplace. Regular reviews will ensure its alignment with evolving legal requirements and best practices.\n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(companypolicies)"
      ],
      "execution_count": 4
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fc2bbfbf-7311-45f3-9ab6-4b776ee89871"
      },
      "source": [
        "It is a long document about a company's policies.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aa058f90-b438-425c-8f4b-55dd8eba5f03"
      },
      "source": [
        "### Document object\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5265d964-6f3c-40f6-9759-f3ff14c0601f"
      },
      "source": [
        "Before introducing the splitters, let's take a look at the document object in LangChain, which is a data structure used to represent and manage text data in RAG process.\n",
        "\n",
        "A Document object in LangChain contains information about some data. It has two attributes:\n",
        "\n",
        "- `page_content: str`: The content of this document. Currently is only a string.\n",
        "- `metadata: dict`: Arbitrary metadata associated with this document. Can track the document id, file name, etc.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4a9169f8-8441-43fb-a913-7fdbb272bd87"
      },
      "source": [
        "Here, you can use an example to guide you through creating a document object. Langchain uses this object type to deal with text/documents.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1d721479-6e12-4145-a1e2-deac5a1085e6",
        "outputId": "c2b632cf-1925-4744-e0e6-cc27679dbd54"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Document(metadata={'my_document_id': 234234, 'my_document_source': 'About Python', 'my_document_create_time': 1680013019}, page_content=\"Python is an interpreted high-level general-purpose programming language. \\n                        Python's design philosophy emphasizes code readability with its notable use of significant indentation.\")"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ],
      "source": [
        "from langchain_core.documents import Document\n",
        "Document(page_content=\"\"\"Python is an interpreted high-level general-purpose programming language.\n",
        "                        Python's design philosophy emphasizes code readability with its notable use of significant indentation.\"\"\",\n",
        "         metadata={\n",
        "             'my_document_id' : 234234,\n",
        "             'my_document_source' : \"About Python\",\n",
        "             'my_document_create_time' : 1680013019\n",
        "         })"
      ],
      "execution_count": 5
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "50347df0-ea9f-4e1a-b18b-7defb2e220fb"
      },
      "source": [
        "### Split by Character\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e44b0778-2bcd-4f85-9cae-efe4ed1ff35d"
      },
      "source": [
        "This is the simplest method, which splits the text based on characters (by default `\"\\n\\n\"`) and measures chunk length by the number of characters.\n",
        "- **How the text is split**: By single character.\n",
        "- **How the chunk size is measured**: By number of characters.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d43090a2-b149-4bce-9881-019bd3d1384a"
      },
      "source": [
        "Let's see how to implement this method using code.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4e47a9eb-e89e-4592-88b7-11a26a8c605c"
      },
      "source": [
        "In the following code, you will use `CharacterTextSplitter` to split the document by character.\n",
        "- Separator: Set to `''`, meaning that any character can act as a separator once the chunk size reaches the set limit.\n",
        "- Chunk size: Set to `200`, meaning that once a chunk reaches 200 characters, it will be split.\n",
        "- Chunk overlap: Set to `20`, meaning there will be `20` characters overlapping between chunks.\n",
        "- Length function: Set to `len`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "70c6eb0a-a909-430f-aeaf-26335d9356b7"
      },
      "outputs": [],
      "source": [
        "from langchain.text_splitter import CharacterTextSplitter\n",
        "\n",
        "text_splitter = CharacterTextSplitter(\n",
        "    separator=\"\",\n",
        "    chunk_size=200,\n",
        "    chunk_overlap=20,\n",
        "    length_function=len,\n",
        ")"
      ],
      "execution_count": 6
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f0c6a37e-24a5-4a35-af27-7a744c9b5774"
      },
      "source": [
        "You will use `split_text` function to operate the split.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4b3d3042-1c13-4f05-8f58-62586611e9a4"
      },
      "outputs": [],
      "source": [
        "texts = text_splitter.split_text(companypolicies)"
      ],
      "execution_count": 7
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b93819ba-5c5d-493b-9cf3-53b08f5414db"
      },
      "source": [
        "Let's take a look how the document has been split.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c2a4bc5c-787b-4954-a7f0-a8898d68e50e",
        "outputId": "bf21cb70-fbe2-4f80-fd32-c4be2c5a6a31"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['1.\\tCode of Conduct\\n\\nOur Code of Conduct outlines the fundamental principles and ethical standards that guide every member of our organization. We are committed to maintaining a workplace that is built',\n",
              " 'kplace that is built on integrity, respect, and accountability.\\nIntegrity: We hold ourselves to the highest ethical standards. This means acting honestly and transparently in all our interactions, whe',\n",
              " 'ur interactions, whether with colleagues, clients, or the broader community. We respect and protect sensitive information, and we avoid conflicts of interest.\\nRespect: We embrace diversity and value e',\n",
              " \"iversity and value each individual's contributions. Discrimination, harassment, or any form of disrespectful behavior is unacceptable. We create an inclusive environment where differences are celebrat\",\n",
              " 'erences are celebrated and everyone is treated with dignity and courtesy.\\nAccountability: We take responsibility for our actions and decisions. We follow all relevant laws and regulations, and we stri',\n",
              " 'lations, and we strive to continuously improve our practices. We report any potential violations of this code and support the investigation of such matters.\\nSafety: We prioritize the safety of our emp',\n",
              " 'he safety of our employees, clients, and the communities we serve. We maintain a culture of safety, including reporting any unsafe conditions or practices.\\nEnvironmental Responsibility: We are committ',\n",
              " \"lity: We are committed to minimizing our environmental footprint and promoting sustainable practices.\\nOur Code of Conduct is not just a set of rules; it is the foundation of our organization's culture\",\n",
              " \"ganization's culture. We expect all employees to uphold these principles and serve as role models for others, ensuring we maintain our reputation for ethical conduct, integrity, and social responsibil\",\n",
              " 'd social responsibility.\\n\\n2.\\tRecruitment Policy\\n\\nOur Recruitment Policy reflects our commitment to attracting, selecting, and onboarding the most qualified and diverse candidates to join our organizat',\n",
              " 'o join our organization. We believe that the success of our company relies on the talents, skills, and dedication of our employees.\\nEqual Opportunity: We are an equal opportunity employer and do not d',\n",
              " 'mployer and do not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, or any other protected status. We actively promote di',\n",
              " 'actively promote diversity and inclusion.\\nTransparency: We maintain transparency in our recruitment processes. All job vacancies are advertised internally and externally when appropriate. Job descrip',\n",
              " 'opriate. Job descriptions and requirements are clear and accurately represent the role.\\nSelection Criteria: Our selection process is based on the qualifications, experience, and skills necessary for t',\n",
              " \"ills necessary for the position. Interviews and assessments are conducted objectively, and decisions are made without bias.\\nData Privacy: We are committed to protecting the privacy of candidates' pers\",\n",
              " \"of candidates' personal information and adhere to all relevant data protection laws and regulations.\\nFeedback: Candidates will receive timely and constructive feedback on their application and interv\",\n",
              " 'plication and interview performance.\\nOnboarding: New employees receive comprehensive onboarding to help them integrate into the organization effectively. This includes information on our culture, poli',\n",
              " 'on our culture, policies, and expectations.\\nEmployee Referrals: We encourage and appreciate employee referrals as they contribute to building a strong and engaged team.\\nOur Recruitment Policy is a fou',\n",
              " 'ment Policy is a foundation for creating a diverse, inclusive, and talented workforce. It ensures that we attract and hire the best candidates who align with our company values and contribute to our c',\n",
              " 'contribute to our continued success. We continuously review and update this policy to reflect evolving best practices in recruitment.\\n\\n3.\\tInternet and Email Policy\\n\\nOur Internet and Email Policy is e',\n",
              " 'nd Email Policy is established to guide the responsible and secure use of these essential tools within our organization. We recognize their significance in daily business operations and the importance',\n",
              " 's and the importance of adhering to principles that maintain security, productivity, and legal compliance.\\nAcceptable Use: Company-provided internet and email services are primarily meant for job-rela',\n",
              " \"y meant for job-related tasks. Limited personal use is allowed during non-work hours, provided it doesn't interfere with work responsibilities.\\nSecurity: Safeguard your login credentials, avoiding the\",\n",
              " 'ntials, avoiding the sharing of passwords. Exercise caution with email attachments and links from unknown sources. Promptly report any unusual online activity or potential security breaches.\\nConfident',\n",
              " 'breaches.\\nConfidentiality: Reserve email for the transmission of confidential information, trade secrets, and sensitive customer data only when encryption is applied. Exercise discretion when discuss',\n",
              " 'cretion when discussing company matters on public forums or social media.\\nHarassment and Inappropriate Content: Internet and email usage must not involve harassment, discrimination, or the distributio',\n",
              " ', or the distribution of offensive or inappropriate content. Show respect and sensitivity to others in all online communications.\\nCompliance: Ensure compliance with all relevant laws and regulations r',\n",
              " 'ws and regulations regarding internet and email usage, including those related to copyright and data protection.\\nMonitoring: The company retains the right to monitor internet and email usage for secur',\n",
              " 'mail usage for security and compliance purposes.\\nConsequences: Policy violations may lead to disciplinary measures, including potential termination.\\nOur Internet and Email Policy aims to promote safe,',\n",
              " 'ims to promote safe, responsible usage of digital communication tools that align with our values and legal obligations. Each employee is expected to understand and follow this policy. Regular reviews',\n",
              " 'cy. Regular reviews ensure its alignment with evolving technology and security standards.\\n\\n4.\\tMobile Phone Policy\\n\\nThe Mobile Phone Policy sets forth the standards and expectations governing the appro',\n",
              " 'governing the appropriate and responsible usage of mobile devices in the organization. The purpose of this policy is to ensure that employees utilize mobile phones in a manner consistent with company',\n",
              " 'sistent with company values and legal compliance.\\nAcceptable Use: Mobile devices are primarily intended for work-related tasks. Limited personal usage is allowed, provided it does not disrupt work obl',\n",
              " 'not disrupt work obligations.\\nSecurity: Safeguard your mobile device and access credentials. Exercise caution when downloading apps or clicking links from unfamiliar sources. Promptly report security',\n",
              " 'tly report security concerns or suspicious activities related to your mobile device.\\nConfidentiality: Avoid transmitting sensitive company information via unsecured messaging apps or emails. Be discre',\n",
              " 'or emails. Be discreet when discussing company matters in public spaces.\\nCost Management: Keep personal phone usage separate from company accounts and reimburse the company for any personal charges on',\n",
              " 'personal charges on company-issued phones.\\nCompliance: Adhere to all pertinent laws and regulations concerning mobile phone usage, including those related to data protection and privacy.\\nLost or Stol',\n",
              " 'rivacy.\\nLost or Stolen Devices: Immediately report any lost or stolen mobile devices to the IT department or your supervisor.\\nConsequences: Non-compliance with this policy may lead to disciplinary act',\n",
              " 'to disciplinary actions, including the potential loss of mobile phone privileges.\\nThe Mobile Phone Policy is aimed at promoting the responsible and secure use of mobile devices in line with legal and',\n",
              " 'line with legal and ethical standards. Every employee is expected to comprehend and abide by these guidelines. Regular reviews of the policy ensure its ongoing alignment with evolving technology and',\n",
              " 'ving technology and security best practices.\\n\\n5.\\tSmoking Policy\\n\\nPolicy Purpose: The Smoking Policy has been established to provide clear guidance and expectations concerning smoking on company premis',\n",
              " 'ng on company premises. This policy is in place to ensure a safe and healthy environment for all employees, visitors, and the general public.\\nDesignated Smoking Areas: Smoking is only permitted in des',\n",
              " 'nly permitted in designated smoking areas, as marked by appropriate signage. These areas have been chosen to minimize exposure to secondhand smoke and to maintain the overall cleanliness of the premis',\n",
              " 'liness of the premises.\\nSmoking Restrictions: Smoking inside company buildings, offices, meeting rooms, and other enclosed spaces is strictly prohibited. This includes electronic cigarettes and vaping',\n",
              " 'igarettes and vaping devices.\\nCompliance with Applicable Laws: All employees and visitors must adhere to relevant federal, state, and local smoking laws and regulations.\\nDisposal of Smoking Materials:',\n",
              " 'f Smoking Materials: Properly dispose of cigarette butts and related materials in designated receptacles. Littering on company premises is prohibited.\\nNo Smoking in Company Vehicles: Smoking is not pe',\n",
              " 's: Smoking is not permitted in company vehicles, whether they are owned or leased, to maintain the condition and cleanliness of these vehicles.\\nEnforcement and Consequences: All employees and visitors',\n",
              " 'ployees and visitors are expected to adhere to this policy. Non-compliance may lead to appropriate disciplinary action, which could include fines, or, in the case of employees, possible termination of',\n",
              " 'sible termination of employment.\\nReview of Policy: This policy will be reviewed periodically to ensure its alignment with evolving legal requirements and best practices for maintaining a healthy and s',\n",
              " 'ning a healthy and safe workplace.\\nWe appreciate your cooperation in maintaining a smoke-free and safe environment for all.\\n\\n6.\\tDrug and Alcohol Policy\\n\\nPolicy Objective: The Drug and Alcohol Policy i',\n",
              " 'and Alcohol Policy is established to establish clear expectations and guidelines for the responsible use of drugs and alcohol within the organization. This policy aims to maintain a safe, healthy, and',\n",
              " 'a safe, healthy, and productive workplace.\\nProhibited Substances: The use, possession, distribution, or sale of illegal drugs or unauthorized controlled substances is strictly prohibited on company pr',\n",
              " 'ibited on company premises or during work-related activities. This includes the misuse of prescription drugs.\\nAlcohol Consumption: The consumption of alcoholic beverages is not allowed during work hou',\n",
              " 'owed during work hours, on company property, or while performing company-related duties. Exception may be made for company-sanctioned events.\\nImpairment: Employees are expected to perform their job du',\n",
              " 'perform their job duties without impairment from drugs or alcohol. The use of substances that could impair job performance or pose a safety risk is prohibited.\\nTesting and Searches: The organization r',\n",
              " ': The organization reserves the right to conduct drug and alcohol testing as per applicable laws and regulations. Employees may be subject to testing in cases of reasonable suspicion, post-accident, o',\n",
              " 'on, post-accident, or as part of routine workplace safety measures.\\nReporting: Employees should report any concerns related to drug or alcohol misuse by themselves or their colleagues, as well as safe',\n",
              " 'ues, as well as safety concerns arising from such misuse.\\nTreatment and Assistance: Employees with substance abuse issues are encouraged to seek help. The organization is committed to providing suppor',\n",
              " 'to providing support, resources, and information to assist those seeking treatment.\\nConsequences: Violation of this policy may result in disciplinary actions, up to and including termination of emplo',\n",
              " 'termination of employment. Legal action may also be pursued when necessary.\\nPolicy Review: This policy will undergo periodic review to ensure its continued relevance and compliance with evolving legal',\n",
              " 'with evolving legal requirements and best practices for a safe and productive work environment.\\nYour adherence to this policy is appreciated as it helps to maintain a safe and drug-free workplace for',\n",
              " 'g-free workplace for all.\\n\\n7.\\tHealth and Safety Policy\\n\\nOur commitment to health and safety is paramount. We prioritize the well-being of our employees, customers, and the public. We diligently comply',\n",
              " 'We diligently comply with all relevant health and safety laws and regulations. Our objective is to maintain a workplace free from hazards, preventing accidents, injuries, and illnesses. Every individu',\n",
              " 'sses. Every individual within our organization is responsible for upholding these standards. We regularly assess and improve our safety measures, provide adequate training, and encourage open communic',\n",
              " 'ourage open communication regarding safety concerns. Through collective dedication, we aim to ensure a safe, healthy, and secure environment for all. Your cooperation is essential in achieving this co',\n",
              " 'in achieving this common goal.\\n\\n8.\\tAnti-discrimination and Harassment Policy\\n\\nThe Anti-Discrimination and Harassment Policy is a testament to the commitment of this organization in fostering a workpla',\n",
              " 'fostering a workplace that is free from discrimination, harassment, and any form of unlawful bias. This policy applies to every individual within the organization, including employees, contractors, v',\n",
              " 'yees, contractors, visitors, and clients.\\nNon-Discrimination: This organization strictly prohibits discrimination based on race, color, religion, gender, national origin, age, disability, sexual orien',\n",
              " 'bility, sexual orientation, or any other legally protected characteristic in all aspects of employment, including recruitment, hiring, compensation, benefits, promotions, and terminations.\\nHarassment:',\n",
              " 'nations.\\nHarassment: Harassment in any form, whether based on the aforementioned characteristics or any other protected status, is unacceptable. This encompasses unwelcome advances, offensive jokes, s',\n",
              " ', offensive jokes, slurs, and other verbal or physical conduct that creates a hostile or intimidating work environment.\\nReporting: Individuals who experience or witness any form of discrimination or h',\n",
              " 'discrimination or harassment are encouraged to promptly report the incident to their supervisor, manager, or the designated HR representative. The organization is committed to a timely and confidenti',\n",
              " 'imely and confidential investigation of such complaints.\\nConsequences: Violation of this policy may result in disciplinary action, including termination of employment. The organization is committed to',\n",
              " 'tion is committed to taking appropriate action against any individual found to be in violation of this policy.\\nReview and Update: This policy is subject to regular review and update to remain aligned',\n",
              " 'e to remain aligned with evolving legal requirements and best practices in preventing discrimination and harassment. This organization considers it a collective responsibility to ensure a workplace fr',\n",
              " 'nsure a workplace free from discrimination and harassment, and it is essential that every individual within the organization plays their part in upholding these principles.\\n\\n9.\\tDiscipline and Terminat',\n",
              " \"cipline and Termination Policy\\n\\nThe Discipline and Termination Policy underscores the organization's commitment to maintaining a productive, ethical, and respectful work environment. This policy appli\",\n",
              " 't. This policy applies to all personnel, including employees, contractors, and temporary staff.\\nPerformance and Conduct Expectations: Employees are expected to meet performance standards and adhere to',\n",
              " 'ndards and adhere to conduct guidelines. The organization will provide clear expectations, feedback, and opportunities for improvement when performance or conduct issues arise.\\nDisciplinary Actions: W',\n",
              " 'ciplinary Actions: When necessary, disciplinary actions will be taken, which may include verbal warnings, written warnings, suspension, or other appropriate measures. Disciplinary actions are designed',\n",
              " \"actions are designed to address issues constructively and maintain performance standards.\\nTermination: In situations where an employee's performance or conduct issues persist, the organization may res\",\n",
              " 'organization may resort to termination. Termination may also occur for reasons such as redundancy, violation of policies, or restructuring.\\nTermination Procedure: The organization will follow appropri',\n",
              " 'will follow appropriate procedures, ensuring fairness and adherence to legal requirements during the termination process. Employees may be eligible for notice periods, severance pay, or other benefits',\n",
              " 'y, or other benefits as per employment agreements and applicable laws.\\nExit Process: The organization will conduct an exit process to ensure a smooth transition for departing employees, including the',\n",
              " 'yees, including the return of company property, final pay, and cancellation of access and benefits.\\nThis policy serves as a framework for handling discipline and termination. The organization recogniz',\n",
              " 'rganization recognizes the importance of fairness and consistency in these processes, and decisions will be made after careful consideration. Every employee is expected to understand and adhere to thi',\n",
              " 'nd and adhere to this policy, contributing to a respectful and productive workplace. Regular reviews will ensure its alignment with evolving legal requirements and best practices.']"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ],
      "source": [
        "texts"
      ],
      "execution_count": 8
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e10af240-c62e-4e84-9043-24b6deed255c"
      },
      "source": [
        "After the split, you'll see that the document has been divided into multiple chunks, with some character overlaps between the chunks.\n",
        "\n",
        "You can see how many chunks you get.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4795048a-b3b5-4106-9332-dd27ee28e847",
        "outputId": "5ce55987-de10-445a-ae94-48eb3b4e803a"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "87"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "len(texts)"
      ],
      "execution_count": 9
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d4f32879-97fd-457a-b83b-1c70c7824082"
      },
      "source": [
        "You get `87` chunks.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4559e76d-8524-404e-9ab5-572b90a92850"
      },
      "source": [
        "You can also use the following code to add metadata to the text, forming it into a `document` object using LangChain.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "948f8be8-3101-4b2b-85a9-de6440a97fc6"
      },
      "outputs": [],
      "source": [
        "texts = text_splitter.create_documents([companypolicies], metadatas=[{\"document\":\"Company Policies\"}])  # pass the metadata as well"
      ],
      "execution_count": 10
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "824a4d7c-9cff-49fb-9026-7d187b699223",
        "outputId": "80ff5cbc-f59b-4bd3-a187-ca091d852a9f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Document(metadata={'document': 'Company Policies'}, page_content='1.\\tCode of Conduct\\n\\nOur Code of Conduct outlines the fundamental principles and ethical standards that guide every member of our organization. We are committed to maintaining a workplace that is built')"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ],
      "source": [
        "texts[0]"
      ],
      "execution_count": 11
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ad82fbc1-19b6-4d92-aefb-0ae4f413af51"
      },
      "source": [
        "### Recursively Split by Character\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9dce3207-12f9-44c3-afb7-6f2adca470e5"
      },
      "source": [
        "This text splitter is the recommended one for generic text. It is parameterized by a list of characters, and it tries to split on them in order until the chunks are small enough. The default list is `[\"\\n\\n\", \"\\n\", \" \", \"\"]`.\n",
        "\n",
        "It processes the large text by attempting to split it by the first character, `\\n\\n`. If the first split by \\n\\n results in chunks that are still too large, it moves to the next character, `\\n`, and attempts to split by it. This process continues through the list of characters until the chunks are less than the specified chunk size.\n",
        "\n",
        "This method aims to keep all paragraphs (then sentences, then words) together as much as possible, as these are generally the most semantically related pieces of text.\n",
        "\n",
        "- **How the text is split**: by list of characters.\n",
        "- **How the chunk size is measured**: by number of characters.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4a07b0a6-e170-41cd-af24-86c9d3dbfacd"
      },
      "source": [
        "The following code is showing how to implement it.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9038bbf4-15a0-4370-8e6c-b13fee73b787"
      },
      "source": [
        "The `RecursiveCharacterTextSplitter` class from LangChain is used to implement it.\n",
        "- You use the default separator list, which is `[\"\\n\\n\", \"\\n\", \" \", \"\"]`.\n",
        "- Chunk size is set to `100`.\n",
        "- Chunk overlap is set to `20`.\n",
        "- And the length function is `len`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ceb557d1-cca1-4b0f-8857-ccbd46ee20b2"
      },
      "outputs": [],
      "source": [
        "from langchain.text_splitter import RecursiveCharacterTextSplitter"
      ],
      "execution_count": 12
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5ead98a3-3044-481d-a23a-39ef9b6cdb3c"
      },
      "outputs": [],
      "source": [
        "text_splitter = RecursiveCharacterTextSplitter(\n",
        "    chunk_size=100,\n",
        "    chunk_overlap=20,\n",
        "    length_function=len,\n",
        ")"
      ],
      "execution_count": 13
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cba03fe8-bdf2-48af-9358-3bbd158f29a6"
      },
      "source": [
        "Here, you are using the same document \"companypolicies.txt\" from earlier as an example to show the performance of `RecursiveCharacterTextSplitter`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4b76de2e-d9ac-47ee-adff-042c716be351"
      },
      "outputs": [],
      "source": [
        "texts = text_splitter.create_documents([companypolicies])"
      ],
      "execution_count": 14
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4cf28c8c-1c96-4b6c-8767-917258cc7ff3",
        "outputId": "e3ff6405-d647-4102-d9c4-e8447906125f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Document(page_content='1.\\tCode of Conduct'),\n",
              " Document(page_content='Our Code of Conduct outlines the fundamental principles and ethical standards that guide every'),\n",
              " Document(page_content='that guide every member of our organization. We are committed to maintaining a workplace that is'),\n",
              " Document(page_content='a workplace that is built on integrity, respect, and accountability.'),\n",
              " Document(page_content='Integrity: We hold ourselves to the highest ethical standards. This means acting honestly and'),\n",
              " Document(page_content='acting honestly and transparently in all our interactions, whether with colleagues, clients, or the'),\n",
              " Document(page_content='clients, or the broader community. We respect and protect sensitive information, and we avoid'),\n",
              " Document(page_content='and we avoid conflicts of interest.'),\n",
              " Document(page_content=\"Respect: We embrace diversity and value each individual's contributions. Discrimination,\"),\n",
              " Document(page_content='Discrimination, harassment, or any form of disrespectful behavior is unacceptable. We create an'),\n",
              " Document(page_content='We create an inclusive environment where differences are celebrated and everyone is treated with'),\n",
              " Document(page_content='is treated with dignity and courtesy.'),\n",
              " Document(page_content='Accountability: We take responsibility for our actions and decisions. We follow all relevant laws'),\n",
              " Document(page_content='all relevant laws and regulations, and we strive to continuously improve our practices. We report'),\n",
              " Document(page_content='We report any potential violations of this code and support the investigation of such matters.'),\n",
              " Document(page_content='Safety: We prioritize the safety of our employees, clients, and the communities we serve. We'),\n",
              " Document(page_content='we serve. We maintain a culture of safety, including reporting any unsafe conditions or practices.'),\n",
              " Document(page_content='Environmental Responsibility: We are committed to minimizing our environmental footprint and'),\n",
              " Document(page_content='footprint and promoting sustainable practices.'),\n",
              " Document(page_content=\"Our Code of Conduct is not just a set of rules; it is the foundation of our organization's culture.\"),\n",
              " Document(page_content='culture. We expect all employees to uphold these principles and serve as role models for others,'),\n",
              " Document(page_content='models for others, ensuring we maintain our reputation for ethical conduct, integrity, and social'),\n",
              " Document(page_content='and social responsibility.'),\n",
              " Document(page_content='2.\\tRecruitment Policy'),\n",
              " Document(page_content='Our Recruitment Policy reflects our commitment to attracting, selecting, and onboarding the most'),\n",
              " Document(page_content='onboarding the most qualified and diverse candidates to join our organization. We believe that the'),\n",
              " Document(page_content='We believe that the success of our company relies on the talents, skills, and dedication of our'),\n",
              " Document(page_content='dedication of our employees.'),\n",
              " Document(page_content='Equal Opportunity: We are an equal opportunity employer and do not discriminate on the basis of'),\n",
              " Document(page_content='on the basis of race, color, religion, sex, sexual orientation, gender identity, national origin,'),\n",
              " Document(page_content='national origin, age, disability, or any other protected status. We actively promote diversity and'),\n",
              " Document(page_content='diversity and inclusion.'),\n",
              " Document(page_content='Transparency: We maintain transparency in our recruitment processes. All job vacancies are'),\n",
              " Document(page_content='job vacancies are advertised internally and externally when appropriate. Job descriptions and'),\n",
              " Document(page_content='descriptions and requirements are clear and accurately represent the role.'),\n",
              " Document(page_content='Selection Criteria: Our selection process is based on the qualifications, experience, and skills'),\n",
              " Document(page_content='and skills necessary for the position. Interviews and assessments are conducted objectively, and'),\n",
              " Document(page_content='objectively, and decisions are made without bias.'),\n",
              " Document(page_content=\"Data Privacy: We are committed to protecting the privacy of candidates' personal information and\"),\n",
              " Document(page_content='information and adhere to all relevant data protection laws and regulations.'),\n",
              " Document(page_content='Feedback: Candidates will receive timely and constructive feedback on their application and'),\n",
              " Document(page_content='application and interview performance.'),\n",
              " Document(page_content='Onboarding: New employees receive comprehensive onboarding to help them integrate into the'),\n",
              " Document(page_content='integrate into the organization effectively. This includes information on our culture, policies,'),\n",
              " Document(page_content='culture, policies, and expectations.'),\n",
              " Document(page_content='Employee Referrals: We encourage and appreciate employee referrals as they contribute to building a'),\n",
              " Document(page_content='to building a strong and engaged team.'),\n",
              " Document(page_content='Our Recruitment Policy is a foundation for creating a diverse, inclusive, and talented workforce.'),\n",
              " Document(page_content='talented workforce. It ensures that we attract and hire the best candidates who align with our'),\n",
              " Document(page_content='who align with our company values and contribute to our continued success. We continuously review'),\n",
              " Document(page_content='continuously review and update this policy to reflect evolving best practices in recruitment.'),\n",
              " Document(page_content='3.\\tInternet and Email Policy'),\n",
              " Document(page_content='Our Internet and Email Policy is established to guide the responsible and secure use of these'),\n",
              " Document(page_content='secure use of these essential tools within our organization. We recognize their significance in'),\n",
              " Document(page_content='significance in daily business operations and the importance of adhering to principles that'),\n",
              " Document(page_content='to principles that maintain security, productivity, and legal compliance.'),\n",
              " Document(page_content='Acceptable Use: Company-provided internet and email services are primarily meant for job-related'),\n",
              " Document(page_content=\"for job-related tasks. Limited personal use is allowed during non-work hours, provided it doesn't\"),\n",
              " Document(page_content=\"provided it doesn't interfere with work responsibilities.\"),\n",
              " Document(page_content='Security: Safeguard your login credentials, avoiding the sharing of passwords. Exercise caution'),\n",
              " Document(page_content='Exercise caution with email attachments and links from unknown sources. Promptly report any unusual'),\n",
              " Document(page_content='report any unusual online activity or potential security breaches.'),\n",
              " Document(page_content='Confidentiality: Reserve email for the transmission of confidential information, trade secrets, and'),\n",
              " Document(page_content='trade secrets, and sensitive customer data only when encryption is applied. Exercise discretion'),\n",
              " Document(page_content='Exercise discretion when discussing company matters on public forums or social media.'),\n",
              " Document(page_content='Harassment and Inappropriate Content: Internet and email usage must not involve harassment,'),\n",
              " Document(page_content='involve harassment, discrimination, or the distribution of offensive or inappropriate content. Show'),\n",
              " Document(page_content='content. Show respect and sensitivity to others in all online communications.'),\n",
              " Document(page_content='Compliance: Ensure compliance with all relevant laws and regulations regarding internet and email'),\n",
              " Document(page_content='internet and email usage, including those related to copyright and data protection.'),\n",
              " Document(page_content='Monitoring: The company retains the right to monitor internet and email usage for security and'),\n",
              " Document(page_content='for security and compliance purposes.'),\n",
              " Document(page_content='Consequences: Policy violations may lead to disciplinary measures, including potential termination.'),\n",
              " Document(page_content='Our Internet and Email Policy aims to promote safe, responsible usage of digital communication'),\n",
              " Document(page_content='communication tools that align with our values and legal obligations. Each employee is expected to'),\n",
              " Document(page_content='is expected to understand and follow this policy. Regular reviews ensure its alignment with'),\n",
              " Document(page_content='its alignment with evolving technology and security standards.'),\n",
              " Document(page_content='4.\\tMobile Phone Policy'),\n",
              " Document(page_content='The Mobile Phone Policy sets forth the standards and expectations governing the appropriate and'),\n",
              " Document(page_content='the appropriate and responsible usage of mobile devices in the organization. The purpose of this'),\n",
              " Document(page_content='The purpose of this policy is to ensure that employees utilize mobile phones in a manner consistent'),\n",
              " Document(page_content='a manner consistent with company values and legal compliance.'),\n",
              " Document(page_content='Acceptable Use: Mobile devices are primarily intended for work-related tasks. Limited personal'),\n",
              " Document(page_content='Limited personal usage is allowed, provided it does not disrupt work obligations.'),\n",
              " Document(page_content='Security: Safeguard your mobile device and access credentials. Exercise caution when downloading'),\n",
              " Document(page_content='when downloading apps or clicking links from unfamiliar sources. Promptly report security concerns'),\n",
              " Document(page_content='security concerns or suspicious activities related to your mobile device.'),\n",
              " Document(page_content='Confidentiality: Avoid transmitting sensitive company information via unsecured messaging apps or'),\n",
              " Document(page_content='messaging apps or emails. Be discreet when discussing company matters in public spaces.'),\n",
              " Document(page_content='Cost Management: Keep personal phone usage separate from company accounts and reimburse the company'),\n",
              " Document(page_content='the company for any personal charges on company-issued phones.'),\n",
              " Document(page_content='Compliance: Adhere to all pertinent laws and regulations concerning mobile phone usage, including'),\n",
              " Document(page_content='usage, including those related to data protection and privacy.'),\n",
              " Document(page_content='Lost or Stolen Devices: Immediately report any lost or stolen mobile devices to the IT department'),\n",
              " Document(page_content='the IT department or your supervisor.'),\n",
              " Document(page_content='Consequences: Non-compliance with this policy may lead to disciplinary actions, including the'),\n",
              " Document(page_content='including the potential loss of mobile phone privileges.'),\n",
              " Document(page_content='The Mobile Phone Policy is aimed at promoting the responsible and secure use of mobile devices in'),\n",
              " Document(page_content='mobile devices in line with legal and ethical standards. Every employee is expected to comprehend'),\n",
              " Document(page_content='to comprehend and abide by these guidelines. Regular reviews of the policy ensure its ongoing'),\n",
              " Document(page_content='ensure its ongoing alignment with evolving technology and security best practices.'),\n",
              " Document(page_content='5.\\tSmoking Policy'),\n",
              " Document(page_content='Policy Purpose: The Smoking Policy has been established to provide clear guidance and expectations'),\n",
              " Document(page_content='and expectations concerning smoking on company premises. This policy is in place to ensure a safe'),\n",
              " Document(page_content='to ensure a safe and healthy environment for all employees, visitors, and the general public.'),\n",
              " Document(page_content='Designated Smoking Areas: Smoking is only permitted in designated smoking areas, as marked by'),\n",
              " Document(page_content='areas, as marked by appropriate signage. These areas have been chosen to minimize exposure to'),\n",
              " Document(page_content='exposure to secondhand smoke and to maintain the overall cleanliness of the premises.'),\n",
              " Document(page_content='Smoking Restrictions: Smoking inside company buildings, offices, meeting rooms, and other enclosed'),\n",
              " Document(page_content='and other enclosed spaces is strictly prohibited. This includes electronic cigarettes and vaping'),\n",
              " Document(page_content='and vaping devices.'),\n",
              " Document(page_content='Compliance with Applicable Laws: All employees and visitors must adhere to relevant federal, state,'),\n",
              " Document(page_content='federal, state, and local smoking laws and regulations.'),\n",
              " Document(page_content='Disposal of Smoking Materials: Properly dispose of cigarette butts and related materials in'),\n",
              " Document(page_content='materials in designated receptacles. Littering on company premises is prohibited.'),\n",
              " Document(page_content='No Smoking in Company Vehicles: Smoking is not permitted in company vehicles, whether they are'),\n",
              " Document(page_content='whether they are owned or leased, to maintain the condition and cleanliness of these vehicles.'),\n",
              " Document(page_content='Enforcement and Consequences: All employees and visitors are expected to adhere to this policy.'),\n",
              " Document(page_content='to this policy. Non-compliance may lead to appropriate disciplinary action, which could include'),\n",
              " Document(page_content='which could include fines, or, in the case of employees, possible termination of employment.'),\n",
              " Document(page_content='Review of Policy: This policy will be reviewed periodically to ensure its alignment with evolving'),\n",
              " Document(page_content='with evolving legal requirements and best practices for maintaining a healthy and safe workplace.'),\n",
              " Document(page_content='We appreciate your cooperation in maintaining a smoke-free and safe environment for all.'),\n",
              " Document(page_content='6.\\tDrug and Alcohol Policy'),\n",
              " Document(page_content='Policy Objective: The Drug and Alcohol Policy is established to establish clear expectations and'),\n",
              " Document(page_content='expectations and guidelines for the responsible use of drugs and alcohol within the organization.'),\n",
              " Document(page_content='the organization. This policy aims to maintain a safe, healthy, and productive workplace.'),\n",
              " Document(page_content='Prohibited Substances: The use, possession, distribution, or sale of illegal drugs or unauthorized'),\n",
              " Document(page_content='or unauthorized controlled substances is strictly prohibited on company premises or during'),\n",
              " Document(page_content='premises or during work-related activities. This includes the misuse of prescription drugs.'),\n",
              " Document(page_content='Alcohol Consumption: The consumption of alcoholic beverages is not allowed during work hours, on'),\n",
              " Document(page_content='work hours, on company property, or while performing company-related duties. Exception may be made'),\n",
              " Document(page_content='may be made for company-sanctioned events.'),\n",
              " Document(page_content='Impairment: Employees are expected to perform their job duties without impairment from drugs or'),\n",
              " Document(page_content='from drugs or alcohol. The use of substances that could impair job performance or pose a safety'),\n",
              " Document(page_content='or pose a safety risk is prohibited.'),\n",
              " Document(page_content='Testing and Searches: The organization reserves the right to conduct drug and alcohol testing as'),\n",
              " Document(page_content='alcohol testing as per applicable laws and regulations. Employees may be subject to testing in'),\n",
              " Document(page_content='to testing in cases of reasonable suspicion, post-accident, or as part of routine workplace safety'),\n",
              " Document(page_content='workplace safety measures.'),\n",
              " Document(page_content='Reporting: Employees should report any concerns related to drug or alcohol misuse by themselves or'),\n",
              " Document(page_content='by themselves or their colleagues, as well as safety concerns arising from such misuse.'),\n",
              " Document(page_content='Treatment and Assistance: Employees with substance abuse issues are encouraged to seek help. The'),\n",
              " Document(page_content='to seek help. The organization is committed to providing support, resources, and information to'),\n",
              " Document(page_content='and information to assist those seeking treatment.'),\n",
              " Document(page_content='Consequences: Violation of this policy may result in disciplinary actions, up to and including'),\n",
              " Document(page_content='up to and including termination of employment. Legal action may also be pursued when necessary.'),\n",
              " Document(page_content='Policy Review: This policy will undergo periodic review to ensure its continued relevance and'),\n",
              " Document(page_content='relevance and compliance with evolving legal requirements and best practices for a safe and'),\n",
              " Document(page_content='for a safe and productive work environment.'),\n",
              " Document(page_content='Your adherence to this policy is appreciated as it helps to maintain a safe and drug-free workplace'),\n",
              " Document(page_content='drug-free workplace for all.'),\n",
              " Document(page_content='7.\\tHealth and Safety Policy'),\n",
              " Document(page_content='Our commitment to health and safety is paramount. We prioritize the well-being of our employees,'),\n",
              " Document(page_content='of our employees, customers, and the public. We diligently comply with all relevant health and'),\n",
              " Document(page_content='relevant health and safety laws and regulations. Our objective is to maintain a workplace free from'),\n",
              " Document(page_content='workplace free from hazards, preventing accidents, injuries, and illnesses. Every individual within'),\n",
              " Document(page_content='individual within our organization is responsible for upholding these standards. We regularly'),\n",
              " Document(page_content='We regularly assess and improve our safety measures, provide adequate training, and encourage open'),\n",
              " Document(page_content='and encourage open communication regarding safety concerns. Through collective dedication, we aim'),\n",
              " Document(page_content='dedication, we aim to ensure a safe, healthy, and secure environment for all. Your cooperation is'),\n",
              " Document(page_content='Your cooperation is essential in achieving this common goal.'),\n",
              " Document(page_content='8.\\tAnti-discrimination and Harassment Policy'),\n",
              " Document(page_content='The Anti-Discrimination and Harassment Policy is a testament to the commitment of this organization'),\n",
              " Document(page_content='this organization in fostering a workplace that is free from discrimination, harassment, and any'),\n",
              " Document(page_content='harassment, and any form of unlawful bias. This policy applies to every individual within the'),\n",
              " Document(page_content='within the organization, including employees, contractors, visitors, and clients.'),\n",
              " Document(page_content='Non-Discrimination: This organization strictly prohibits discrimination based on race, color,'),\n",
              " Document(page_content='on race, color, religion, gender, national origin, age, disability, sexual orientation, or any'),\n",
              " Document(page_content='orientation, or any other legally protected characteristic in all aspects of employment, including'),\n",
              " Document(page_content='including recruitment, hiring, compensation, benefits, promotions, and terminations.'),\n",
              " Document(page_content='Harassment: Harassment in any form, whether based on the aforementioned characteristics or any'),\n",
              " Document(page_content='or any other protected status, is unacceptable. This encompasses unwelcome advances, offensive'),\n",
              " Document(page_content='advances, offensive jokes, slurs, and other verbal or physical conduct that creates a hostile or'),\n",
              " Document(page_content='a hostile or intimidating work environment.'),\n",
              " Document(page_content='Reporting: Individuals who experience or witness any form of discrimination or harassment are'),\n",
              " Document(page_content='or harassment are encouraged to promptly report the incident to their supervisor, manager, or the'),\n",
              " Document(page_content='manager, or the designated HR representative. The organization is committed to a timely and'),\n",
              " Document(page_content='to a timely and confidential investigation of such complaints.'),\n",
              " Document(page_content='Consequences: Violation of this policy may result in disciplinary action, including termination of'),\n",
              " Document(page_content='termination of employment. The organization is committed to taking appropriate action against any'),\n",
              " Document(page_content='action against any individual found to be in violation of this policy.'),\n",
              " Document(page_content='Review and Update: This policy is subject to regular review and update to remain aligned with'),\n",
              " Document(page_content='remain aligned with evolving legal requirements and best practices in preventing discrimination and'),\n",
              " Document(page_content='discrimination and harassment. This organization considers it a collective responsibility to ensure'),\n",
              " Document(page_content='to ensure a workplace free from discrimination and harassment, and it is essential that every'),\n",
              " Document(page_content='that every individual within the organization plays their part in upholding these principles.'),\n",
              " Document(page_content='9.\\tDiscipline and Termination Policy'),\n",
              " Document(page_content=\"The Discipline and Termination Policy underscores the organization's commitment to maintaining a\"),\n",
              " Document(page_content='to maintaining a productive, ethical, and respectful work environment. This policy applies to all'),\n",
              " Document(page_content='applies to all personnel, including employees, contractors, and temporary staff.'),\n",
              " Document(page_content='Performance and Conduct Expectations: Employees are expected to meet performance standards and'),\n",
              " Document(page_content='standards and adhere to conduct guidelines. The organization will provide clear expectations,'),\n",
              " Document(page_content='clear expectations, feedback, and opportunities for improvement when performance or conduct issues'),\n",
              " Document(page_content='or conduct issues arise.'),\n",
              " Document(page_content='Disciplinary Actions: When necessary, disciplinary actions will be taken, which may include verbal'),\n",
              " Document(page_content='may include verbal warnings, written warnings, suspension, or other appropriate measures.'),\n",
              " Document(page_content='measures. Disciplinary actions are designed to address issues constructively and maintain'),\n",
              " Document(page_content='and maintain performance standards.'),\n",
              " Document(page_content=\"Termination: In situations where an employee's performance or conduct issues persist, the\"),\n",
              " Document(page_content='issues persist, the organization may resort to termination. Termination may also occur for reasons'),\n",
              " Document(page_content='occur for reasons such as redundancy, violation of policies, or restructuring.'),\n",
              " Document(page_content='Termination Procedure: The organization will follow appropriate procedures, ensuring fairness and'),\n",
              " Document(page_content='fairness and adherence to legal requirements during the termination process. Employees may be'),\n",
              " Document(page_content='Employees may be eligible for notice periods, severance pay, or other benefits as per employment'),\n",
              " Document(page_content='as per employment agreements and applicable laws.'),\n",
              " Document(page_content='Exit Process: The organization will conduct an exit process to ensure a smooth transition for'),\n",
              " Document(page_content='transition for departing employees, including the return of company property, final pay, and'),\n",
              " Document(page_content='final pay, and cancellation of access and benefits.'),\n",
              " Document(page_content='This policy serves as a framework for handling discipline and termination. The organization'),\n",
              " Document(page_content='The organization recognizes the importance of fairness and consistency in these processes, and'),\n",
              " Document(page_content='processes, and decisions will be made after careful consideration. Every employee is expected to'),\n",
              " Document(page_content='is expected to understand and adhere to this policy, contributing to a respectful and productive'),\n",
              " Document(page_content='and productive workplace. Regular reviews will ensure its alignment with evolving legal'),\n",
              " Document(page_content='with evolving legal requirements and best practices.')]"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ],
      "source": [
        "texts"
      ],
      "execution_count": 15
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fb480058-f35f-4561-a5fb-baf2742aefc0"
      },
      "source": [
        "From the split results, you can see that the splitter uses recursion as the core strategy to divide the document into chunks.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "31721def-7ee8-4dc6-999d-52f639437399"
      },
      "source": [
        "You can also see how many chunks you get.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9aeecbe0-b5d0-42f5-8668-ae1213fb810b",
        "outputId": "b47b7549-3596-44fd-8b7f-2a63dc6d79c5"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "215"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ],
      "source": [
        "len(texts)"
      ],
      "execution_count": 16
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cf978837-81af-47d8-94d1-4cb9b63a690d"
      },
      "source": [
        "You get `215` chunks.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b24ca269-e865-4239-90c3-e05b84d0aa07"
      },
      "source": [
        "### Split Code\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1640e255-0935-405e-b4aa-eee4a35b49dc"
      },
      "source": [
        "The `CodeTextSplitter` allows you to split your code, supporting multiple programming languages. It is based on the `RecursiveCharacterTextSplitter` strategy. Simply import enum `Language` and specify the language.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2b5318ba-9070-4bf2-ac87-c9f807529cea"
      },
      "outputs": [],
      "source": [
        "from langchain.text_splitter import Language, RecursiveCharacterTextSplitter"
      ],
      "execution_count": 17
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "841d9014-8736-489d-9415-3952b9b93ebb"
      },
      "source": [
        "Use the following to see a list of codes it supports.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b32a2c26-59bc-4a09-8d50-c7512174281b",
        "outputId": "827e1c54-4e04-458f-f35f-eabaa15b6ec6"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['cpp',\n",
              " 'go',\n",
              " 'java',\n",
              " 'kotlin',\n",
              " 'js',\n",
              " 'ts',\n",
              " 'php',\n",
              " 'proto',\n",
              " 'python',\n",
              " 'rst',\n",
              " 'ruby',\n",
              " 'rust',\n",
              " 'scala',\n",
              " 'swift',\n",
              " 'markdown',\n",
              " 'latex',\n",
              " 'html',\n",
              " 'sol',\n",
              " 'csharp',\n",
              " 'cobol',\n",
              " 'c',\n",
              " 'lua',\n",
              " 'perl',\n",
              " 'haskell',\n",
              " 'elixir']"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ],
      "source": [
        "[e.value for e in Language]"
      ],
      "execution_count": 18
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d9bc0250-e5c6-4d90-972c-228a2209cf0b"
      },
      "source": [
        "Use the following code to see what default separators it uses, for example, for Python.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ad4262f6-678d-4889-80da-07aea7570e1d",
        "outputId": "9248489e-be15-4dd7-de59-f159da464701"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['\\nclass ', '\\ndef ', '\\n\\tdef ', '\\n\\n', '\\n', ' ', '']"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ],
      "source": [
        "RecursiveCharacterTextSplitter.get_separators_for_language(Language.PYTHON)"
      ],
      "execution_count": 19
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "af969559-8fb5-4e3e-a674-a8c4693463c3"
      },
      "source": [
        "#### Python\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "217cc677-74e2-4423-9709-a81daf2376cd"
      },
      "source": [
        "The following demonstrates how to split Python code using the `RecursiveCharacterTextSplitter` class.\n",
        "\n",
        "The main difference between splitting code and using the original `RecursiveCharacterTextSplitter` is that you need to call `.from_language` after the `RecursiveCharacterTextSplitter` and specify the `language`. The other parameter settings remain the same as before.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8026a3e1-e843-474b-9047-40ea0846d9a3",
        "outputId": "2866caf2-ce08-4ee1-d9d7-bf96860e49d9"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Document(page_content='def hello_world():'),\n",
              " Document(page_content='print(\"Hello, World!\")'),\n",
              " Document(page_content='# Call the function\\n    hello_world()')]"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ],
      "source": [
        "PYTHON_CODE = \"\"\"\n",
        "    def hello_world():\n",
        "        print(\"Hello, World!\")\n",
        "\n",
        "    # Call the function\n",
        "    hello_world()\n",
        "\"\"\"\n",
        "python_splitter = RecursiveCharacterTextSplitter.from_language(\n",
        "    language=Language.PYTHON, chunk_size=50, chunk_overlap=0\n",
        ")\n",
        "python_docs = python_splitter.create_documents([PYTHON_CODE])\n",
        "python_docs"
      ],
      "execution_count": 20
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "49d158af-7052-41e0-98fc-d1bdda494d9e"
      },
      "source": [
        "#### Javascript\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bd1ebbf4-6d16-4b5b-bf0a-b67700c86931"
      },
      "source": [
        "Let's see the separators for JSON language.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "b5174f59-d1aa-4751-98e4-70b9c72a4232",
        "outputId": "601444fb-9ac5-4986-e82c-f81352395cec"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['\\nfunction ',\n",
              " '\\nconst ',\n",
              " '\\nlet ',\n",
              " '\\nvar ',\n",
              " '\\nclass ',\n",
              " '\\nif ',\n",
              " '\\nfor ',\n",
              " '\\nwhile ',\n",
              " '\\nswitch ',\n",
              " '\\ncase ',\n",
              " '\\ndefault ',\n",
              " '\\n\\n',\n",
              " '\\n',\n",
              " ' ',\n",
              " '']"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ],
      "source": [
        "RecursiveCharacterTextSplitter.get_separators_for_language(Language.JS)"
      ],
      "execution_count": 21
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "903dd831-66f1-4679-bd86-ba4f4cf3c96f"
      },
      "source": [
        "The following code is used to separate the JSON language code.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5b9457a3-3732-4009-90e5-192c6b2002e8",
        "outputId": "10c4e8dd-69de-473e-c7fb-53a7fca042f7"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Document(page_content='function helloWorld() {'),\n",
              " Document(page_content='console.log(\"Hello, World!\");\\n    }'),\n",
              " Document(page_content='// Call the function\\n    helloWorld();')]"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ],
      "source": [
        "JS_CODE = \"\"\"\n",
        "    function helloWorld() {\n",
        "      console.log(\"Hello, World!\");\n",
        "    }\n",
        "\n",
        "    // Call the function\n",
        "    helloWorld();\n",
        "\"\"\"\n",
        "\n",
        "js_splitter = RecursiveCharacterTextSplitter.from_language(\n",
        "    language=Language.JS, chunk_size=60, chunk_overlap=0\n",
        ")\n",
        "js_docs = js_splitter.create_documents([JS_CODE])\n",
        "js_docs"
      ],
      "execution_count": 22
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1cff1a3b-f9e9-4edb-8aa5-d686e263b15e"
      },
      "source": [
        "For more information about applying to other languages, you can refer [here](https://python.langchain.com/v0.1/docs/modules/data_connection/document_transformers/code_splitter/).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "73aa7af6-a6e5-4e79-b8a3-287182eaf8a9"
      },
      "source": [
        "### Markdown Header Text Splitter\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "76215112-0a0e-4a97-a14f-8aefa3f87ba5"
      },
      "source": [
        "As mentioned, chunking often aims to keep text with a common context together. With this in mind, you might want to specifically honor the structure of the document itself. For example, a Markdown file is organized by headers. Creating chunks within specific header groups is an intuitive approach.\n",
        "\n",
        "To address this challenge, you can use `MarkdownHeaderTextSplitter`. This splitter will divide a Markdown file based on a specified set of headers.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "46cd797a-65c0-455a-a51a-91cbbb34c443"
      },
      "outputs": [],
      "source": [
        "from langchain.text_splitter import MarkdownHeaderTextSplitter"
      ],
      "execution_count": 23
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "921474ad-b896-403e-ad41-245d67a7bb15"
      },
      "source": [
        "For example, if you want to split this markdown:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3752a8e8-6b28-488a-94d4-d95e1985322e"
      },
      "outputs": [],
      "source": [
        "md = \"# Foo\\n\\n## Bar\\n\\nHi this is Jim\\n\\nHi this is Joe\\n\\n### Boo \\n\\nHi this is Lance \\n\\n## Baz\\n\\nHi this is Molly\""
      ],
      "execution_count": 24
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2c3f3ea9-558d-4c6b-b0fc-14df1b962359"
      },
      "source": [
        "You can specify the headers to split on:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4bd31a1f-7ce0-44af-9dc2-a9594b6c5e8d"
      },
      "outputs": [],
      "source": [
        "headers_to_split_on = [\n",
        "    (\"#\", \"Header 1\"),\n",
        "    (\"##\", \"Header 2\"),\n",
        "    (\"###\", \"Header 3\"),\n",
        "]"
      ],
      "execution_count": 25
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "586c260f-8455-4fb1-b7ba-83154aa757e2",
        "outputId": "70867c1b-0ec9-41db-c748-4525d14d3f6d"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}, page_content='Hi this is Jim  \\nHi this is Joe'),\n",
              " Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}, page_content='Hi this is Lance'),\n",
              " Document(metadata={'Header 1': 'Foo', 'Header 2': 'Baz'}, page_content='Hi this is Molly')]"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ],
      "source": [
        "markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)\n",
        "md_header_splits = markdown_splitter.split_text(md)\n",
        "md_header_splits"
      ],
      "execution_count": 26
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "17b36403-40eb-4d4c-8e9f-0d707e34115c"
      },
      "source": [
        "From the split results, you can see that the Markdown file is divided into several chunks formatted as document objects. The `page_content` contains the text under the headings, and the `metadata` contains the header information corresponding to the `page_content`.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6a7be672-fc86-41db-9a42-f92e7647e474"
      },
      "source": [
        "If you want the headers appears in the page_content as well, you can specify `strip_headers=False` when you call the `MarkdownHeaderTextSplitter`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "edcf07d6-60d3-4777-8946-0daed60b509c",
        "outputId": "2fa59110-7c37-4eba-b134-c7f6d4ff0fc3"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar'}, page_content='# Foo  \\n## Bar  \\nHi this is Jim  \\nHi this is Joe'),\n",
              " Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar', 'Header 3': 'Boo'}, page_content='### Boo  \\nHi this is Lance'),\n",
              " Document(metadata={'Header 1': 'Foo', 'Header 2': 'Baz'}, page_content='## Baz  \\nHi this is Molly')]"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ],
      "source": [
        "markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on, strip_headers=False)\n",
        "md_header_splits = markdown_splitter.split_text(md)\n",
        "md_header_splits"
      ],
      "execution_count": 27
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "42df7765-de3a-42fc-be95-808ba38f5472"
      },
      "source": [
        "### Split by HTML\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "05ee83df-1fdc-4f82-b3e7-bd91695f906f"
      },
      "source": [
        "#### Split by HTML header\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0c652ceb-ee01-4f5f-82e5-474584682ca3"
      },
      "source": [
        "Similar in concept to the `MarkdownHeaderTextSplitter`, the HTMLHeaderTextSplitter is a \"structure-aware\" chunker that splits text at the element level and adds metadata for each header \"relevant\" to any given chunk. It can return chunks element by element or combine elements with the same metadata, with the objectives of (a) keeping related text grouped (more or less) semantically and (b) preserving context-rich information encoded in document structures. It can be used with other text splitters as part of a chunking pipeline.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "28cb2816-0973-4ad5-a36f-012d26928ede"
      },
      "outputs": [],
      "source": [
        "from langchain_text_splitters import HTMLHeaderTextSplitter"
      ],
      "execution_count": 28
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5a2b8bd1-766e-4390-8479-301bbe07de18"
      },
      "source": [
        "Assume you have the following HTML code that you want to split.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "09aebef4-f660-42fa-8268-d2f49445bb88"
      },
      "outputs": [],
      "source": [
        "html_string = \"\"\"\n",
        "    <!DOCTYPE html>\n",
        "    <html>\n",
        "    <body>\n",
        "        <div>\n",
        "            <h1>Foo</h1>\n",
        "            <p>Some intro text about Foo.</p>\n",
        "            <div>\n",
        "                <h2>Bar main section</h2>\n",
        "                <p>Some intro text about Bar.</p>\n",
        "                <h3>Bar subsection 1</h3>\n",
        "                <p>Some text about the first subtopic of Bar.</p>\n",
        "                <h3>Bar subsection 2</h3>\n",
        "                <p>Some text about the second subtopic of Bar.</p>\n",
        "            </div>\n",
        "            <div>\n",
        "                <h2>Baz</h2>\n",
        "                <p>Some text about Baz</p>\n",
        "            </div>\n",
        "            <br>\n",
        "            <p>Some concluding text about Foo</p>\n",
        "        </div>\n",
        "    </body>\n",
        "    </html>\n",
        "\"\"\""
      ],
      "execution_count": 29
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d6e11310-6071-43ae-901f-d1cf63d571e6"
      },
      "source": [
        "Set up the header to split.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fbd9b083-4fe0-409d-8eee-5cb3de13c90d"
      },
      "outputs": [],
      "source": [
        "headers_to_split_on = [\n",
        "    (\"h1\", \"Header 1\"),\n",
        "    (\"h2\", \"Header 2\"),\n",
        "    (\"h3\", \"Header 3\"),\n",
        "]"
      ],
      "execution_count": 30
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "04cdc309-9fac-49d7-847f-8b220c0b83a5"
      },
      "source": [
        "Split the HTML string using `HTMLHeaderTextSplitter`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9e4cb5f1-66b0-4acd-b1c5-fc144af42862",
        "outputId": "2cf5a3cd-3d1f-4bd9-c26f-9f1900a87fdf"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Document(page_content='Foo'),\n",
              " Document(metadata={'Header 1': 'Foo'}, page_content='Some intro text about Foo.  \\nBar main section Bar subsection 1 Bar subsection 2'),\n",
              " Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar main section'}, page_content='Some intro text about Bar.'),\n",
              " Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar main section', 'Header 3': 'Bar subsection 1'}, page_content='Some text about the first subtopic of Bar.'),\n",
              " Document(metadata={'Header 1': 'Foo', 'Header 2': 'Bar main section', 'Header 3': 'Bar subsection 2'}, page_content='Some text about the second subtopic of Bar.'),\n",
              " Document(metadata={'Header 1': 'Foo'}, page_content='Baz'),\n",
              " Document(metadata={'Header 1': 'Foo', 'Header 2': 'Baz'}, page_content='Some text about Baz'),\n",
              " Document(metadata={'Header 1': 'Foo'}, page_content='Some concluding text about Foo')]"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ],
      "source": [
        "html_splitter = HTMLHeaderTextSplitter(headers_to_split_on=headers_to_split_on)\n",
        "html_header_splits = html_splitter.split_text(html_string)\n",
        "html_header_splits"
      ],
      "execution_count": 31
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "63c6479b-948e-4446-acc5-97ead29a9ba4"
      },
      "source": [
        "From the split results, you can see that the context under the headings is extracted and put in the `page_content` parameter. The `metatdata` contains the header information.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5353036c-4868-4948-b631-03b5dc1770d3"
      },
      "source": [
        "#### Split by HTML section\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cc09d8f4-e8bf-4be2-8f48-f244b2aef765"
      },
      "source": [
        "Similar to the `HTMLHeaderTextSplitter`, the `HTMLSectionSplitter` is also a \"structure-aware\" chunker that splits text section by section based on headings.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "21675da3-77ca-4115-a706-786f80d55cd0"
      },
      "source": [
        "The following code is used to implement it.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "138b6dc6-1632-4ee9-930f-fb5cc02a83b4",
        "outputId": "b65c9a9c-8709-4432-8346-0443b53c18d4"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Document(metadata={'Header 1': 'Foo'}, page_content='Foo \\n Some intro text about Foo.'),\n",
              " Document(metadata={'Header 2': 'Bar main section'}, page_content='Bar main section \\n Some intro text about Bar.'),\n",
              " Document(metadata={'Header 3': 'Bar subsection 1'}, page_content='Bar subsection 1 \\n Some text about the first subtopic of Bar.'),\n",
              " Document(metadata={'Header 3': 'Bar subsection 2'}, page_content='Bar subsection 2 \\n Some text about the second subtopic of Bar.'),\n",
              " Document(metadata={'Header 2': 'Baz'}, page_content='Baz \\n Some text about Baz \\n \\n \\n Some concluding text about Foo')]"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ],
      "source": [
        "from langchain_text_splitters import HTMLSectionSplitter\n",
        "\n",
        "html_string = \"\"\"\n",
        "    <!DOCTYPE html>\n",
        "    <html>\n",
        "    <body>\n",
        "        <div>\n",
        "            <h1>Foo</h1>\n",
        "            <p>Some intro text about Foo.</p>\n",
        "            <div>\n",
        "                <h2>Bar main section</h2>\n",
        "                <p>Some intro text about Bar.</p>\n",
        "                <h3>Bar subsection 1</h3>\n",
        "                <p>Some text about the first subtopic of Bar.</p>\n",
        "                <h3>Bar subsection 2</h3>\n",
        "                <p>Some text about the second subtopic of Bar.</p>\n",
        "            </div>\n",
        "            <div>\n",
        "                <h2>Baz</h2>\n",
        "                <p>Some text about Baz</p>\n",
        "            </div>\n",
        "            <br>\n",
        "            <p>Some concluding text about Foo</p>\n",
        "        </div>\n",
        "    </body>\n",
        "    </html>\n",
        "\"\"\"\n",
        "\n",
        "headers_to_split_on = [(\"h1\", \"Header 1\"), (\"h2\", \"Header 2\"), (\"h3\", \"Header 3\")]\n",
        "\n",
        "html_splitter = HTMLSectionSplitter(headers_to_split_on=headers_to_split_on)\n",
        "html_header_splits = html_splitter.split_text(html_string)\n",
        "html_header_splits"
      ],
      "execution_count": 32
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d1121236-4873-426e-9633-05dd6eb84827"
      },
      "source": [
        "# Exercises\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cf4221f3-7eea-484c-9bc1-549820336602"
      },
      "source": [
        "### Exercise 1 - Changing separator for CharacterTextSplitter\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9ce405c0-eb85-4147-8d5f-5279325b27b0"
      },
      "source": [
        "Try to change to use another separator, for example `\"\\n\"` to see how it affect the split and chunks.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c33c168b-66fd-4e7a-a5af-92e093f018c1",
        "outputId": "da953be7-5a89-476e-a409-b43d1537a26c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:langchain_text_splitters.base:Created a chunk of size 223, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 274, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 274, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 262, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 282, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 262, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 270, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 224, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 215, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 325, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 300, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 216, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 226, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 242, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 235, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 300, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 294, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 234, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 321, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 256, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 241, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 248, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 249, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 246, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 211, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 267, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 206, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 694, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 323, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 326, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 290, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 296, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 233, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 421, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 243, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 260, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 272, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 229, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 290, which is longer than the specified 200\n",
            "WARNING:langchain_text_splitters.base:Created a chunk of size 208, which is longer than the specified 200\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['1.\\tCode of Conduct',\n",
              " 'Our Code of Conduct outlines the fundamental principles and ethical standards that guide every member of our organization. We are committed to maintaining a workplace that is built on integrity, respect, and accountability.',\n",
              " 'Integrity: We hold ourselves to the highest ethical standards. This means acting honestly and transparently in all our interactions, whether with colleagues, clients, or the broader community. We respect and protect sensitive information, and we avoid conflicts of interest.',\n",
              " \"Respect: We embrace diversity and value each individual's contributions. Discrimination, harassment, or any form of disrespectful behavior is unacceptable. We create an inclusive environment where differences are celebrated and everyone is treated with dignity and courtesy.\",\n",
              " 'Accountability: We take responsibility for our actions and decisions. We follow all relevant laws and regulations, and we strive to continuously improve our practices. We report any potential violations of this code and support the investigation of such matters.',\n",
              " 'Safety: We prioritize the safety of our employees, clients, and the communities we serve. We maintain a culture of safety, including reporting any unsafe conditions or practices.',\n",
              " 'Environmental Responsibility: We are committed to minimizing our environmental footprint and promoting sustainable practices.',\n",
              " \"Our Code of Conduct is not just a set of rules; it is the foundation of our organization's culture. We expect all employees to uphold these principles and serve as role models for others, ensuring we maintain our reputation for ethical conduct, integrity, and social responsibility.\",\n",
              " '2.\\tRecruitment Policy',\n",
              " 'Our Recruitment Policy reflects our commitment to attracting, selecting, and onboarding the most qualified and diverse candidates to join our organization. We believe that the success of our company relies on the talents, skills, and dedication of our employees.',\n",
              " 'Equal Opportunity: We are an equal opportunity employer and do not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, or any other protected status. We actively promote diversity and inclusion.',\n",
              " 'Transparency: We maintain transparency in our recruitment processes. All job vacancies are advertised internally and externally when appropriate. Job descriptions and requirements are clear and accurately represent the role.',\n",
              " 'Selection Criteria: Our selection process is based on the qualifications, experience, and skills necessary for the position. Interviews and assessments are conducted objectively, and decisions are made without bias.',\n",
              " \"Data Privacy: We are committed to protecting the privacy of candidates' personal information and adhere to all relevant data protection laws and regulations.\",\n",
              " 'Feedback: Candidates will receive timely and constructive feedback on their application and interview performance.',\n",
              " 'Onboarding: New employees receive comprehensive onboarding to help them integrate into the organization effectively. This includes information on our culture, policies, and expectations.',\n",
              " 'Employee Referrals: We encourage and appreciate employee referrals as they contribute to building a strong and engaged team.',\n",
              " 'Our Recruitment Policy is a foundation for creating a diverse, inclusive, and talented workforce. It ensures that we attract and hire the best candidates who align with our company values and contribute to our continued success. We continuously review and update this policy to reflect evolving best practices in recruitment.',\n",
              " '3.\\tInternet and Email Policy',\n",
              " 'Our Internet and Email Policy is established to guide the responsible and secure use of these essential tools within our organization. We recognize their significance in daily business operations and the importance of adhering to principles that maintain security, productivity, and legal compliance.',\n",
              " \"Acceptable Use: Company-provided internet and email services are primarily meant for job-related tasks. Limited personal use is allowed during non-work hours, provided it doesn't interfere with work responsibilities.\",\n",
              " 'Security: Safeguard your login credentials, avoiding the sharing of passwords. Exercise caution with email attachments and links from unknown sources. Promptly report any unusual online activity or potential security breaches.',\n",
              " 'Confidentiality: Reserve email for the transmission of confidential information, trade secrets, and sensitive customer data only when encryption is applied. Exercise discretion when discussing company matters on public forums or social media.',\n",
              " 'Harassment and Inappropriate Content: Internet and email usage must not involve harassment, discrimination, or the distribution of offensive or inappropriate content. Show respect and sensitivity to others in all online communications.',\n",
              " 'Compliance: Ensure compliance with all relevant laws and regulations regarding internet and email usage, including those related to copyright and data protection.',\n",
              " 'Monitoring: The company retains the right to monitor internet and email usage for security and compliance purposes.',\n",
              " 'Consequences: Policy violations may lead to disciplinary measures, including potential termination.',\n",
              " 'Our Internet and Email Policy aims to promote safe, responsible usage of digital communication tools that align with our values and legal obligations. Each employee is expected to understand and follow this policy. Regular reviews ensure its alignment with evolving technology and security standards.',\n",
              " '4.\\tMobile Phone Policy',\n",
              " 'The Mobile Phone Policy sets forth the standards and expectations governing the appropriate and responsible usage of mobile devices in the organization. The purpose of this policy is to ensure that employees utilize mobile phones in a manner consistent with company values and legal compliance.',\n",
              " 'Acceptable Use: Mobile devices are primarily intended for work-related tasks. Limited personal usage is allowed, provided it does not disrupt work obligations.',\n",
              " 'Security: Safeguard your mobile device and access credentials. Exercise caution when downloading apps or clicking links from unfamiliar sources. Promptly report security concerns or suspicious activities related to your mobile device.',\n",
              " 'Confidentiality: Avoid transmitting sensitive company information via unsecured messaging apps or emails. Be discreet when discussing company matters in public spaces.',\n",
              " 'Cost Management: Keep personal phone usage separate from company accounts and reimburse the company for any personal charges on company-issued phones.',\n",
              " 'Compliance: Adhere to all pertinent laws and regulations concerning mobile phone usage, including those related to data protection and privacy.',\n",
              " 'Lost or Stolen Devices: Immediately report any lost or stolen mobile devices to the IT department or your supervisor.',\n",
              " 'Consequences: Non-compliance with this policy may lead to disciplinary actions, including the potential loss of mobile phone privileges.',\n",
              " 'The Mobile Phone Policy is aimed at promoting the responsible and secure use of mobile devices in line with legal and ethical standards. Every employee is expected to comprehend and abide by these guidelines. Regular reviews of the policy ensure its ongoing alignment with evolving technology and security best practices.',\n",
              " '5.\\tSmoking Policy',\n",
              " 'Policy Purpose: The Smoking Policy has been established to provide clear guidance and expectations concerning smoking on company premises. This policy is in place to ensure a safe and healthy environment for all employees, visitors, and the general public.',\n",
              " 'Designated Smoking Areas: Smoking is only permitted in designated smoking areas, as marked by appropriate signage. These areas have been chosen to minimize exposure to secondhand smoke and to maintain the overall cleanliness of the premises.',\n",
              " 'Smoking Restrictions: Smoking inside company buildings, offices, meeting rooms, and other enclosed spaces is strictly prohibited. This includes electronic cigarettes and vaping devices.',\n",
              " 'Compliance with Applicable Laws: All employees and visitors must adhere to relevant federal, state, and local smoking laws and regulations.',\n",
              " 'Disposal of Smoking Materials: Properly dispose of cigarette butts and related materials in designated receptacles. Littering on company premises is prohibited.',\n",
              " 'No Smoking in Company Vehicles: Smoking is not permitted in company vehicles, whether they are owned or leased, to maintain the condition and cleanliness of these vehicles.',\n",
              " 'Enforcement and Consequences: All employees and visitors are expected to adhere to this policy. Non-compliance may lead to appropriate disciplinary action, which could include fines, or, in the case of employees, possible termination of employment.',\n",
              " 'Review of Policy: This policy will be reviewed periodically to ensure its alignment with evolving legal requirements and best practices for maintaining a healthy and safe workplace.',\n",
              " 'We appreciate your cooperation in maintaining a smoke-free and safe environment for all.\\n6.\\tDrug and Alcohol Policy',\n",
              " 'Policy Objective: The Drug and Alcohol Policy is established to establish clear expectations and guidelines for the responsible use of drugs and alcohol within the organization. This policy aims to maintain a safe, healthy, and productive workplace.',\n",
              " 'Prohibited Substances: The use, possession, distribution, or sale of illegal drugs or unauthorized controlled substances is strictly prohibited on company premises or during work-related activities. This includes the misuse of prescription drugs.',\n",
              " 'Alcohol Consumption: The consumption of alcoholic beverages is not allowed during work hours, on company property, or while performing company-related duties. Exception may be made for company-sanctioned events.',\n",
              " 'Impairment: Employees are expected to perform their job duties without impairment from drugs or alcohol. The use of substances that could impair job performance or pose a safety risk is prohibited.',\n",
              " 'Testing and Searches: The organization reserves the right to conduct drug and alcohol testing as per applicable laws and regulations. Employees may be subject to testing in cases of reasonable suspicion, post-accident, or as part of routine workplace safety measures.',\n",
              " 'Reporting: Employees should report any concerns related to drug or alcohol misuse by themselves or their colleagues, as well as safety concerns arising from such misuse.',\n",
              " 'Treatment and Assistance: Employees with substance abuse issues are encouraged to seek help. The organization is committed to providing support, resources, and information to assist those seeking treatment.',\n",
              " 'Consequences: Violation of this policy may result in disciplinary actions, up to and including termination of employment. Legal action may also be pursued when necessary.',\n",
              " 'Policy Review: This policy will undergo periodic review to ensure its continued relevance and compliance with evolving legal requirements and best practices for a safe and productive work environment.',\n",
              " 'Your adherence to this policy is appreciated as it helps to maintain a safe and drug-free workplace for all.\\n7.\\tHealth and Safety Policy',\n",
              " 'Our commitment to health and safety is paramount. We prioritize the well-being of our employees, customers, and the public. We diligently comply with all relevant health and safety laws and regulations. Our objective is to maintain a workplace free from hazards, preventing accidents, injuries, and illnesses. Every individual within our organization is responsible for upholding these standards. We regularly assess and improve our safety measures, provide adequate training, and encourage open communication regarding safety concerns. Through collective dedication, we aim to ensure a safe, healthy, and secure environment for all. Your cooperation is essential in achieving this common goal.',\n",
              " '8.\\tAnti-discrimination and Harassment Policy',\n",
              " 'The Anti-Discrimination and Harassment Policy is a testament to the commitment of this organization in fostering a workplace that is free from discrimination, harassment, and any form of unlawful bias. This policy applies to every individual within the organization, including employees, contractors, visitors, and clients.',\n",
              " 'Non-Discrimination: This organization strictly prohibits discrimination based on race, color, religion, gender, national origin, age, disability, sexual orientation, or any other legally protected characteristic in all aspects of employment, including recruitment, hiring, compensation, benefits, promotions, and terminations.',\n",
              " 'Harassment: Harassment in any form, whether based on the aforementioned characteristics or any other protected status, is unacceptable. This encompasses unwelcome advances, offensive jokes, slurs, and other verbal or physical conduct that creates a hostile or intimidating work environment.',\n",
              " 'Reporting: Individuals who experience or witness any form of discrimination or harassment are encouraged to promptly report the incident to their supervisor, manager, or the designated HR representative. The organization is committed to a timely and confidential investigation of such complaints.',\n",
              " 'Consequences: Violation of this policy may result in disciplinary action, including termination of employment. The organization is committed to taking appropriate action against any individual found to be in violation of this policy.',\n",
              " 'Review and Update: This policy is subject to regular review and update to remain aligned with evolving legal requirements and best practices in preventing discrimination and harassment. This organization considers it a collective responsibility to ensure a workplace free from discrimination and harassment, and it is essential that every individual within the organization plays their part in upholding these principles.',\n",
              " '9.\\tDiscipline and Termination Policy',\n",
              " \"The Discipline and Termination Policy underscores the organization's commitment to maintaining a productive, ethical, and respectful work environment. This policy applies to all personnel, including employees, contractors, and temporary staff.\",\n",
              " 'Performance and Conduct Expectations: Employees are expected to meet performance standards and adhere to conduct guidelines. The organization will provide clear expectations, feedback, and opportunities for improvement when performance or conduct issues arise.',\n",
              " 'Disciplinary Actions: When necessary, disciplinary actions will be taken, which may include verbal warnings, written warnings, suspension, or other appropriate measures. Disciplinary actions are designed to address issues constructively and maintain performance standards.',\n",
              " \"Termination: In situations where an employee's performance or conduct issues persist, the organization may resort to termination. Termination may also occur for reasons such as redundancy, violation of policies, or restructuring.\",\n",
              " 'Termination Procedure: The organization will follow appropriate procedures, ensuring fairness and adherence to legal requirements during the termination process. Employees may be eligible for notice periods, severance pay, or other benefits as per employment agreements and applicable laws.',\n",
              " 'Exit Process: The organization will conduct an exit process to ensure a smooth transition for departing employees, including the return of company property, final pay, and cancellation of access and benefits.',\n",
              " 'This policy serves as a framework for handling discipline and termination. The organization recognizes the importance of fairness and consistency in these processes, and decisions will be made after careful consideration. Every employee is expected to understand and adhere to this policy, contributing to a respectful and productive workplace. Regular reviews will ensure its alignment with evolving legal requirements and best practices.']"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ],
      "source": [
        "# Your code here\n",
        "\n",
        "text_splitter = CharacterTextSplitter(\n",
        "    separator=\"\\n\",\n",
        "    chunk_size=200,\n",
        "    chunk_overlap=20,\n",
        "    length_function=len,\n",
        ")\n",
        "texts = text_splitter.split_text(companypolicies)\n",
        "texts"
      ],
      "execution_count": 33
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "419169bd-be3f-4e4d-a804-42a842426a3c"
      },
      "source": [
        "<details>\n",
        "    <summary>Click here for Solution</summary>\n",
        "\n",
        "```python\n",
        "text_splitter = CharacterTextSplitter(\n",
        "    separator=\"\\n\",\n",
        "    chunk_size=200,\n",
        "    chunk_overlap=20,\n",
        "    length_function=len,\n",
        ")\n",
        "texts = text_splitter.split_text(companypolicies)\n",
        "texts\n",
        "```\n",
        "\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "53e607cc-779e-4d00-9e00-a43b842ec3e8"
      },
      "source": [
        "### Exercise 2 - Splitting Latex code\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ebfe3ee-e47c-4575-8425-7820d1728390"
      },
      "source": [
        "Here is an example of Latex code. Try to split it.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a6c64094-7c08-4af2-a399-aa00f0a3c5e3"
      },
      "outputs": [],
      "source": [
        "latex_text = \"\"\"\n",
        "    \\documentclass{article}\n",
        "\n",
        "    \\begin{document}\n",
        "\n",
        "    \\maketitle\n",
        "\n",
        "    \\section{Introduction}\n",
        "    Large language models (LLMs) are a type of machine learning model that can be trained on vast amounts of text data to generate human-like language. In recent years, LLMs have made significant advances in a variety of natural language processing tasks, including language translation, text generation, and sentiment analysis.\n",
        "\n",
        "    \\subsection{History of LLMs}\n",
        "    The earliest LLMs were developed in the 1980s and 1990s, but they were limited by the amount of data that could be processed and the computational power available at the time. In the past decade, however, advances in hardware and software have made it possible to train LLMs on massive datasets, leading to significant improvements in performance.\n",
        "\n",
        "    \\subsection{Applications of LLMs}\n",
        "    LLMs have many applications in industry, including chatbots, content creation, and virtual assistants. They can also be used in academia for research in linguistics, psychology, and computational linguistics.\n",
        "\n",
        "    \\end{document}\n",
        "\"\"\""
      ],
      "execution_count": 34
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "da1429cb-215b-4b3e-a385-90dfc42e83ae",
        "outputId": "674f6fb5-905e-43fb-8633-0185b87567c5"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Document(page_content='\\\\documentclass{article}\\n    \\n    \\x08egin{document}'),\n",
              " Document(page_content='\\\\maketitle\\n    \\n    \\\\section{Introduction}\\n    Large'),\n",
              " Document(page_content='language models (LLMs) are a type of machine learning model'),\n",
              " Document(page_content='that can be trained on vast amounts of text data to'),\n",
              " Document(page_content='generate human-like language. In recent years, LLMs have'),\n",
              " Document(page_content='made significant advances in a variety of natural language'),\n",
              " Document(page_content='processing tasks, including language translation, text'),\n",
              " Document(page_content='generation, and sentiment analysis.'),\n",
              " Document(page_content='\\\\subsection{History of LLMs}\\n    The earliest LLMs were'),\n",
              " Document(page_content='developed in the 1980s and 1990s, but they were limited by'),\n",
              " Document(page_content='the amount of data that could be processed and the'),\n",
              " Document(page_content='computational power available at the time. In the past'),\n",
              " Document(page_content='decade, however, advances in hardware and software have'),\n",
              " Document(page_content='made it possible to train LLMs on massive datasets, leading'),\n",
              " Document(page_content='to significant improvements in performance.'),\n",
              " Document(page_content='\\\\subsection{Applications of LLMs}\\n    LLMs have many'),\n",
              " Document(page_content='applications in industry, including chatbots, content'),\n",
              " Document(page_content='creation, and virtual assistants. They can also be used in'),\n",
              " Document(page_content='academia for research in linguistics, psychology, and'),\n",
              " Document(page_content='computational linguistics.\\n    \\n    \\\\end{document}')]"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ],
      "source": [
        "# Your code here\n",
        "\n",
        "latex_splitter = RecursiveCharacterTextSplitter.from_language(\n",
        "    language=Language.LATEX, chunk_size=60, chunk_overlap=0\n",
        ")\n",
        "latex_docs = latex_splitter.create_documents([latex_text])\n",
        "latex_docs"
      ],
      "execution_count": 35
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d4b4c694-cf8d-44db-9e26-58bcfb83afa1"
      },
      "source": [
        "<details>\n",
        "    <summary>Click here for Solution</summary>\n",
        "\n",
        "```python\n",
        "latex_splitter = RecursiveCharacterTextSplitter.from_language(\n",
        "    language=Language.LATEX, chunk_size=60, chunk_overlap=0\n",
        ")\n",
        "latex_docs = latex_splitter.create_documents([latex_text])\n",
        "latex_docs\n",
        "```\n",
        "\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7130ffcf-0365-4f99-9783-49fa3e4f0360"
      },
      "source": [
        "## Authors\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3491ba22-4c9c-4a13-84fb-bcb0e3332480"
      },
      "source": [
        "[Kang Wang](https://author.skills.network/instructors/kang_wang)\n",
        "\n",
        "Kang Wang is a Data Scientist in IBM. He is also a PhD Candidate in the University of Waterloo.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "086f68d1-68f6-4179-aa3e-83acced88146"
      },
      "source": [
        "### Other Contributors\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4ad205f8-8796-40ce-8b7a-6984abd73d08"
      },
      "source": [
        "[Joseph Santarcangelo](https://author.skills.network/instructors/joseph_santarcangelo)\n",
        "\n",
        "Joseph has a Ph.D. in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f09faef9-23bd-435a-885e-6b89f7a568cf"
      },
      "source": [
        "```{## Change Log}\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "28cba324-e768-452a-871a-159c2662ba39"
      },
      "source": [
        "```{Date (YYYY-MM-DD)|Version|Changed By|Change Description||-|-|-|-||2024-07-16|0.1|Kang Wang|Create the lab|}\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fb179f46-8ee2-4a29-91bb-285e64a77bc9"
      },
      "source": [
        "© Copyright IBM Corporation. All rights reserved.\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.9"
    },
    "prev_pub_hash": "b4650da252dcbdb793dcbe2597a46559505ee5c9b43bcef40bf9fe09bef8765b",
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}