Topic_Modeling.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyOYmZbkC1S8RStykh+sA/SO",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/epythonlab/PythonLab/blob/master/Topic_Modeling.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Topic Modeling and It's Practical Uses"
      ],
      "metadata": {
        "id": "K_UJfhNnyAlK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Introduction:"
      ],
      "metadata": {
        "id": "Bob6ukKj5OEO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Hello everyone, and welcome to today's tutorial on topic modeling and its practical uses! I'm excited to have you join me as we explore this fascinating topic together.\n",
        "\n",
        "Have you ever wondered how computers can understand and extract meaning from large volumes of text data? That's where topic modeling comes in! Today, I'll dive into the world of natural language processing and uncover the hidden thematic structures within documents."
      ],
      "metadata": {
        "id": "PI3nLxaH5VaG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## What is Topic Modeling?"
      ],
      "metadata": {
        "id": "v-HU7lKS5kOF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's start by defining topic modeling. Imagine you have a collection of documents—articles, emails, or social media posts. Topic modeling allows you to uncover the underlying themes or topics present in these documents."
      ],
      "metadata": {
        "id": "pG50whtO5nRF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Understanding Latent Dirichlet Allocation (LDA)"
      ],
      "metadata": {
        "id": "q8GxpVYV6TNR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "One of the most popular algorithms for topic modeling is Latent Dirichlet Allocation, or LDA for short. LDA assumes that each document is a mixture of topics, and each topic is a distribution over words. It's like solving a puzzle to find the hidden themes!"
      ],
      "metadata": {
        "id": "m-LbUcaM6XVp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Implementing Topic Modeling with Python"
      ],
      "metadata": {
        "id": "1VM4AzfK6ik4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now, let's roll up our sleeves and dive into some Python coding! I'll use libraries like `gensim` and `scikit-learn` to perform topic modeling step by step. Whether you're using sample data or your own dataset, the process remains the same."
      ],
      "metadata": {
        "id": "G_db3daa82rd"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Generating Sample Data(Optional)"
      ],
      "metadata": {
        "id": "foXE-23i-n72"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "But before you dive into coding, let's have some fun generating sample data! Don't worry—if you already have your own dataset, you can skip this step and use your data instead. Let's explore the power of topic modeling with your very own simulated documents!"
      ],
      "metadata": {
        "id": "n9tAkKJo-w3d"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "\n",
        "def generate_sample_documents(doc_dir):\n",
        "    # Create the directory if it doesn't exist\n",
        "    if not os.path.exists(doc_dir):\n",
        "        os.makedirs(doc_dir)\n",
        "\n",
        "    # Sample topics and corresponding text\n",
        "    topics = [\"Technology\", \"Health\", \"Finance\"]\n",
        "    topic_texts = [\n",
        "        \"In today's rapidly advancing world of technology, staying updated with the latest innovations is crucial for success. From artificial intelligence to blockchain technology, there are endless possibilities for businesses to leverage technology to gain a competitive edge.\",\n",
        "        \"Maintaining good health is essential for a happy and fulfilling life. Regular exercise, balanced nutrition, and proper sleep are key factors in preventing illness and promoting overall well-being. Remember to prioritize your health and make healthy choices every day.\",\n",
        "        \"Financial planning is an important aspect of securing your future. Whether it's saving for retirement, investing in the stock market, or managing debt, having a solid financial strategy can help you achieve your long-term goals. Make informed decisions and seek professional advice when necessary.\"\n",
        "    ]\n",
        "\n",
        "    # Generate and save sample documents\n",
        "    for i, topic in enumerate(topics):\n",
        "        filename = os.path.join(doc_dir, f\"Document_{i}.txt\")\n",
        "        with open(filename, \"w\", encoding=\"utf-8\") as file:\n",
        "            file.write(topic_texts[i])\n",
        "\n",
        "# Call the function to generate sample documents\n",
        "generate_sample_documents(\"sample_documents\")\n"
      ],
      "metadata": {
        "id": "1uFvrpMOCGJP"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Preprocessing Text Data"
      ],
      "metadata": {
        "id": "73G6T_Tr-28G"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Before diving into topic modeling, you need to prepare your text data. This includes tasks like **tokenization**, **removing stopwords**, and **stemming** or **lemmatization**. But fear not! I'll walk you through each step."
      ],
      "metadata": {
        "id": "1FYGIcaA-rSD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import nltk # import nltk for preprocessing text data\n",
        "from nltk.tokenize import word_tokenize  # Importing word_tokenize for tokenization\n",
        "from nltk.corpus import stopwords  # Importing stopwords from NLTK\n",
        "from nltk.stem import WordNetLemmatizer  # Importing WordNetLemmatizer for lemmatization\n",
        "\n",
        "# Download NLTK resources (if not already downloaded)\n",
        "# Download NLTK resources (if not already downloaded)\n",
        "nltk.download('punkt')  # Downloading the 'punkt' tokenizer\n",
        "nltk.download('stopwords')  # Downloading the stopwords corpus\n",
        "nltk.download('wordnet')  # Downloading the WordNet corpus for lemmatization\n",
        "\n",
        "def preprocess_text(text):\n",
        "    \"\"\"\n",
        "    Preprocesses a text by tokenizing, removing stopwords, and lemmatizing the tokens.\n",
        "\n",
        "    Args:\n",
        "        text (str): Input text to be preprocessed.\n",
        "\n",
        "    Returns:\n",
        "        str: Preprocessed text.\n",
        "    \"\"\"\n",
        "    # Tokenize text\n",
        "    tokens = word_tokenize(text)  # Tokenizing the input text\n",
        "\n",
        "    # Remove stopwords\n",
        "    stop_words = set(stopwords.words('english'))  # Getting English stopwords\n",
        "    filtered_tokens = [word for word in tokens if word.lower() not in stop_words]  # Removing stopwords\n",
        "\n",
        "    # Lemmatize tokens\n",
        "    lemmatizer = WordNetLemmatizer()  # Initializing WordNetLemmatizer\n",
        "    lemmatized_tokens = [lemmatizer.lemmatize(token) for token in filtered_tokens]  # Lemmatizing tokens\n",
        "\n",
        "    # Join tokens back into a single string\n",
        "    preprocessed_text = ' '.join(lemmatized_tokens)  # Joining tokens into a single string\n",
        "\n",
        "    return preprocessed_text\n",
        "\n",
        "def preprocess_documents(doc_dir):\n",
        "    \"\"\"\n",
        "    Preprocesses all documents in a directory.\n",
        "\n",
        "    Args:\n",
        "        doc_dir (str): Path to the directory containing documents.\n",
        "\n",
        "    Returns:\n",
        "        list: List of preprocessed documents.\n",
        "    \"\"\"\n",
        "    preprocessed_documents = []\n",
        "    for filename in os.listdir(doc_dir):\n",
        "        if filename.endswith('.txt'):\n",
        "            file_path = os.path.join(doc_dir, filename)\n",
        "            with open(file_path, 'r', encoding='utf-8') as file:\n",
        "                text = file.read()\n",
        "                preprocessed_text = preprocess_text(text)\n",
        "                preprocessed_documents.append(preprocessed_text)\n",
        "    return preprocessed_documents\n",
        "\n",
        "# Call the function to preprocess sample documents\n",
        "preprocessed_docs = preprocess_documents(\"sample_documents\")\n",
        "\n",
        "# Print preprocessed documents\n",
        "for idx, doc in enumerate(preprocessed_docs):\n",
        "    print(f\"Preprocessed Document {idx+1}:\\n{doc}\\n\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GOPpkpWCCrDS",
        "outputId": "ae5e0d7b-f018-4ecb-baa1-3c4657b0d620"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Preprocessed Document 1:\n",
            "Financial planning important aspect securing future . Whether 's saving retirement , investing stock market , managing debt , solid financial strategy help achieve long-term goal . Make informed decision seek professional advice necessary .\n",
            "\n",
            "Preprocessed Document 2:\n",
            "today 's rapidly advancing world technology , staying updated latest innovation crucial success . artificial intelligence blockchain technology , endless possibility business leverage technology gain competitive edge .\n",
            "\n",
            "Preprocessed Document 3:\n",
            "Maintaining good health essential happy fulfilling life . Regular exercise , balanced nutrition , proper sleep key factor preventing illness promoting overall well-being . Remember prioritize health make healthy choice every day .\n",
            "\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n",
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Package stopwords is already up-to-date!\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training the LDA Model"
      ],
      "metadata": {
        "id": "nKuUrVS1AHBO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Next, I'll train the **LDA** model using my preprocessed text data. This involves setting parameters such as the **number of topics** and **the number of iterations**. Once trained, the model will reveal the underlying topics present in your documents."
      ],
      "metadata": {
        "id": "rpewImsgAJPr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.decomposition import LatentDirichletAllocation\n",
        "import pandas as pd\n",
        "def perform_topic_modeling(num_topics, documents_path, output_csv):\n",
        "    \"\"\"\n",
        "    Performs topic modeling using LDA and generates a document-topic matrix CSV.\n",
        "\n",
        "    Args:\n",
        "        num_topics: Number of topics to model.\n",
        "        documents_path: Path to a directory containing documents.\n",
        "        output_csv: Path to the output CSV file.\n",
        "    \"\"\"\n",
        "    # Preprocess documents\n",
        "    documents = preprocess_documents(documents_path)\n",
        "\n",
        "    # Vectorize documents using TF-IDF\n",
        "    vectorizer = TfidfVectorizer(max_df=0.95, min_df=2, stop_words='english')\n",
        "    tfidf_matrix = vectorizer.fit_transform(documents)\n",
        "\n",
        "    # Train LDA model\n",
        "    lda_model = LatentDirichletAllocation(n_components=num_topics, random_state=1)\n",
        "    lda_model.fit(tfidf_matrix)\n",
        "\n",
        "    # Get document topic matrix\n",
        "    document_topic_matrix = lda_model.transform(tfidf_matrix)\n",
        "\n",
        "    # Create pandas dataframe\n",
        "    df = pd.DataFrame(document_topic_matrix, columns=[f\"Topic_{i}\" for i in range(num_topics)])\n",
        "    df.index = pd.Index(range(len(documents)))\n",
        "\n",
        "    # Save to CSV\n",
        "    df.to_csv(output_csv, index=True)\n",
        "\n",
        "    print(f\"Document-topic matrix saved to: {output_csv}\")\n",
        "\n",
        "# Example usage\n",
        "num_topics = 5  # Change this to the desired number of topics\n",
        "documents_path = \"sample_documents\"  # Replace with your documents directory\n",
        "output_csv = \"document_topics.csv\"\n",
        "\n",
        "perform_topic_modeling(num_topics, documents_path, output_csv)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5sIjIZCcHOQU",
        "outputId": "e1638927-02ea-4340-cbf5-f2de42dc8c67"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Document-topic matrix saved to: document_topics.csv\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "The expected output of the provided code is a CSV file containing a document-topic matrix generated by performing topic modeling using Latent Dirichlet Allocation (LDA) on a set of preprocessed documents.\n",
        "\n",
        "The document-topic matrix represents the distribution of topics across each document. Each row in the matrix corresponds to a document, and each column corresponds to a topic. The values in the matrix represent the weight or probability of each topic in the respective document.\n",
        "\n",
        "After running the code, you should see a message indicating that the document-topic matrix has been saved to a CSV file specified by the output_csv parameter. The CSV file will contain the document-topic matrix, with each row representing a document and each column representing a topic."
      ],
      "metadata": {
        "id": "Bo5_MEGvLFBA"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Recommendations"
      ],
      "metadata": {
        "id": "C1cbZr8wMXmT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now that I've successfully generated document-topic matrix,next dive into analyzing the results!"
      ],
      "metadata": {
        "id": "7ElMLSLvMbQt"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The document-topic matrix provides you with a wealth of information about the underlying themes present in your documents. Let's explore some next steps you can take to gain deeper insights."
      ],
      "metadata": {
        "id": "0PCiShHYMybh"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "1. **Topic Interpretation**: you'll begin by examining the topics identified by the model. Analyzing the top words associated with each topic will help you understand the main themes present in your documents.\n",
        "\n",
        "2. **Document Classification:** Next, you'll explore how to classify or categorize your documents based on their dominant topics. This step will help you organize and label your documents for further analysis.\n",
        "\n",
        "3. **Visualization:** Visualizing the document-topic matrix can provide you with a clearer understanding of the relationships between documents and topics. You'll explore techniques such as **heatmaps** or multidimensional scaling to bring your data to life.\n",
        "\n",
        "4. **Evaluation:** It's important to assess the quality of your topic modeling results. You'll delve into various evaluation metrics, including **coherence scores and human judgment**, to determine how effective your model is at capturing the underlying themes."
      ],
      "metadata": {
        "id": "voKRqMEYM_KP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Practical Applications of Topic Modeling"
      ],
      "metadata": {
        "id": "lquUlVisAhFU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Topic modeling isn't just a theoretical concept—it has real-world applications across various domains. Whether it's recommending relevant content, analyzing customer feedback, or tracking trends on social media, topic modeling can unlock valuable insights for your projects."
      ],
      "metadata": {
        "id": "U3afrbLwA2fG"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Conclusion"
      ],
      "metadata": {
        "id": "7tJS98LfA_uI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "As I conclude today's tutorial, I hope you've gained a deeper understanding of topic modeling and its practical uses. Whether you're a beginner or an experienced data scientist, topic modeling opens up a world of possibilities for extracting insights from text data."
      ],
      "metadata": {
        "id": "szGJFng-BCL7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Closing Remark"
      ],
      "metadata": {
        "id": "40BLDHi-BTqJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Thank you all for joining me today! Whether you're using sample data or your own dataset, I encourage you to explore topic modeling further and apply it to your projects. If you enjoyed this tutorial, don't forget to like, comment, and subscribe for more exciting content on data science and machine learning. Until next time, happy learning!"
      ],
      "metadata": {
        "id": "UwYWOiaNBVc5"
      }
    }
  ]
}