Large_Language_Models_Article_Separation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyN8MxU868OagAbRX3uguOkR",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ieg-dhr/NLP-Course4Humanities_2024/blob/main/Large_Language_Models_Article_Separation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Large Language Models and Article Extraction\n",
        "\n",
        "\n",
        "Created by Sarah Oberbichler [![ORCID](https://info.orcid.org/wp-content/uploads/2019/11/orcid_16x16.png)](https://orcid.org/0000-0002-1031-2759)\n",
        "\n",
        "###Using LLMs via APIs\n",
        "\n",
        "For this course, we utilize the NVIDIA API, which provides up to 4,000 free credits to access the open-source model llama-3.1-nemotron-70b-instruct via NVIDIA's GPU infrastructure. When using larger models outside of chatbot applications, they demand significant computational resources.\n",
        "While APIs offer a solution for accessing models and GPU power through third parties where no local computer power is available, they typically:\n",
        "\n",
        "*   Require payment beyond free trial credits\n",
        "*   Should not be used with sensitive data\n",
        "*   Should not be used with copyright restricted data\n",
        "\n",
        "\n",
        "### Using LLMs via APIs for the Analysis of Historical Newspapers\n",
        "Historical newspapers published before 1940 are generally free from copyright protection and, when accessed through public newspaper platforms, are not classified as sensitive data. However, important considerations include:\n",
        "\n",
        "*   Library licensing agreements may restrict usage\n",
        "*   Cultural heritage institutions might have specific terms of use\n",
        "*   Access and processing policies may vary by institution\n",
        "\n",
        "When using API's provided by third parties, make sure to check the licensing agreements of the data provider (e.g. library). For example, newspapers makred with **Public Domain Mark 1.0 Universell** don't have any restrictions."
      ],
      "metadata": {
        "id": "LX69IhPEGqhv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!git clone https://github.com/ieg-dhr/NLP-Course4Humanities_2024.git"
      ],
      "metadata": {
        "id": "abStdgpVAYfY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Setting up the Large Language Model\n",
        "\n",
        "In order to use the large language model via API, you need to get an API key: https://build.nvidia.com/nvidia/llama-3_1-nemotron-70b-instruct. Add your private key to you Colab Notebook under *Secrets* as NVIDIA_TOKEN. Run the next cell and see if everything worked as intended."
      ],
      "metadata": {
        "id": "vzUjqdnmXvg1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip uninstall -y httpx\n",
        "!pip install httpx==0.27.2\n",
        "from openai import OpenAI\n",
        "from google.colab import userdata\n",
        "\n",
        "client = OpenAI(\n",
        "    base_url=\"https://integrate.api.nvidia.com/v1\",\n",
        "    api_key=userdata.get('NVIDIA_TOKEN'),\n",
        "    # Remove any default timeout settings\n",
        "    timeout=None\n",
        ")\n",
        "\n",
        "\n",
        "completion = client.chat.completions.create(\n",
        "  model=\"nvidia/llama-3.1-nemotron-70b-instruct\",\n",
        "  messages=[{\"role\":\"user\",\"content\":f\"\"\"Hello?\"\"\"\n",
        "}],\n",
        "  temperature=0.3,\n",
        "  top_p=1,\n",
        "  max_tokens=10024,\n",
        "  stream=True\n",
        ")\n",
        "\n",
        "for chunk in completion:\n",
        "  if chunk.choices[0].delta.content is not None:\n",
        "    print(chunk.choices[0].delta.content, end=\"\")"
      ],
      "metadata": {
        "id": "gaQcmvOhAmuq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Importing the Dataset"
      ],
      "metadata": {
        "id": "yBC3w7zJEIzb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "\n",
        "# Replace 'your_file.xlsx' with the actual path to your Excel file.\n",
        "df = pd.read_excel('/content/NLP-Course4Humanities_2024/datasets/Süddeutsche_Zeitung_Messina.xlsx')\n",
        "\n",
        "# Display the first few rows of the DataFrame to verify it's loaded correctly.\n",
        "df=df[:4]\n",
        "df.head()"
      ],
      "metadata": {
        "id": "6xy-00_EDB1e"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "from typing import List, Dict\n",
        "from openai import OpenAI\n",
        "\n",
        "# Initialize OpenAI client with NVIDIA API settings\n",
        "client = OpenAI(\n",
        "    base_url=\"https://integrate.api.nvidia.com/v1\",\n",
        "    api_key=userdata.get('NVIDIA_TOKEN')\n",
        ")\n",
        "\n",
        "def analyze_dataframe(df: pd.DataFrame, text_column: str) -> pd.DataFrame:\n",
        "    def analyze_text(text: str) -> str:\n",
        "        system_prompt = \"\"\"\n",
        "# System Instructions\n",
        "You are an expert text analyst and information retrieval specialist and hate summarization as well as enumerations.\n",
        "Your task is to carefully analyze given texts and extract complete articles that contain specific themes. You never change original texts.\n",
        "\n",
        "Classify as relevant if the text contains:\n",
        "- Primary earthquake terminology from the 19th and 20th century\n",
        "- Official earthquake reports\n",
        "- geology and seismology\n",
        "- Impact descriptions\n",
        "- Solution description\n",
        "- Technical description\n",
        "- Aid\n",
        "- Honorations\n",
        "- Political discussion and opinions on earthquake\n",
        "- Stories from victims and refugees\n",
        "- reportings on refugees and victims\n",
        "- Live of victims\n",
        "- historical references\n",
        "- comparisons\n",
        "\n",
        "Your output should consist of nothing else but the the xml structure <article></article><verification></verification><human_verification_needed></human_verification_needed>\n",
        "\n",
        "Maintain a neutral, objective stance throughout the analysis. Focus on accuracy and completeness in your extractions\n",
        "\"\"\"\n",
        "        user_prompt = f\"\"\"\n",
        "Bitte befolgen Sie diese Spezifikationen:\n",
        "1. Definition eines Artikels: Ein Artikel ist eine semantische Einheit im Text, die sich deutlich von vorangehendem und nachfolgendem Inhalt abgrenzt (z.B. durch eine eigene Überschrift).\n",
        "3. Antwortformat:\n",
        "- Wenn ein oder mehrere relevante Artikel gefunden werden, strukturieren Sie Ihre Antwort mit XML-Tags wie im folgenden Beispiel, unter Verwendung der Tags article, verification und human_verification_needed (True oder False): <article>vollständiger extrahierter Artikelinhalt</article><verification>Ist die Einheit kohärent? Ist das Thema vorhanden? Ist der Artikel vollständig? Wurden alle Artikel gefunden?</verification><human_verification_needed>False</human_verification_needed>\n",
        "- Gebe alle relevanten Artikel in ihrer Originalform zurück, ohne Ergänzungen, Auslassungen, Korrekturen oder Kommentare.\n",
        "- Wenn keine relevanten Artikel gefunden werden, ist keine besondere Strukturierung erforderlich; gebe einfach \"Kein relevanter Artikel gefunden.\" ohne weitere Erklärungen zurück.\n",
        "4. Hinweise zur Segmentierung:\n",
        "- Stelle sicher, dass über mehrere Absätze verteilte Artikel als eine Einheit behandelt werden.\n",
        "5. Menschliche Überprüfung notwendig:\n",
        "- Kann die Werte \"True\" oder \"False\" haben\n",
        "- False: Wenn Sie glauben, den Artikel korrekt segmentiert und seine Relevanz richtig eingeschätzt zu haben.\n",
        "- True: Wenn du unsicher bist, ob du den vollständigen Inhalt des Artikels, wie er im Zeitungsdokument enthalten ist, erfasst hast oder ob er relevant ist.\n",
        "\n",
        "Hier ist das Zeitungsdokument:\n",
        "\n",
        "{text}\n",
        "\"\"\"\n",
        "        try:\n",
        "            messages = [\n",
        "                {\n",
        "                    'role': 'system',\n",
        "                    'content': system_prompt\n",
        "                },\n",
        "                {\n",
        "                    'role': 'user',\n",
        "                    'content': user_prompt\n",
        "                }\n",
        "            ]\n",
        "\n",
        "            completion = client.chat.completions.create(\n",
        "                model=\"nvidia/llama-3.1-nemotron-70b-instruct\",\n",
        "                messages=messages,\n",
        "                temperature=0.0,\n",
        "                max_tokens=20000\n",
        "            )\n",
        "            return completion.choices[0].message.content\n",
        "        except Exception as e:\n",
        "            print(f\"Error in API call: {str(e)}\")\n",
        "            return \"\"\n",
        "\n",
        "    # Apply the analysis to each row in the DataFrame\n",
        "    df['separated_articles'] = df[text_column].apply(lambda x: analyze_text(x) if pd.notna(x) else \"\")\n",
        "\n",
        "    return df\n",
        "\n",
        "# Usage example (assuming df is your input DataFrame)\n",
        "if __name__ == \"__main__\":\n",
        "    # Process the DataFrame\n",
        "    text_column = 'plainpagefulltext'  # or your text column name\n",
        "    result_df = analyze_dataframe(df, text_column)\n",
        "\n",
        "    # Save the results\n",
        "    result_df.to_excel('analyzed_results.xlsx', index=False)\n",
        "\n",
        "    # Display sample results\n",
        "    print(\"\\nSample of processed articles:\")\n",
        "    print(result_df['separated_articles'].head())"
      ],
      "metadata": {
        "id": "G6GHbkcUb0hR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Write a prompt for OCR Post-Correction"
      ],
      "metadata": {
        "id": "2bigMCHLEwix"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "from openai import OpenAI\n",
        "\n",
        "# Initialize OpenAI client with NVIDIA API settings\n",
        "client = OpenAI(\n",
        "    base_url=\"https://integrate.api.nvidia.com/v1\",\n",
        "    api_key = userdata.get('NVIDIA_TOKEN')\n",
        ")\n",
        "\n",
        "# Process the DataFrame\n",
        "all_articles = []\n",
        "for index, row in result_df.iterrows():\n",
        "    try:\n",
        "        # Make API call\n",
        "        completion = client.chat.completions.create(\n",
        "            model=\"nvidia/llama-3.1-nemotron-70b-instruct\",\n",
        "            messages=[\n",
        "                {\n",
        "                    'role': 'system',\n",
        "                    'content': \"\"\" System Instructions: \"\"\"\n",
        "                },\n",
        "                {\n",
        "                    'role': 'user',\n",
        "                    'content': f\"\"\"# Task Instructions:\n",
        "Text to analyze:\n",
        "{row['extracted_article']}\"\"\"\n",
        "                }\n",
        "            ],\n",
        "            temperature=0.0,\n",
        "            max_tokens=20000\n",
        "        )\n",
        "\n",
        "        content = completion.choices[0].message.content\n",
        "\n",
        "        # Process articles\n",
        "        if content and \"Keine Artikel mit dem angegebenen Thema gefunden.\" not in content:\n",
        "            new_row = row.to_dict()\n",
        "            new_row['article_corrected'] = content.strip()\n",
        "            all_articles.append(new_row)\n",
        "\n",
        "    except Exception as e:\n",
        "        print(f\"Error processing row {index}: {str(e)}\")\n",
        "        continue\n",
        "\n",
        "# Create final DataFrame\n",
        "result_2_df = pd.DataFrame(all_articles)\n",
        "\n",
        "# Save to Excel\n",
        "result_2_df.to_excel('test_2.xlsx', index=False)\n",
        "\n",
        "# Display results\n",
        "print(result_2_df.head())"
      ],
      "metadata": {
        "id": "-6S_983uGveC"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}