2-gpt-4o-mini.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GPT-4o-mini (details 08/08/24)\n",
    "https://platform.openai.com/docs/api-reference/introduction\n",
    "\n",
    "This experiment used GPT4o\n",
    "\n",
    "### User tier 3 \n",
    "Check your tier here: https://platform.openai.com/settings/organization/limits. API usage is subject to rate limits applied on tokens per minute (TPM), requests per minute or day (RPM/RPD), and other model-specific limits. Your organization's rate limits are listed below.\n",
    "- Token Limits = 800,000 TPM\n",
    "- Request and other limits\t= 5,000 RPM\n",
    "- Batch queue limits = 100,000,000 TPD\n",
    "\n",
    "### GPT-4o and GPT-4o mini pricing\n",
    "https://openai.com/api/pricing/\n",
    "\n",
    "GPT-4o is our most advanced multimodal model that’s faster and cheaper than GPT-4 Turbo with stronger vision capabilities. The model has 128K context and an October 2023 knowledge cutoff.\n",
    "\n",
    "GPT-4o mini is our most cost-efficient small model that’s smarter and cheaper than GPT-3.5 Turbo, and has vision capabilities. The model has 128K context and an October 2023 knowledge cutoff.\n",
    "- gpt-4o\n",
    "    - $5.00 / 1M input tokens\n",
    "    - $15.00 / 1M output tokens\n",
    "    - Vision pricing: $0.003825 per image (1600x1200)\n",
    "- gpt-4o-2024-08-06\n",
    "    - $2.50 / 1M input tokens\n",
    "    - $10.00 / 1M output tokens\n",
    "    - Vision pricing: $0.001913 per image (1600x1200)\n",
    "- gpt-4o-mini \n",
    "    - $0.150 / 1M input tokens\n",
    "    - $0.600 / 1M output tokens\n",
    "    - Vision pricing: $0.003825 per image (1600x1200)\n",
    "\n",
    "\n",
    "### *We will be testing on gpt-4o-mini*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Structured Outputs vs JSON mode\n",
    "Structured Outputs is the evolution of JSON mode. While both ensure valid JSON is produced, only Structured Outputs ensure schema adherance. Both Structured Outputs and JSON mode are supported in the Chat Completions API, Assistants API, Fine-tuning API and Batch API.\n",
    "\n",
    "We recommend always using Structured Outputs instead of JSON mode when possible.\n",
    "\n",
    "However, Structured Outputs with response_format: {type: \"json_schema\", ...} is only supported with the gpt-4o-mini, gpt-4o-mini-2024-07-18, and gpt-4o-2024-08-06 model snapshots and later.\n",
    "\n",
    "Structured Outputs\tJSON Mode\n",
    "Outputs valid JSON\tYes\tYes\n",
    "Adheres to schema\tYes (see supported schemas)\tNo\n",
    "Compatible models\tgpt-4o-mini, gpt-4o-2024-08-06, and later\tgpt-3.5-turbo, gpt-4-* and gpt-4o-* models\n",
    "Enabling\tresponse_format: { type: \"json_schema\", json_schema: {\"strict\": true, \"schema\": ...} }\tresponse_format: { type: \"json_object\" }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "^C\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Requirement already satisfied: openai in c:\\users\\ucesnjo\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (1.40.3)\n",
      "Requirement already satisfied: anyio<5,>=3.5.0 in c:\\users\\ucesnjo\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from openai) (4.2.0)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in c:\\users\\ucesnjo\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from openai) (1.9.0)\n",
      "Requirement already satisfied: httpx<1,>=0.23.0 in c:\\users\\ucesnjo\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from openai) (0.26.0)\n",
      "Requirement already satisfied: jiter<1,>=0.4.0 in c:\\users\\ucesnjo\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from openai) (0.5.0)\n",
      "Requirement already satisfied: pydantic<3,>=1.9.0 in c:\\users\\ucesnjo\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from openai) (2.5.3)\n",
      "Requirement already satisfied: sniffio in c:\\users\\ucesnjo\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from openai) (1.3.0)\n",
      "Requirement already satisfied: tqdm>4 in c:\\users\\ucesnjo\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from openai) (4.65.0)\n",
      "Requirement already satisfied: typing-extensions<5,>=4.11 in c:\\users\\ucesnjo\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from openai) (4.12.2)\n",
      "Requirement already satisfied: idna>=2.8 in c:\\users\\ucesnjo\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from anyio<5,>=3.5.0->openai) (3.4)\n",
      "Requirement already satisfied: certifi in c:\\users\\ucesnjo\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from httpx<1,>=0.23.0->openai) (2023.5.7)\n",
      "Requirement already satisfied: httpcore==1.* in c:\\users\\ucesnjo\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from httpx<1,>=0.23.0->openai) (1.0.2)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in c:\\users\\ucesnjo\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from httpcore==1.*->httpx<1,>=0.23.0->openai) (0.14.0)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in c:\\users\\ucesnjo\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from pydantic<3,>=1.9.0->openai) (0.6.0)\n",
      "Requirement already satisfied: pydantic-core==2.14.6 in c:\\users\\ucesnjo\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from pydantic<3,>=1.9.0->openai) (2.14.6)\n",
      "Requirement already satisfied: colorama in c:\\users\\ucesnjo\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from tqdm>4->openai) (0.4.6)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "[notice] A new release of pip is available: 24.1.2 -> 24.2\n",
      "[notice] To update, run: python.exe -m pip install --upgrade pip\n"
     ]
    }
   ],
   "source": [
    "# %pip uninstall openai\n",
    "!pip install openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "import os\n",
    "import pandas as pd\n",
    "import PIL.Image\n",
    "import json\n",
    "from io import StringIO\n",
    "import time\n",
    "import typing\n",
    "import base64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instruction (text input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can experiment with 'local_context' and change 'country' as needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add the local context to the input data\n",
    "country = \"//country//\"\n",
    "local_context = \"//local_context//\"\n",
    "\n",
    "# country and local_context that we used in the experiment\n",
    "country = \"Thailand\"\n",
    "local_context = \"Speed limits in {country} are a set of maximum speeds that apply to all roads in the country. For <Attribute> = 'Speed limit', the maximum limits are as follows: 80 km/h within built-up areas and in Bangkok, 90 km/h outside built-up areas, and 120 km/h on motorways.\\nFor the <Attribute> = 'Motorcycle speed limit', the limits are 80 km/h in Bangkok and other provinces' built-up areas, and 90 km/h on other highways (motorcycles are not allowed on motorways). For the <Attribute> = 'Truck speed limit', the limits are 60 km/h in Bangkok and other provinces' built-up areas, 80 km/h on highways, and 100 km/h on motorways. <Attribute>: Carriageway answer should be Divided carriageway= 'Carriageway A of a divided road'.\\nFor highways in Thailand <Attribute> 'Lane width' is 'Wide \\u22653.25m'. Moreover, <Attribute>: 'Area type' in Bangkok and Phatum thani should be 'Urban'; therefore, <Attribute>: 'Upgrade cost' should be 'High' and <Attribute>: 'Street lighting' should be 'Present' in Bangkok and Phatum thani area.\\nThe common <Attribute>: Roadside severity - driver-side object are 'Safety barrier - concrete' and 'Safety barrier - metal' that should carefully look at the image. The common <Attribute>: Roadside severity - passenger-side object are 'Safety barrier - concrete', 'Safety barrier - metal', 'Deep drainage ditch', 'Rigid sign, post or pole \\u226510cm' and 'Unprotected safety barrier end' that should carefully look at the image. In Thailand highway, there should not be 'No object' for <Attribute>: Roadside severity - passenger-side object and <Attribute>: Roadside severity - driver-side object.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make sure the json file is in the correct path\n",
    "json_file = './/text//prompts.json'\n",
    "\n",
    "def format_attributes_to_json(json_file, image_id=\"image_id\"):\n",
    "    with open(json_file, 'r') as file:\n",
    "        data = json.load(file)\n",
    "\n",
    "    formatted_data = {\"image_id\": image_id}\n",
    "    \n",
    "    for attribute in data[\"attributes\"]:\n",
    "        item_name = attribute.get('Item', 'Unknown Item')\n",
    "        categories = [category.get('Category', 'Unknown Category') for category in attribute.get('categories', [])]\n",
    "        formatted_data[item_name] = categories\n",
    "    \n",
    "    return json.dumps(formatted_data, indent=4)\n",
    "output_json = format_attributes_to_json(json_file)\n",
    "\n",
    "\n",
    "prompt_instruction = f\"\"\"\n",
    "You are a road safety assessment coder from the International Road Assessment Programme (iRAP). Your task is to analyse images of road sections taken in {country} and accurately assess 52 road safety attributes. For each attribute, follow these steps:\n",
    "\n",
    "1. Analyse the Image: Examine the road section in the image, focusing on all relevant elements that correspond to the 52 '<Attribute>'s you need to assess.\n",
    "2. Read the <Attribute Description>: For each of the 52 attributes, read the '<Attribute description>' to understand what specific aspect of the image you need to evaluate.\n",
    "3. Refer to Categories: For each attribute, refer to the possible '<Category class>' options provided. If a '<Category description>' is available, read it to understand the specific criteria for each category.\n",
    "4. Select the Most Matching Category: Based on your analysis of the image and understanding of the attribute and category descriptions, select the single <Category class> that best matches what you observe in the image. If multiple categories are equally relevant, choose the category that appears first in the provided list.\n",
    "5. Output the Results in JSON Format: Return the results in JSON format, with each attribute associated with a single <Category class> value that you assess to be the most appropriate based on the image.\n",
    "\n",
    "Local context:\n",
    "Please use <location> to understand the local context. '<driver-side>' and '<passenger-side>' are used throughout the <Attribute> and <Attribute description>. Driver-side refers to the side of the road corresponding with the driver of a vehicle travelling in the direction of the survey, and the passenger-side is the other side. If the country drives on the left (e.g., the UK), the passenger side is on the left of the image, and the driver side is on the right of the image. {local_context}\n",
    "\"\"\"\n",
    "\n",
    "output_format = f\"\"\"\n",
    "Output Format: Return the results in JSON format, where each attribute is associated with a single <Category class> value that best matches your analysis of the image. If multiple categories seem equally relevant, select the category that appears first in the provided list.\n",
    "\n",
    "JSON structure:\n",
    "{output_json}\n",
    "Ensure that each attribute in the JSON output contains only one selected <Category class> that you determine to be the most appropriate based on the image.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Path configurations for ThaiRAP\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Path configurations\n",
    "image_folder_path = './/image//ThaiRAP'\n",
    "csv_file_path = './Validation.csv'\n",
    "save_path = './/result//gpt4o-mini_thairap.csv'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Path configurations for Mapillary\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Path configurations\n",
    "image_folder_path = './/image//Mapillary_processed' #used image folder as the image folder path (images were already processed (cropped, resized, and renamed))\n",
    "csv_file_path = './/image//Mapillary_processed//mapillary.csv'\n",
    "save_path = './/result//gpt4o-mini__mapillary.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Single prompt\n",
    "This mean, in a request/ask VLM on 1 image and its infomation, however, the aim is to get 52 answers ('attributes')\n",
    "\n",
    "These below functions used to set up prommpts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Read the CSV file\n",
    "df = pd.read_csv(csv_file_path)\n",
    "\n",
    "# Function to generate a prompt for a single image\n",
    "def generate_single_image_prompt(image_id, df):\n",
    "    row = df[df['image_id'] == int(image_id)]\n",
    "    if not row.empty:\n",
    "        lat = row['Latitude start'].values[0]\n",
    "        lon = row['Longitude start'].values[0]\n",
    "        return f\"<image_id>: {image_id} and <location>: {{{lat},{lon}}}\"\n",
    "    else:\n",
    "        return None \n",
    "\n",
    "# Function to convert JSON to text prompt\n",
    "def json2text(include_attribute_description=True, include_category_description=True):\n",
    "    with open(json_file, 'r') as file:\n",
    "        data = json.load(file)\n",
    "    attributes = data[\"attributes\"]\n",
    "    formatted_descriptions = []\n",
    "    \n",
    "    for idx, attribute in enumerate(attributes, 1):\n",
    "        item = attribute.get('Item', 'Unknown Item')\n",
    "        attribute_description = attribute.get('Attribute description', 'No description available.')\n",
    "        \n",
    "        if idx > 1:\n",
    "            formatted_descriptions.append(\"\")  # Add a blank line before each new item\n",
    "        \n",
    "        formatted_descriptions.append(f\"{idx}. <Attribute>: {item}\")\n",
    "        if include_attribute_description:\n",
    "            formatted_descriptions.append(f\"<Attribute description>: {attribute_description}\")\n",
    "        \n",
    "        categories_details = []\n",
    "        \n",
    "        for category in attribute.get('categories', []):\n",
    "            cat_id = category.get('Category', 'N/A')\n",
    "            category_name = category.get('Category', 'Unknown Category')\n",
    "            category_description = category.get('Category_description', '')\n",
    "            \n",
    "            if include_category_description and category_description:\n",
    "                categories_details.append(f\" <Category class>:{cat_id}, <Category description>: {category_description}\")\n",
    "            else:\n",
    "                categories_details.append(f\" <Category class>:{cat_id}\")\n",
    "        \n",
    "        # Format the categories list\n",
    "        formatted_descriptions.append(f\"<Categories>: [{', '.join([category['Category'] for category in attribute.get('categories', [])])}]\")\n",
    "        \n",
    "        # Append detailed category descriptions\n",
    "        formatted_descriptions.extend(categories_details)\n",
    "        \n",
    "    return \"\\n\".join(formatted_descriptions)\n",
    "\n",
    "def format_attributes_to_json(json_file, image_id=\"image_id\"):\n",
    "    with open(json_file, 'r') as file:\n",
    "        data = json.load(file)\n",
    "\n",
    "    formatted_data = {\"image_id\": image_id}\n",
    "    \n",
    "    for attribute in data[\"attributes\"]:\n",
    "        item_name = attribute.get('Item', 'Unknown Item')\n",
    "        categories = [category.get('Category', 'Unknown Category') for category in attribute.get('categories', [])]\n",
    "        formatted_data[item_name] = categories\n",
    "    \n",
    "    return json.dumps(formatted_data, indent=4)\n",
    "output_json = format_attributes_to_json(json_file)\n",
    "\n",
    "fields = [\n",
    "    \"image_id\",\n",
    "    \"Carriageway\",\n",
    "    \"Upgrade cost\",\n",
    "    \"Motorcycle observed flow\",\n",
    "    \"Bicycle observed flow\",\n",
    "    \"Pedestrian observed flow across the road\",\n",
    "    \"Pedestrian observed flow along the road driver-side\",\n",
    "    \"Pedestrian observed flow along the road passenger-side\",\n",
    "    \"Land use - driver-side\",\n",
    "    \"Land use - passenger-side\",\n",
    "    \"Area type\",\n",
    "    \"Speed limit\",\n",
    "    \"Motorcycle speed limit\",\n",
    "    \"Truck speed limit\",\n",
    "    \"Differential speed limits\",\n",
    "    \"Median type\",\n",
    "    \"Centreline rumble strips\",\n",
    "    \"Roadside severity - driver-side distance\",\n",
    "    \"Roadside severity - driver-side object\",\n",
    "    \"Roadside severity - passenger-side distance\",\n",
    "    \"Roadside severity - passenger-side object\",\n",
    "    \"Shoulder rumble strips\",\n",
    "    \"Paved shoulder - driver-side\",\n",
    "    \"Paved shoulder - passenger-side\",\n",
    "    \"Intersection type\",\n",
    "    \"Intersection channelisation\",\n",
    "    \"Intersecting road volume\",\n",
    "    \"Intersection quality\",\n",
    "    \"Property access points\",\n",
    "    \"Number of lanes\",\n",
    "    \"Lane width\",\n",
    "    \"Curvature\",\n",
    "    \"Quality of curve\",\n",
    "    \"Grade\",\n",
    "    \"Road condition\",\n",
    "    \"Skid resistance / grip\",\n",
    "    \"Delineation\",\n",
    "    \"Street lighting\",\n",
    "    \"Pedestrian crossing facilities - inspected road\",\n",
    "    \"Pedestrian crossing quality\",\n",
    "    \"Pedestrian crossing facilities - intersecting road\",\n",
    "    \"Pedestrian fencing\",\n",
    "    \"Speed management / traffic calming\",\n",
    "    \"Vehicle parking\",\n",
    "    \"Sidewalk - driver-side\",\n",
    "    \"Sidewalk - passenger-side\",\n",
    "    \"Service road\",\n",
    "    \"Facilities for motorised two wheelers\",\n",
    "    \"Facilities for bicycles\",\n",
    "    \"Roadworks\",\n",
    "    \"Sight distance\",\n",
    "    \"School zone warning\",\n",
    "    \"School zone crossing supervisor\"\n",
    "]\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choose image_id here\n",
    "This can be used for both ThaiRAP and Mapillary\n",
    "- ThaiRAP 2037 images\n",
    "- Mapillary 168 images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# all the image ids\n",
    "#choose_image_id = range(1, 2038) # test processing image 1 to 2037\n",
    "\n",
    "# test\n",
    "choose_image_id = range(1, 2) # test processing image 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GPT configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set your OpenAI API key directly\n",
    "api_key = \"//api_key_here//\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run to get the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processed image 1 - 1.jpg\n"
     ]
    }
   ],
   "source": [
    "# connect to the OpenAI API\n",
    "client = OpenAI(api_key=api_key)\n",
    "\n",
    "# Initialize an empty DataFrame if the save file doesn't exist\n",
    "if not os.path.exists(save_path):\n",
    "    # Create a new DataFrame with the appropriate columns\n",
    "    output_df = pd.DataFrame(columns=fields)\n",
    "    output_df.to_csv(save_path, index=False)  # Save the new empty CSV with headers\n",
    "else:\n",
    "    # Load the existing file with keep_default_na=False to avoid converting empty strings to NaN\n",
    "    output_df = pd.read_csv(save_path, keep_default_na=False)\n",
    "\n",
    "# Process each image\n",
    "for image_id in choose_image_id:  # Adjust the range as necessary\n",
    "    image_file = f\"{image_id}.jpg\"\n",
    "    image_path = os.path.join(image_folder_path, image_file)\n",
    "    \n",
    "    # Generate prompt for the current image\n",
    "    image_prompt = generate_single_image_prompt(image_id, df)\n",
    "    if image_prompt is None:\n",
    "        print(f\"Skipping image {image_id} - No data found in CSV\")\n",
    "        continue\n",
    "\n",
    "    def encode_image(image_path):\n",
    "        with open(image_path, \"rb\") as image_file:\n",
    "            return base64.b64encode(image_file.read()).decode('utf-8')\n",
    "\n",
    "\n",
    "    # Getting the base64 string\n",
    "    base64_image = encode_image(image_path)\n",
    "\n",
    "    # Construct the prompt to send to OpenAI's API\n",
    "    prompt_system = f\"{prompt_instruction}\\n{json2text()}\\n\\n{output_format}\"\n",
    "\n",
    "    # Call the OpenAI API to generate completions\n",
    "    response = client.chat.completions.create(\n",
    "    model=\"gpt-4o-mini\",\n",
    "    messages=[\n",
    "        {\n",
    "            \"role\": \"system\", \n",
    "            \"content\": prompt_system\n",
    "        },\n",
    "        {\n",
    "            \"role\": \"user\", \n",
    "            \"content\": [\n",
    "                {\n",
    "                    \"type\": \"text\",\n",
    "                    \"text\": image_prompt\n",
    "                },\n",
    "                {\n",
    "                    \"type\": \"image_url\",\n",
    "                    \"image_url\": {\n",
    "                        \"url\":f\"data:image/jpeg;base64,{base64_image}\",\n",
    "                        \"detail\": \"high\"\n",
    "                        }\n",
    "                },\n",
    "            ]\n",
    "        }\n",
    "    ],\n",
    "    temperature=0,\n",
    "    #max_tokens=16384\n",
    "    #response_format={type: \"V_RoAst_schema\"}\n",
    "    )\n",
    "\n",
    "    response_text = response.choices[0].message.content\n",
    "    response_text = response_text[8:-3].strip()\n",
    "    response_dict = json.loads(response_text)\n",
    "\n",
    "    df_result = pd.DataFrame.from_dict(response_dict, orient='index').transpose()\n",
    "    df_result = df_result.applymap(lambda x: x[0] if isinstance(x, list) and len(x) > 0 else x)\n",
    "\n",
    "    for field in fields:\n",
    "        if field not in df.columns:\n",
    "            df_result[field] = None  # Add missing fields with None\n",
    "\n",
    "    # Reorder the DataFrame to match the fields list\n",
    "    df_result = df_result[fields]\n",
    "    print(f\"Processed image {image_id} - {image_file}\")\n",
    "    \n",
    "    # Append the new row to the existing DataFrame\n",
    "    output_df = pd.concat([output_df, df_result], ignore_index=True)\n",
    "\n",
    "    # Save the updated DataFrame back to the CSV after processing each image\n",
    "    output_df.to_csv(save_path, index=False)\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}