GPT_4_Turbo_Vision.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNvOBFMDuplhQxEalgR6sJy",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/PradipNichite/Youtube-Tutorials/blob/main/GPT_4_Turbo_Vision.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "##Agenda\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "*   GPT-4 Turbo Vision Introduction\n",
        "*   Generate Image Description using image url or image file\n",
        "*   Asking Questions Based on Image\n",
        "*   Passing Multiple Images and complex question\n",
        "*   Calculating Cost and Usage limit\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "yI6TQoeqozRI"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##GPT-4 Turbo Vision Introduction\n",
        "\n",
        "https://platform.openai.com/docs/guides/vision/introduction\n",
        "\n",
        "GPT-4 with Vision, sometimes referred to as GPT-4V or gpt-4-vision-preview in the API, allows the model to take in images and answer questions about them.\n",
        "\n",
        "Note that the Assistants API does not currently support image inputs."
      ],
      "metadata": {
        "id": "P4IZLF983O_m"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install openai -q"
      ],
      "metadata": {
        "id": "MqfaMupxhkHy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "api_key = \"sk-\"\n",
        "from openai import OpenAI\n",
        "client = OpenAI(api_key=api_key)"
      ],
      "metadata": {
        "id": "3g6b9LuTiNaR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "response = client.chat.completions.create(\n",
        "  model=\"gpt-3.5-turbo\",\n",
        "  messages=[\n",
        "    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
        "    {\"role\": \"user\", \"content\": \"Who won the world series in 2020?\"},\n",
        "    {\"role\": \"assistant\", \"content\": \"The Los Angeles Dodgers won the World Series in 2020.\"},\n",
        "    {\"role\": \"user\", \"content\": \"Where was it played?\"}\n",
        "  ]\n",
        ")\n",
        "response.choices[0].message.content"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "id": "Jd6KNWgLiTYJ",
        "outputId": "1146052e-ffe7-4a4a-878e-23ad5f79e421"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'The 2020 World Series was played in Arlington, Texas, at the Globe Life Field, which was the new home stadium for the Texas Rangers.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Generate Image Description using image url or image file"
      ],
      "metadata": {
        "id": "D9eDSz-93ism"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "image_1 = \"https://images.pexels.com/photos/4079293/pexels-photo-4079293.jpeg\""
      ],
      "metadata": {
        "id": "oRZYnU0RsqCH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ee9IYAwchh7q",
        "outputId": "e674ef40-a1a6-4d22-8a09-f2e1db6daca5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The laptop in the image has a distinctive logo on its lid that resembles an apple with a bite taken out, which is the logo of Apple Inc. This indicates that the laptop is likely a MacBook, which is a line of Macintosh laptop computers designed and marketed by Apple Inc.\n"
          ]
        }
      ],
      "source": [
        "image_1 = \"https://images.pexels.com/photos/4079293/pexels-photo-4079293.jpeg\"\n",
        "response = client.chat.completions.create(\n",
        "  model=\"gpt-4-vision-preview\",\n",
        "  messages=[\n",
        "    {\n",
        "      \"role\": \"user\",\n",
        "      \"content\": [\n",
        "        {\"type\": \"text\", \"text\": \"what is brand of laptop\"},\n",
        "        {\n",
        "          \"type\": \"image_url\",\n",
        "          \"image_url\": {\n",
        "            \"url\": image_1,\n",
        "            \"detail\": \"high\"\n",
        "          },\n",
        "        },\n",
        "      ],\n",
        "    }\n",
        "  ],\n",
        "  max_tokens=300,\n",
        "  )\n",
        "\n",
        "print(response.choices[0].message.content)"
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gUGF-M7g9zN2",
        "outputId": "970595be-3b93-40af-a44e-ab1e11582666"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The image shows a brightly lit indoor scene with a woman sitting at a modern dining table. She appears to be in a casual, comfortable setting, possibly at home. The woman has blonde hair and is smiling gently while holding an orange with her right hand and gently cradling a baby with her left arm. The baby, whose face is not visible, seems to be leaning towards her.\n",
            "\n",
            "In front of the woman is an open laptop, indicating she might be working or browsing. To her right, there are various items on the table including a smartphone, a cup, and a plate of fruits with bananas and apples. Just next to the fruits, there's another plate with croissants, suggesting a breakfast setting.\n",
            "\n",
            "Large windows allow ample natural light to enter the room, and through the glass, greenery from outside is visible, suggesting a pleasant environment. The woman seems to be relaxing or taking a break, enjoying the moment with her child, while perhaps contemplating her next activity.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "id": "jF_sdbJ_9_C7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import base64\n",
        "\n",
        "# Function to encode the image\n",
        "def encode_image(image_path):\n",
        "  with open(image_path, \"rb\") as image_file:\n",
        "    return base64.b64encode(image_file.read()).decode('utf-8')\n",
        "\n",
        "# Path to your image\n",
        "image_path = \"/content/sample_image.jpg\"\n",
        "\n",
        "# Getting the base64 string\n",
        "base64_image = encode_image(image_path)\n",
        "\n",
        "\n",
        "response = client.chat.completions.create(\n",
        "  model=\"gpt-4-vision-preview\",\n",
        "  messages=[\n",
        "    {\n",
        "      \"role\": \"user\",\n",
        "      \"content\": [\n",
        "        {\"type\": \"text\", \"text\": \"What’s in this image?\"},\n",
        "        {\n",
        "          \"type\": \"image_url\",\n",
        "          \"image_url\": {\n",
        "            \"url\": f\"data:image/jpeg;base64,{base64_image}\",\n",
        "          },\n",
        "        },\n",
        "      ],\n",
        "    }\n",
        "  ],\n",
        "  max_tokens=300,\n",
        ")\n",
        "\n",
        "response.choices[0].message.content"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 87
        },
        "id": "RJE045-EtQtN",
        "outputId": "a7d897c6-d043-44fa-eb3a-50c296d29d02"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'This image features a pear-shaped diamond ring with a halo setting placed on a textured, dark background that looks like a piece of coal or a rough rock. The diamond at the center is prominently displayed, surrounded by smaller diamonds in a halo formation and along the band of the ring, which appears to be made of a precious metal, possibly gold or platinum with a rose or white gold finish. The contrasting backdrop accentuates the sparkle and clarity of the diamonds.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Asking Questions Based on Image"
      ],
      "metadata": {
        "id": "glm-3pSz3pE0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Passing Multiple Images"
      ],
      "metadata": {
        "id": "DKgjO4Qf3syW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "https://www.jagranjosh.com/general-knowledge/spot-the-difference-can-you-spot-8-differences-between-the-two-images-in-26-seconds-1676378058-1"
      ],
      "metadata": {
        "id": "gu8WeJ804ksg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "image_sample_1 = \"/content/sample_image_diff_1.png\"\n",
        "image_sample_2 = \"/content/sample_image_diff_2.png\"\n",
        "\n",
        "base64_image_1 = encode_image(image_sample_1)\n",
        "\n",
        "base64_image_2 = encode_image(image_sample_2)\n",
        "response = client.chat.completions.create(\n",
        "  model=\"gpt-4-vision-preview\",\n",
        "  messages=[\n",
        "    {\n",
        "      \"role\": \"user\",\n",
        "      \"content\": [\n",
        "        {\n",
        "          \"type\": \"text\",\n",
        "          \"text\": \"Understand both images and Spot Differences in these two images\"\n",
        "        },\n",
        "        {\n",
        "          \"type\": \"image_url\",\n",
        "          \"image_url\": {\n",
        "            \"url\": f\"data:image/jpeg;base64,{base64_image_1}\"\n",
        "          }\n",
        "        },\n",
        "        {\n",
        "          \"type\": \"image_url\",\n",
        "          \"image_url\": {\n",
        "            \"url\": f\"data:image/jpeg;base64,{base64_image_2}\"\n",
        "          },\n",
        "        },\n",
        "      ],\n",
        "    }\n",
        "  ],\n",
        "  max_tokens=300,\n",
        ")\n",
        "print(response.choices[0].message.content)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xyfV-xOiuXcv",
        "outputId": "e11e77a5-82d8-4b91-b1a9-6f0677b3e582"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "In these images, there are several differences that we can identify by comparing them:\n",
            "\n",
            "1. The direction of the ball: In the first image, the soccer ball is moving to the left, towards the goalpost. In the second image, the direction of the ball is to the right, away from the goalpost.\n",
            "2. The player's jersey number: In the background, the red player's jersey number changes. It is number \"9\" in the first image and number \"6\" in the second image.\n",
            "3. The goalkeeper's glove: In the first image, the goalkeeper is wearing his gloves, but in the second image, his left glove is missing.\n",
            "4. The blue team player's socks: The player in blue, who is facing us, has different patterns on his socks. In the first image, the socks have stripes, while in the second image, the stripes are missing.\n",
            "5. The shoe of the red player: In the second image, the shoe of the running red player (number 7) has an additional detail on it, which looks like a white strap or band, that is not present in the first image.\n",
            "\n",
            "These are the visible differences between the two images.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "##Calculating Cost and Usage limit\n",
        "\n",
        "https://platform.openai.com/docs/guides/rate-limits/usage-tiers?context=tier-five\n",
        "\n",
        "https://platform.openai.com/docs/guides/vision/calculating-costs\n",
        "\n",
        "https://openai.com/pricing#language-models"
      ],
      "metadata": {
        "id": "Xdd3-N9a3y5E"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "NnoCkdHN0inD"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}