WebUI_colab.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "authorship_tag": "ABX9TyMBCtDViKbuNPUplQJ1ueh/",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU",
    "gpuClass": "standard"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/tgieruc/Heritage-in-the-digital-age/blob/main/WebUI_colab.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<h1 align=\"center\">Heritage in the digital age</h1>\n",
        "<h3 align=\"center\"><em>A Step-by-Step Guide to Augmenting Digitized Historical Images </em></h3>\n",
        "<h4 align=\"center\"><i>Semester Project @ CVlab &amp; EPFL+ECAL Lab </i></h4>\n",
        "<h4 align=\"center\"><a href=\"https://tgieruc.github.io/Heritage-in-the-digital-age/\">Visit the Website for the gallery and an explanation of the project!</a></h4>\n",
        "<h4 align=\"center\"><a href=\"https://github.com/tgieruc/Heritage-in-the-digital-age/\">Visit the Github repo!</a></h4>\n",
        "<br/>"
      ],
      "metadata": {
        "id": "2hCo-F-G7wBr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "This Google Colab Notebook runs the Web UI application, allowing to go through the whole pipeline with your own data.\n",
        "\n",
        "A Google Colab instance with GPU is recommanded.\n",
        "\n",
        "## Capabilities\n",
        "This application can translate captions, preprocess them, do phrase grounding and agnostic segmentation. \n",
        "\n",
        "For now, it uses [Grounding DINO](https://github.com/IDEA-Research/GroundingDINO) for Phrase Grounding and [Segment Anything](https://github.com/facebookresearch/segment-anything) or Agnostic Segment for segmentation.\n",
        "\n",
        "## Data structure\n",
        "Your data needs to be in the following format:\n",
        "\n",
        "* A pandas dataframe serialized as a pickle file.\n",
        "For example: \n",
        "\n",
        "| **image_id** | **caption**                      |\n",
        "|--------------|----------------------------------|\n",
        "| cat_1        | \"a cat sitting on the pavement\" |\n",
        "| cat_2        | \"a group of cats\"                |\n",
        "\n",
        "* A zip file containing all the images. For example\n",
        "```\n",
        "cats.zip/\n",
        "├── cat_1.png\n",
        "└── cat_2.png\n",
        "```\n",
        "\n",
        "\n",
        "An example dataframe and zip file can be created with this script:"
      ],
      "metadata": {
        "id": "zMPCVOA99D9c"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import files\n",
        "import pandas as pd \n",
        "\n",
        "# Create a list of tuples with image_id and caption\n",
        "data = [('cat_1', 'cats eating food in the street', 'Des chats à la rue Augustin, Fribourg'),\n",
        "        ('cat_2', 'a cat on a laptop', 'Un chat blanc regardant Wikipedia')]\n",
        "# Create a pandas DataFrame with the data and column names\n",
        "df = pd.DataFrame(data, columns=['image_id', 'caption', 'title'])\n",
        "df.to_pickle('df.pkl')\n",
        "\n",
        "files.download('df.pkl')"
      ],
      "metadata": {
        "id": "1MVuuOQbkTLz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!wget -q https://upload.wikimedia.org/wikipedia/commons/thumb/9/94/Street_cats_%281%29.jpg/640px-Street_cats_%281%29.jpg -O cat_1.jpg\n",
        "!wget -q https://upload.wikimedia.org/wikipedia/commons/a/aa/White_cat_watching_Wikipedia.jpg -O cat_2.jpg\n",
        "!zip -q imgs.zip cat_1.jpg cat_2.jpg\n",
        "files.download('imgs.zip')"
      ],
      "metadata": {
        "id": "-9t-AsTEqHXN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "⚠ You might need to autorise your browser to download several files ⚠"
      ],
      "metadata": {
        "id": "CdO4zcA5zn90"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jDZmRuTF0iRM"
      },
      "outputs": [],
      "source": [
        "!git clone --recurse-submodules https://github.com/tgieruc/Heritage-in-the-digital-age.git"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%cd Heritage-in-the-digital-age/\n",
        "!bash setup.sh"
      ],
      "metadata": {
        "id": "P_4m1l3p1SYT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "**Open the public URL**"
      ],
      "metadata": {
        "id": "JkQqqPVR-k-K"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!python webui.py"
      ],
      "metadata": {
        "id": "75rx5h1o5anf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "xRejm5qFz9BR"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}