app.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional Neural Networks\n",
    "\n",
    "## Project: Write an Algorithm for Landmark Classification\n",
    "\n",
    "### A simple app\n",
    "\n",
    "In this notebook we build a very simple app that uses our exported model.\n",
    "\n",
    "> <img src=\"static_images/icons/noun-info-2558213.png\" alt=\"?\" style=\"width:25px\"/> Note how we are not importing anything from our source code (we do not use any module from the ``src`` directory). This is because the exported model, differently from the model weights, is a standalone serialization of our model and therefore it does not need anything else. You can ship that file to anybody, and as long as they can import ``torch``, they will be able to use your model. This is very important for releasing pytorch models to production.\n",
    "\n",
    "### Test your app\n",
    "Go to a search engine for images (like Google Images) and search for images of some of the landmarks, like the Eiffel Tower, the Golden Gate Bridge, Machu Picchu and so on. Save a few examples locally, then upload them to your app to see how your model behaves!\n",
    "\n",
    "The app will show the top 5 classes that the model think are most relevant for the picture you have uploaded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Defaulting to user installation because normal site-packages is not writeable\r\n"
     ]
    }
   ],
   "source": [
    "# Install requirements\n",
    "!pip install -r requirements.txt | grep -v \"already satisfied\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Defaulting to user installation because normal site-packages is not writeable\n",
      "Requirement already satisfied: ipywidgets in /root/.local/lib/python3.7/site-packages (7.6.5)\n",
      "Requirement already satisfied: nbformat>=4.2.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets) (5.0.4)\n",
      "Requirement already satisfied: ipython>=4.0.0; python_version >= \"3.3\" in /opt/conda/lib/python3.7/site-packages (from ipywidgets) (7.13.0)\n",
      "Requirement already satisfied: jupyterlab-widgets>=1.0.0; python_version >= \"3.6\" in /root/.local/lib/python3.7/site-packages (from ipywidgets) (3.0.10)\n",
      "Requirement already satisfied: ipykernel>=4.5.1 in /opt/conda/lib/python3.7/site-packages (from ipywidgets) (5.1.4)\n",
      "Requirement already satisfied: ipython-genutils~=0.2.0 in /opt/conda/lib/python3.7/site-packages (from ipywidgets) (0.2.0)\n",
      "Requirement already satisfied: traitlets>=4.3.1 in /opt/conda/lib/python3.7/site-packages (from ipywidgets) (4.3.3)\n",
      "Requirement already satisfied: widgetsnbextension~=3.5.0 in /root/.local/lib/python3.7/site-packages (from ipywidgets) (3.5.2)\n",
      "Requirement already satisfied: jupyter-core in /opt/conda/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets) (4.6.3)\n",
      "Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /opt/conda/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets) (3.2.0)\n",
      "Requirement already satisfied: jedi>=0.10 in /opt/conda/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets) (0.16.0)\n",
      "Requirement already satisfied: pickleshare in /opt/conda/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets) (0.7.5)\n",
      "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets) (3.0.3)\n",
      "Requirement already satisfied: pygments in /opt/conda/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets) (2.5.2)\n",
      "Requirement already satisfied: decorator in /opt/conda/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets) (4.4.2)\n",
      "Requirement already satisfied: backcall in /opt/conda/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets) (0.1.0)\n",
      "Requirement already satisfied: pexpect; sys_platform != \"win32\" in /opt/conda/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets) (4.8.0)\n",
      "Requirement already satisfied: setuptools>=18.5 in /opt/conda/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets) (45.2.0.post20200209)\n",
      "Requirement already satisfied: jupyter-client in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets) (6.0.0)\n",
      "Requirement already satisfied: tornado>=4.2 in /opt/conda/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets) (5.1.1)\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from traitlets>=4.3.1->ipywidgets) (1.16.0)\n",
      "Requirement already satisfied: notebook>=4.4.1 in /opt/conda/lib/python3.7/site-packages (from widgetsnbextension~=3.5.0->ipywidgets) (5.7.4)\n",
      "Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets) (19.3.0)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets) (0.15.7)\n",
      "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /opt/conda/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets) (1.5.0)\n",
      "Requirement already satisfied: parso>=0.5.2 in /opt/conda/lib/python3.7/site-packages (from jedi>=0.10->ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets) (0.6.1)\n",
      "Requirement already satisfied: wcwidth in /opt/conda/lib/python3.7/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets) (0.1.8)\n",
      "Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.7/site-packages (from pexpect; sys_platform != \"win32\"->ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets) (0.6.0)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /opt/conda/lib/python3.7/site-packages (from jupyter-client->ipykernel>=4.5.1->ipywidgets) (2.8.1)\n",
      "Requirement already satisfied: pyzmq>=13 in /opt/conda/lib/python3.7/site-packages (from jupyter-client->ipykernel>=4.5.1->ipywidgets) (19.0.0)\n",
      "Requirement already satisfied: nbconvert in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (5.6.1)\n",
      "Requirement already satisfied: jinja2 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (2.11.1)\n",
      "Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.7.1)\n",
      "Requirement already satisfied: Send2Trash in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (1.5.0)\n",
      "Requirement already satisfied: terminado>=0.8.1 in /opt/conda/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.8.3)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata; python_version < \"3.8\"->jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets) (3.0.0)\n",
      "Requirement already satisfied: bleach in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (3.1.1)\n",
      "Requirement already satisfied: defusedxml in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.6.0)\n",
      "Requirement already satisfied: mistune<2,>=0.8.1 in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.8.4)\n",
      "Requirement already satisfied: testpath in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.4.4)\n",
      "Requirement already satisfied: entrypoints>=0.2.2 in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.3)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /opt/conda/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (1.4.2)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from jinja2->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (1.1.1)\n",
      "Requirement already satisfied: webencodings in /opt/conda/lib/python3.7/site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets) (0.5.1)\n",
      "Enabling notebook extension jupyter-js-widgets/extension...\n",
      "      - Validating: \u001b[32mOK\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install ipywidgets\n",
    "!jupyter nbextension enable --py widgetsnbextension"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c5ec5ebcf77e4be29467988d056693af",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='Please upload a picture of a landmark'), FileUpload(value={}, description='Upload'…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ipywidgets import VBox, Button, FileUpload, Output, Label\n",
    "from PIL import Image\n",
    "from IPython.display import display\n",
    "import io\n",
    "import numpy as np\n",
    "import torchvision\n",
    "import torchvision.transforms as T\n",
    "import torch\n",
    "\n",
    "# Decide which model you want to use among the ones exported\n",
    "learn_inf = torch.jit.load(\"checkpoints/transfer_exported.pt\") # YOUR CODE HERE\n",
    "\n",
    "def on_click_classify(change):\n",
    "\n",
    "    # Load image that has been uploaded\n",
    "    fn = io.BytesIO(btn_upload.data[-1])\n",
    "\n",
    "    img = Image.open(fn)\n",
    "    img.load()\n",
    "\n",
    "    # Let's clear the previous output (if any)\n",
    "    out_pl.clear_output()\n",
    "\n",
    "    # Display the image\n",
    "    with out_pl:\n",
    "\n",
    "        ratio = img.size[0] / img.size[1]\n",
    "        c = img.copy()\n",
    "        c.thumbnail([ratio * 200, 200])\n",
    "        display(c)\n",
    "\n",
    "    # Transform to tensor\n",
    "    timg = T.ToTensor()(img).unsqueeze_(0)\n",
    "\n",
    "    # Calling the model\n",
    "    softmax = learn_inf(timg).data.cpu().numpy().squeeze()\n",
    "    \n",
    "    # Get the indexes of the classes ordered by softmax\n",
    "    # (larger first)\n",
    "    idxs = np.argsort(softmax)[::-1]\n",
    "    \n",
    "    # Loop over the classes with the largest softmax\n",
    "    for i in range(5):\n",
    "        # Get softmax value\n",
    "        p = softmax[idxs[i]]\n",
    "    \n",
    "        # Get class name\n",
    "        landmark_name = learn_inf.class_names[idxs[i]]\n",
    "        \n",
    "        labels[i].value = f\"{landmark_name} (prob: {p:.2f})\"\n",
    "\n",
    "\n",
    "# Putting back btn_upload to a widget for next cell\n",
    "btn_upload = FileUpload()\n",
    "\n",
    "btn_run = Button(description=\"Classify\")\n",
    "btn_run.on_click(on_click_classify)\n",
    "\n",
    "labels = []\n",
    "for _ in range(5):\n",
    "    labels.append(Label())\n",
    "\n",
    "out_pl = Output()\n",
    "out_pl.clear_output()\n",
    "\n",
    "wgs = [Label(\"Please upload a picture of a landmark\"), btn_upload, btn_run, out_pl]\n",
    "wgs.extend(labels)\n",
    "\n",
    "VBox(wgs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (optional) Standalone app or web app\n",
    "\n",
    "You can run this notebook as a standalone app on your computer by following these steps:\n",
    "\n",
    "1. Download this notebook in a directory on your machine\n",
    "2. Download the model export (for example, ``checkpoints/transfer_exported.pt``) in a subdirectory called ``checkpoints`` within the directory where you save the app.ipynb notebook\n",
    "3. Install voila if you don't have it already (``pip install voila``)\n",
    "4. Run your app: ``voila app.ipynb --show_tracebacks=True``\n",
    "5. Customize your notebook to make your app prettier and rerun voila\n",
    "\n",
    "You can also deploy this app as a website using Binder: https://voila.readthedocs.io/en/stable/deploy.html#deployment-on-binder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create your submission archive\n",
    "\n",
    "Now that you are done with your project, please run the following cell. It will generate a file containing all the code you have written, as well as the notebooks. Please submit that file to complete your project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "executing: jupyter nbconvert --to html cnn_from_scratch.ipynb\n",
      "[NbConvertApp] Converting notebook cnn_from_scratch.ipynb to html\n",
      "[NbConvertApp] Writing 1062252 bytes to cnn_from_scratch.html\n",
      "executing: jupyter nbconvert --to html app.ipynb\n",
      "[NbConvertApp] Converting notebook app.ipynb to html\n",
      "[NbConvertApp] Writing 286652 bytes to app.html\n",
      "executing: jupyter nbconvert --to html transfer_learning.ipynb\n",
      "[NbConvertApp] Converting notebook transfer_learning.ipynb to html\n",
      "[NbConvertApp] Writing 401746 bytes to transfer_learning.html\n",
      "Adding files to submission_2024-05-19T04h53m.tar.gz\n",
      "src/predictor.py\n",
      "src/model.py\n",
      "src/__init__.py\n",
      "src/helpers.py\n",
      "src/transfer.py\n",
      "src/train.py\n",
      "src/data.py\n",
      "src/optimization.py\n",
      "src/create_submit_pkg.py\n",
      "cnn_from_scratch.ipynb\n",
      "app.ipynb\n",
      "transfer_learning.ipynb\n",
      "app.html\n",
      "transfer_learning.html\n",
      "cnn_from_scratch.html\n",
      "\n",
      "----------------------------------------------------------------\n",
      "Done. Please submit the file submission_2024-05-19T04h53m.tar.gz\n",
      "----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "!python src/create_submit_pkg.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}