Roboflow_EfficientDet_v2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Roboflow-EfficientDet-v2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dA0CmSKv-7y-",
        "colab_type": "text"
      },
      "source": [
        "For the most up to date version of this notebook, please copy from this link\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n",
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1ZmbeTro4SqT7h_TfW63MLdqbrCUk_1br#scrollTo=KwDS9qqBbMQa)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sAs6vn4Rukct",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "\n",
        "# Overview\n",
        "\n",
        "💡 Recommendation: Open this blog post on [how to train EfficientDet](https://towardsdatascience.com/training-efficientdet-object-detection-model-with-a-custom-dataset-25fb0f190555) to continue.\n",
        "\n",
        "In this notebook we show an example of how to train EfficientDet using a pytorch implementation on a custom dataset that has been uploaded through RoboFlow. The example provides a flexible framework, so you can apply it to your own dataset with a custom number of classes and a different objective. We we tackle chess here. \n",
        "\n",
        "![Chess Example](https://i.imgur.com/nkjobw1.png)\n",
        "\n",
        "### **Our Data and Roboflow**\n",
        "\n",
        "Our dataset of 289 chess images (and 2894 annotations!) is hosted publicly on Roboflow [here](https://public.roboflow.ai/object-detection/chess-full). Roboflow also hosts many other public datasets and you can easily upload your own custom dataset for your use case, augment, and export in flexible formats. Our tutorial uses Coco Json, but you might have another format (say tfrecord). No problem! Upload your dataset and we will export it in the required format.\n",
        "\n",
        "### **Model and Training**\n",
        "\n",
        "For a deep dive on the EfficientDet model please see [the paper](https://arxiv.org/abs/1911.09070). For a shorter look, here is a great [blog post](https://towardsdatascience.com/efficientdet-scalable-and-efficient-object-detection-review-4472ffc34fd9)! \n",
        "\n",
        "We use a pytorch implementation of EfficientDet using the [image detection library](https://github.com/roboflow-ai/Monk_Object_Detection) from Tessellate-Imaging for object detection. Our implementation uses the base version of EfficientDet-d0.  We train from the EfficientNet base backbone, without using a pretrained checkpoint for the detector.\n",
        "\n",
        "### **Inference**\n",
        "\n",
        "We witness some fast inference on a few basic examples from our test set to see that our approach is heading in the right direction.\n",
        "\n",
        "### **Export**\n",
        "\n",
        "We export our model weights to google drive for future utilization.\n",
        "\n",
        "### **Next Steps**\n",
        "\n",
        "We will be exploring evaluation on custom RoboFlow datasets and objectives compared to yoloV3, including training time, inference time, model size, and performance. \n",
        "\n",
        "We will also explore comparing performance from the Coco pretrained checkpoint!\n",
        "\n",
        "## **Stay in touch!**\n",
        "\n",
        "If you run into any hurdles on your own data set or just want to share some cool results in your own domain, [reach out to us](roboflow.ai)! \n",
        "\n",
        "#### ![Roboflow Workmark](https://i.imgur.com/WHFqYSJ.png)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8SJBVVCyvObX",
        "colab_type": "text"
      },
      "source": [
        "# Setting up our envionment"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X2EDGrtsba1E",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "93606bc3-6668-4921-9de3-2ccca567ea2d"
      },
      "source": [
        "#our fork of the Tessellate-Imaging image detection library\n",
        "#!rm -rf Monk_Object_Detection\n",
        "! git clone https://github.com/roboflow-ai/Monk_Object_Detection.git"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "fatal: destination path 'Monk_Object_Detection' already exists and is not an empty directory.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EIu1PTPlcAuU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 180
        },
        "outputId": "a89aa291-6055-4c5e-8e24-271500b4a3f6"
      },
      "source": [
        "# For colab use the command below\n",
        "# Set up library requirments\n",
        "! cd Monk_Object_Detection/3_mxrcnn/installation && cat requirements_colab.txt | xargs -n 1 -L 1 pip install"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: mxnet-cu100 in /usr/local/lib/python3.6/dist-packages (1.5.1.post0)\n",
            "Requirement already satisfied: graphviz<0.9.0,>=0.8.1 in /usr/local/lib/python3.6/dist-packages (from mxnet-cu100) (0.8.4)\n",
            "Requirement already satisfied: requests<3,>=2.20.0 in /usr/local/lib/python3.6/dist-packages (from mxnet-cu100) (2.21.0)\n",
            "Requirement already satisfied: numpy<2.0.0,>1.16.0 in /usr/local/lib/python3.6/dist-packages (from mxnet-cu100) (1.18.3)\n",
            "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.20.0->mxnet-cu100) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.20.0->mxnet-cu100) (2020.4.5.1)\n",
            "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.20.0->mxnet-cu100) (2.8)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests<3,>=2.20.0->mxnet-cu100) (3.0.4)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N2YxNg_6ftEC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#fixed version of tqdm output for Colab\n",
        "!pip install --force https://github.com/chengs/tqdm/archive/colab.zip\n",
        "#IGNORE restart runtime warning, it is indeed installed\n",
        "#missing a few extra packages that we will need later! \n",
        "!pip install efficientnet_pytorch\n",
        "!pip install tensorboardX"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8PdgWcjbc0hj",
        "colab_type": "text"
      },
      "source": [
        "# Let's get some data! \n",
        "\n",
        "The best part about Roboflow is the efficient management of your datasets. [Upload you dataset](roboflow.ai) and you will recieve a fresh curl code to ouput it in whatever augmented and annotated format you need. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QK73cnbNcMey",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#fresh curl courtesy of roboflow.ai, outputing our dataset in Coco Json format\n",
        "\n",
        "##THIS CELL NEEDS CHANGED##\n",
        "\n",
        "!curl -L [YOUR LINK HERE] > roboflow.zip; unzip roboflow.zip; rm roboflow.zip"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nspgbsjTdSjZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#let's take a look at our directory\n",
        "#notice the data came down in train, valid, test, splits - this is pre set during the dataset upload process\n",
        "%ls"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4tZCaH-bdUnP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#let's take a peak in train\n",
        "#jpg images and some coco json annotations\n",
        "%ls train"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ArXUsWVwdbdQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#let's take a peek at the annotations\n",
        "#class categories, and bounding boxes are provided for our train set\n",
        "#%pycat train/_annotations.coco.json"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5DtDu-7Hlk_O",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#in the next three cells, we move the data into a structure that the image detection library will be expecting\n",
        "#but no file data manipulation is necessary\n",
        "#images can also be segmented into class folders, but we combine all classes here\n",
        "!mkdir Chess\n",
        "!mkdir Chess/annotations\n",
        "!mkdir Chess/Annotations\n",
        "!mkdir Chess/Images"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3uZzGiw_mrRI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cp train/_annotations.coco.json Chess/annotations/instances_Images.json"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NLzZXkkhrvx4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%cp train/*.jpg Chess/Images/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wBi3RS1_fd4N",
        "colab_type": "text"
      },
      "source": [
        "# Training\n",
        "\n",
        "In this section we set up the efficientDet-d0 model from backbone and train to our custom case"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fvfo6x7Vdw6i",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import sys\n",
        "sys.path.append(\"Monk_Object_Detection/4_efficientdet/lib/\");"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "21o9flNqfP-8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from train_detector import Detector"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c35tC0dCfoky",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gtf = Detector();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aYKeGqpqgvN4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#directs the model towards file structure\n",
        "root_dir = \"./\";\n",
        "coco_dir = \"Chess\";\n",
        "img_dir = \"./\";\n",
        "set_dir = \"Images\";"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T6epk9-HkBwJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#smells like some free compute from Colab, nice\n",
        "gtf.Train_Dataset(root_dir, coco_dir, img_dir, set_dir, batch_size=8, image_size=512, use_gpu=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O0-xLOmisU9P",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gtf.Model();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y3XsPHdxsXsH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gtf.Set_Hyperparams(lr=0.0001, val_interval=1, es_min_delta=0.0, es_patience=0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d7vtQ28fsc9K",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%time\n",
        "gtf.Train(num_epochs=100, model_output_dir=\"trained/\");"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1cj9gst-hRDg",
        "colab_type": "text"
      },
      "source": [
        "# Inference"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZC263x69sfai",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import sys\n",
        "sys.path.append(\"Monk_Object_Detection/4_efficientdet/lib/\");"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IsxWRWL8tcgP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from infer_detector import Infer"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VEuuSjWYte5-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gtf = Infer();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YWbOZk-rthzW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#our trained model weights are in here in onxx format\n",
        "gtf.Model(model_dir=\"trained/\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CH2Tv7gstkP_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#extract class list from our annotations\n",
        "import json\n",
        "with open('train/_annotations.coco.json') as json_file:\n",
        "    data = json.load(json_file)\n",
        "class_list = []\n",
        "for category in data['categories']:\n",
        "  class_list.append(category['name'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qMLy8LIcuRQS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class_list"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Oc87-3NPummz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%time\n",
        "test_images = [f for f in os.listdir('test') if f.endswith('.jpg')]\n",
        "import random\n",
        "img_path = \"test/\" + random.choice(test_images);\n",
        "duration, scores, labels, boxes = gtf.Predict(img_path, class_list, vis_threshold=0.2);"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KbJbgLggvGM_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from IPython.display import Image\n",
        "Image(filename='output.jpg') "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WfyWosXWrbWN",
        "colab_type": "text"
      },
      "source": [
        "# Export Trained Weights"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KwDS9qqBbMQa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#export trained model\n",
        "# mount Google Drive\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WSqXvZhlbfCc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%mkdir trained_export\n",
        "%cp ./trained/signatrix_efficientdet_coco.onnx ./trained_export/signatrix_efficientdet_coco_$(date +%F-%H:%M).onnx\n",
        "%cp ./trained/signatrix_efficientdet_coco.pth ./trained_export/signatrix_efficientdet_coco_$(date +%F-%H:%M).pth\n",
        "%mv ./trained_export/* /content/drive/My\\ Drive/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ieAIXwm11CO",
        "colab_type": "text"
      },
      "source": [
        "# Reloading Trained Weights after Export\n",
        "\n",
        "Imagine you have exported your trained model and would like to reaccess it later. This portion of the notebook picks up the trained model and starts at inference\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fJ5__F0haiqP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#export trained model\n",
        "# mount Google Drive\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nhyNJ7GsatcD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#our fork of the Tessellate-Imaging image detection library\n",
        "#!rm -rf Monk_Object_Detection\n",
        "! git clone https://github.com/roboflow-ai/Monk_Object_Detection.git"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pTVWQohaa2WU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# For colab use the command below\n",
        "# Set up library requirments\n",
        "! cd Monk_Object_Detection/3_mxrcnn/installation && cat requirements_colab.txt | xargs -n 1 -L 1 pip install\n",
        "\n",
        "#fixed version of tqdm output for Colab\n",
        "!pip install --force https://github.com/chengs/tqdm/archive/colab.zip\n",
        "#IGNORE restart runtime warning, it is indeed installed\n",
        "#missing a few extra packages that we will need later! \n",
        "!pip install efficientnet_pytorch\n",
        "!pip install tensorboardX"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_rKXFWlLbZlf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#recover trained weights\n",
        "!mkdir '/trained'\n",
        "#note these are old weights that I didn't train for too long\n",
        "!cp '/content/drive/My Drive/signatrix_efficientdet_coco_2020-04-10-15:18.onnx' '/trained/signatrix_efficientdet_coco.onnx'\n",
        "!cp '/content/drive/My Drive/signatrix_efficientdet_coco_2020-04-10-15:18.pth' '/trained/signatrix_efficientdet_coco.pth'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "maetU4VddJNC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "import sys\n",
        "sys.path.append(\"Monk_Object_Detection/4_efficientdet/lib/\");"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C3Jrco_TdkhU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from infer_detector import Infer\n",
        "gtf = Infer();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GgZKzmI1egUa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#our trained model weights are in here in onxx format\n",
        "gtf.Model(model_dir=\"/trained\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lEksZQC5fMQ2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#download some test data\n",
        "!curl -L [YOUR LINK HERE] | jar -x"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZOOGlzad1Kln",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!ls test"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M0qOyjPh1MOs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#extract class list from our annotations\n",
        "#in your application you will probably already have this saved\n",
        "import json\n",
        "with open('train/_annotations.coco.json') as json_file:\n",
        "    data = json.load(json_file)\n",
        "class_list = []\n",
        "for category in data['categories']:\n",
        "  class_list.append(category['name'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xW8EEb111csu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class_list"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ObmqK4EI1ga0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%%time\n",
        "#bang!\n",
        "img_path = \"test/2f6fb003bb89cd401322a535acb42f65_jpg.rf.49b342a7b1f6de3f0e328beaf094a945.jpg\";\n",
        "duration, scores, labels, boxes = gtf.Predict(img_path, class_list, vis_threshold=0.2);"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AINktQgG1kmc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from IPython.display import Image\n",
        "Image(filename='output.jpg') "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FHIdI0d81m_N",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}