Traffic_Sign_Classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project: Build a Traffic Sign Recognition Program\n",
    "\n",
    "[//]: # (Image References)\n",
    "[step-1-lenet-best]: ./images/step-1-lenet-best.png \"LeNet - best results\"\n",
    "[step-1-lenet-t2]: ./images/step-1-lenet-t2.png \"LeNet - graph with best results\"\n",
    "\n",
    "Overview\n",
    "---\n",
    "\n",
    "In this project we are going to train a network to recognize traffic signs.\\\n",
    "The dataset of traffic signs come from [German Traffic Sign Dataset](http://benchmark.ini.rub.de/?section=gtsrb&subsection=dataset) and this dataset will be used to train the neural network recognizing traffic signs.\\\n",
    "\\\n",
    "To track results of all training experiments I use [Neptune.ml](http://neptune.ml) tool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 0: Load The Data\n",
    "\n",
    "At the beginning we need to load the dataset. Dataset is kept in the folder `data` and it contains following files:\\\n",
    "```bash\n",
    "total 311760\n",
    "-rw-r--r--@ 1 grzegorz.tyminski  staff   38888118 Nov  7  2016 test.p\n",
    "-rw-r--r--@ 1 grzegorz.tyminski  staff  107146452 Feb  2  2017 train.p\n",
    "-rw-r--r--@ 1 grzegorz.tyminski  staff   13578712 Feb  2  2017 valid.p\n",
    "```\n",
    "Class loading this dataset is implemented in the file [traffic_sign_dataset.py](traffic_sign_dataset.py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from traffic_sign_dataset import TrafficData\n",
    "\n",
    "# initiate and load dataset\n",
    "dataset = TrafficData()\n",
    "\n",
    "# normalize dataset --> change values of pixels from 0..255 to 0..1\n",
    "dataset.normalize_data()\n",
    "# randomize the orderd of images in dataset\n",
    "dataset.shuffle_dataset()\n",
    "    \n",
    "X_train, y_train = dataset.get_training_dataset()\n",
    "X_valid, y_valid = dataset.get_validation_dataset()\n",
    "X_test, y_test = dataset.get_testing_dataset()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Number of training examples\n",
    "n_train = len(X_train)\n",
    "\n",
    "# TODO: Number of validation examples\n",
    "n_validation = len(X_valid)\n",
    "\n",
    "# TODO: Number of testing examples.\n",
    "n_test = len(X_test)\n",
    "\n",
    "# TODO: What's the shape of an traffic sign image?\n",
    "image_shape = [32, 32, 3]\n",
    "\n",
    "# TODO: How many unique classes/labels there are in the dataset.\n",
    "n_classes = 43\n",
    "\n",
    "print(\"Number of training examples =\", n_train)\n",
    "print(\"Number of testing examples =\", n_test)\n",
    "print(\"Image data shape =\", image_shape)\n",
    "print(\"Number of classes =\", n_classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.preview_random()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(figsize=(16,4))\n",
    "plt.title(\"Number of images per class in dataset\")\n",
    "\n",
    "plt.hist(y_train,bins = n_classes, alpha=0.5, label = 'train')\n",
    "plt.hist(y_test,bins = n_classes, alpha=0.5, label='test')\n",
    "plt.hist(y_valid,bins = n_classes, alpha=0.5, label='valid')\n",
    "\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(dataset.label_for(0))\n",
    "print(dataset.label_for(7))\n",
    "print(dataset.label_for(16))\n",
    "print(dataset.label_for(19))\n",
    "print(dataset.label_for(24))\n",
    "print(dataset.label_for(27))\n",
    "print(dataset.label_for(29))\n",
    "print(dataset.label_for(32))\n",
    "print(dataset.label_for(37))\n",
    "print(dataset.label_for(39))\n",
    "print(dataset.label_for(41))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Train same LeNet as in MNIST example (v1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from LeNet import LeNet\n",
    "import tensorflow as tf\n",
    "\n",
    "lenet = LeNet(output_classes=43)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lenet.set_hiperparams(epochs=10, batch_size=64, learn_rate=0.002)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lenet.train(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lenet.model_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Train modified LeNet with dropouts (v2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from LeNet2 import LeNet2\n",
    "import tensorflow as tf\n",
    "\n",
    "lenet = LeNet2(output_classes=43)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lenet.set_hiperparams(epochs=10, batch_size=64, learn_rate=0.002, dropout_val=0.25)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lenet.train(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lenet.model_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Improve dataset normalization\n",
    "\n",
    "- improve brightness\n",
    "- to grayscale\n",
    "\n",
    "... and training __with__ Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from traffic_sign_dataset import TrafficData\n",
    "\n",
    "# initiate and load dataset\n",
    "dataset = TrafficData()\n",
    "\n",
    "# normalize dataset --> change values of pixels from 0..255 to 0..1\n",
    "dataset.normalize_data(brightness=True, grayscale=True)\n",
    "# randomize the orderd of images in dataset\n",
    "dataset.shuffle_dataset()\n",
    "    \n",
    "X_train, y_train = dataset.get_training_dataset()\n",
    "X_valid, y_valid = dataset.get_validation_dataset()\n",
    "X_test, y_test = dataset.get_testing_dataset()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.preview_random(grayscale=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(figsize=(16,4))\n",
    "plt.title(\"Traffic Signs classes' distribution\")\n",
    "\n",
    "plt.hist(y_train,bins = n_classes, alpha=0.5, label = 'train')\n",
    "plt.hist(y_test,bins = n_classes, alpha=0.5, label='test')\n",
    "plt.hist(y_valid,bins = n_classes, alpha=0.5, label='valid')\n",
    "\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from LeNet3 import LeNet3\n",
    "import tensorflow as tf\n",
    "\n",
    "lenet = LeNet3(output_classes=43)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lenet.set_hiperparams(epochs=15, batch_size=64, learn_rate=0.002, dropout_val=1.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lenet.train(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lenet.model_summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Dataset with normalization as before\n",
    "\n",
    "... but training __without__ Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from LeNet4 import LeNet4\n",
    "import tensorflow as tf\n",
    "\n",
    "lenet = LeNet4(output_classes=43, channels=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lenet.set_hiperparams(epochs=10, batch_size=64, learn_rate=0.002)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lenet.train(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lenet.model_summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from traffic_sign_dataset import TrafficData\n",
    "\n",
    "# initiate and load dataset\n",
    "dataset = TrafficData()\n",
    "\n",
    "# normalize dataset --> change values of pixels from 0..255 to 0..1\n",
    "dataset.normalize_data(brightness=True, grayscale=True)\n",
    "# randomize the orderd of images in dataset\n",
    "dataset.shuffle_dataset()\n",
    "    \n",
    "X_train, y_train = dataset.get_training_dataset()\n",
    "X_valid, y_valid = dataset.get_validation_dataset()\n",
    "X_test, y_test = dataset.get_testing_dataset()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from LeNet4 import LeNet4\n",
    "import tensorflow as tf\n",
    "\n",
    "lenet = LeNet4(output_classes=43, channels=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lenet.set_hiperparams(epochs=10, batch_size=4, learn_rate=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lenet.train(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's check on `test` dataset where the model has problems to recognize correctly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "n_classes = 43\n",
    "\n",
    "result = lenet.predict(X_test)\n",
    "counter = 0\n",
    "\n",
    "wrong = [list({}) for i in range(n_classes)]\n",
    "\n",
    "for ind in range(len(X_test)):\n",
    "    image = X_test[ind]\n",
    "    label = dataset.label_for(y_test[ind])\n",
    "    pred_label = result[ind]\n",
    "    if pred_label != label:\n",
    "        counter = counter+1\n",
    "        cls_no = y_test[ind]\n",
    "\n",
    "        cls_list = wrong[cls_no]\n",
    "        if cls_list is None:\n",
    "            cls_list = []\n",
    "        cls_list.append({ind, pred_label})\n",
    "        wrong[cls_no] = cls_list\n",
    "        # print(ind, ',', result[ind], ',', label)\n",
    "print('wrong predictions total - ', counter)\n",
    "print()\n",
    "print('wrong predictions per class')\n",
    "for cls_no in range(n_classes):\n",
    "    label = dataset.label_for(y_test[cls_no])\n",
    "    counter = len(wrong[cls_no])\n",
    "    print(label, \"=\", counter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5 - augment dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from traffic_sign_dataset import TrafficData\n",
    "\n",
    "# initiate and load dataset\n",
    "dataset = TrafficData()\n",
    "# augment dataset to get 2x bigger dataset\n",
    "dataset.augment_dataset()\n",
    "# augment dataset again to get 4x bigger dataset\n",
    "dataset.augment_dataset()\n",
    "\n",
    "# normalize dataset --> change values of pixels from 0..255 to 0..1\n",
    "dataset.normalize_data(brightness=False, grayscale=True)\n",
    "# randomize the orderd of images in dataset\n",
    "dataset.shuffle_dataset()\n",
    "    \n",
    "X_train, y_train = dataset.get_training_dataset()\n",
    "X_valid, y_valid = dataset.get_validation_dataset()\n",
    "X_test, y_test = dataset.get_testing_dataset()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from LeNet4 import LeNet4\n",
    "import tensorflow as tf\n",
    "\n",
    "lenet = LeNet4(output_classes=43, channels=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lenet.set_hiperparams(epochs=10, batch_size=32, learn_rate=0.002)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for batch_size in [8, 16, 32, 64]:\n",
    "    for learn_rate in [0.001, 0.002]:\n",
    "        lenet.set_hiperparams(epochs=10, batch_size=batch_size, learn_rate=learn_rate)\n",
    "        lenet.train(dataset, neptune_tags=['augmentation'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------\n",
    "\n",
    "## Step 6 - verify model on random images from internet\n",
    "\n",
    "Let's first download some images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.image as mpimg\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from scipy.special import softmax\n",
    "import pprint\n",
    "from traffic_sign_dataset import TrafficData\n",
    "from graph import Graph\n",
    "dataset = TrafficData()\n",
    "\n",
    "for index in range(1, 11, 1):\n",
    "    print(\"------------------------------\")\n",
    "    image_name = f\"input/sign_{index}.png\"\n",
    "    orig_image = mpimg.imread(image_name)\n",
    "    image = dataset.__normalize_image__(orig_image, brightness=True, grayscale=True)\n",
    "\n",
    "    result = lenet.predict_image(image)\n",
    "    result_softmax = softmax(result)\n",
    "    #print(np.max(result_softmax))\n",
    "    result_max = np.argmax(result)\n",
    "    result_sorted = np.argsort(result)[:,::-1]    \n",
    "    result_top3 = result_sorted[:,:3]\n",
    "    #print(result_softmax)\n",
    "\n",
    "    print(f\"File {image_name}\")\n",
    "    plt.figure(figsize=(2,2))\n",
    "    plt.imshow(orig_image, cmap=\"gray\")\n",
    "    plt.show()\n",
    "\n",
    "    for i,classid in enumerate(result_top3[0]):\n",
    "        print(f\"TOP{i}(prob={result_softmax[0,classid]:0.2f}): {dataset.label_for(classid)} (classid={classid}) \")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "def preview_class(classid: int = 1, max_cells: int = 10):\n",
    "    max_cells = 10\n",
    "    indexes = np.where(y_train == classid)[0]\n",
    "    indexes = random.sample(set(indexes), max_cells)\n",
    "    f, axes = plt.subplots(1, max_cells, sharey=True, figsize=(12,12))\n",
    "    counter = 0\n",
    "    for index in indexes:\n",
    "        if counter < max_cells:\n",
    "            image = X_train[index]\n",
    "            img = np.reshape(image, [32, 32])\n",
    "            axes[counter].axis('off')\n",
    "            axes[counter].imshow(img, cmap='gray')\n",
    "            counter = counter + 1\n",
    "        elif counter == max_cells:\n",
    "            pass\n",
    "        \n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "    pass\n",
    "\n",
    "preview_class(16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}