Crack_Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Crack Classification with Deep Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deep Learning (DL) is a subset of Machine Learning that uses Neural Network inspired architecture to make predictions. Convolutional Neural Networks (CNN) are a type of DL model that is effective in learning patterns in 2-dimensional data such as images. Images of concrete with and without cracks are used to train a classifier to identify cracks. See [Machine Learning for Engineers course website](https://apmonitor.com/pds) for the source code to [Image Classification: Bits and Cracks](https://apmonitor.com/pds/index.php/Main/BitClassification). The webpage starts with a tutorial on bit classification and then moves to crack classification.\n",
    "\n",
    "<img align=left width=400px src='https://apmonitor.com/pds/uploads/Main/bit_classification.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the following Python modules. Use ```pip``` to install any missing packages. For example, if there is an error: ```ModuleNotFoundError: No module named 'cv2'```, add a new cell and run the following command in another cell: ```pip install opencv-python```. An online search *how to install python cv2* may be needed if the name of the install package isn't the same as the import name. See additional information on [how to install and manage Python packages](https://apmonitor.com/pds/index.php/Main/InstallPythonPackages)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yF35Ku0Ms-O6"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import zipfile\n",
    "import urllib.request\n",
    "import cv2\n",
    "import re\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "from tensorflow.keras.preprocessing import image\n",
    "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Activation, Flatten\n",
    "from tensorflow.keras.layers import Conv2D, MaxPooling2D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pip install opencv-python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Labeled Photos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download photos from [Crack Images (concrete_cracks.zip)](http://apmonitor.com/pds/uploads/Main/concrete_crack.zip). Download and extract the images by either selecting the [link](http://apmonitor.com/pds/uploads/Main/concrete_cracks.zip) or running the code cell below to download and unzip the archive.\n",
    "\n",
    "<img align=left width=400px src='https://apmonitor.com/pds/uploads/Main/concrete_cracks.png'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# download concrete_cracks.zip\n",
    "file = 'concrete_cracks.zip'\n",
    "url = 'http://apmonitor.com/pds/uploads/Main/'+file\n",
    "urllib.request.urlretrieve(url, file)\n",
    "\n",
    "# extract archive and remove concrete_cracks.zip\n",
    "with zipfile.ZipFile(file, 'r') as zip_ref:\n",
    "    zip_ref.extractall('./')\n",
    "os.remove(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The archive contains two folders, a test folder and train folder with subdirectories corresponding to the possible image types (Positive or Negative). The images are found within each subdirectory. The tree structure of the folders is:\n",
    "\n",
    "```\n",
    "  ├───test\n",
    "  │   ├───Positive\n",
    "  │   └───Negative\n",
    "  └───train\n",
    "      ├───Positive\n",
    "      └───Negative\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the photos into the Python session. The first step is to process the images into a format that 1) makes the data readable to the model, and 2) provides more training material for the model to learn. For example, the ```train_processor``` variable scales the data so that it can be a feature (input) for the model, but also takes each images and augments it so that the model can learn from multiple variations of the same picture. It flips it horizontally, rotates it, and shifts it, and more to make sure the model learns from the shape of the crack rather than the orientation or size. \n",
    "\n",
    "**Change:** The original image is 227x277 and the image is rescaled to 128x128 without significant loss in classifier accuracy. The [solution video](https://youtu.be/UotUJuBb0qc) uses 227x227 but there is an error with later versions of TensorFlow that the image pixels must be a multiple of the layer size.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 23726,
     "status": "ok",
     "timestamp": 1618065873113,
     "user": {
      "displayName": "Peter Van Katwyk",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjhknOaA2vSO9txrYBB1U01ff4hmNe3wj5xrjP9Nzg=s64",
      "userId": "16032387266776692701"
     },
     "user_tz": 360
    },
    "id": "topbCwEWsr3q",
    "outputId": "a4116caa-a435-41f2-d2d8-71ca1e307d8c"
   },
   "outputs": [],
   "source": [
    "# Data processing\n",
    "train_processor = ImageDataGenerator(rescale = 1./255, \\\n",
    "    horizontal_flip = True, zoom_range = 0.2,          \\\n",
    "    rotation_range = 10, shear_range = 0.2,            \\\n",
    "    height_shift_range = 0.1, width_shift_range = 0.1)\n",
    "test_processor = ImageDataGenerator(rescale = 1./255)\n",
    "\n",
    "# Load data\n",
    "train = train_processor.flow_from_directory('train',   \\\n",
    "    target_size = (128,128), batch_size = 32,         \\\n",
    "    class_mode = 'categorical', shuffle = True)\n",
    "\n",
    "test = test_processor.flow_from_directory('test',      \\\n",
    "    target_size = (128,128), batch_size = 32,         \\\n",
    "    class_mode = 'categorical', shuffle = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Building"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next step is to build the CNN model with options for building the model. This includes the number of convolutional layers, fully connected dense layers, the number of nodes in each layer, and the number of training epochs. For more information on these parameters and Convolutional Neural Networks in general, see [Computer Vision with Deep Learning](https://apmonitor.com/pds/index.php/Main/VisionDeepLearning).  Change these parameters and analyze the performance of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 3288116,
     "status": "ok",
     "timestamp": 1618026147600,
     "user": {
      "displayName": "Peter Van Katwyk",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjhknOaA2vSO9txrYBB1U01ff4hmNe3wj5xrjP9Nzg=s64",
      "userId": "16032387266776692701"
     },
     "user_tz": 360
    },
    "id": "9-jCm9Y3Ehbs",
    "outputId": "c99cfeb8-dd44-4da7-e0b7-e97ba1455875"
   },
   "outputs": [],
   "source": [
    "# choose model parameters\n",
    "num_conv_layers = 2\n",
    "num_dense_layers = 1\n",
    "layer_size = 16\n",
    "num_training_epochs = 15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initiate model variable\n",
    "model = Sequential()\n",
    "\n",
    "# begin adding properties to model variable\n",
    "# e.g. add a convolutional layer\n",
    "model.add(Conv2D(layer_size, (3,3), input_shape=(128,128,3)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "\n",
    "# add additional convolutional layers based on num_conv_layers\n",
    "for _ in range(num_conv_layers-1):\n",
    "    model.add(Conv2D(layer_size, (3, 3)))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "\n",
    "# reduce dimensionality\n",
    "model.add(Flatten())\n",
    "\n",
    "# add fully connected \"dense\" layers if specified\n",
    "for _ in range(num_dense_layers):\n",
    "    model.add(Dense(layer_size))\n",
    "    model.add(Activation('relu'))\n",
    "\n",
    "# add output layer\n",
    "model.add(Dense(2))\n",
    "model.add(Activation('softmax'))\n",
    "\n",
    "# compile the sequential model with all added properties\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "                optimizer='adam',\n",
    "                metrics=['accuracy'],\n",
    "                )\n",
    "\n",
    "# use the data already loaded previously to train/tune the model\n",
    "model.fit(train,\n",
    "            epochs=num_training_epochs,\n",
    "            validation_data = test)\n",
    "\n",
    "# save the trained model\n",
    "model.save(f'cracks.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model is trained and saved as an `h5` file. The last line of the printed output contains the accuracy for both the training and testing data.\n",
    "\n",
    "```\n",
    "Epoch 14/15\n",
    "2/2 [====] - 3s 2s/step - loss: 0.8785 - accuracy: 0.6202 - val_loss: 0.9091 - val_accuracy: 0.6000\n",
    "Epoch 15/15\n",
    "2/2 [====] - 3s 2s/step - loss: 0.9028 - accuracy: 0.5721 - val_loss: 0.8648 - val_accuracy: 0.6667\n",
    "```\n",
    "\n",
    "The ```val_accuracy``` is the accuracy on the test images that are not included in the training. Hyperparameter optimization can be used to improve the accuracy by adjusting the CNN architecture, training selections, or other parameters. The function ```make_prediction``` takes the file path to a crack photo as an input and produces a classification result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 286
    },
    "executionInfo": {
     "elapsed": 3080,
     "status": "ok",
     "timestamp": 1617992171243,
     "user": {
      "displayName": "Peter Van Katwyk",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GjhknOaA2vSO9txrYBB1U01ff4hmNe3wj5xrjP9Nzg=s64",
      "userId": "16032387266776692701"
     },
     "user_tz": 360
    },
    "id": "Chpl4ZrSPtHc",
    "outputId": "cb9bd334-7945-4a5d-d765-6abd8f0f0eec",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "btype = ['Negative','Positive'] # possible output values\n",
    "def make_prediction(image_fp):\n",
    "    im = cv2.imread(image_fp) # load image\n",
    "    plt.imshow(im)\n",
    "    img = image.load_img(image_fp, target_size = (128,128))\n",
    "    img = image.img_to_array(img)\n",
    "\n",
    "    image_array = img / 255. # scale the image\n",
    "    img_batch = np.expand_dims(image_array, axis = 0)\n",
    "    \n",
    "    predicted_value = btype[model.predict(img_batch).argmax()]\n",
    "    true_value = re.search(r'(Negative)|(Positive)', image_fp)[0]\n",
    "    \n",
    "    out = f\"\"\"Predicted Crack Type: {predicted_value}\n",
    "    True Crack Type: {true_value}\n",
    "    Correct?: {predicted_value == true_value}\"\"\"\n",
    "    \n",
    "    return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# randomly select type (0-1) and image number (1-5)\n",
    "i = random.randint(0,1); j = random.randint(19901,20000)\n",
    "b = btype[i]; im = str(j) + '.jpg'\n",
    "test_image_filepath = r'./test/'+b+'/'+im\n",
    "print(make_prediction(test_image_filepath))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The validation accuracy as well as individual testing shows that there are misclassifications.\n",
    "\n",
    "Here are a few things that can improve the accuracy for this application:  \n",
    "- <b>More photos!</b> This is the most important thing, Machine Learning typically requires many photos that are representative of what the classifiers see. At this point, there are not nearly enough photos for the model to learn each crack type.\n",
    "- <b>Background Clutter.</b> Most of the images in this set have the background removed. To train a classifier to identify crack types in the field, more photos are needed.\n",
    "- <b>Hyperparameter Optimization.</b> To make the best model, the best parameters must be selected to maximize the accuracy (hyperparameter optimization). Packages such as [Hyperopt](http://hyperopt.github.io/hyperopt/) can intelligently try different parameter combinations to increase the accuracy without more data."
   ]
  }
 ],
 "metadata": {
  "accelerator": "TPU",
  "colab": {
   "authorship_tag": "ABX9TyNhoK7FQHGoNgSPJqWd8qt+",
   "collapsed_sections": [],
   "name": "Crack_Classification.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}