Training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finery Image Recognition/Classification\n",
    "\n",
    "This is the code for loading images, training an transfered image recognition model, testing on single images, and also testing on batch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Dependencies\n",
    "\n",
    "We will need tensorflow, keras and numpy for this notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import keras\n",
    "import numpy as np\n",
    "\n",
    "from keras import backend as K\n",
    "from keras.preprocessing import image\n",
    "from keras.applications import resnet50\n",
    "from keras.applications.imagenet_utils import preprocess_input, decode_predictions\n",
    "from keras.layers import Dense,GlobalAveragePooling2D\n",
    "from keras.models import Model\n",
    "from keras.callbacks import EarlyStopping, ModelCheckpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checking if GPU is recognized by tensorflow\n",
    "\n",
    "if GPU present and recognized, it will show: `\"/device:GPU:0\"` in the output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.python.client import device_lib\n",
    "print(device_lib.list_local_devices())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Limit GPU memory to 30% capacity per process, setting too high might hang the program, or may crash the machine!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.backend.tensorflow_backend import set_session\n",
    "config = tf.ConfigProto()\n",
    "config.gpu_options.per_process_gpu_memory_fraction = 0.3\n",
    "set_session(tf.Session(config=config))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the training and testing data using ImageDataGenerator\n",
    "\n",
    "*If you made changes to the images' directory, here is the place to update them*\n",
    "\n",
    "General structure of the images:\n",
    "\n",
    "`./images/train/110/...jpg\n",
    "    |           120\n",
    "    |           130\n",
    "    |           140\n",
    "    |           150\n",
    "    |           160\n",
    "    |           ...\n",
    "    ----->/test/110/...jpg\n",
    "                120\n",
    "                130\n",
    "                140\n",
    "                150\n",
    "                160\n",
    "                ...`\n",
    "\n",
    "You can also specify the rotation of images (for improved feature extraction), and different preprocess to the images at this step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_datagen = image.ImageDataGenerator(preprocessing_function=preprocess_input) # define a generator\n",
    "train_generator = train_datagen.flow_from_directory('images/train/', # the training image file path\n",
    "                                                 target_size=(224,224), # resnet takes input of size 244 by 244\n",
    "                                                 color_mode='rgb', # top-less resnet takes input size of (3,244,244)\n",
    "                                                 batch_size=32, # batch size for loading by generator\n",
    "                                                 class_mode='categorical',\n",
    "                                                 shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_datagen = image.ImageDataGenerator(preprocessing_function=preprocess_input)\n",
    "test_generator = test_datagen.flow_from_directory('images/test/',\n",
    "                                                 target_size=(224,224),\n",
    "                                                 color_mode='rgb',\n",
    "                                                 batch_size=32,\n",
    "                                                 class_mode='categorical',\n",
    "                                                 shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These creates the training data as a generator class. From this you can implement preprocessing to the training and testing images, such as resizing, rotation, labeling, etc.\n",
    "\n",
    "It is a good practice to check that both generator has the same number of categories."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load a \"top-less\" ResNet 50 model\n",
    "\n",
    "This command will load a resnet 50 model without the final prediction layer, allowing us to append our own prediction layer at the end of the convolutional layer output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resnet_model = resnet50.ResNet50(weights='imagenet', include_top=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make a few layers of fully connected layer that attaches to resnet model output\n",
    "\n",
    "The purpose of these layers is to serve as prediciton layers to train our image on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = resnet_model.output # make a new layer that attaches to the resnet model output\n",
    "x = GlobalAveragePooling2D()(x) # flattens the output using global average\n",
    "x = Dense(1024, activation='relu')(x)\n",
    "x = Dense(512, activation='relu')(x)\n",
    "final = Dense(12, activation='softmax')(x) # output layer, make sure the number\n",
    "                                           # is same # of classes from above block\n",
    "\n",
    "model=Model(inputs=resnet_model.input,outputs=final) # model takes input of resnet and output from our final layer,\n",
    "                                                     # there is a function in keras that prints the layers of the\n",
    "                                                     # model in a clean and concise way, have to check"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check the model structure, Locate the newly created layers\n",
    "\n",
    "For **ResNet**, layer 1 to 174 are supposed to be frozen and layers from 174 on are supposed to be set to trainable, depends on how many layers we created in the previous step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i,layer in enumerate(model.layers):\n",
    "  print(i,layer.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, the following code shows a pretty model structure, input, output size and number of parameters of the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set the convolutional layers not trainable, retaining their weights, Force the newly created layers to be trainable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for layer in model.layers[:174]: # freeze layer 0 to 174\n",
    "    layer.trainable=False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for layer in model.layers[174:]: # make the layers we created trainable\n",
    "    layer.trainable=True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defined Metrics:\n",
    "\n",
    "We define these metrics so that we can use them at the callback stage for each epoch.\n",
    "\n",
    "### AUC, area under curve, area under ROC curve\n",
    "\n",
    "AUC is an appropriate measure of model accuracy for binary classifications. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def auc(y_true, y_pred):\n",
    "     auc = tf.metrics.auc(y_true, y_pred)[1]\n",
    "     K.get_session().run(tf.local_variables_initializer())\n",
    "     return auc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $F_1$ Score\n",
    "\n",
    "$F_1$ score is the harmonic average of precision and recall, another metric for accuracy.\n",
    "\n",
    "It is defined as: $F_1 = 2 \\cdot \\frac{\\text{precision} \\cdot \\text{recall}}{\\text{precision} + \\text{recall}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f1(y_true, y_pred):\n",
    "     f1 = tf.contrib.metrics.f1_score(y_true, y_pred)[1]\n",
    "     K.get_session().run(tf.local_variables_initializer())\n",
    "     return f1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top One Accuracy\n",
    "\n",
    "This is a built-in metric for categorical classification in Keras. This is essentially the same as accuracy but we specify k=1 so that it calculates the micro-average accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def top_1(x,y):\n",
    "    return keras.metrics.top_k_categorical_accuracy(x,y,k=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top Three Accuracy\n",
    "\n",
    "This is a built-in metric for categorical classification in Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def top_3(x,y):\n",
    "    return keras.metrics.top_k_categorical_accuracy(x,y,k=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compile the model proper optimizers, loss function, and metric\n",
    "\n",
    "More experiments can be done on this part, we are just choosing the optimizers, loss and metrics that, through testing, makes the most sense here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer='Adam',loss='categorical_crossentropy',metrics=[top_1,top_3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate penalizing weights using negative frequencies.\n",
    "\n",
    "Negative frequencies $W_c$ is defined as the log smoothed ratio of the total number of samples $N$ to number of samples in specific category $n_c$:\n",
    "\n",
    "$W_c = \\log \\left( \\frac{N}{n_c} \\right) \\text{if } > 1 \\text{, else }1$.\n",
    "\n",
    "Which means if this smoothed ratio is less than 1, the weight for that particular category will be 1, else being the calculated ratio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cw ={0:1, 1:1, 2:1.9659, 3:1.168, 4:1.339, 5:1.3629, 6:2.3743, 7:1, 8:1, 9:1.0239, 10:1, 11:1.3792}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define callback precedures after each epoch end\n",
    "\n",
    "Implement early stopping by using the validation set's loss as a stopping metric, `patiance=2` means stopping after two epoch if the loss does not decrease. This is quite strict and might require tuning when sample size is large or undersampling is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "early = EarlyStopping(monitor=\"val_loss\", mode=\"min\", patience=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement checkpoint by monitoring the validation set's loss, only saves the best model achieve to avoid occupying the hard drive too much.\n",
    "\n",
    "This callback also creates the model structure and weights file in `*.hdf5`, we can load this weight directly without training again later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint = ModelCheckpoint(filepath='best.hdf5', monitor='val_loss', save_best_only=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the batch size of training\n",
    "\n",
    "Here we use the total number of samples divide by the batch size of the image generator. Change this when sample size is large, otherwise will hang."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "step_size_train=train_generator.n//train_generator.batch_size\n",
    "step_size_test=test_generator.n//test_generator.batch_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit the image data on the model using GPU\n",
    "\n",
    "Here we use the test set as the validation set. Not only do we see test accuracy after every epoch, we also can have early stopping so the model won't overfit. Use `epoch=20` since we want the model to be fully trained without stopping too early. However from our experience, the training usually stops at epoch 4 to 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.device('/gpu:0'):\n",
    "    model.fit_generator(generator=train_generator,\n",
    "                        validation_data=test_generator,\n",
    "                        steps_per_epoch=step_size_train,\n",
    "                        validation_steps=step_size_test,\n",
    "                        epochs=20,\n",
    "                        class_weight=cw,\n",
    "                        callbacks=[early, checkpoint])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## To see accuracy of test data in batch\n",
    "\n",
    "There will be several outputs, the first will always be loss, the next few will be the metric you defined in order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.evaluate_generator(generator=test_generator, steps=step_size_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "# Model using VGG16:\n",
    "\n",
    "Same structure just different conv. layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.applications import vgg16\n",
    "vgg_model = vgg16.VGG16(weights='imagenet', include_top=False)\n",
    "\n",
    "x = vgg_model.output\n",
    "x = GlobalAveragePooling2D()(x)\n",
    "x = Dense(1024, activation='relu')(x)\n",
    "x = Dense(512, activation='relu')(x)\n",
    "final = Dense(12, activation='softmax')(x)\n",
    "model2=Model(inputs=vgg_model.input,outputs=final)\n",
    "\n",
    "for layer in model2.layers[:18]:\n",
    "    layer.trainable=False\n",
    "for layer in model2.layers[18:]:\n",
    "    layer.trainable=True\n",
    "    \n",
    "model2.compile(optimizer='Adam',loss='categorical_crossentropy',metrics=[top_1,top_3])\n",
    "\n",
    "checkpoint2 = ModelCheckpoint(filepath='best_vgg.hdf5', monitor='val_loss', save_best_only=True)\n",
    "\n",
    "with tf.device('/gpu:0'):\n",
    "    model2.fit_generator(generator=train_generator,\n",
    "                         validation_data=test_generator,\n",
    "                         steps_per_epoch=step_size_train,\n",
    "                         validation_steps=step_size_test,\n",
    "                         epochs=20,\n",
    "                         class_weight=cw,\n",
    "                         callbacks=[early, checkpoint2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "# Binary classification case >>misleading results<<\n",
    "\n",
    "Images in train and test will be put in `non-wardrobe` and `wardrove` folders."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_datagen = image.ImageDataGenerator(preprocessing_function=preprocess_input)\n",
    "train_generator = train_datagen.flow_from_directory('hot_dog/train/',\n",
    "                                                 target_size=(224,224),\n",
    "                                                 color_mode='rgb',\n",
    "                                                 batch_size=32,\n",
    "                                                 class_mode='categorical',\n",
    "                                                 shuffle=True)\n",
    "test_datagen = image.ImageDataGenerator(preprocessing_function=preprocess_input)\n",
    "test_generator = test_datagen.flow_from_directory('hot_dog/test/',\n",
    "                                                 target_size=(224,224),\n",
    "                                                 color_mode='rgb',\n",
    "                                                 batch_size=32,\n",
    "                                                 class_mode='categorical',\n",
    "                                                 shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = resnet_model.output\n",
    "x = GlobalAveragePooling2D()(x)\n",
    "x = Dense(1024, activation='relu')(x)\n",
    "x = Dense(512, activation='relu')(x)\n",
    "final = Dense(2, activation='softmax')(x)\n",
    "model3=Model(inputs=resnet_model.input,outputs=final)\n",
    "\n",
    "for layer in model3.layers[:174]:\n",
    "    layer.trainable=False\n",
    "for layer in model3.layers[174:]:\n",
    "    layer.trainable=True\n",
    "    \n",
    "model3.compile(optimizer='Adam',loss='categorical_crossentropy',metrics=[auc,f1])\n",
    "\n",
    "cw ={0:1.3792, 1:1}\n",
    "\n",
    "checkpoint_hotdog = ModelCheckpoint(filepath='best_hotdog.hdf5', monitor='val_loss', save_best_only=True)\n",
    "\n",
    "step_size_train = train_generator.n//train_generator.batch_size\n",
    "step_size_test = test_generator.n//test_generator.batch_size\n",
    "\n",
    "with tf.device('/gpu:0'):\n",
    "    model.fit_generator(generator=train_generator,\n",
    "                        validation_data=test_generator,\n",
    "                        steps_per_epoch=step_size_train,\n",
    "                        validation_steps=step_size_test,\n",
    "                        epochs=20,\n",
    "                        class_weight=cw,\n",
    "                        callbacks=[early, checkpoint_hotdog])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "$\\space$\n",
    "\n",
    "# Classifying on all 57 categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_datagen = image.ImageDataGenerator(preprocessing_function=preprocess_input)\n",
    "train_generator = train_datagen.flow_from_directory('images/train/', \n",
    "                                                 target_size=(224,224),\n",
    "                                                 color_mode='rgb',\n",
    "                                                 batch_size=32,\n",
    "                                                 class_mode='categorical',\n",
    "                                                 shuffle=True)\n",
    "\n",
    "test_datagen = image.ImageDataGenerator(preprocessing_function=preprocess_input)\n",
    "test_generator = test_datagen.flow_from_directory('images/test/',\n",
    "                                                 target_size=(224,224),\n",
    "                                                 color_mode='rgb',\n",
    "                                                 batch_size=32,\n",
    "                                                 class_mode='categorical',\n",
    "                                                 shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = resnet_model.output\n",
    "x = GlobalAveragePooling2D()(x)\n",
    "x = Dense(1024, activation='relu')(x)\n",
    "x = Dense(512, activation='relu')(x)\n",
    "final = Dense(57, activation='softmax')(x)\n",
    "model = Model(inputs=resnet_model.input,outputs=final)\n",
    "\n",
    "for layer in model5.layers[:174]:\n",
    "    layer.trainable=False\n",
    "for layer in model5.layers[174:]:\n",
    "    layer.trainable=True\n",
    "    \n",
    "model.compile(optimizer='Adam',loss='categorical_crossentropy',metrics=[top_1,top_3])\n",
    "\n",
    "checkpoint5 = ModelCheckpoint(filepath='best_resnet_cat.hdf5', monitor='val_loss', save_best_only=True)\n",
    "\n",
    "cw = {0:1.50131362, 1:1.1820588, 2:1.53955363, 3:1.74326729, 4:1.48907916,\n",
    "      5:1.80859256, 6:1.81749496, 7:1.88701917, 8:1.63040832, 9:1.53752579,\n",
    "      10:2.0139732, 11:3.06765806, 12:2.33781149, 13:2.23246831, 14:2.24604111,\n",
    "      15:1.20684739, 16:3.02419236, 17:2.82789772, 18:3.11596274, 19:2.06311043,\n",
    "      20:2.14501453, 21:1.74543876, 22:2.3129879, 23:2.79010916, 24:2.11852495,\n",
    "      25:1.69416532, 26:2.18504366, 27:3.34641166, 28:1.75870069, 29:3.26723041,\n",
    "      30:3.8692904, 31:3.50131362, 32:2.38737027, 33:1.38594088, 34:1.30108868,\n",
    "      35:1.71700206, 36:1.5837331, 37:1.61161183, 38:1.5807431 , 39:1.78531028,\n",
    "      40:1.85786094, 41:1.19259679, 42:2.05415559, 43:1.99422914, 44:1.75091544,\n",
    "      45:2.00995193, 46:1.2827031, 47:2.13958578, 48:2.05194543, 49:2.0346578,\n",
    "      50:2.85504997, 51:2.1703204, 52:2.65621558, 53:2.89925363, 54:1,\n",
    "      55:3.06765806, 56:1.36008788}\n",
    "\n",
    "step_size_train = train_generator.n//train_generator.batch_size\n",
    "step_size_test = test_generator.n//test_generator.batch_size\n",
    "\n",
    "with tf.device('/gpu:0'):\n",
    "    model.fit_generator(generator=train_generator,\n",
    "                        validation_data=test_generator,\n",
    "                        steps_per_epoch=step_size_train,\n",
    "                        validation_steps=step_size_test,\n",
    "                        epochs=100,\n",
    "                        class_weight=cw,\n",
    "                        callbacks=[early, checkpoint5])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}