person_for_the_blog.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6lWk3Y17JlZ0"
   },
   "source": [
    "# PersonAttributes Classifier - training a multi-task neural network to detect attributes like age, gender, ..., emotion "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vfWl-VHrKSJY"
   },
   "source": [
    "The goal of this challenge is to detect a person's `gender, age, weight, carryingbag, footwear, emotion, bodypose` as well as the `imagequality` all from a single image. Any neural network architecture is allowed, but the **training has to be done from scratch, that means no pre-trained weights, and no transfer learning.**<br>\n",
    "To complete this successfully we will require a convolutional neural network with multiple outputs, each output catering to a prediction.<br> For this challenge I ended up using a [**Densenet121**](https://towardsdatascience.com/understanding-and-visualizing-densenets-7f688092391a) architecture, but any modern parallel network with skip connections should provide good results.<br> The training is done using **Keras**. I also tried **one-cycle policy and cyclic LR in tensorflow**, but found the keras solution simple and effective. <br>\n",
    "Before we take a look at the proposed network architecture let's take a look at the dataset we are dealing with."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jgHABCjmly9k"
   },
   "source": [
    "You can download the dataset from [person-data-gdrive](https://drive.google.com/file/d/1Abe336Tzvi8BC6bVGTN3unlAtj1XdiTT/view) for experimentation.<br> Also find the [person-classifier-github-link](https://github.com/baronrogers5/person-classifier) which contains the notebook that can directly be run on [google-colab](https://colab.research.google.com/).<br>\n",
    "If you are running the notebook on colab, just make sure to download the data and put it in your google drive's `My Drive` as `hvc_data.zip`. <br>\n",
    "All the required files visualized below are present in the `hvc_data.zip` file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "O0BRp8TUKSHb"
   },
   "source": [
    "## Visualizing the data\n",
    "\n",
    "Each image is labelled according to the dataframe given below:\n",
    "\n",
    "![person-df](https://raw.githubusercontent.com/baronrogers5/person-classifier/master/images/df_head.png)\n",
    "\n",
    "Some examples of the images we have.\n",
    "\n",
    "![raw-data](https://raw.githubusercontent.com/baronrogers5/person-classifier/master/images/display_images.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "N0xxOwMngPdB"
   },
   "source": [
    "Let's see all possible predictions for each category\n",
    "\n",
    "```\n",
    "gender\n",
    "--------------------\n",
    "['male' 'female']\n",
    "\n",
    "imagequality\n",
    "--------------------\n",
    "['Average' 'Good' 'Bad']\n",
    "\n",
    "age\n",
    "--------------------\n",
    "['35-45' '45-55' '25-35' '15-25' '55+']\n",
    "\n",
    "weight\n",
    "--------------------\n",
    "['normal-healthy' 'over-weight' 'slightly-overweight' 'underweight']\n",
    "\n",
    "carryingbag\n",
    "--------------------\n",
    "['Grocery/Home/Plastic Bag' 'None' 'Daily/Office/Work Bag']\n",
    "\n",
    "footwear\n",
    "--------------------\n",
    "['Normal' 'CantSee' 'Fancy']\n",
    "\n",
    "emotion\n",
    "--------------------\n",
    "['Neutral' 'Angry/Serious' 'Happy' 'Sad']\n",
    "\n",
    "bodypose\n",
    "--------------------\n",
    "['Front-Frontish' 'Side' 'Back']\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "KJdpaIn8EmIM"
   },
   "source": [
    "## Viewing the network's head\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JwD1ePkNuqr6"
   },
   "source": [
    "\n",
    "As promised let's have a look at the head (where the output's arrive) of the network, so that we can understand the code better.<br> Some connections are not-visible to post a clearer image. Find the full network image [here](https://github.com/baronrogers5/person-classifier/blob/master/images/model.png).<br><br>\n",
    "\n",
    "![model-head](https://raw.githubusercontent.com/baronrogers5/person-classifier/master/images/model_head.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "l81fiTTEgPaV"
   },
   "source": [
    "Now that we have an idea about the problem at hand, let's start with the actual steps required to train an awesome classifier from scratch.\n",
    "\n",
    "- Data Preprocessing -\n",
    "    - Converting to one-hot encoded.\n",
    "    - Normalizing using the mean and std-dev.\n",
    "    - Data Augmentation([cutout](https://github.com/yu4u/cutout-random-erasing))\n",
    "    - Building a keras Sequence to feed to the training loop.\n",
    "- Designing the architecture -\n",
    "    - Choosing a backbone\n",
    "    - Constructing the tower.\n",
    "    - Constructing heads [final predictions] for each class.\n",
    "- Defining the training callbacks\n",
    "- Actual training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ntf52-1J2LKh"
   },
   "source": [
    "## Data PreProcessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pCfkFR42Dz5q"
   },
   "source": [
    "### Converting to one-hot encoded\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XJgKTbl80vQR"
   },
   "source": [
    "\n",
    "Neural networks expect the prediction labels to be one-hot encoded. So that the predicted probabilities [equal to the number of classes] can be directly matched against the correct labels.\n",
    "\n",
    "Hence, we need to convert all our prediction labels to one-hot encoded *with a prefix*, so that it becomes easier to identify them.\n",
    "\n",
    "```python\n",
    "one_hot_df = pd.concat([\n",
    "    df[[\"image_path\"]],\n",
    "    pd.get_dummies(df.gender, prefix=\"gender\"),\n",
    "    pd.get_dummies(df.imagequality, prefix=\"imagequality\"),\n",
    "    pd.get_dummies(df.age, prefix=\"age\"),\n",
    "    pd.get_dummies(df.weight, prefix=\"weight\"),\n",
    "    pd.get_dummies(df.carryingbag, prefix=\"carryingbag\"),\n",
    "    pd.get_dummies(df.footwear, prefix=\"footwear\"),\n",
    "    pd.get_dummies(df.emotion, prefix=\"emotion\"),\n",
    "    pd.get_dummies(df.bodypose, prefix=\"bodypose\"),\n",
    "], axis = 1)\n",
    "\n",
    "df.shape\n",
    "```\n",
    "##### (13573, 9)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DGpSPJZI2ASC"
   },
   "source": [
    "The one-hot encoded data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 917
    },
    "colab_type": "code",
    "id": "iIhWQvhA187a",
    "outputId": "9382193e-74d9-4808-fe21-5ee189bdf96e"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>image_path</th>\n",
       "      <td>resized/1.jpg</td>\n",
       "      <td>resized/2.jpg</td>\n",
       "      <td>resized/3.jpg</td>\n",
       "      <td>resized/4.jpg</td>\n",
       "      <td>resized/5.jpg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gender_female</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gender_male</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imagequality_Average</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imagequality_Bad</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imagequality_Good</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age_15-25</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age_25-35</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age_35-45</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age_45-55</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age_55+</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight_normal-healthy</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight_over-weight</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight_slightly-overweight</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>weight_underweight</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>carryingbag_Daily/Office/Work Bag</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>carryingbag_Grocery/Home/Plastic Bag</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>carryingbag_None</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>footwear_CantSee</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>footwear_Fancy</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>footwear_Normal</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>emotion_Angry/Serious</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>emotion_Happy</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>emotion_Neutral</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>emotion_Sad</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bodypose_Back</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bodypose_Front-Frontish</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bodypose_Side</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  0  ...              4\n",
       "image_path                            resized/1.jpg  ...  resized/5.jpg\n",
       "gender_female                                     0  ...              1\n",
       "gender_male                                       1  ...              0\n",
       "imagequality_Average                              1  ...              0\n",
       "imagequality_Bad                                  0  ...              0\n",
       "imagequality_Good                                 0  ...              1\n",
       "age_15-25                                         0  ...              0\n",
       "age_25-35                                         0  ...              0\n",
       "age_35-45                                         1  ...              1\n",
       "age_45-55                                         0  ...              0\n",
       "age_55+                                           0  ...              0\n",
       "weight_normal-healthy                             1  ...              0\n",
       "weight_over-weight                                0  ...              0\n",
       "weight_slightly-overweight                        0  ...              1\n",
       "weight_underweight                                0  ...              0\n",
       "carryingbag_Daily/Office/Work Bag                 0  ...              0\n",
       "carryingbag_Grocery/Home/Plastic Bag              1  ...              0\n",
       "carryingbag_None                                  0  ...              1\n",
       "footwear_CantSee                                  0  ...              1\n",
       "footwear_Fancy                                    0  ...              0\n",
       "footwear_Normal                                   1  ...              0\n",
       "emotion_Angry/Serious                             0  ...              0\n",
       "emotion_Happy                                     0  ...              0\n",
       "emotion_Neutral                                   1  ...              1\n",
       "emotion_Sad                                       0  ...              0\n",
       "bodypose_Back                                     0  ...              0\n",
       "bodypose_Front-Frontish                           1  ...              1\n",
       "bodypose_Side                                     0  ...              0\n",
       "\n",
       "[28 rows x 5 columns]"
      ]
     },
     "execution_count": 108,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "one_hot_df.head().T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WWgGJH9qrmE5"
   },
   "source": [
    "### Normalizing using the mean and std-*dev*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-Lt-9_lArmBm"
   },
   "source": [
    "```python\n",
    "images = np.array([cv2.resize(io.imread(p), (200, 200)) for p in df.iloc[:, -1]])\n",
    "mean = []\n",
    "std = []\n",
    "for i in range(images.shape[-1]):\n",
    "    pixels = images[:, :, :, i].ravel()\n",
    "    mean.append(np.mean(pixels))\n",
    "    std.append(np.std(pixels))\n",
    "\n",
    "print(mean, std)\n",
    "\n",
    "# Output\n",
    "([46.56585295255286, 41.46895223605688, 41.246545649451114],\n",
    " [69.21208428987939, 63.936488294726296, 63.29494237674264])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DPxh63mOAeEW"
   },
   "source": [
    "The original size of the images were (200x200), but our dataset contains resized versions of them at (224x224).\n",
    "\n",
    "Since, we will be using a [densenet](https://towardsdatascience.com/review-densenet-image-classification-b6631a8ef803) as our architecture without the head, we don't need the extra computation and hence we resize the images to (200x200).\n",
    "\n",
    "This resizing is done in the PersonDataGenerator class.\n",
    "\n",
    "Let's create the keras Sequence which can be called to give a sequence of batches when called by `fit_generator`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Z31aZNhE53RS"
   },
   "source": [
    "### Store attribute column names in variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CI1MHVvF6Gkb"
   },
   "source": [
    "We prefixed columns with attribute names while creating one-hot encoded versions, so that we could collate columns with specific attribute names together, this will help identify which columns does a specific target belong to.\n",
    "\n",
    "```python\n",
    "_gender_cols_ = [col for col in one_hot_df.columns if col.startswith(\"gender\")]\n",
    "_imagequality_cols_ = [col for col in one_hot_df.columns if col.startswith(\"imagequality\")]\n",
    "_age_cols_ = [col for col in one_hot_df.columns if col.startswith(\"age\")]\n",
    "_weight_cols_ = [col for col in one_hot_df.columns if col.startswith(\"weight\")]\n",
    "_carryingbag_cols_ = [col for col in one_hot_df.columns if col.startswith(\"carryingbag\")]\n",
    "_footwear_cols_ = [col for col in one_hot_df.columns if col.startswith(\"footwear\")]\n",
    "_emotion_cols_ = [col for col in one_hot_df.columns if col.startswith(\"emotion\")]\n",
    "_bodypose_cols_ = [col for col in one_hot_df.columns if col.startswith(\"bodypose\")]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "omfSBWIcDtny"
   },
   "source": [
    "### Building a keras Sequence to feed to the training loop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lt5s5KX2AeBp"
   },
   "source": [
    "\n",
    "\n",
    "Let's build the sequence class which will give batches of data as a generator, shuffle it and apply data augmentations if specified. \n",
    "\n",
    "```python\n",
    "class PersonDataGenerator(keras.utils.Sequence):\n",
    "    \"\"\"Ground truth data generator\"\"\"\n",
    "\n",
    "    def __init__(self, df, batch_size=32, image_size=200, shuffle=True, augmentation=None):\n",
    "        self.df = df\n",
    "        self.batch_size=batch_size\n",
    "        self.shuffle = shuffle\n",
    "        self.augmentation = augmentation\n",
    "        self.image_size = image_size\n",
    "        self.normalize_image = lambda x: (x - mean) / std\n",
    "        self.on_epoch_end()\n",
    "        \n",
    "\n",
    "    def __len__(self):\n",
    "        return int(np.floor(self.df.shape[0] / self.batch_size))\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        \"\"\"fetch batched images and targets\"\"\"\n",
    "        batch_slice = slice(index * self.batch_size, (index + 1) * self.batch_size)\n",
    "        items = self.df.iloc[batch_slice]\n",
    "        \n",
    "        # resizing the image to (self.image_size, self.image_size)\n",
    "        image = np.stack([self.normalize_image(cv2.resize(cv2.imread(item[\"image_path\"]), \n",
    "                                            (self.image_size, self.image_size))) for _, item in items.iterrows()])\n",
    "\n",
    "        if self.augmentation is not None:\n",
    "            # This is required for featurewise-center and featurewise-stdnorm\n",
    "            # self.augmentation.fit(image)\n",
    "            image = self.augmentation.flow(image, shuffle=False).next()\n",
    "\n",
    "        target = {\n",
    "            \"gender_output\": items[_gender_cols_].values,\n",
    "            \"image_quality_output\": items[_imagequality_cols_].values,\n",
    "            \"age_output\": items[_age_cols_].values,\n",
    "            \"weight_output\": items[_weight_cols_].values,\n",
    "            \"bag_output\": items[_carryingbag_cols_].values,\n",
    "            \"pose_output\": items[_bodypose_cols_].values,\n",
    "            \"footwear_output\": items[_footwear_cols_].values,\n",
    "            \"emotion_output\": items[_emotion_cols_].values,\n",
    "        }\n",
    "        return image, target\n",
    "\n",
    "    def on_epoch_end(self):\n",
    "        \"\"\"Updates indexes after each epoch\"\"\"\n",
    "        if self.shuffle == True:\n",
    "            self.df = self.df.sample(frac=1).reset_index(drop=True)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "obEfTJ5wAd9q"
   },
   "source": [
    "Let's quickly call the PersonDataGenerator class to create the train and test generators. To implement cutout we use\n",
    "[get_random_generator](https://github.com/yu4u/cutout-random-erasing/blob/master/random_eraser.py) function.\n",
    "\n",
    "```python\n",
    "# create train and validation data generators\n",
    "train_gen = PersonDataGenerator(train_df, batch_size=32, image_size=200, augmentation=ImageDataGenerator(\n",
    "    width_shift_range=0.2,\n",
    "    height_shift_range=0.1,\n",
    "    zoom_range=0.2,\n",
    "    horizontal_flip=True,\n",
    "    preprocessing_function=get_random_eraser(v_h=1, s_l=0.01, s_h=0.2, r_1=0.1, r_2=1)\n",
    "))\n",
    "\n",
    "valid_gen = PersonDataGenerator(val_df, batch_size=32, image_size=200, shuffle=False)\n",
    "\n",
    "# get number of output units from data\n",
    "images, targets = next(iter(train_gen))\n",
    "# num_units actually tells the number of one-hot encoded values / choices for a category\n",
    "num_units = { k.split(\"_output\")[0]:v.shape[1] for k, v in targets.items()}\n",
    "\n",
    "print(num_units)\n",
    "\n",
    "# output\n",
    "\n",
    "{'age': 5,\n",
    " 'bag': 3,\n",
    " 'emotion': 4,\n",
    " 'footwear': 3,\n",
    " 'gender': 2,\n",
    " 'image_quality': 3,\n",
    " 'pose': 3,\n",
    " 'weight': 4}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Bc83jBV7Ad5Z"
   },
   "source": [
    "## Designing the architecture "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JaAeN5HcEEg-"
   },
   "source": [
    "### Choosing a backbone"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hcygh7lFEEbt"
   },
   "source": [
    "We are not using pre-trained weights and starting from scratch as that was one of the requirements of this exercise.\n",
    "We take a **DenseNet121** backbone, without the head. This was chosen over **Resnet50**, but that should also provide comparable results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "g5F-9NXBEEZp"
   },
   "source": [
    "Since this is a multi-task classification. The final model we are going for would be something like this:\n",
    "\n",
    "\n",
    "```\n",
    "inputs -> \n",
    "        densenet121 + GAP -> (\n",
    "                            tower + gender_head,\n",
    "                            tower + age_head,\n",
    "                            tower + emotion_head, ...\n",
    "                                                    ) -> outputs \n",
    "\n",
    "```\n",
    "1. Choose an architecture as a backbone (here we are choosing densenet121, as it worked better than resnet50 in intial tests), do not include the head, as we would build our own.\n",
    "2. Build towers for each class, the architecture of the towers would largely remain the same.\n",
    "3. Build the respective heads which are the outputs for each class.\n",
    "4. Construct the overall model, specifying the inputs and the outputs.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "V-n9ec75FAzU"
   },
   "source": [
    "Code for the backbone"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jual-lOhAd21"
   },
   "source": [
    "```python\n",
    "backbone = DenseNet121(\n",
    "    include_top=False,\n",
    "    input_shape=(200, 200, 3),\n",
    "    weights= None\n",
    ")\n",
    "\n",
    "model = Model(inputs=backbone.input, outputs=backbone.output)\n",
    "model.summary()\n",
    "```\n",
    "\n",
    "![backbone head](https://raw.githubusercontent.com/baronrogers5/person-classifier/master/images/backbone_head.png \"model summary\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i7t9VDj7FMCw"
   },
   "source": [
    "The final shape after relu is (6, 6, 1024). \n",
    "\n",
    "This is passed through GlobalAveragePooling (further referenced as GAP) to average each feature map across the channel dimension. This brings the tensor shape to a flat (None, 1024).\n",
    "\n",
    " I strongly believe this can be improved if instead of only using GAP, we also use GlobalMaxPooling and concatenate the outputs. That results in 2048 rank 1 tensor, which would have advantages of both mean and max results.\n",
    "\n",
    "Here is a representation of what I mean:\n",
    "\n",
    "```python\n",
    "# original\n",
    "Input (None, 6, 6, 1024) --GAP--> (None, 1024)\n",
    "\n",
    "# proposed \n",
    "Input (None, 6, 6, 1024) --GAP--> (None, 1024) # call this out1, here we have the mean of features.\n",
    "Input (None, 6, 6, 1024) --GMP--> (None, 1024) # call this out2, here we have the max value of features.\n",
    "\n",
    "concatenate(out1 + out2) --> out (None, 20148)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yA09i6tlEEVd"
   },
   "source": [
    "### Constructing the tower"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lTYMKFZ9ExCS"
   },
   "source": [
    "The tower adds a batchnorm after GAP, to normalize the results after GAP and also a small amount of dropout, to improve resilience.<br> It is followed by a densely connected layer that reduces the nodes to 128, on which each head builds.\n",
    "\n",
    "```python\n",
    "def build_tower(in_layer):\n",
    "    neck = BatchNormalization()(in_layer)\n",
    "    neck = Dropout(0.1)(neck)\n",
    "    neck = Dense(128, activation='relu')(neck)\n",
    "    return neck\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Se8p55LBEx1z"
   },
   "source": [
    "### Constructing the heads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Dd6IKWUOExwY"
   },
   "source": [
    "The head is the final layer that produces outputs.<br> Let's take **pose**(one of the attributes we need to predict), it has 3 possible outputs, namely **(front-frontish, back and side)**, so each possible output needs a final node which represents the probability of it's occurrence.<br> `num_units` contains the mapping of each name and the number of categories present. `build_head` builds the heads of each category assigning appropriate final nodes as per `num_units`.  \n",
    "\n",
    "```\n",
    "{\n",
    "    'age': 5,\n",
    "    'bag': 3,\n",
    "    'emotion': 4,\n",
    "    'footwear': 3,\n",
    "    'gender': 2,\n",
    "    'image_quality': 3,\n",
    "    'pose': 3,\n",
    "    'weight': 4\n",
    " }\n",
    " ```\n",
    "\n",
    " ```python\n",
    " def build_head(name, in_layer):\n",
    "    return Dense(num_units[name], activation='softmax', name=f'{name}_output')(in_layer)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Xg1KNlTnExtd"
   },
   "source": [
    "### Code to build the complete Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0ieYtgxAExbG"
   },
   "source": [
    "```python\n",
    "backbone = DenseNet121(\n",
    "    include_top=False,\n",
    "    input_shape=(200, 200, 3),\n",
    "    weights=None,                     # No pre-trained weights, for imagenet weights, pass `imagenet` as arg. \n",
    ")\n",
    "\n",
    "# \n",
    "neck = GlobalAveragePooling2D()(backbone.output)\n",
    "\n",
    "def build_tower(in_layer):\n",
    "    neck = BatchNormalization()(in_layer)\n",
    "    neck = Dropout(0.1)(neck)\n",
    "    neck = Dense(128, activation='relu')(neck)\n",
    "    return neck\n",
    "\n",
    "def build_head(name, in_layer):\n",
    "    return Dense(num_units[name], activation='softmax', name=f'{name}_output')(in_layer)\n",
    "\n",
    "#heads\n",
    "gender = build_head(\"gender\", build_tower(neck))\n",
    "image_quality = build_head(\"image_quality\", build_tower(neck))\n",
    "age = build_head(\"age\", build_tower(neck))\n",
    "weight = build_head(\"weight\", build_tower(neck))\n",
    "bag = build_head(\"bag\", build_tower(neck))\n",
    "footwear = build_head(\"footwear\", build_tower(neck))\n",
    "emotion = build_head(\"emotion\", build_tower(neck))\n",
    "pose = build_head(\"pose\", build_tower(neck))\n",
    "\n",
    "\n",
    "model = Model(\n",
    "    inputs=backbone.input,\n",
    "    outputs=[gender, image_quality, age, weight, bag, footwear, pose, emotion]\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lQ3qizVeKqmp"
   },
   "source": [
    "You can check out how the complete network looks [here](https://github.com/baronrogers5/person-classifier/blob/master/images/model.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nF8uyclVGDh2"
   },
   "source": [
    "## Defining the training callbacks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hb-jyDfAGDf7"
   },
   "source": [
    "\n",
    "1. `ReduceLROnPlateau` -> To reduce learning_rate when the model val_loss does not improve by `min_delta` for some `patience` epochs\n",
    "2. `ModelCheckpoint` -> To save the model weights, in specified directory.\n",
    "3. `EarlyStopping` -> To stop training if `val_loss` does not improve by `min_delta` for `patience` epochs, restore the `best_weights` during training after stopping. \n",
    "\n",
    "```python\n",
    "# reduce lr on plateau\n",
    "reduce_lr_on_plateau = ReduceLROnPlateau(\n",
    "    monitor='val_loss',\n",
    "    factor=0.7,\n",
    "    patience=3,\n",
    "    min_lr=1e-6,\n",
    "    verbose=1,\n",
    "    min_delta=1e-3\n",
    ")\n",
    "\n",
    "# save the model\n",
    "save_dir = os.path.join(os.getcwd(), 'gdrive/My Drive/Person_MultiClass')\n",
    "model_name = 'person_multiclass_model_densenet_200_normalized_frozen_reduce_dropout.h5'\n",
    "\n",
    "if not os.path.isdir(save_dir):\n",
    "    os.makedirs(save_dir)\n",
    "filepath = os.path.join(save_dir, model_name)\n",
    "\n",
    "# checkpoint callback\n",
    "checkpoint = ModelCheckpoint(filepath=filepath,\n",
    "                             monitor='val_loss',\n",
    "                             verbose=1,\n",
    "                             save_best_only=True)\n",
    "\n",
    "# Early Stopping\n",
    "early_stopping = EarlyStopping(\n",
    "    patience=8,\n",
    "    verbose=1,\n",
    "    min_delta=1e-3,\n",
    "    restore_best_weights=True\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XAnEIxXMGDbm"
   },
   "source": [
    "Let's compile the model before we start the actual training.\n",
    "\n",
    "\n",
    "The model compilation step has been implemented as a function, so that it can be called from any cell, and on any model (backbone or partial models)\n",
    "\n",
    "```python\n",
    "def compile_model(model):\n",
    "    model.compile(\n",
    "        optimizer=Adam(lr=1e-3),\n",
    "        loss = 'categorical_crossentropy',\n",
    "        metrics=['accuracy']\n",
    "    )\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "J0ZFJZVuGDZV"
   },
   "source": [
    "## Let the training begin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IXYvj-ZWGDXQ"
   },
   "source": [
    "Initial rounds of training were done with a smaller image size of (100x100), so that the model gets to adjust it's initial weights. <br> This helps when data of size (200x200) is introduced, as it acts as completely new data for the model. And, we can do the fine-tuned training with the larger images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rN3U_2eMGDTh"
   },
   "source": [
    "The training was carried on in the following sequence:\n",
    "\n",
    "1. (100x100) images with all layers trainable.\n",
    "2. (100x100) images with backbone frozen, so that the final layers could be fine-tuned.\n",
    "3. (100x100) images with all trainable layers but aggressive data augmentation with a `lr = 1e-4`.\n",
    "4. (200x200) images with all trainable layers.\n",
    "5. (200x200) images with backbone frozen.\n",
    "\n",
    "All the training loops had `EarlyStopping` enabled, so the number of epochs ranged from 8-15 in each case, after which the training stopped and the next step was run."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "t2uDm19QGDO1"
   },
   "source": [
    "For the purpose of brevity, I will not include all training steps here, but you can always check the [actual colab file](https://github.com/baronrogers5/person-classifier/blob/master/Person_using_DenseNet.ipynb), which has all the steps with logs.\n",
    "\n",
    "If you wish to continue training, and wish to use my pre-trained weights, you can find them in the [github repo models folder](https://github.com/baronrogers5/person-classifier/tree/master/models)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "59RACgtgGDLG"
   },
   "source": [
    "```python\n",
    "compile_model(model)\n",
    "\n",
    "model.fit_generator(\n",
    "    generator=train_gen,\n",
    "    validation_data=valid_gen,\n",
    "    use_multiprocessing=True,\n",
    "    workers=6, \n",
    "    epochs=50,\n",
    "    verbose=1,\n",
    "    callbacks=[reduce_lr_on_plateau, checkpoint, early_stopping]\n",
    ")\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nJD_WeCPGDIx"
   },
   "source": [
    "## Results\n",
    "\n",
    "```python\n",
    "def show_results(model: keras.Model, show_loss=False):\n",
    "    data = []\n",
    "    results = model.evaluate_generator(valid_gen, verbose=1)\n",
    "    for m, r in zip(model.metrics_names, results):\n",
    "        if show_loss:\n",
    "            data.append({'metric_name': m, 'accuracy': r})            \n",
    "        \n",
    "        elif 'acc' in m:\n",
    "                data.append({'metric_name': m, 'accuracy': r})\n",
    "    \n",
    "    df = pd.DataFrame(data, columns=['metric_name', 'accuracy'])\n",
    "    print(df)\n",
    "\n",
    "\n",
    "compile_model(model)\n",
    "model.load_weights('/content/gdrive/My Drive/Person_MultiClass/person_multiclass_model_densenet_200_normalized_frozen_augmented.h5')\n",
    "show_results(model, show_loss=True)\n",
    "\n",
    "# Output\n",
    "\n",
    "63/63 [==============================] - 16s 249ms/step\n",
    "                  metric_name  accuracy\n",
    "0                        loss  6.113962\n",
    "1          gender_output_loss  0.267376\n",
    "2   image_quality_output_loss  0.883270\n",
    "3             age_output_loss  1.261447\n",
    "4          weight_output_loss  0.901494\n",
    "5             bag_output_loss  0.746320\n",
    "6        footwear_output_loss  0.760049\n",
    "7            pose_output_loss  0.419371\n",
    "8         emotion_output_loss  0.874636\n",
    "\n",
    "9           gender_output_acc  0.888889\n",
    "10   image_quality_output_acc  0.584821\n",
    "11             age_output_acc  0.447421\n",
    "12          weight_output_acc  0.652778\n",
    "13             bag_output_acc  0.697917\n",
    "14        footwear_output_acc  0.670635\n",
    "15            pose_output_acc  0.857143\n",
    "16         emotion_output_acc  0.704365\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6wSZMHUcGDGA"
   },
   "source": [
    "## Further experiments / Ways to improve the model  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SNw9FywiK_Qy"
   },
   "source": [
    "There can be a lot of ways to improve the results that we achieved here. I will keep updating the actual [github repo](https://github.com/baronrogers5/person-classifier) with some of these recommendations down the line. If you liked the article or would like further updates, please `watch` the github repo or `star` it. It would motivate me to work on other similar articles. \n",
    "\n",
    "So without further ado, I'm listing some recommendations in no particular order. \n",
    "\n",
    "1. Better normalization and weight initialization.\n",
    "2. Loss weights for each individual class.\n",
    "3. Better image augmentation especially [imgaug](https://github.com/aleju/imgaug).\n",
    "\n",
    "<center>\n",
    "    <img src=\"https://raw.githubusercontent.com/aleju/imgaug-doc/master/readme_images/augmenter_videos/coarsedropout_p_0_2.gif\" alt=\"coarse dropout\" align=\"middle\">\n",
    "</center>\n",
    "\n",
    "4. Try different architectures example Inceptionv4, EfficientNet, ResNext.\n",
    "5. Loose the regularizations such as dropout.\n",
    "6. Try One Cycle Policy and Cyclic Learning Rate, especially the implemenation by [fast.ai](https://www.fast.ai/).\n",
    "7. Instead of only GAP use `concatenate(GAP, GMP)`, and build tower after that.\n",
    "\n",
    "\n",
    "I hope to implement some of these and see the loss decrease and get better results, also if any of you implement and see some interesting results, would love to hear more about it.\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "person-for-the-blog.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}