tp1_neural_net.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TP1: Réseau de neurones à deux couches, fonction de perte **Entropie croisée**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "from utils.data_utils import load_CIFAR10\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# pour automatiquement recharger les modules externes\n",
    "# voir http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=5>MLP à deux couches</font>\n",
    "\n",
    "Ici, nous développerons un **réseau pleinement connecté** à deux couches avec **softmax** et **entropie croisée**.  En principe, vous devriez avoir pris connaissance du notebook \"tp1_simple_neural_net.ipynb\".  Le but ici est d'enchasser les différents éléments d'un réseau de neurones dans des **classes**.\n",
    "\n",
    "Pour commencer, nous commence avec un petit réseau à \n",
    "\n",
    "* **10 neurones cachées**\n",
    "* **3 classes** \n",
    "* un vecteur d'entrée de taille **4**.\n",
    "\n",
    "Avant de commencer, vous devez vous familier avec les classes **model.Model**, **layers.Dense** et la loss **cross_entropy_loss** (cette dernière fonction comprend le softmax + la loss)\n",
    "\n",
    "Les fonctions que vous devez modifier sont \n",
    "\n",
    "* dans la classe **Dense** (répertoire layer)\n",
    "* dans la fonction **cross_entropy_loss_npdl** (dans utils.model_loss)\n",
    "* dans le fichier **activations.py** (répetoire utils)\n",
    "\n",
    "À ce point-ci, l'entropie croisée (et son gradient) que vous avez codé précédemment **doit être fonctionnelle**.  Ce code doit donc être récupéré ici!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##############################################################################\n",
    "# On crée un modèle jouet pour rapidement tester les différentes composantes #\n",
    "# La classe Model encapsule et relie les différentes couches de notre        #\n",
    "# réseau de neurones. Son fonctionnement est très similaire à celui de la    #\n",
    "# classe Sequential de Keras: https://keras.io/api/models/sequential/        #\n",
    "##############################################################################\n",
    "from model.Model import Model\n",
    "from layers.Dense import Dense\n",
    "from utils.model_loss import cross_entropy_loss_npdl\n",
    "\n",
    "hidden_size = 10\n",
    "num_classes = 3\n",
    "input_size = 4\n",
    "\n",
    "def create_toy_model():\n",
    "    np.random.seed(0)\n",
    "    model = Model()\n",
    "    model.add(Dense(input_size, hidden_size, weight_scale=1e-1, activation='relu'))\n",
    "    model.add(Dense(hidden_size, num_classes, weight_scale=1e-1))\n",
    "    model.add_loss(cross_entropy_loss_npdl)\n",
    "    return model\n",
    "\n",
    "model = create_toy_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_random_toy_data(nb_elements, input_size, nb_classes):\n",
    "    np.random.seed(1)\n",
    "    \n",
    "    X = 10 * np.random.randn(nb_elements, input_size)\n",
    "    y = np.random.randint(nb_classes, size=nb_elements)\n",
    "        \n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x,y=create_random_toy_data(1, 5, 3)\n",
    "x.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Commençons avec la prédiction d'**un seul vecteur d'entrée** de taille 4 et dont la cible est 0.  Ici, le **score** est la sortie du réseau avant la **softmax**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##############################################################################\n",
    "# TODO: Implémenter la méthode forward (propagation avant) de la classe de   #\n",
    "# couche Dense.                                                              #\n",
    "##############################################################################\n",
    "X, y = create_random_toy_data(1, input_size, num_classes)\n",
    "\n",
    "scores = model.forward_npdl(X)\n",
    "print('Your scores:')\n",
    "print(scores)\n",
    "print()\n",
    "print('correct scores:')\n",
    "correct_scores = np.asarray([[-0.81233741, -1.27654624, -0.70335995]])\n",
    "print(correct_scores)\n",
    "print()\n",
    "\n",
    "# La différence devrait être assez basse, en principe inférieure à 1e-7.\n",
    "print('Difference between your scores and correct scores:')\n",
    "print(np.sum(np.abs(scores - correct_scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##############################################################################\n",
    "# TODO: Implémenter le calcul de perte vectorisé adapté pour un réseau de    #\n",
    "# neurones dans la fonction softmax_loss. La différence avec la softmax      #\n",
    "# implémentée dans la partie précédente est que la dérivée retournée par     #\n",
    "# softmax_loss est en fonction des scores, et non en fonction des poids      #\n",
    "##############################################################################\n",
    "\n",
    "loss, _, softmax = model.calculate_loss(scores, y, 0.1)\n",
    "print('Softmax: ', softmax)\n",
    "print('Loss: ', loss)\n",
    "\n",
    "correct_loss = 0.938286073616293\n",
    "correct_softmax = np.asarray([0.3644621, 0.22911264, 0.40642526])\n",
    "\n",
    "print('\\nCorrect loss: ', correct_loss)\n",
    "print('Difference between your loss and correct loss:')\n",
    "# on devrait obtenir une erreur de loss inférieure à 1e-9.\n",
    "print(np.sum(np.abs(loss - correct_loss)))\n",
    "\n",
    "# on devrait obtenir une erreur de softmax inférieure à 1e-7.\n",
    "print('\\nDifference between your softmax and correct softmax:')\n",
    "print(np.sum(np.abs(correct_softmax - softmax)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Maintenant allons-y avec la prédiction de **5 vecteurs d'entrée** de taille 4 et dont la cible est 0.  Ici, le **score** est la sortie du réseau avant la **softmax**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##############################################################################\n",
    "# TODO: Implémenter la méthode forward (propagation avant) de la classe de   #\n",
    "# couche Dense.                                                              #\n",
    "##############################################################################\n",
    "X, y = create_random_toy_data(5, input_size, num_classes)\n",
    "\n",
    "scores = model.forward_npdl(X)\n",
    "print('Your scores:')\n",
    "print(scores)\n",
    "print()\n",
    "print('Correct scores:')\n",
    "correct_scores = np.asarray([\n",
    "  [-0.81233741, -1.27654624, -0.70335995],\n",
    "  [-0.17129677, -1.18803311, -0.47310444],\n",
    "  [-0.51590475, -1.01354314, -0.8504215 ],\n",
    "  [-0.15419291, -0.48629638, -0.52901952],\n",
    "  [-0.00618733, -0.12435261, -0.15226949]])\n",
    "print(correct_scores)\n",
    "print()\n",
    "\n",
    "# La différence devrait être assez basse, en principe inférieure à 1e-7.\n",
    "print('Difference between your scores and correct scores:')\n",
    "print(np.sum(np.abs(scores - correct_scores)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##############################################################################\n",
    "# TODO: Implémenter le calcul de perte vectorisé adapté pour un réseau de    #\n",
    "# neurones dans la fonction softmax_loss. La différence avec la softmax      #\n",
    "# implémentée dans la partie précédente est que la dérivée retournée par     #\n",
    "# softmax_loss est en fonction des scores, et non en fonction des poids      #\n",
    "##############################################################################\n",
    "\n",
    "loss, _, softmax = model.calculate_loss(scores, y, 0.1)\n",
    "print('Softmax: ', softmax)\n",
    "print('Loss: ', loss)\n",
    "\n",
    "correct_loss = 1.06368300738\n",
    "\n",
    "correct_softmax = np.asarray([\n",
    "    [0.3644621,  0.22911264, 0.40642526],\n",
    "    [0.47590629, 0.17217039, 0.35192332],\n",
    "    [0.43035767, 0.26164229, 0.30800004],\n",
    "    [0.41583127, 0.2983228,  0.28584593],\n",
    "    [0.36328815, 0.32279939, 0.31391246]])\n",
    "\n",
    "# on devrait obtenir une erreur de loss inférieure à 1e-10.\n",
    "print('\\nCorrect loss: ', correct_loss)\n",
    "print('Difference between your loss and correct loss:')\n",
    "print(np.sum(np.abs(loss - correct_loss)))\n",
    "\n",
    "# on devrait obtenir une erreur de softmax inférieure à 1e-7.\n",
    "print('\\nDifference between your softmax and correct softmax:')\n",
    "print(np.sum(np.abs(softmax - correct_softmax)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rétropropagation\n",
    "\n",
    "Maintenant vous devez rédiger la **rétro-progatation** de la classe **model**.  Vos gradients seront testés avec un gradient numérique."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##############################################################################\n",
    "# TODO: Implémenter la méthode backward (rétro-propagation) de la classe de  #\n",
    "# couche Dense, ainsi que la méthode backward de la classe Model.            #\n",
    "##############################################################################\n",
    "loss, dScores, softmax = model.calculate_loss(scores, y, 0.1)\n",
    "\n",
    "_ = model.backward_npdl(dScores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.gradients import evaluate_numerical_gradient\n",
    "\n",
    "# Retourne l'erreur relative maximale des matrices de gradients passées en paramètre.\n",
    "# Pour chaque paramètre, l'erreur relative devrait être inférieure à environ 1e-8.\n",
    "def rel_error(x, y):\n",
    "    rel = np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y)))\n",
    "    return np.max(rel)\n",
    "\n",
    "gradients = model.gradients()\n",
    "model_params = model.parameters()\n",
    "\n",
    "# Si tout va bien, vous devriez avoir des erreurs inférieurs à 1e-8\n",
    "for layer_name, layer_params in model_params.items():\n",
    "    for param_name, _ in layer_params.items():\n",
    "        grad_num = evaluate_numerical_gradient(X, y, model, layer_name, param_name, reg=0.1)\n",
    "        max_error = rel_error(grad_num, gradients[layer_name][param_name])\n",
    "        print('%s max relative error: %e' % (layer_name + '-' + param_name, max_error))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Entraînement\n",
    "\n",
    "Il est maintanant temps de rédiger le code de la descente de gradient devant permettre d'entraîner le modèle.  Le code à rédiger est dans le fichier **model.Solver**.  Si votre code fonctionne, le graphique de la loss devrait **descendre**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from model.Solver import solver_npdl, SGD\n",
    "\n",
    "model = create_toy_model()\n",
    "sgd_optimizer = SGD(1e-1, model)\n",
    "\n",
    "loss_history, _, _ = solver_npdl(X, y, X, y, 1e-2, sgd_optimizer, num_iter=100)\n",
    "\n",
    "print('Final training loss: ', loss_history[-1])\n",
    "\n",
    "from visualization.utils import visualize_loss\n",
    "\n",
    "# Visualisation de l'historique de perte lors de l'entraînement\n",
    "visualize_loss(loss_history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"5\">Classifieur d'images</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_CIFAR10_data(num_training=49000, num_validation=1000, num_test=1000, num_dev=500):\n",
    "    \"\"\"\n",
    "    Charger la banque de données CIFAR-10, prétraiter les images et ajouter une dimension pour le biais.\n",
    "    \n",
    "    Input :\n",
    "    - num_training : nombre d'images à mettre dans l'ensemble d'entrainement\n",
    "    - num_validation : nombre d'images à mettre dans l'ensemble de validation\n",
    "    - num_test : nombre d'images à mettre dans l'ensemble de test\n",
    "    - num_dev : d'images à mettre dans l'ensemble dev\n",
    "    \n",
    "    Output :\n",
    "    - X_train, y_train : données et cibles d'entrainement\n",
    "    - X_val, y_val: données et cibles de validation\n",
    "    - X_test y_test: données et cibles de test \n",
    "    - X_dev, y_dev: données et cicles dev\n",
    "    \"\"\"\n",
    "    # Charger les données CIFAR-10\n",
    "    cifar10_dir = 'datasets/cifar-10-batches-py'\n",
    "    X_train, y_train, X_test, y_test = load_CIFAR10(cifar10_dir)\n",
    "  \n",
    "    # Séparer en ensembles d'entraînement, de validation, de test et de dev\n",
    "    mask = range(num_training, num_training + num_validation)\n",
    "    X_val = X_train[mask]\n",
    "    y_val = y_train[mask]\n",
    "    mask = range(num_training)\n",
    "    X_train = X_train[mask]\n",
    "    y_train = y_train[mask]\n",
    "    mask = range(num_test)\n",
    "    X_test = X_test[mask]\n",
    "    y_test = y_test[mask]\n",
    "    mask = np.random.choice(num_training, num_dev, replace=False)\n",
    "    X_dev = X_train[mask]\n",
    "    y_dev = y_train[mask]\n",
    "\n",
    "    X_train = np.reshape(X_train, (X_train.shape[0], -1))\n",
    "    X_val = np.reshape(X_val, (X_val.shape[0], -1))\n",
    "    X_test = np.reshape(X_test, (X_test.shape[0], -1))\n",
    "    X_dev = np.reshape(X_dev, (X_dev.shape[0], -1))\n",
    "\n",
    "    # Normalisation\n",
    "    X_train -= np.mean(X_train, axis = 0)\n",
    "    X_val -= np.mean(X_val, axis = 0)\n",
    "    X_test -= np.mean(X_test, axis = 0)\n",
    "    X_dev -= np.mean(X_dev, axis = 0)\n",
    "\n",
    "    return X_train, y_train, X_val, y_val, X_test, y_test, X_dev, y_dev"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Charger les images pour les différents ensembles de données\n",
    "X_train, y_train, X_val, y_val, X_test, y_test, X_dev, y_dev = get_CIFAR10_data()\n",
    "print('Train data shape: ', X_train.shape)\n",
    "print('Train labels shape: ', y_train.shape)\n",
    "print('Validation data shape: ', X_val.shape)\n",
    "print('Validation labels shape: ', y_val.shape)\n",
    "print('Test data shape: ', X_test.shape)\n",
    "print('Test labels shape: ', y_test.shape)\n",
    "print('dev data shape: ', X_dev.shape)\n",
    "print('dev labels shape: ', y_dev.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Si le modèle et toutes ses composantes ont bien été implémentées\n",
    "# à l'étape du modèle jouet,\n",
    "def create_model(hidden_size):\n",
    "    model = Model()\n",
    "    model.add(Dense(dim_input=3*32*32, dim_output=hidden_size, activation='relu'))\n",
    "    model.add(Dense(dim_input=hidden_size, dim_output=10))\n",
    "    model.add_loss(cross_entropy_loss_npdl)\n",
    "    return model\n",
    "    \n",
    "model = create_model(50)\n",
    "\n",
    "scores = model.forward_npdl(X_dev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss, dScores, softmax = model.calculate_loss(scores, y_dev, 0.5)\n",
    "\n",
    "# La loss d'un modèle non-entrainé devrait s'approcher de -log(0.1).\n",
    "print('loss: %f' % loss)\n",
    "print('sanity check loss: %f' % (-np.log(0.1)))\n",
    "\n",
    "print('dScores shape: ', dScores.shape)\n",
    "print('softmax shape: ', softmax.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = SGD(8e-5, model)\n",
    "\n",
    "loss_history, train_accuracy, val_accuracy = solver_npdl(X_train, y_train, X_val, y_val, 0.5, optimizer, lr_decay=0.98, num_iter=3000)\n",
    "\n",
    "print('Final training loss: ', loss_history[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=5>Visualisation</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from visualization.utils import visualize_loss\n",
    "\n",
    "# Visualisation de l'historique de loss de l'entraînement\n",
    "visualize_loss(loss_history)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from visualization.utils import visualize_accuracy\n",
    "\n",
    "# Visualisation de l'historique d'accuracy lors de l'entraînement.\n",
    "# Ceci inclut la accuracy d'entraînement et la accuracy de validation\n",
    "visualize_accuracy(train_accuracy, val_accuracy)\n",
    "\n",
    "# Si la accuracy de validation est proche de 0.4, c'est bon signe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from visualization.utils import visualize_as_grid\n",
    "\n",
    "def show_net_weights(model):\n",
    "  W1 = model.parameters()['L0']['W']\n",
    "  W1 = W1.reshape(32, 32, 3, -1).transpose(3, 0, 1, 2)\n",
    "  plt.imshow(visualize_as_grid(W1, padding=3).astype('uint8'))\n",
    "  plt.gca().axis('off')\n",
    "  plt.show()\n",
    "\n",
    "show_net_weights(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_model = None \n",
    "best_val_acc = 0.0\n",
    "best_loss = None\n",
    "best_train_acc = 0.0\n",
    "best_params = None\n",
    "\n",
    "from itertools import product as itprod\n",
    "\n",
    "lr_list = 10 ** np.linspace(-4, -3, 4)\n",
    "reg_list = 10 ** np.linspace(-7, -3, 4)\n",
    "hs = 200\n",
    "ni = 1000\n",
    "\n",
    "best_params = None\n",
    "\n",
    "for lr, reg in itprod(lr_list, reg_list):\n",
    "    model = create_model(hs)\n",
    "    optimizer = SGD(lr, model)\n",
    "    loss_history, train_accuracy, val_accuracy = solver_npdl(X_train, y_train, X_val, y_val, reg, optimizer, lr_decay=0.98, num_iter=ni)\n",
    "\n",
    "    params = (lr, reg, hs, ni)\n",
    "    if np.mean(val_accuracy) > np.mean(best_val_acc):\n",
    "        best_val_acc = val_accuracy\n",
    "        best_train_acc = train_accuracy\n",
    "        best_model = model\n",
    "        best_loss = loss_history\n",
    "        best_params = params\n",
    "    print('{} -- Tacc: {:>5.3f}  Vacc: {:>5.3f}'.format(params, np.mean(train_accuracy), np.mean(val_accuracy))) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Best: {}'.format(best_params))\n",
    "# Visualisation de l'historique de perte lors de l'entraînement\n",
    "visualize_loss(best_loss)\n",
    "\n",
    "# Visualisation de l'historique d'accuracy lors de l'entraînement.\n",
    "# Ceci inclut la accuracy d'entraînement et la accuracy de validation\n",
    "visualize_accuracy(best_train_acc, best_val_acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_net_weights(best_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_acc = (best_model.predict(X_test) == y_test).mean()\n",
    "print('Test accuracy: ', test_acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}