Differential_Privacy_using_PATE_Analysis.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Differential Privacy using PATE Analysis.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/AJ-54/Differential-Privacy/blob/master/Differential_Privacy_using_PATE_Analysis.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ce4miP-4n0qk",
        "colab_type": "text"
      },
      "source": [
        "# Differential Privacy for MNIST Dataset\n",
        "*Project Background:* <br>\n",
        "We, as a student, has a labeled private dataset and an unlabeled public dataset. We would like to collaberate with certain number of teachers to label our public dataset by training their similar datasets. In order to protect the data privacy of the teachers, we agree to add global noise to the labels they return for the public dataset. Then we label our public dataset based on the labels that most teachers agree. We combine the new labels with the public dataset and train with our private dataset. <br>\n",
        "\n",
        "*PATE analysis* is performed on teachers' predictions and the new labels with certain level of epsilon indicating how much information leakage is allowed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lv4rJtShKWC6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# install syft package to use Private Aggregation of Teacher Ensembles (PATE)\n",
        "# !pip install syft"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HpCdyDIok8oZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 63
        },
        "outputId": "902ed279-1eb8-42fb-a5c6-c998ed5631ff"
      },
      "source": [
        "# import necessary packages\n",
        "%matplotlib inline\n",
        "%config InlineBackend.figure_format = 'retina'\n",
        "\n",
        "import numpy as np\n",
        "import torch\n",
        "from torchvision import datasets, transforms\n",
        "from torch.utils.data import Subset, TensorDataset\n",
        "from torch.utils.data.dataset import random_split\n",
        "from torch import nn, optim\n",
        "import torch.nn.functional as F\n",
        "from syft.frameworks.torch.differential_privacy import pate\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<p style=\"color: red;\">\n",
              "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n",
              "We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n",
              "or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n",
              "<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eUytC-A-m2Zg",
        "colab_type": "text"
      },
      "source": [
        "### Section 1: Download and load MNIST dataset\n",
        "\n",
        "*   **Trainset**: divide among teachers without overlapping\n",
        "*   **Testset**: split into private(labeled) and public(unlabeled) datasets; use models trained by teachers to label public dataset and train with the private dataset\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ckTIiYM5mcjr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# tranform for data normalization\n",
        "transform = transforms.Compose([transforms.ToTensor(),\n",
        "                                transforms.Normalize((0.5, ), (0.5, )),    # MNIST data is 2D\n",
        "                                ])\n",
        "\n",
        "# download training and test data\n",
        "trainset = datasets.MNIST('~/.pytorch/MNIST_data/', download=True, train=True, transform=transform)\n",
        "testset = datasets.MNIST('~/.pytorch/MNIST_data/', download=True, train=False, transform=transform)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "71jTLYE8mr-n",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "17e97b77-cf76-45f1-aa57-fffae88acd77"
      },
      "source": [
        "# training and test data info\n",
        "print('num training data: {0}'.format(len(trainset)))\n",
        "print('num test data: {0}'.format(len(testset)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "num training data: 60000\n",
            "num test data: 10000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Yx9_2Y5og5C",
        "colab_type": "text"
      },
      "source": [
        "### Section 2: Functions\n",
        "\n",
        "\n",
        "*   `teacher_data_allocator`: evenly allocate training dataset to the number of teachers as their private datasets\n",
        "*   `private_public_allocator`: split test dataset as the student's private dataset and public dataset\n",
        "*   `train`: model training with training and valid datasets\n",
        "*   `get_label_testset`: get labels of the student's public dataset from all teachers\n",
        "*   `add_global_noise`: add global noise to the labels that most teachers agree\n",
        "*   `create_labeled_public_db`: extract images from the public dataset and combine these images with the new labels; create a new dataloader for this new dataset\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "laCE8KjVpfyv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# divide unique training datasets among teachers\n",
        "# split each dataset into training and valid sets\n",
        "def teacher_data_allocator(trainset, batch_size=64, num_teachers=10, valid_train_split=0.3):\n",
        "  data_per_teacher = len(trainset) // num_teachers    # round to the nearest integer\n",
        "  train_loaders = []\n",
        "  valid_loaders = []\n",
        "  for i in range(num_teachers):\n",
        "    if i == num_teachers - 1:\n",
        "      ind = list(range(i * data_per_teacher, len(trainset)))\n",
        "    else:\n",
        "      ind = list(range(i * data_per_teacher, (i + 1) * data_per_teacher))\n",
        "    data = Subset(trainset, ind)\n",
        "    split = int(valid_train_split * len(data))\n",
        "    valid_data, train_data = random_split(data, [split, len(data) - split])\n",
        "    train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size, shuffle=True)\n",
        "    valid_loader = torch.utils.data.DataLoader(valid_data, batch_size=batch_size, shuffle=False)\n",
        "    train_loaders.append(train_loader)\n",
        "    valid_loaders.append(valid_loader)\n",
        "  return train_loaders, valid_loaders"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oO2s8OsywWDE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# divide testset into private and public\n",
        "def private_public_allocator(dataset, batch_size=64, split=0.7):\n",
        "  split_ind = int(len(dataset) * split)\n",
        "  \n",
        "  # validset and testset\n",
        "  # validset is smaller because dataset of each teacher is small\n",
        "  private_ind = list(range(0, split_ind))\n",
        "  public_ind = list(range(split_ind, len(dataset)))\n",
        "  private_set = Subset(dataset, private_ind)\n",
        "  public_set = Subset(dataset, public_ind)\n",
        "\n",
        "  # validloader and testloader\n",
        "  private_loader = torch.utils.data.DataLoader(private_set, batch_size=batch_size, shuffle=True)\n",
        "  public_loader = torch.utils.data.DataLoader(public_set, batch_size=batch_size, shuffle=False)\n",
        "  return private_loader, public_loader"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nHv26UDqE14c",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# method for training\n",
        "def train(model, criterion, optimizer, trainloader, validloader, epochs=10):\n",
        "  model = model\n",
        "  criterion = criterion\n",
        "  optimizer = optimizer\n",
        "  epochs = epochs\n",
        "\n",
        "  train_losses, valid_losses = [], []\n",
        "  for e in range(epochs):\n",
        "    running_loss = 0\n",
        "    for inputs, labels in trainloader:\n",
        "\n",
        "      optimizer.zero_grad()\n",
        "\n",
        "      log_ps = model(inputs)\n",
        "      loss = criterion(log_ps, labels)\n",
        "      loss.backward()\n",
        "      optimizer.step()\n",
        "\n",
        "      running_loss += loss.item()\n",
        "  \n",
        "    else:\n",
        "      valid_loss = 0\n",
        "      acc = 0\n",
        "\n",
        "      # turn off gradients for validation, saving memory and computations\n",
        "      with torch.no_grad():\n",
        "        model.eval()\n",
        "        for inputs, labels in validloader:\n",
        "\n",
        "          log_ps = model(inputs)\n",
        "          valid_loss += criterion(log_ps, labels)\n",
        "\n",
        "          ps = torch.exp(log_ps)\n",
        "          top_p, top_class = ps.topk(1, dim=1)\n",
        "          equals = top_class == labels.view(*top_class.shape)\n",
        "          acc += torch.mean(equals.type(torch.FloatTensor))\n",
        "\n",
        "      model.train()\n",
        "\n",
        "      train_losses.append(running_loss / len(trainloader))\n",
        "      valid_losses.append(valid_loss / len(validloader))\n",
        "\n",
        "      print('Epoch: {}/{}.. '.format(e+1, epochs),\n",
        "            'Training Loss: {:.3f}.. '.format(running_loss / len(trainloader)),\n",
        "            'Valid Loss: {:.3f}.. '.format(valid_loss / len(validloader)),\n",
        "            'Valid Accuracy: {:.3f} '.format(acc / len(validloader)),\n",
        "            '')\n",
        "  return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6TZYgHy2GVsR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# label testset by models\n",
        "def get_label_testset(models, testloader):\n",
        "  test_labels = []\n",
        "  for model in models:\n",
        "    test_label = []\n",
        "    for inputs, _ in testloader:\n",
        "      with torch.no_grad():\n",
        "        log_ps = model(inputs)\n",
        "        ps = torch.exp(log_ps)\n",
        "      top_p, top_class = ps.topk(1, dim=1)\n",
        "      test_label.append(top_class.squeeze().tolist())\n",
        "    test_label = sum(test_label, [])\n",
        "    test_labels.append(test_label)\n",
        "  return test_labels"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "WxmgZtRAC4fO",
        "colab": {}
      },
      "source": [
        "# get predictions with global noise\n",
        "def add_global_noise(preds, epsilon=0.1):\n",
        "  labels_with_noise = []\n",
        "  for pred in preds:    # go thru entries\n",
        "    label_counts = np.bincount(pred, minlength=preds.shape[1])\n",
        "\n",
        "    epsilon = epsilon\n",
        "    beta = 1 / epsilon\n",
        "\n",
        "    for i in range(len(label_counts)):\n",
        "      label_counts[i] += np.random.laplace(0, beta, 1)\n",
        "  \n",
        "    new_label = np.argmax(label_counts)\n",
        "    labels_with_noise.append(new_label)\n",
        "  return np.array(labels_with_noise)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YdzQ8GZre3UJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# create a new dataloader for the public dataset\n",
        "# using the new labels added global noise\n",
        "def create_labeled_public_db(dataloader, labels, batch_size=64):\n",
        "  image_list = []\n",
        "  for image, _ in dataloader:\n",
        "    image_list.append(image)\n",
        "  data = np.vstack(image_list)\n",
        "  new_dataloader = list(zip(data, labels))\n",
        "  new_dataloader = torch.utils.data.DataLoader(new_dataloader, shuffle=False, batch_size=batch_size)\n",
        "  return new_dataloader"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pz7op5yp7Xsx",
        "colab_type": "text"
      },
      "source": [
        "### Section 3: Define neural network model structure"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XA7zKRJb1IGM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Classifier for model creation\n",
        "class Classifier(nn.Module):\n",
        "  def __init__(self):\n",
        "    super().__init__()\n",
        "    self.fc1 = nn.Linear(784, 256)\n",
        "    self.fc2 = nn.Linear(256, 128)\n",
        "    self.fc3 = nn.Linear(128, 64)\n",
        "    self.fc4 = nn.Linear(64, 10)\n",
        "\n",
        "    # dropout module with 0.2 drop probability\n",
        "    self.dropout = nn.Dropout(p=0.2)\n",
        "  \n",
        "  def forward(self, x):\n",
        "    # flatten input tensor\n",
        "    x = x.view(x.shape[0], -1)\n",
        "\n",
        "    x = self.dropout(F.relu(self.fc1(x)))\n",
        "    x = self.dropout(F.relu(self.fc2(x)))\n",
        "    x = self.dropout(F.relu(self.fc3(x)))\n",
        "\n",
        "    x = F.log_softmax(self.fc4(x), dim=1)\n",
        "\n",
        "    return x"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ezGTqiDvC9xV"
      },
      "source": [
        "### Section 4: Model training and PATE analysis\n",
        "We first ask the teachers to train their data and use these models to label the student's public data. After adding the global noise to create the new labels for the public dataset, the student then trains his/her private (training set) and public (valid set) data. PATE analysis returns independent and dependent epsilons show how much information has been leaked and how much teachers agree with each other."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rrJ8gF33CpXi",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "36f24e3c-ac01-4c38-98af-0f2eff7e02ac"
      },
      "source": [
        "# allocate training dataset to teachers\n",
        "# split each of these datasets into training and valid datasets\n",
        "teachers_train, teachers_valid = teacher_data_allocator(trainset)\n",
        "print(len(teachers_train))\n",
        "print(len(teachers_valid))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "10\n",
            "10\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "OuQI9wMoCjVd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "320394ba-a325-4d43-b980-e588c28b644a"
      },
      "source": [
        "# allocate test dataset to the student\n",
        "# split the dataset into private and public datasets\n",
        "private_loader, public_loader = private_public_allocator(testset)\n",
        "print(len(private_loader))\n",
        "print(len(public_loader))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "110\n",
            "47\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YXMsrC6WGctw",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "0715c550-fce6-4835-9c6b-eac515922bfe"
      },
      "source": [
        "# get models trained by each teacher\n",
        "teacher_models = []\n",
        "for teacher_train, teacher_valid in zip(teachers_train, teachers_valid):\n",
        "  model = Classifier()\n",
        "  criterion = nn.NLLLoss()\n",
        "  optimizer = optim.Adam(model.parameters(), lr=0.003)\n",
        "  teacher_model = train(model, criterion, optimizer, teacher_train, teacher_valid)\n",
        "  teacher_models.append(teacher_model)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch: 1/10..  Training Loss: 1.349..  Valid Loss: 0.518..  Valid Accuracy: 0.847  \n",
            "Epoch: 2/10..  Training Loss: 0.592..  Valid Loss: 0.354..  Valid Accuracy: 0.894  \n",
            "Epoch: 3/10..  Training Loss: 0.447..  Valid Loss: 0.345..  Valid Accuracy: 0.894  \n",
            "Epoch: 4/10..  Training Loss: 0.383..  Valid Loss: 0.315..  Valid Accuracy: 0.907  \n",
            "Epoch: 5/10..  Training Loss: 0.347..  Valid Loss: 0.307..  Valid Accuracy: 0.905  \n",
            "Epoch: 6/10..  Training Loss: 0.323..  Valid Loss: 0.283..  Valid Accuracy: 0.917  \n",
            "Epoch: 7/10..  Training Loss: 0.289..  Valid Loss: 0.269..  Valid Accuracy: 0.924  \n",
            "Epoch: 8/10..  Training Loss: 0.290..  Valid Loss: 0.228..  Valid Accuracy: 0.934  \n",
            "Epoch: 9/10..  Training Loss: 0.248..  Valid Loss: 0.221..  Valid Accuracy: 0.939  \n",
            "Epoch: 10/10..  Training Loss: 0.261..  Valid Loss: 0.229..  Valid Accuracy: 0.929  \n",
            "Epoch: 1/10..  Training Loss: 1.423..  Valid Loss: 0.634..  Valid Accuracy: 0.805  \n",
            "Epoch: 2/10..  Training Loss: 0.655..  Valid Loss: 0.415..  Valid Accuracy: 0.874  \n",
            "Epoch: 3/10..  Training Loss: 0.482..  Valid Loss: 0.433..  Valid Accuracy: 0.858  \n",
            "Epoch: 4/10..  Training Loss: 0.429..  Valid Loss: 0.435..  Valid Accuracy: 0.859  \n",
            "Epoch: 5/10..  Training Loss: 0.378..  Valid Loss: 0.295..  Valid Accuracy: 0.916  \n",
            "Epoch: 6/10..  Training Loss: 0.340..  Valid Loss: 0.283..  Valid Accuracy: 0.917  \n",
            "Epoch: 7/10..  Training Loss: 0.313..  Valid Loss: 0.240..  Valid Accuracy: 0.935  \n",
            "Epoch: 8/10..  Training Loss: 0.276..  Valid Loss: 0.260..  Valid Accuracy: 0.923  \n",
            "Epoch: 9/10..  Training Loss: 0.248..  Valid Loss: 0.270..  Valid Accuracy: 0.928  \n",
            "Epoch: 10/10..  Training Loss: 0.267..  Valid Loss: 0.308..  Valid Accuracy: 0.905  \n",
            "Epoch: 1/10..  Training Loss: 1.373..  Valid Loss: 0.694..  Valid Accuracy: 0.760  \n",
            "Epoch: 2/10..  Training Loss: 0.639..  Valid Loss: 0.507..  Valid Accuracy: 0.843  \n",
            "Epoch: 3/10..  Training Loss: 0.473..  Valid Loss: 0.428..  Valid Accuracy: 0.866  \n",
            "Epoch: 4/10..  Training Loss: 0.416..  Valid Loss: 0.392..  Valid Accuracy: 0.885  \n",
            "Epoch: 5/10..  Training Loss: 0.376..  Valid Loss: 0.409..  Valid Accuracy: 0.877  \n",
            "Epoch: 6/10..  Training Loss: 0.357..  Valid Loss: 0.374..  Valid Accuracy: 0.886  \n",
            "Epoch: 7/10..  Training Loss: 0.327..  Valid Loss: 0.369..  Valid Accuracy: 0.899  \n",
            "Epoch: 8/10..  Training Loss: 0.300..  Valid Loss: 0.370..  Valid Accuracy: 0.884  \n",
            "Epoch: 9/10..  Training Loss: 0.284..  Valid Loss: 0.339..  Valid Accuracy: 0.899  \n",
            "Epoch: 10/10..  Training Loss: 0.252..  Valid Loss: 0.342..  Valid Accuracy: 0.909  \n",
            "Epoch: 1/10..  Training Loss: 1.235..  Valid Loss: 0.522..  Valid Accuracy: 0.839  \n",
            "Epoch: 2/10..  Training Loss: 0.546..  Valid Loss: 0.367..  Valid Accuracy: 0.890  \n",
            "Epoch: 3/10..  Training Loss: 0.413..  Valid Loss: 0.359..  Valid Accuracy: 0.884  \n",
            "Epoch: 4/10..  Training Loss: 0.383..  Valid Loss: 0.347..  Valid Accuracy: 0.894  \n",
            "Epoch: 5/10..  Training Loss: 0.313..  Valid Loss: 0.357..  Valid Accuracy: 0.892  \n",
            "Epoch: 6/10..  Training Loss: 0.289..  Valid Loss: 0.326..  Valid Accuracy: 0.906  \n",
            "Epoch: 7/10..  Training Loss: 0.288..  Valid Loss: 0.296..  Valid Accuracy: 0.924  \n",
            "Epoch: 8/10..  Training Loss: 0.239..  Valid Loss: 0.305..  Valid Accuracy: 0.914  \n",
            "Epoch: 9/10..  Training Loss: 0.212..  Valid Loss: 0.335..  Valid Accuracy: 0.907  \n",
            "Epoch: 10/10..  Training Loss: 0.217..  Valid Loss: 0.278..  Valid Accuracy: 0.926  \n",
            "Epoch: 1/10..  Training Loss: 1.368..  Valid Loss: 0.684..  Valid Accuracy: 0.755  \n",
            "Epoch: 2/10..  Training Loss: 0.654..  Valid Loss: 0.454..  Valid Accuracy: 0.862  \n",
            "Epoch: 3/10..  Training Loss: 0.498..  Valid Loss: 0.398..  Valid Accuracy: 0.872  \n",
            "Epoch: 4/10..  Training Loss: 0.434..  Valid Loss: 0.468..  Valid Accuracy: 0.843  \n",
            "Epoch: 5/10..  Training Loss: 0.392..  Valid Loss: 0.295..  Valid Accuracy: 0.902  \n",
            "Epoch: 6/10..  Training Loss: 0.320..  Valid Loss: 0.295..  Valid Accuracy: 0.914  \n",
            "Epoch: 7/10..  Training Loss: 0.313..  Valid Loss: 0.275..  Valid Accuracy: 0.915  \n",
            "Epoch: 8/10..  Training Loss: 0.303..  Valid Loss: 0.371..  Valid Accuracy: 0.893  \n",
            "Epoch: 9/10..  Training Loss: 0.263..  Valid Loss: 0.291..  Valid Accuracy: 0.919  \n",
            "Epoch: 10/10..  Training Loss: 0.279..  Valid Loss: 0.301..  Valid Accuracy: 0.916  \n",
            "Epoch: 1/10..  Training Loss: 1.278..  Valid Loss: 0.631..  Valid Accuracy: 0.773  \n",
            "Epoch: 2/10..  Training Loss: 0.604..  Valid Loss: 0.475..  Valid Accuracy: 0.850  \n",
            "Epoch: 3/10..  Training Loss: 0.504..  Valid Loss: 0.374..  Valid Accuracy: 0.895  \n",
            "Epoch: 4/10..  Training Loss: 0.412..  Valid Loss: 0.415..  Valid Accuracy: 0.867  \n",
            "Epoch: 5/10..  Training Loss: 0.365..  Valid Loss: 0.324..  Valid Accuracy: 0.900  \n",
            "Epoch: 6/10..  Training Loss: 0.327..  Valid Loss: 0.293..  Valid Accuracy: 0.914  \n",
            "Epoch: 7/10..  Training Loss: 0.329..  Valid Loss: 0.338..  Valid Accuracy: 0.905  \n",
            "Epoch: 8/10..  Training Loss: 0.308..  Valid Loss: 0.275..  Valid Accuracy: 0.921  \n",
            "Epoch: 9/10..  Training Loss: 0.298..  Valid Loss: 0.292..  Valid Accuracy: 0.914  \n",
            "Epoch: 10/10..  Training Loss: 0.274..  Valid Loss: 0.275..  Valid Accuracy: 0.916  \n",
            "Epoch: 1/10..  Training Loss: 1.329..  Valid Loss: 0.643..  Valid Accuracy: 0.798  \n",
            "Epoch: 2/10..  Training Loss: 0.671..  Valid Loss: 0.430..  Valid Accuracy: 0.865  \n",
            "Epoch: 3/10..  Training Loss: 0.504..  Valid Loss: 0.368..  Valid Accuracy: 0.878  \n",
            "Epoch: 4/10..  Training Loss: 0.404..  Valid Loss: 0.321..  Valid Accuracy: 0.904  \n",
            "Epoch: 5/10..  Training Loss: 0.392..  Valid Loss: 0.282..  Valid Accuracy: 0.919  \n",
            "Epoch: 6/10..  Training Loss: 0.381..  Valid Loss: 0.333..  Valid Accuracy: 0.908  \n",
            "Epoch: 7/10..  Training Loss: 0.334..  Valid Loss: 0.342..  Valid Accuracy: 0.897  \n",
            "Epoch: 8/10..  Training Loss: 0.312..  Valid Loss: 0.290..  Valid Accuracy: 0.919  \n",
            "Epoch: 9/10..  Training Loss: 0.293..  Valid Loss: 0.341..  Valid Accuracy: 0.901  \n",
            "Epoch: 10/10..  Training Loss: 0.273..  Valid Loss: 0.285..  Valid Accuracy: 0.918  \n",
            "Epoch: 1/10..  Training Loss: 1.451..  Valid Loss: 0.612..  Valid Accuracy: 0.803  \n",
            "Epoch: 2/10..  Training Loss: 0.641..  Valid Loss: 0.502..  Valid Accuracy: 0.846  \n",
            "Epoch: 3/10..  Training Loss: 0.537..  Valid Loss: 0.461..  Valid Accuracy: 0.865  \n",
            "Epoch: 4/10..  Training Loss: 0.446..  Valid Loss: 0.377..  Valid Accuracy: 0.890  \n",
            "Epoch: 5/10..  Training Loss: 0.401..  Valid Loss: 0.359..  Valid Accuracy: 0.889  \n",
            "Epoch: 6/10..  Training Loss: 0.363..  Valid Loss: 0.381..  Valid Accuracy: 0.882  \n",
            "Epoch: 7/10..  Training Loss: 0.320..  Valid Loss: 0.349..  Valid Accuracy: 0.903  \n",
            "Epoch: 8/10..  Training Loss: 0.304..  Valid Loss: 0.320..  Valid Accuracy: 0.899  \n",
            "Epoch: 9/10..  Training Loss: 0.281..  Valid Loss: 0.333..  Valid Accuracy: 0.903  \n",
            "Epoch: 10/10..  Training Loss: 0.282..  Valid Loss: 0.315..  Valid Accuracy: 0.912  \n",
            "Epoch: 1/10..  Training Loss: 1.256..  Valid Loss: 0.720..  Valid Accuracy: 0.750  \n",
            "Epoch: 2/10..  Training Loss: 0.603..  Valid Loss: 0.437..  Valid Accuracy: 0.865  \n",
            "Epoch: 3/10..  Training Loss: 0.463..  Valid Loss: 0.397..  Valid Accuracy: 0.874  \n",
            "Epoch: 4/10..  Training Loss: 0.390..  Valid Loss: 0.366..  Valid Accuracy: 0.882  \n",
            "Epoch: 5/10..  Training Loss: 0.354..  Valid Loss: 0.374..  Valid Accuracy: 0.885  \n",
            "Epoch: 6/10..  Training Loss: 0.324..  Valid Loss: 0.315..  Valid Accuracy: 0.907  \n",
            "Epoch: 7/10..  Training Loss: 0.305..  Valid Loss: 0.301..  Valid Accuracy: 0.908  \n",
            "Epoch: 8/10..  Training Loss: 0.300..  Valid Loss: 0.267..  Valid Accuracy: 0.922  \n",
            "Epoch: 9/10..  Training Loss: 0.282..  Valid Loss: 0.303..  Valid Accuracy: 0.906  \n",
            "Epoch: 10/10..  Training Loss: 0.242..  Valid Loss: 0.348..  Valid Accuracy: 0.906  \n",
            "Epoch: 1/10..  Training Loss: 1.237..  Valid Loss: 0.562..  Valid Accuracy: 0.820  \n",
            "Epoch: 2/10..  Training Loss: 0.529..  Valid Loss: 0.392..  Valid Accuracy: 0.874  \n",
            "Epoch: 3/10..  Training Loss: 0.410..  Valid Loss: 0.390..  Valid Accuracy: 0.887  \n",
            "Epoch: 4/10..  Training Loss: 0.350..  Valid Loss: 0.305..  Valid Accuracy: 0.912  \n",
            "Epoch: 5/10..  Training Loss: 0.290..  Valid Loss: 0.274..  Valid Accuracy: 0.915  \n",
            "Epoch: 6/10..  Training Loss: 0.260..  Valid Loss: 0.259..  Valid Accuracy: 0.921  \n",
            "Epoch: 7/10..  Training Loss: 0.245..  Valid Loss: 0.256..  Valid Accuracy: 0.919  \n",
            "Epoch: 8/10..  Training Loss: 0.232..  Valid Loss: 0.223..  Valid Accuracy: 0.930  \n",
            "Epoch: 9/10..  Training Loss: 0.230..  Valid Loss: 0.237..  Valid Accuracy: 0.924  \n",
            "Epoch: 10/10..  Training Loss: 0.214..  Valid Loss: 0.272..  Valid Accuracy: 0.928  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FuUFTUtyTrDT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# predict labels for testset from each teacher\n",
        "# row = num of test data\n",
        "# col = predictions from each teacher\n",
        "preds = get_label_testset(teacher_models, public_loader)\n",
        "preds = np.array([np.array(p) for p in preds]).transpose(1, 0)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mAGGbiy9pV7e",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "2f5b246c-b718-464a-aad6-e2f580fafab2"
      },
      "source": [
        "preds_with_noise = add_global_noise(preds, epsilon=0.6)\n",
        "len(preds_with_noise)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3000"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PoGCUUNLJb9Q",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "bc9807b1-c301-44ee-815e-3afcfabeb4e9"
      },
      "source": [
        "# PATE analysis\n",
        "data_dep_eps, data_ind_eps = pate.perform_analysis(teacher_preds=preds.T, indices=preds_with_noise, noise_eps=0.1, delta=1e-5)\n",
        "print('Data dependent epsilon:', data_dep_eps)\n",
        "print('Data independent epsilon:', data_ind_eps)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Data dependent epsilon: 131.5129254649778\n",
            "Data independent epsilon: 131.51292546497027\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ifLpJDqlNcVS",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "d65c72dd-eb95-42e5-85e1-f91f01561d6e"
      },
      "source": [
        "# create new dataloader for the public data\n",
        "# using the original function\n",
        "public_loader_labeled = create_labeled_public_db(public_loader, preds_with_noise)\n",
        "len(public_loader_labeled)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "47"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tENIpozxYkhe",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        },
        "outputId": "f4b38a45-1529-4489-fe1d-d736f85fc815"
      },
      "source": [
        "# train model using private and public datasets\n",
        "my_model = Classifier()\n",
        "criterion = nn.NLLLoss()\n",
        "optimizer = optim.Adam(my_model.parameters(), lr=1e-3)\n",
        "train(model=my_model, criterion=criterion, optimizer=optimizer, trainloader=private_loader, validloader=public_loader_labeled)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch: 1/10..  Training Loss: 1.247..  Valid Loss: 0.735..  Valid Accuracy: 0.810  \n",
            "Epoch: 2/10..  Training Loss: 0.558..  Valid Loss: 0.612..  Valid Accuracy: 0.859  \n",
            "Epoch: 3/10..  Training Loss: 0.420..  Valid Loss: 0.575..  Valid Accuracy: 0.884  \n",
            "Epoch: 4/10..  Training Loss: 0.349..  Valid Loss: 0.578..  Valid Accuracy: 0.894  \n",
            "Epoch: 5/10..  Training Loss: 0.310..  Valid Loss: 0.538..  Valid Accuracy: 0.909  \n",
            "Epoch: 6/10..  Training Loss: 0.264..  Valid Loss: 0.574..  Valid Accuracy: 0.902  \n",
            "Epoch: 7/10..  Training Loss: 0.254..  Valid Loss: 0.601..  Valid Accuracy: 0.910  \n",
            "Epoch: 8/10..  Training Loss: 0.230..  Valid Loss: 0.630..  Valid Accuracy: 0.904  \n",
            "Epoch: 9/10..  Training Loss: 0.212..  Valid Loss: 0.614..  Valid Accuracy: 0.905  \n",
            "Epoch: 10/10..  Training Loss: 0.186..  Valid Loss: 0.620..  Valid Accuracy: 0.912  \n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Classifier(\n",
              "  (fc1): Linear(in_features=784, out_features=256, bias=True)\n",
              "  (fc2): Linear(in_features=256, out_features=128, bias=True)\n",
              "  (fc3): Linear(in_features=128, out_features=64, bias=True)\n",
              "  (fc4): Linear(in_features=64, out_features=10, bias=True)\n",
              "  (dropout): Dropout(p=0.2, inplace=False)\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    }
  ]
}