real_classification_experiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fair Dummies: Classification Example\n",
    "\n",
    "This notebook implements a the fair dummies framework for learning predictive models that approximately satisfy the equalized odds notion of fairness.\n",
    "\n",
    "Paper: \"Achieving Equalized Odds by Resampling Sensitive Attributes,\" Y. Romano, S. Bates, and E. J. Candès, 2020\n",
    "\n",
    "### Proposed approach\n",
    "\n",
    "__Core idea__: fit a regression function, minimizing\n",
    "\n",
    "$$ \\text{loss = prediction error + distance to equalized odds}$$\n",
    "\n",
    "\n",
    "__Input__: $  \\{(X_i,A_i,Y_i)\\}_{i=1}^n \\sim P_{XAY}$ training data\n",
    "    \n",
    "\n",
    "__Step 1__: sample dummy protected attributes\n",
    "$$\n",
    "\\tilde{A}_i \\sim P_{A|Y}(A \\mid Y=y_{i}) \\quad \\forall \\ i=1,2,\\dots,n \\nonumber\n",
    "$$\n",
    "\n",
    "$A \\in \\{0,1\\}$? generate $\\tilde{A}$ using a biased coin-flip, with\n",
    "$$\n",
    "P\\{A=1|Y=y\\} = \\frac{P\\{y \\mid A=1\\}P\\{A=1\\}}{P\\{y \\mid A=1\\}P\\{A=1\\} + P\\{y \\mid A=0\\}P\\{A=0\\}}\n",
    "$$\n",
    "    \n",
    "__Step 2__: fit a regression function on $\\{(X_i, A_i, Y_i)\\}_{i=1}^n$\n",
    "\n",
    "$$\n",
    "        \\hat{f}(\\cdot) \\,= \\, \\underset{f \\in \\mathcal{F}}{\\mathrm{arg min}} \\, \\frac{1-\\lambda}{n} \\sum_{i=1}^n \\text{cross-entropy}(Y_i,f(X_i))  + \\lambda \\mathcal{D}\\left( [\\hat{\\mathbf{Y}}, \\mathbf{A}, \\mathbf{Y}] , [\\hat{\\mathbf{Y}}, \\tilde{\\mathbf{A}}, \\mathbf{Y}] \\right) \\nonumber\n",
    "$$\n",
    "\n",
    "where\n",
    "    \n",
    "$$\n",
    "\\hat{\\mathbf{Y}} = \\left[f(X_{1}), f(X_{2}), \\dots, f(X_{n})\\right]^T \\ ; \\ \\mathbf{A} = \\left[A_{1}, A_{2}, \\dots, A_{n}\\right]^T \\ ; \\ \\tilde{\\mathbf{A}} = \\left[\\tilde{A}_{1}, \\tilde{A}_{2}, \\dots, \\tilde{A}_{n}\\right]^T \\ ; \\ \\mathbf{Y} = \\left[Y_{1}, Y_{2}, \\dots, Y_{n}\\right]^T\n",
    "$$\n",
    "\n",
    "and $\\mathcal{D}\\left( \\mathbf{Z}_1, \\mathbf{Z}_2 \\right)$ tests whether $P_{Z_1} = \\ P_{Z_2}$ given $\\mathbf{Z}_1, \\mathbf{Z}_2$,  here it implemented as a classifier two-sample test + second moment penalty (see paper for more details)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "base_path = os.getcwd() + '/data/'\n",
    "\n",
    "import torch\n",
    "import random\n",
    "import get_dataset\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from fair_dummies import utility_functions\n",
    "from fair_dummies import fair_dummies_learning\n",
    "\n",
    "seed = 123"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.seed(seed)\n",
    "np.random.seed(seed)\n",
    "torch.manual_seed(seed)\n",
    "if torch.cuda.is_available():\n",
    "    torch.cuda.manual_seed_all(seed)\n",
    "\n",
    "dataset = \"nursery\"\n",
    "\n",
    "# use minibatch sgd\n",
    "batch_size = 32\n",
    "\n",
    "# step size to minimize loss\n",
    "lr_loss = 0.5\n",
    "\n",
    "# step size used to fit binary classifier (discriminator)\n",
    "lr_dis = 0.5\n",
    "\n",
    "# pre-train discriminator and predictive model\n",
    "pre_training_steps = 2\n",
    "\n",
    "# inner epochs to fit loss\n",
    "loss_steps = 1\n",
    "\n",
    "# inner epochs to fit discriminator classifier\n",
    "dis_steps = 1\n",
    "\n",
    "# equalized odds penalty\n",
    "mu_val = 0.9\n",
    "second_scale = 0.00001\n",
    "\n",
    "\n",
    "# total number of epochs\n",
    "epochs = 50\n",
    "\n",
    "# utility loss\n",
    "cost_pred = torch.nn.CrossEntropyLoss()\n",
    "\n",
    "# base predictive model\n",
    "model_type = \"deep_model\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dataset: nursery\n",
      "training samples (used to fit predictive model) = 7774 p = 13\n",
      "holdout samples (used to fit fair-dummies test statistics function) = 2592\n",
      "test samples = 2592\n"
     ]
    }
   ],
   "source": [
    "print(\"dataset: \" + dataset)\n",
    "\n",
    "X, A, Y, X_cal, A_cal, Y_cal, X_test, A_test, Y_test = get_dataset.get_train_test_data(base_path, dataset, seed)\n",
    "\n",
    "in_shape = X.shape[1]\n",
    "num_classes = len(np.unique(Y))\n",
    "\n",
    "print(\"training samples (used to fit predictive model) = \" + str(X.shape[0]) + \" p = \" + str(X.shape[1]))\n",
    "print(\"holdout samples (used to fit fair-dummies test statistics function) = \" + str(X_cal.shape[0]))\n",
    "print(\"test samples = \" + str(X_test.shape[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/romano/opt/anaconda3/lib/python3.7/site-packages/torch/nn/_reduction.py:43: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.\n",
      "  warnings.warn(warning.format(ret))\n"
     ]
    }
   ],
   "source": [
    "model = fair_dummies_learning.EquiClassLearner(lr=lr_loss,\n",
    "                                               pretrain_pred_epochs=pre_training_steps,\n",
    "                                               pretrain_dis_epochs=pre_training_steps,\n",
    "                                               epochs=epochs,\n",
    "                                               loss_steps=loss_steps,\n",
    "                                               dis_steps=dis_steps,\n",
    "                                               cost_pred=cost_pred,\n",
    "                                               in_shape=in_shape,\n",
    "                                               batch_size=batch_size,\n",
    "                                               model_type=model_type,\n",
    "                                               lambda_vec=mu_val,\n",
    "                                               second_moment_scaling=second_scale,\n",
    "                                               num_classes=num_classes)\n",
    "\n",
    "input_data_train = np.concatenate((A[:,np.newaxis],X),1)\n",
    "model.fit(input_data_train, Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_data_cal = np.concatenate((A_cal[:,np.newaxis],X_cal),1)\n",
    "Yhat_out_cal = model.predict(input_data_cal)\n",
    "\n",
    "input_data_test = np.concatenate((A_test[:,np.newaxis],X_test),1)\n",
    "Yhat_out_test = model.predict(input_data_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "misclassification error = 0.2569444444444444\n",
      "Init Loss = 0.6796401\n",
      "Final Loss = 0.45065048\n",
      "Fair dummies test (classification score), p-value: 0.5534465534465535\n"
     ]
    }
   ],
   "source": [
    "misclassification = 1.0-utility_functions.compute_acc_numpy(Yhat_out_test, Y_test)\n",
    "print(\"misclassification error = \" + str(misclassification))\n",
    "\n",
    "p_val = utility_functions.fair_dummies_test_classification(Yhat_out_cal,\n",
    "                                                           A_cal,\n",
    "                                                           Y_cal,\n",
    "                                                           Yhat_out_test,\n",
    "                                                           A_test,\n",
    "                                                           Y_test,\n",
    "                                                           num_reps = 1,\n",
    "                                                           num_p_val_rep=1000,\n",
    "                                                           reg_func_name=\"Net\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}