label_errors/Lab - Label Errors.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zwfWqPeA1zX0"
   },
   "source": [
    "# Lab — Label Errors\n",
    "\n",
    "This lab highlights data-centric AI techniques (using [confident learning](https://jair.org/index.php/jair/article/view/12125)) to improve the accuracy of an XGBoost classifier on a noisy dataset that has label errors.\n",
    "\n",
    "The DCAI techniques demonstrated in this lab involve optimizing the dataset itself rather than altering the model's architecture or hyperparameters. As a result, it is possible to achieve further improvements in accuracy by fine-tuning the model in conjunction with the newly enhanced data, but that is not the focus of this lab.\n",
    "\n",
    "In this lab, we will:\n",
    "\n",
    "- Establish a baseline [XGBoost](https://xgboost.readthedocs.io/) model accuracy on the original data\n",
    "- Automatically find mislabeled data points by:\n",
    "    - Computing out-of-sample predicted probabilities\n",
    "    - Estimating the number of label errors using confident learning\n",
    "    - Ranking errors, using the number of label errors as a cutoff in identifying issues\n",
    "- Remove the bad data\n",
    "- Retrain the exact same XGBoost model to see the improvement in test accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Software installation\n",
    "\n",
    "This lab relies on a couple PyPI packages. If you don't have them installed, run the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install xgboost==1.7 scikit-learn pandas cleanlab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "je7P55z4RwX_"
   },
   "source": [
    "## Setup and Data Processing\n",
    "\n",
    "Let's take a look at the dataset used in this lab, a tabular dataset of student grades.\n",
    "\n",
    "The data includes three exam scores (numerical features), a written note (categorical feature with missing values), and a (noisy) letter grade (categorical label). Our aim is to train a model to classify the grade for each student based on the other features.\n",
    "\n",
    "In this dataset, 20% of the grade labels are actually incorrect (the `noisy_letter_grade` column). Synthetic noise was added to this dataset for the purpose of this lab. In this lab, we have access to the true letter grade each student should have received (the `letter_grade` column), which we use for evaluating both the underlying accuracy of model predictions and how well our approach detects which data are mislabeled. We are careful to only use these true grades for evaluation, not for model training.\n",
    "\n",
    "In the real world, you don't have access to the true labels (you only observe the `noisy_letter_grade`, not the true `letter_grade`). So when evaluating models in the real world, you have to be careful to make sure that your test set is free of error (using methods like those covered in this lab, ideally combined with human review)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "nQVmMBQOS43j",
    "outputId": "54a45659-ecb6-47fe-acfa-d0424027af47"
   },
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "from sklearn import preprocessing\n",
    "from sklearn.model_selection import cross_val_predict\n",
    "from sklearn.metrics import accuracy_score\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "df = pd.read_csv(\"./student-grades.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_c = df.copy()\n",
    "# Transform letter grades and notes to categorical numbers.\n",
    "# Necessary for XGBoost.\n",
    "df['letter_grade'] = preprocessing.LabelEncoder().fit_transform(df['letter_grade'])\n",
    "df['noisy_letter_grade'] = preprocessing.LabelEncoder().fit_transform(df['noisy_letter_grade'])\n",
    "df['notes'] = preprocessing.LabelEncoder().fit_transform(df[\"notes\"])\n",
    "df['notes'] = df['notes'].astype('category')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dVH_iciASD9F"
   },
   "source": [
    "# Get What We Need\n",
    "\n",
    "To apply confident learning (the technique explained in today's lecture), we need to obtain [**out-of-sample** predicted probabilities](https://docs.cleanlab.ai/stable/tutorials/pred_probs_cross_val.html#out-of-sample-predicted-probabilities) for all of our data. To do this, we can use K-fold cross validation: for each fold, we will train on some subset of our data and get predictions on the rest of the data that was _not_ used for training.\n",
    "\n",
    "We need to choose a model in order to do this. For this lab, we'll use [XGBoost](https://xgboost.readthedocs.io/), a library implementing gradient-boosted decision trees, a class of model commonly used for tabular data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gCS19IqJsQUL",
    "outputId": "fd89b945-6793-4b3a-a017-7b19f8e6a29b"
   },
   "outputs": [],
   "source": [
    "# Prepare training data (remove labels from the dataframe) and labels\n",
    "data = df.drop(['stud_ID', 'letter_grade', 'noisy_letter_grade'], axis=1)\n",
    "labels = df['noisy_letter_grade']\n",
    "\n",
    "# XGBoost(experimental) supports categorical data.\n",
    "# Here we use default hyperparameters for simplicity.\n",
    "# Get out-of-sample predicted probabilities and check model accuracy.\n",
    "model = XGBClassifier(tree_method=\"hist\", enable_categorical=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 1: getting out-of-sample predicted probabilities\n",
    "\n",
    "Compute out-of-sample predicted probabilities for every data point. You can do this manually using for loops and multiple invocations of model training and prediction, or you can use scikit-learn's [cross_val_predict](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_predict.html) (if you're using this function, take a look at the documentations, and in particular, the `method=` keyword argument)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pred_probs should be a Nx5 matrix of out-of-sample predicted probabilities, with N = len(data)\n",
    "\n",
    "pred_probs = None # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checking model accuracy on original data\n",
    "\n",
    "Now that we have out-of-sample predicted probabilities, we can also check the model's (cross-val) accuracy on the original (noisy) data, so we'll have a baseline to compare our final results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = np.argmax(pred_probs, axis=1)\n",
    "acc_original = accuracy_score(preds, labels)\n",
    "print(f\"Accuracy with original data: {round(acc_original*100,1)}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding label issues automatically\n",
    "\n",
    "We count label issues using confident learning. First, we need to compute class thresholds for the different classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 2: computing class thresholds\n",
    "\n",
    "Implement the Confident Learning algorithm for computing class thresholds for the 5 classes. You can refer to slide 26 from today's lecture or see equation 2 in [this paper](https://jair.org/index.php/jair/article/view/12125).\n",
    "\n",
    "The class threshold for each class is the model's expected (average) self-confidence for each class. In other words, to compute the threshold for a particular class, you can average the predicted probability for that class, for all datapoints that are labeled with that particular class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_class_thresholds(pred_probs: np.ndarray, labels: np.ndarray) -> np.ndarray:\n",
    "    # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# should be a numpy array of length 5\n",
    "thresholds = compute_class_thresholds(pred_probs, labels.to_numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 3: constructing the confident joint\n",
    "\n",
    "Next, we compute the confident joint, a matrix that counts the number of label errors for each noisy label $\\tilde{y}$ and true label $y^*$. You can follow the algorithm that we walked through in slide 27 from today's lecture, or see equation 1 in [this paper](https://jair.org/index.php/jair/article/view/12125).\n",
    "\n",
    "The confident joint C is a K x K matrix (with K = 5 for this dataset), where `C[i][j]` is an estimate of the count of the number of data points with noisy label `i` and true label `j`. From lecture, recall that we put a data point in bin `(i, j)` if its given label is `i`, and its predicted probability for class `j` is above the threshold for class `j` (`thresholds[j]`). Each data point should only go in a single bin; if a data point's predicted probability is above the class threshold for multiple classes, it goes in the bin for which it has the highest predicted probability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_confident_joint(pred_probs: np.ndarray, labels: np.ndarray, thresholds: np.ndarray) -> np.ndarray:\n",
    "    # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "C = compute_confident_joint(pred_probs, labels.to_numpy(), thresholds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 4: count the number of label issues\n",
    "\n",
    "Now that we have the confident joint C, we can count the estimated number of label issues in our dataset. Recall that this is the sum of the off-diagonal entries (the cases where we estimate that a label has been flipped)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_label_issues = None # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Estimated noise rate: {:.1f}%'.format(100*num_label_issues / pred_probs.shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 5: filter out label issues\n",
    "\n",
    "In this lab, our approach to identifying issues is to rank the data points by a score (\"self-confidence\", the model's predicted probability for a data point's given label) and then take the top `num_label_issues` of those.\n",
    "\n",
    "First, we want to compute the model's _self-confidence_ for each data point. For a data point `i`, that is `pred_probs[i, labels[i]]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this should be a numpy array of length 941 of probabilities\n",
    "self_confidences = None # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we rank the _indices_ of the data points by the self-confidence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this should be a numpy array of length 941 of integer indices\n",
    "ranked_indices = None # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's compute the indices of label issues as the top `num_label_issues` items in the `ranked_indices`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "issue_idx = None # YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at a couple of the highest-ranked data points (most likely to be label issues):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df_c.iloc[ranked_indices[:5]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PrvJHkPzSq6Q"
   },
   "source": [
    "# How'd We Do?\n",
    "\n",
    "Let's go a step further and see how we did at automatically identifying which data points are mislabeled. If we take the intersection of the labels errors identified by Confident Learning and the true label errors, we see that our approach was able to identify 75% of the label errors correctly (based on predictions from a model that is only 67% accurate). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9O2a6urWc1DA",
    "outputId": "f88b5ce6-33f2-4ef6-e774-19013c33f0e8"
   },
   "outputs": [],
   "source": [
    "# Computing percentage of true errors identified. \n",
    "true_error_idx = df[df.letter_grade != df.noisy_letter_grade].index.values\n",
    "cl_acc = len(set(true_error_idx).intersection(set(issue_idx)))/len(true_error_idx)\n",
    "print(f\"Percentage of errors found: {round(cl_acc*100,1)}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YzxXoDOqSzn-"
   },
   "source": [
    "# Train a More Robust Model\n",
    "\n",
    "Now that we have the indices of potential label errors within our data, let's remove them from our data, retrain our model, and see what improvement we can gain.\n",
    "\n",
    "Keep in mind that our baseline model from above, trained on the original data using the `noisy_letter_grade` as the prediction label, achieved a cross-validation accuracy of 67%.\n",
    "\n",
    "Let's use a very simple method to handle these label errors and just drop them entirely from the data and retrain our exact same `XGBClassifier`. In a real-world application, a better approach might be to have humans review the issues and _correct_ the labels rather than dropping the data points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FsQFmy7xgSUa",
    "outputId": "a33e17ab-c197-4f95-c9c1-0473c16af313"
   },
   "outputs": [],
   "source": [
    "# Remove the label errors found by Confident Learning\n",
    "data = df.drop(issue_idx)\n",
    "clean_labels = data['noisy_letter_grade']\n",
    "data = data.drop(['stud_ID', 'letter_grade', 'noisy_letter_grade'], axis=1)\n",
    "\n",
    "# Train a more robust classifier with less erroneous data\n",
    "model = XGBClassifier(tree_method=\"hist\", enable_categorical=True)\n",
    "clean_pred_probs = cross_val_predict(model, data, clean_labels, method='predict_proba')\n",
    "clean_preds = np.argmax(clean_pred_probs, axis=1)\n",
    "\n",
    "acc_clean = accuracy_score(clean_preds, clean_labels)\n",
    "print(f\"Accuracy with original data: {round(acc_original*100, 1)}%\")\n",
    "print(f\"Accuracy with errors found by Confident Learning removed: {round(acc_clean*100, 1)}%\")\n",
    "\n",
    "# Compute reduction in error.\n",
    "err = ((1-acc_original)-(1-acc_clean))/(1-acc_original)\n",
    "print(f\"Reduction in error: {round(err*100,1)}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9J9clVf1UzQZ"
   },
   "source": [
    "After removing the suspected label issues, our model's new cross-validation accuracy is now 90%, which means we **reduced the error-rate of the model by 70%** (the original model had 67% accuracy). \n",
    "\n",
    "**Note: throughout this entire process we never changed any code related to model architecture/hyperparameters, training, or data preprocessing!  This improvement is strictly coming from increasing the quality of our data which leaves additional room for additional optimizations on the modeling side.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-W-Lo82SVp7I"
   },
   "source": [
    "# Conclusion\n",
    "\n",
    "For the student grades dataset, we found that simply dropping identified label errors and retraining the model resulted in a 70% reduction in prediction error on our classification problem (with accuracy improving from 67% to 90%).\n",
    "\n",
    "An implementation of the Confident Learning algorithm (and much more) is available in the [cleanlab](https://github.com/cleanlab/cleanlab) library on GitHub. This is how today's lab assignment can be done in a single line of code with Cleanlab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cleanlab\n",
    "\n",
    "cl_issue_idx = cleanlab.filter.find_label_issues(labels, pred_probs, return_indices_ranked_by='self_confidence')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_c.iloc[cl_issue_idx[:5]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Advanced topic_: you might notice that the above `cl_issue_idx` differs in length (by a little bit) from our `issue_idx`. The reason for this is that we implemented a slightly simplified version of the algorithm in this lab. We skipped a calibration step after computing the confident joint that makes the confident joint have the true noisy prior $p(labels)$ (summed over columns for each row) and also add up to the total number of examples. If you're interested in the details of this, see equation 3 and the subsequent explanation in the [paper](https://jair.org/index.php/jair/article/view/12125)."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}