4 Decision Tree.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "56733ad6-2e01-440a-b1f2-ab5fdb36a2c2",
   "metadata": {},
   "source": [
    "# Decision tree model\n",
    "Author: Roddy Jaques <br>\n",
    "*NHS Blood and Transplant*\n",
    "***\n",
    "## Assessing the predictive ability of a decision tree model\n",
    "A decision tree model was chosen to be compared to the benchmark logistic regression model. A decision tree was chosen because decision trees model categorical data well and can be graphically displayed, which makes the model explainable and transparent. Explainability and transparency are important for any models being used in a medical or clinical setting.\n",
    "<br><br>\n",
    "First the data is imported and split into testing and training sets..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "349c70cd-a4aa-4d91-b5ba-ffeb4c7594af",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import relevant libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.tree import DecisionTreeClassifier, plot_tree\n",
    "import sklearn.metrics as mets\n",
    "%matplotlib inline\n",
    "\n",
    "# Function to print confusion matrix, balanced accuracy and accuracy for a set of actual and predicted labels\n",
    "def show_metrics(actual,predict):\n",
    "    \"\"\" Prints the confusion matrix, balanced accuracy and accuracy given datasets of actual and predicted labels\n",
    "    \n",
    "    Arguments:\n",
    "        actual - Dataset of actual labels\n",
    "        predict - Dataset of predicted labels\n",
    "     \"\"\"\n",
    "    cm = mets.confusion_matrix(actual, predict)\n",
    "    \n",
    "    print(\"********* MODEL METRIC REPORT *********\\n\\nConfusion matrix:\\n\")\n",
    "\n",
    "    print(\"TP  FN\\nFP  TN\\n\") #this is a reminder of what each part of the confusion matrix means e.g. TP = True Positive\n",
    "    \n",
    "    # print the confusion matrix\n",
    "    print(str(int(cm[0,0])) + \"    \" + str(int(cm[0,1])))\n",
    "    print(str(int(cm[1,0])) + \"    \" + str(int(cm[1,1])) + \"\\n\") \n",
    "\n",
    "    # classification report for DBD model\n",
    "    print(\"Classification report:\\n\")\n",
    "    print(mets.classification_report(actual, predict))\n",
    "\n",
    "    print(\"Balanced accuracy: \" + str(round(mets.balanced_accuracy_score(actual, predict),2)))\n",
    "\n",
    "    print(\"Accuracy: \" + str(round(mets.accuracy_score(actual, predict),2)))\n",
    "    \n",
    "    # Predicted vs actual consent rates\n",
    "    cons_rate = int(100 * len(actual[actual==\"Consent\"]) / len(actual) )\n",
    "    print(\"\\nActual consent rate: \" + str(cons_rate))\n",
    "    \n",
    "    pred_rate = int(100 * len(predict[predict==\"Consent\"]) / len(predict) )\n",
    "    print(\"Predicted consent rate: \" + str(pred_rate))\n",
    "    \n",
    "    pass\n",
    " \n",
    "# Function to format consent column from integer code to text\n",
    "def format_consent(x):\n",
    "    if x == 2:\n",
    "        return \"Consent\"\n",
    "    if x == 1:\n",
    "        return \"Non-consent\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2971600f-bfdd-4da6-84a6-99da4e72fd10",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read in datasets \n",
    "dbd_model_data = pd.read_csv(\"Data/dbd_model_data.csv\")\n",
    "dcd_model_data = pd.read_csv(\"Data/dcd_model_data.csv\")\n",
    "\n",
    "# Columns used to create DBD model\n",
    "dbd_cols = [\"wish\", \"FORMAL_APR_WHEN\", \"donation_mentioned\", \"app_nature\", \"eth_grp\", \"religion_grp\", \"GENDER\", \"FAMILY_WITNESS_BSDT\", \"DTC_PRESENT_BSD_CONV\", \n",
    "            \"acorn_new\", \"adult\",\"FAMILY_CONSENT\"]\n",
    "\n",
    "dbd_model_data2 = pd.get_dummies(data=dbd_model_data,columns=dbd_cols[:-1],drop_first=True)\n",
    "\n",
    "dbd_features = dbd_model_data2.drop(\"FAMILY_CONSENT\",axis=1)\n",
    "dbd_consents = dbd_model_data2[\"FAMILY_CONSENT\"].apply(format_consent)\n",
    "\n",
    "# Columns used to create DCD model in paper\n",
    "dcd_cols = [\"wish\", \"donation_mentioned\", \n",
    "            \"app_nature\", \"eth_grp\", \"religion_grp\", \"GENDER\", \"DTC_WD_TRTMENT_PRESENT\", \n",
    "            \"acorn_new\", \"adult\",\"cod_neuro\",\"FAMILY_CONSENT\"]\n",
    "\n",
    "dcd_model_data2 = pd.get_dummies(data=dcd_model_data,columns=dcd_cols[:-1],drop_first=True)\n",
    "\n",
    "dcd_features = dcd_model_data2.drop(\"FAMILY_CONSENT\",axis=1)\n",
    "dcd_consents = dcd_model_data2[\"FAMILY_CONSENT\"].apply(format_consent)\n",
    "\n",
    "# creating a train and testing dataset for DBD and DCD approaches\n",
    "DBD_X_train, DBD_X_test, DBD_y_train, DBD_y_test = train_test_split(dbd_features,dbd_consents, test_size=0.33, random_state=10)\n",
    "\n",
    "DCD_X_train, DCD_X_test, DCD_y_train, DCD_y_test = train_test_split(dcd_features,dcd_consents, test_size=0.33, random_state=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4343c9fb-c138-440a-a4b0-b19f92e18a49",
   "metadata": {},
   "source": [
    "A decision tree model is fit to the DBD and DCD data with defualt hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "67b8d836-2fa2-419a-b68d-345ec189ab18",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a tree model with defaut hyperparameters\n",
    "tree_model = DecisionTreeClassifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "5c6ada3a-8786-4e64-93b7-ce465e4dad2a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "********* MODEL METRIC REPORT *********\n",
      "\n",
      "Confusion matrix:\n",
      "\n",
      "TP  FN\n",
      "FP  TN\n",
      "\n",
      "1091    291\n",
      "343    275\n",
      "\n",
      "Classification report:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "     Consent       0.76      0.79      0.77      1382\n",
      " Non-consent       0.49      0.44      0.46       618\n",
      "\n",
      "    accuracy                           0.68      2000\n",
      "   macro avg       0.62      0.62      0.62      2000\n",
      "weighted avg       0.68      0.68      0.68      2000\n",
      "\n",
      "Balanced accuracy: 0.62\n",
      "Accuracy: 0.68\n",
      "\n",
      "Actual consent rate: 69\n",
      "Predicted consent rate: 71\n"
     ]
    }
   ],
   "source": [
    "# fit tree using dbd training data\n",
    "DBD_tree = tree_model.fit(DBD_X_train,DBD_y_train)\n",
    "\n",
    "DBD_preds = DBD_tree.predict(DBD_X_test)\n",
    "\n",
    "show_metrics(DBD_y_test,DBD_preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "6cb3d982-8a27-4319-a4de-1c89f23b4760",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "********* MODEL METRIC REPORT *********\n",
      "\n",
      "Confusion matrix:\n",
      "\n",
      "TP  FN\n",
      "FP  TN\n",
      "\n",
      "1432    433\n",
      "525    714\n",
      "\n",
      "Classification report:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "     Consent       0.73      0.77      0.75      1865\n",
      " Non-consent       0.62      0.58      0.60      1239\n",
      "\n",
      "    accuracy                           0.69      3104\n",
      "   macro avg       0.68      0.67      0.67      3104\n",
      "weighted avg       0.69      0.69      0.69      3104\n",
      "\n",
      "Balanced accuracy: 0.67\n",
      "Accuracy: 0.69\n",
      "\n",
      "Actual consent rate: 60\n",
      "Predicted consent rate: 63\n"
     ]
    }
   ],
   "source": [
    "# repeat above for DCD data\n",
    "DCD_tree = tree_model.fit(DCD_X_train,DCD_y_train)\n",
    "\n",
    "DCD_preds = DCD_tree.predict(DCD_X_test)\n",
    "\n",
    "show_metrics(DCD_y_test,DCD_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1493acef-1ad2-4f5a-aa99-943b7bc6b57f",
   "metadata": {},
   "source": [
    "***\n",
    "#### DBD model\n",
    "The DBD model has a balanced accuracy 0.05 lower than the logistic regression model. The recall for the non-consent class is 0.01 higher than the logisitic regression model. The loss in accuracy comes from the recall for consents being 0.11 lower.\n",
    "<br>\n",
    "The predicted and actual consent rates are more similar in this model than the logistic regression, though this is due to more consents being incorrectly classified as non-consents than it is for any increase in accuracy.\n",
    "\n",
    "#### DCD model\n",
    "The DCD model has a balanced accuracy 0.04 lower than the logistic regression model. The recall and precision for both classes are slightly lower in this model too. \n",
    "\n",
    "***\n",
    "<br>\n",
    "\n",
    "As a decision tree with default hyperparameters was no better than the logistic regression on any metric, the model hyperparameters will be tuned to see if this can improve model performance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "852d5b8f-f96c-4ac8-8c6b-58a891533edb",
   "metadata": {},
   "source": [
    "## Tuning the decision tree\n",
    "\n",
    "A 5-fold cross validated grid search will be used to tune hyperparameters. Cross validation will allow the model to be tuned without overfitting to optimise performance for a single dataset.<br>\n",
    "The CV grid search will optimise for balanced accuracy, as this will avoid the model being optimised to overestimate the number of consents.<br>\n",
    "For both models the hyper parameters and range of values to be explored are: <br>\n",
    "* max_depth - the maximum tree depth. From 10 to 35 in increments of 1.\n",
    "* min_samples_split - the minimum number of samples needed in a leaf. From 75 to 250 in increments of 25.\n",
    "* class weight - a weighting parameter to members of the non-consent class so consents aren't overfit. From 1 to 4 in increments of 0.25."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "766becd7-3618-41c8-a4f6-07b8e1557cbf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'class_weight': {'Non-consent': 2.25, 'Consent': 1}, 'max_depth': 17, 'min_samples_split': 225}\n",
      "0.7454541438516957\n"
     ]
    }
   ],
   "source": [
    "# create tree model \n",
    "tree_model = DecisionTreeClassifier(random_state=66)\n",
    "\n",
    "# create list of dictionaries with non-consent group weights\n",
    "weights = []\n",
    "for w in np.arange(1,4,step=0.25):\n",
    "    w_dic = {\"Non-consent\":w,\"Consent\":1}\n",
    "    weights.append(w_dic)\n",
    "\n",
    "# use dictionary of parameters in a CV grid search to find tree which optimises balanced accuracy \n",
    "params = {'max_depth':np.arange(10,35,step=1),'min_samples_split':np.arange(75,250,step=25),'class_weight':weights}\n",
    "\n",
    "gs_tree_model = GridSearchCV(tree_model, param_grid=params, scoring=\"balanced_accuracy\",cv=5)\n",
    "\n",
    "gs_tree_model.fit(DBD_X_train,DBD_y_train)\n",
    "\n",
    "gs_tree_model.score(DBD_X_train,DBD_y_train)\n",
    "\n",
    "\n",
    "# print hyperparameter which optimise balanced accuracy and the balanced accuracy\n",
    "print(gs_tree_model.best_params_)\n",
    "\n",
    "print(gs_tree_model.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "22e76b2b-b5ae-43f7-8e30-d9684cfa3e1b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "********* MODEL METRIC REPORT *********\n",
      "\n",
      "Confusion matrix:\n",
      "\n",
      "TP  FN\n",
      "FP  TN\n",
      "\n",
      "902    480\n",
      "95    523\n",
      "\n",
      "Classification report:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "     Consent       0.90      0.65      0.76      1382\n",
      " Non-consent       0.52      0.85      0.65       618\n",
      "\n",
      "    accuracy                           0.71      2000\n",
      "   macro avg       0.71      0.75      0.70      2000\n",
      "weighted avg       0.79      0.71      0.72      2000\n",
      "\n",
      "Balanced accuracy: 0.75\n",
      "Accuracy: 0.71\n",
      "\n",
      "Actual consent rate: 69\n",
      "Predicted consent rate: 49\n"
     ]
    }
   ],
   "source": [
    "DBD_preds = gs_tree_model.predict(DBD_X_test)\n",
    "\n",
    "show_metrics(DBD_y_test,DBD_preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "4f01c0c7-6818-4bc1-b79d-3f7a8474224f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'class_weight': {'Non-consent': 1.75, 'Consent': 1}, 'max_depth': 10, 'min_samples_split': 125}\n",
      "0.7293181937241343\n"
     ]
    }
   ],
   "source": [
    "# create tree model \n",
    "tree_model = DecisionTreeClassifier(random_state=66)\n",
    "\n",
    "# create list of dictionaries with non-consent group weights\n",
    "weights = []\n",
    "for w in np.arange(1,4,step=0.25):\n",
    "    w_dic = {\"Non-consent\":w,\"Consent\":1}\n",
    "    weights.append(w_dic)\n",
    "\n",
    "# use dictionary of parameters in a CV grid search to find tree which optimises balanced accuracy \n",
    "params = {'max_depth':np.arange(10,100,step=10),'min_samples_split':np.arange(75,250,step=25),'class_weight':weights}\n",
    "\n",
    "gs_tree_model = GridSearchCV(tree_model, param_grid=params, scoring=\"balanced_accuracy\",cv=5)\n",
    "\n",
    "gs_tree_model.fit(DCD_X_train,DCD_y_train)\n",
    "\n",
    "gs_tree_model.score(DCD_X_train,DCD_y_train)\n",
    "\n",
    "\n",
    "# print hyperparameter which optimise balanced accuracy and the balanced accuracy\n",
    "print(gs_tree_model.best_params_)\n",
    "\n",
    "print(gs_tree_model.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "afe0315d-ff6c-4783-b5b5-c6e7a1c9ef0d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "********* MODEL METRIC REPORT *********\n",
      "\n",
      "Confusion matrix:\n",
      "\n",
      "TP  FN\n",
      "FP  TN\n",
      "\n",
      "1019    846\n",
      "133    1106\n",
      "\n",
      "Classification report:\n",
      "\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "     Consent       0.88      0.55      0.68      1865\n",
      " Non-consent       0.57      0.89      0.69      1239\n",
      "\n",
      "    accuracy                           0.68      3104\n",
      "   macro avg       0.73      0.72      0.68      3104\n",
      "weighted avg       0.76      0.68      0.68      3104\n",
      "\n",
      "Balanced accuracy: 0.72\n",
      "Accuracy: 0.68\n",
      "\n",
      "Actual consent rate: 60\n",
      "Predicted consent rate: 37\n"
     ]
    }
   ],
   "source": [
    "DCD_preds = gs_tree_model.predict(DCD_X_test)\n",
    "\n",
    "show_metrics(DCD_y_test,DCD_preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f84ac4aa-41aa-4ab9-a277-66e288fcb655",
   "metadata": {},
   "source": [
    "***\n",
    "#### Tuned DBD model\n",
    "Tuning the decision tree model has increased the balanced accuracy to 0.75, an improvement on both the untuned decision tree model model and logistic regression model. The recall for the non-consent class in the tuned model is significantly improved too, but the recall for consents is reduced to 0.65, much lower than 0.90 in the logistic regression.  \n",
    "\n",
    "The predicted consent rate is also much lower than the actual consent rate.\n",
    "\n",
    "#### Tuned DCD model\n",
    "\n",
    "The balanced accuracy for the DCD model is 0.72, a 0.01 improvement on the logistic regression. As with the tuned DBD model the non-consent recall is significantly improved, at the expense of the recall for consents and the predicted consent rate is much lower than the actual consent rate.\n",
    "\n",
    "\n",
    "***\n",
    "\n",
    "Decision tree models were an improvement the balanced accuracy compared to the logistic regression and the ability to tune the model allowed more non-consents to be correctly identified. In the next notebook random forest models will be used to try and improve further on the performance of the decision tree models.\n",
    "\n",
    "***"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}