examples/Example2-PROC_LOGISTIC_with_Variable_Selection-SGF2021-Simple_and_Efficient_Bootstrap_Validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Framework for Simple and Efficient Bootstrap Validation in SAS®, with Examples\n",
    "\n",
    "\n",
    "## Example 2: PROC LOGISTIC with Variable Selection &nbsp;&bullet;&nbsp; Invited Paper &nbsp;&bullet;&nbsp;  SAS Global Forum 2021\n",
    "\n",
    "This notebook contains an executable version of the example in Appendix C of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-example Setup Part 1\n",
    "\n",
    "The code below creates macro variables encapsulating model parameters. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only SAS log output should be created.\n",
    "\n",
    "**Note**: You may also wish to change the values of the macro variables in order to explore bootstrap validation for different models. This example is adapted from Module 10 at https://wwwn.cdc.gov/nchs/nhanes/tutorials/samplecode.aspx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%let response_variable_condition = bpxsar >= 140 OR bpxdar >= 90 OR bpq050a = 1;\n",
    "%let response_variable = hyper;\n",
    "%let outcome = &response_variable.(EVENT='1');\n",
    "%let class_variables = bpq100d dmq051 dmd110;\n",
    "%let predictor_variables = lbxtc bpq100d bmxbmi ridageyr lbxtr dmq051 dmd110 indhhinc indfmpir;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-example Setup Part 2\n",
    "\n",
    "The code below downloads the example dataset for Module 10 at https://wwwn.cdc.gov/nchs/nhanes/tutorials/samplecode.aspx. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only SAS log output should be created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename tempfile \"%sysfunc(pathname(work))/analysis_data.sas7bdat\";\n",
    "proc http\n",
    "        url='https://wwwn.cdc.gov/nchs/data/tutorials/analysis_data.sas7bdat'\n",
    "        method='get'\n",
    "        out=tempfile\n",
    "    ;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-example Setup Part 3\n",
    "\n",
    "The code below subsets the dataset downloaded above to rows with no missing values for the predicator variables, as well as creating a response variable. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only SAS log output should be created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data example_dataset;\n",
    "    set analysis_data;\n",
    "    \n",
    "    * Create outcome variable;\n",
    "    if (&response_variable_condition.) then &response_variable. = 1;\n",
    "    else &response_variable. = 0;\n",
    "    \n",
    "    * Subset to observations with no missing values for outcome variable or predictor variables;\n",
    "    if nmiss(&response_variable., %sysfunc(tranwrd(&predictor_variables.,%str( ),%str(,)))) = 0;\n",
    "    \n",
    "    keep &response_variable. &predictor_variables.;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Train a Model using PROC LOGISTIC with Variable Selection\n",
    "\n",
    "See page 13 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only PROC LOGISTIC output should be created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ods output Association=model_association_table(\n",
    "    where=(Label2='c')\n",
    "    keep=Label2 nValue2\n",
    "    rename=(nValue2=original_model_c_statistic)\n",
    ");\n",
    "proc logistic data=example_dataset;\n",
    "    class &class_variables.;\n",
    "    model &outcome. = &predictor_variables.\n",
    "        / selection=backward slstay=0.1 fast\n",
    "    ;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Train a Model using PROC LOGISTIC with Variable Selection (additional optional step)\n",
    "\n",
    "The code below shows the log from the last cell executed. Please execute the cell by clicking anywhere in it and typing Shift-Enter.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Generate Bootstrap Samples\n",
    "\n",
    "See page 5 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only PROC SURVEYSELECT output should be created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "proc surveyselect\n",
    "        data=example_dataset\n",
    "        out=bootstrap_samples\n",
    "        seed=1354687\n",
    "        method=urs\n",
    "        outhits\n",
    "        rep=500\n",
    "        samprate=1\n",
    "    ;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Generate Bootstrap Samples (additional optional step)\n",
    "\n",
    "The code below shows the log from the last cell executed. Please execute the cell by clicking anywhere in it and typing Shift-Enter.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Train Models in Each Bootstrap using PROC LOGISTIC with Variable Selection\n",
    "\n",
    "See pages 13-14 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. (Depending on the version of the SAS Kernel being used by Jupyter, you may see either empty output or the SAS log.)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "* turn off all output;\n",
    "ods graphics off;\n",
    "ods exclude all;\n",
    "ods noresults;\n",
    "\n",
    "* Trains models on all 500 bootstrap samples, capturing the resulting C-statistics;\n",
    "ods output Association=bootstrap_association_table(\n",
    "    where=(Label2='c')\n",
    "    keep=Replicate Label2 nValue2\n",
    "    rename=(nValue2=c_statistic_value)\n",
    ");\n",
    "proc logistic data=bootstrap_samples outmodel=bootstap_models;\n",
    "    by Replicate;\n",
    "    class &class_variables.;\n",
    "    model &outcome. = &predictor_variables.\n",
    "        / selection=backward slstay=0.1 fast\n",
    "    ;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Train Models in Each Bootstrap using PROC LOGISTIC with Variable Selection\n",
    "\n",
    "If the log for the previous cell did not appear, the code below can be used to view it. Please execute the cell by clicking anywhere in it and typing Shift-Enter.\n",
    "\n",
    "**Warning**: This log entry will be particularly long since each of the 500 iterations of PROC LOGISITIC produces its own log entries.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Test Bootstrap Models\n",
    "\n",
    "See page 7 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. (Depending on the version of the SAS Kernel being used by Jupyter, you may see either empty output or the SAS log.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "* Score original dataset with bootstap models;\n",
    "ods output Scorefitstat=bootstrap_scores(\n",
    "    keep=Replicate AUC\n",
    "    rename=(AUC=c_statistic_value)\n",
    ");\n",
    "proc logistic inmodel=bootstap_models;\n",
    "    score\n",
    "        data=example_dataset\n",
    "        fitstat\n",
    "    ;\n",
    "    by Replicate;\n",
    "run;\n",
    "\n",
    "* turn output back on;\n",
    "ods results;\n",
    "ods select all;\n",
    "ods graphics on;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Test Bootstrap Models (additional optional step)\n",
    "\n",
    "If the log for the previous cell did not appear, the code below can be used to view it. Please execute the cell by clicking anywhere in it and typing Shift-Enter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Estimate Optimism\n",
    "\n",
    "See page 8 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only SAS log output should be created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "proc sql;\n",
    "    create table model_optimism as\n",
    "        select\n",
    "            avg(A.c_statistic_value - B.c_statistic_value) as optimism\n",
    "        from\n",
    "            bootstrap_association_table as A\n",
    "            inner join\n",
    "            bootstrap_scores as B\n",
    "            on A.Replicate = B.Replicate\n",
    "    ;\n",
    "quit;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: Adjust Performance with Optimism\n",
    "\n",
    "See page 9 of the paper at https://github.com/saspy-bffs/sgf-2021-bootstrap-validation. Please execute the cell by clicking anywhere in it and typing Shift-Enter. Only PROC PRINT output should be created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "* assemble C-statistic for original model, optimism, and corrected C-statistic into a 1x3 table;\n",
    "data corrected_model_evaluation;\n",
    "    set model_association_table;\n",
    "    set model_optimism;\n",
    "    \n",
    "    corrected_c_statistic = original_model_c_statistic - optimism;\n",
    "\n",
    "    label\n",
    "        original_model_c_statistic = 'Naive C-Statistic'\n",
    "        optimism = 'Optimism'\n",
    "        corrected_c_statistic = 'Optimism-Corrected C-Statistic'\n",
    "    ;\n",
    "    \n",
    "    keep original_model_c_statistic optimism corrected_c_statistic;\n",
    "run;\n",
    "\n",
    "* print final results;\n",
    "proc print\n",
    "        data=corrected_model_evaluation\n",
    "        noobs\n",
    "        label\n",
    "    ;\n",
    "run;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: Adjust Performance with Optimism (additional optional step)\n",
    "\n",
    "The code below shows the log from the last cell executed. Please execute the cell by clicking anywhere in it and typing Shift-Enter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%showLog"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "SAS",
   "language": "sas",
   "name": "sas"
  },
  "language_info": {
   "codemirror_mode": "sas",
   "file_extension": ".sas",
   "mimetype": "text/x-sas",
   "name": "sas"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}