AI4I_Radiomics_Experts.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "AI4I_Radiomics_Experts.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMcHGdBy46aPdANgCeyMZ+S",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/lavrovaliz/dlab-scripts/blob/main/AI4I_Radiomics_Experts.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Radiomics for experts\n",
        "  \n",
        "<b>'Building a radiomics classifier that distinguishes between benign and malignant lesions will certainly improve clinical handling of suspicious lesions, and minimize the need for invasive procedures' - the aim of all the radiomics researchers.</b>\n",
        "\n",
        "## Introduction\n",
        "\n",
        "In this script, you will use a privately collected contrast enhanced spectral mammography (CESM) of 1058 patients at Maastricht University Medical Center (MUMC). You will build a classifier to differentiate between malignant and benign breast lesions. Such a model will reduce the necessity for invasive tissue biopsies. If you find this task, or parts of it, too hard, you can \"cheat\" by looking at the beginners notebook, as most tasks are very similar. \n",
        "\n",
        "<b>Note:</b> this dataset is private and allowed to be used only for the purpose of this course! Nevertheless, for this notebook, we provide you with the features extracted from the data as an invert transformation is not possible. If you want to extract the features yourself, you will receive the dedicated instructions at the practical. In this case, please make sure you delete imaging data after the workshop. For this, you can use a precision-medicine-toolbox: https://github.com/primakov/precision-medicine-toolbox. For the hints, you can have a look at the beginners notebook: https://colab.research.google.com/drive/1K-sg8g2Po6Czq5utFS2F6CtetXcrwpIt?usp=sharing. \n",
        "\n",
        "<b>Brief description</b>\n",
        "\n",
        "Mammography is the standard imaging tool for breast-cancer screening, but is prone to false positives and lowered sensitivity in high-density breasts. Contrast enhanced spectral mammography (CESM) depicts breast lesions with more accuracy and with higher sensitivity than regular mammograms. This technique uses an iodine contrast agent in addition to a high energy and low energy mammogram to cancel out background noise and enhance the tumor. The advantage of CESM compared to regular mammography is that breast density does not affect image quality, and it boasts a sensitivity of 93-100%, compared to 75%-85% of mammography on regular breasts, and lower than 50% in dense breasts.   \n",
        "\n",
        "![picture](https://drive.google.com/uc?export=view&id=1GRt1gWtxPDXGycX6TpDAbH1XAieMQc0t)      \n",
        "A typical contrast-enhanced spectral mammography (CESM) examination (only right mediolateral oblique view shown), consisting of a low-energy (a), high-energy (b) and recombined (c) image. A suspicious lesion is seen on the low-energy image, showing enhancement on the recombined image (white arrows). Histopathology showed invasive ductal carcinoma. The high-energy image is not for diagnostic purposes but is used for construction of the recombined image (Lalji, U.C., Jeukens, C.R.L.P.N., Houben, I. et al. Evaluation of low-energy contrast-enhanced spectral mammography images by comparing them to full-field digital mammography using EUREF image quality criteria. Eur Radiol 25, 2813–2820 (2015). https://doi.org/10.1007/s00330-015-3695-2)\n",
        "\n",
        "<b>Problem</b>  \n",
        "\n",
        "Though CESM improves the detection of breast lesions, there is still a challenge in distinguishing between benign and malignant lesions, as the specificity of CESM is estimated to be 70%. This stands to improve using machine-learning (ML) algorithms. \n",
        "\n",
        "<b>Notebook structure</b>  \n",
        "\n",
        "The Python script will take you through the following steps:  \n",
        " - Installing packages and importing libraries,\n",
        " - Reading the data and assigning the outcomes, \n",
        " - Removing redundant features and selecting predictive features, \n",
        " - Creating the classification models. \n",
        "If you still have time, go back and try balancing your data, and fine-tune your models.\n",
        "\n",
        "## Getting started\n",
        "\n",
        "This is an interactive Python notebook. To run it, you don't need to install anything on your PC since the script is executed in the cloud. On the left tab, you can see the 'Files' folder, that contains the data. Results of the script execution will be saved in this folder.  \n",
        "  \n",
        "The information about the curated imaging dataset will be distributed separately. To open DICOM files, you can use 3D Slicer (https://www.slicer.org/), RadiAnt viewer (https://www.radiantviewer.com/), or MicroDicom viewer (https://www.microdicom.com/downloads.html). As feature extraction process takes time, we prepared the Excel tables with the pre-extracted features to work with. They are available at the shared folder: https://drive.google.com/drive/folders/1GsrUvY60XRpBEjdpFTolZ3gIEow6yD2P?usp=sharing. To make the folder accessable for your notebook, right-click it and select 'Add to My Drive'. In the pop-up window, put the label into 'My Drive' location. After mounting your Google Drive (will be explained later), you will have an access to these files.\n",
        "\n",
        "Please note, that all the files you upload to this notebook as well as the files, produced while executing the script, are automatically deleted as soon as you end the session.\n",
        "\n",
        "First of all, the needed Python packages have to be uploaded. Some of them are not installed in the environment of this notebook, so the installation is needed with '!pip install <i>name-of-the-package</i>' command. We recommend you get acquainted with getting documentation and help on these packages. For example, google 'python sklearn' and you will get to the documentation quickly. Importing libraries is a necessary step with most progamming languages, not only Python."
      ],
      "metadata": {
        "id": "XiGMHG8iYoWV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# installing some packages, which are not part of the present Google Collab environment\n",
        "\n",
        "!pip install precision-medicine-toolbox"
      ],
      "metadata": {
        "id": "60z4gB0WdAnj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "179c17b5-2161-4529-c3bb-41645be1677f"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting precision-medicine-toolbox\n",
            "  Downloading precision-medicine-toolbox-0.4.tar.gz (23 kB)\n",
            "Collecting SimpleITK\n",
            "  Downloading SimpleITK-2.1.1.2-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (48.4 MB)\n",
            "\u001b[K     |████████████████████████████████| 48.4 MB 22 kB/s \n",
            "\u001b[?25hRequirement already satisfied: PyWavelets in /usr/local/lib/python3.7/dist-packages (from precision-medicine-toolbox) (1.3.0)\n",
            "Collecting pykwalify\n",
            "  Downloading pykwalify-1.8.0-py2.py3-none-any.whl (24 kB)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from precision-medicine-toolbox) (1.15.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from precision-medicine-toolbox) (4.64.0)\n",
            "Collecting pydicom\n",
            "  Downloading pydicom-2.3.0-py3-none-any.whl (2.0 MB)\n",
            "\u001b[K     |████████████████████████████████| 2.0 MB 26.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from precision-medicine-toolbox) (1.3.5)\n",
            "Collecting pyradiomics\n",
            "  Downloading pyradiomics-3.0.1-cp37-cp37m-manylinux1_x86_64.whl (188 kB)\n",
            "\u001b[K     |████████████████████████████████| 188 kB 38.0 MB/s \n",
            "\u001b[?25hRequirement already satisfied: scikit-image in /usr/local/lib/python3.7/dist-packages (from precision-medicine-toolbox) (0.18.3)\n",
            "Requirement already satisfied: ipywidgets in /usr/local/lib/python3.7/dist-packages (from precision-medicine-toolbox) (7.7.0)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from precision-medicine-toolbox) (3.2.2)\n",
            "Requirement already satisfied: Pillow in /usr/local/lib/python3.7/dist-packages (from precision-medicine-toolbox) (7.1.2)\n",
            "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from precision-medicine-toolbox) (1.0.2)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from precision-medicine-toolbox) (1.4.1)\n",
            "Requirement already satisfied: plotly in /usr/local/lib/python3.7/dist-packages (from precision-medicine-toolbox) (5.5.0)\n",
            "Collecting mkdocstrings\n",
            "  Downloading mkdocstrings-0.19.0-py3-none-any.whl (25 kB)\n",
            "Requirement already satisfied: statsmodels in /usr/local/lib/python3.7/dist-packages (from precision-medicine-toolbox) (0.10.2)\n",
            "Requirement already satisfied: opencv-python in /usr/local/lib/python3.7/dist-packages (from precision-medicine-toolbox) (4.1.2.30)\n",
            "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->precision-medicine-toolbox) (1.1.0)\n",
            "Requirement already satisfied: widgetsnbextension~=3.6.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->precision-medicine-toolbox) (3.6.0)\n",
            "Requirement already satisfied: ipython>=4.0.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->precision-medicine-toolbox) (5.5.0)\n",
            "Requirement already satisfied: ipykernel>=4.5.1 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->precision-medicine-toolbox) (4.10.1)\n",
            "Requirement already satisfied: traitlets>=4.3.1 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->precision-medicine-toolbox) (5.1.1)\n",
            "Requirement already satisfied: ipython-genutils~=0.2.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->precision-medicine-toolbox) (0.2.0)\n",
            "Requirement already satisfied: nbformat>=4.2.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets->precision-medicine-toolbox) (5.4.0)\n",
            "Requirement already satisfied: tornado>=4.0 in /usr/local/lib/python3.7/dist-packages (from ipykernel>=4.5.1->ipywidgets->precision-medicine-toolbox) (5.1.1)\n",
            "Requirement already satisfied: jupyter-client in /usr/local/lib/python3.7/dist-packages (from ipykernel>=4.5.1->ipywidgets->precision-medicine-toolbox) (5.3.5)\n",
            "Requirement already satisfied: pickleshare in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipywidgets->precision-medicine-toolbox) (0.7.5)\n",
            "Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipywidgets->precision-medicine-toolbox) (0.8.1)\n",
            "Requirement already satisfied: pygments in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipywidgets->precision-medicine-toolbox) (2.6.1)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipywidgets->precision-medicine-toolbox) (4.4.2)\n",
            "Requirement already satisfied: pexpect in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipywidgets->precision-medicine-toolbox) (4.8.0)\n",
            "Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipywidgets->precision-medicine-toolbox) (1.0.18)\n",
            "Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipywidgets->precision-medicine-toolbox) (57.4.0)\n",
            "Requirement already satisfied: jupyter-core in /usr/local/lib/python3.7/dist-packages (from nbformat>=4.2.0->ipywidgets->precision-medicine-toolbox) (4.10.0)\n",
            "Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.7/dist-packages (from nbformat>=4.2.0->ipywidgets->precision-medicine-toolbox) (2.15.3)\n",
            "Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.7/dist-packages (from nbformat>=4.2.0->ipywidgets->precision-medicine-toolbox) (4.3.3)\n",
            "Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema>=2.6->nbformat>=4.2.0->ipywidgets->precision-medicine-toolbox) (5.7.1)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from jsonschema>=2.6->nbformat>=4.2.0->ipywidgets->precision-medicine-toolbox) (4.1.1)\n",
            "Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema>=2.6->nbformat>=4.2.0->ipywidgets->precision-medicine-toolbox) (21.4.0)\n",
            "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema>=2.6->nbformat>=4.2.0->ipywidgets->precision-medicine-toolbox) (0.18.1)\n",
            "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from jsonschema>=2.6->nbformat>=4.2.0->ipywidgets->precision-medicine-toolbox) (4.11.4)\n",
            "Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.7/dist-packages (from importlib-resources>=1.4.0->jsonschema>=2.6->nbformat>=4.2.0->ipywidgets->precision-medicine-toolbox) (3.8.0)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=4.0.0->ipywidgets->precision-medicine-toolbox) (0.2.5)\n",
            "Requirement already satisfied: notebook>=4.4.1 in /usr/local/lib/python3.7/dist-packages (from widgetsnbextension~=3.6.0->ipywidgets->precision-medicine-toolbox) (5.3.1)\n",
            "Requirement already satisfied: nbconvert in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->precision-medicine-toolbox) (5.6.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->precision-medicine-toolbox) (2.11.3)\n",
            "Requirement already satisfied: terminado>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->precision-medicine-toolbox) (0.13.3)\n",
            "Requirement already satisfied: Send2Trash in /usr/local/lib/python3.7/dist-packages (from notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->precision-medicine-toolbox) (1.8.0)\n",
            "Requirement already satisfied: pyzmq>=13 in /usr/local/lib/python3.7/dist-packages (from jupyter-client->ipykernel>=4.5.1->ipywidgets->precision-medicine-toolbox) (23.1.0)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from jupyter-client->ipykernel>=4.5.1->ipywidgets->precision-medicine-toolbox) (2.8.2)\n",
            "Requirement already satisfied: ptyprocess in /usr/local/lib/python3.7/dist-packages (from terminado>=0.8.1->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->precision-medicine-toolbox) (0.7.0)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->precision-medicine-toolbox) (2.0.1)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->precision-medicine-toolbox) (3.0.9)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->precision-medicine-toolbox) (1.4.3)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->precision-medicine-toolbox) (0.11.0)\n",
            "Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.7/dist-packages (from matplotlib->precision-medicine-toolbox) (1.21.6)\n",
            "Collecting mkdocs-autorefs>=0.3.1\n",
            "  Downloading mkdocs_autorefs-0.4.1-py3-none-any.whl (9.8 kB)\n",
            "Collecting mkdocs>=1.2\n",
            "  Downloading mkdocs-1.3.0-py3-none-any.whl (3.7 MB)\n",
            "\u001b[K     |████████████████████████████████| 3.7 MB 34.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: Markdown>=3.3 in /usr/local/lib/python3.7/dist-packages (from mkdocstrings->precision-medicine-toolbox) (3.3.7)\n",
            "Collecting pymdown-extensions>=6.3\n",
            "  Downloading pymdown_extensions-9.5-py3-none-any.whl (217 kB)\n",
            "\u001b[K     |████████████████████████████████| 217 kB 43.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: click>=3.3 in /usr/local/lib/python3.7/dist-packages (from mkdocs>=1.2->mkdocstrings->precision-medicine-toolbox) (7.1.2)\n",
            "Collecting pyyaml-env-tag>=0.1\n",
            "  Downloading pyyaml_env_tag-0.1-py3-none-any.whl (3.9 kB)\n",
            "Collecting watchdog>=2.0\n",
            "  Downloading watchdog-2.1.9-py3-none-manylinux2014_x86_64.whl (78 kB)\n",
            "\u001b[K     |████████████████████████████████| 78 kB 6.2 MB/s \n",
            "\u001b[?25hCollecting mergedeep>=1.3.4\n",
            "  Downloading mergedeep-1.3.4-py3-none-any.whl (6.4 kB)\n",
            "Collecting ghp-import>=1.0\n",
            "  Downloading ghp_import-2.1.0-py3-none-any.whl (11 kB)\n",
            "Requirement already satisfied: packaging>=20.5 in /usr/local/lib/python3.7/dist-packages (from mkdocs>=1.2->mkdocstrings->precision-medicine-toolbox) (21.3)\n",
            "Requirement already satisfied: PyYAML>=3.10 in /usr/local/lib/python3.7/dist-packages (from mkdocs>=1.2->mkdocstrings->precision-medicine-toolbox) (3.13)\n",
            "Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->precision-medicine-toolbox) (0.8.4)\n",
            "Requirement already satisfied: bleach in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->precision-medicine-toolbox) (5.0.0)\n",
            "Requirement already satisfied: testpath in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->precision-medicine-toolbox) (0.6.0)\n",
            "Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->precision-medicine-toolbox) (0.4)\n",
            "Requirement already satisfied: defusedxml in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->precision-medicine-toolbox) (0.7.1)\n",
            "Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.7/dist-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->precision-medicine-toolbox) (1.5.0)\n",
            "Requirement already satisfied: webencodings in /usr/local/lib/python3.7/dist-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.6.0->ipywidgets->precision-medicine-toolbox) (0.5.1)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas->precision-medicine-toolbox) (2022.1)\n",
            "Requirement already satisfied: tenacity>=6.2.0 in /usr/local/lib/python3.7/dist-packages (from plotly->precision-medicine-toolbox) (8.0.1)\n",
            "Collecting ruamel.yaml>=0.16.0\n",
            "  Downloading ruamel.yaml-0.17.21-py3-none-any.whl (109 kB)\n",
            "\u001b[K     |████████████████████████████████| 109 kB 9.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: docopt>=0.6.2 in /usr/local/lib/python3.7/dist-packages (from pykwalify->precision-medicine-toolbox) (0.6.2)\n",
            "Collecting ruamel.yaml.clib>=0.2.6\n",
            "  Downloading ruamel.yaml.clib-0.2.6-cp37-cp37m-manylinux1_x86_64.whl (546 kB)\n",
            "\u001b[K     |████████████████████████████████| 546 kB 43.7 MB/s \n",
            "\u001b[?25hRequirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.7/dist-packages (from scikit-image->precision-medicine-toolbox) (2021.11.2)\n",
            "Requirement already satisfied: imageio>=2.3.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image->precision-medicine-toolbox) (2.4.1)\n",
            "Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.7/dist-packages (from scikit-image->precision-medicine-toolbox) (2.6.3)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->precision-medicine-toolbox) (3.1.0)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->precision-medicine-toolbox) (1.1.0)\n",
            "Requirement already satisfied: patsy>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from statsmodels->precision-medicine-toolbox) (0.5.2)\n",
            "Building wheels for collected packages: precision-medicine-toolbox\n",
            "  Building wheel for precision-medicine-toolbox (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for precision-medicine-toolbox: filename=precision_medicine_toolbox-0.4-py3-none-any.whl size=21979 sha256=4f01176a019cde19c1fd56ba05917a691f5f2e80c73701ca6e4abdbc19582ee9\n",
            "  Stored in directory: /root/.cache/pip/wheels/0a/1f/a8/9772e6ce8660e2333ce4afa90cdfbdd2a788b2c66ba1792450\n",
            "Successfully built precision-medicine-toolbox\n",
            "Installing collected packages: watchdog, ruamel.yaml.clib, pyyaml-env-tag, mergedeep, ghp-import, ruamel.yaml, mkdocs, SimpleITK, pymdown-extensions, pykwalify, mkdocs-autorefs, pyradiomics, pydicom, mkdocstrings, precision-medicine-toolbox\n",
            "Successfully installed SimpleITK-2.1.1.2 ghp-import-2.1.0 mergedeep-1.3.4 mkdocs-1.3.0 mkdocs-autorefs-0.4.1 mkdocstrings-0.19.0 precision-medicine-toolbox-0.4 pydicom-2.3.0 pykwalify-1.8.0 pymdown-extensions-9.5 pyradiomics-3.0.1 pyyaml-env-tag-0.1 ruamel.yaml-0.17.21 ruamel.yaml.clib-0.2.6 watchdog-2.1.9\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "AwuI_iH4UELF",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2759ffe4-b6a9-4600-e8b2-400dfd53cf77"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
            "  import pandas.util.testing as tm\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from pmtool.AnalysisBox import AnalysisBox\n",
        "from pmtool.ToolBox import ToolBox\n",
        "import matplotlib.pyplot as plt \n",
        "import seaborn as sns\n",
        "from sklearn.feature_selection import RFE\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.metrics import balanced_accuracy_score, classification_report, confusion_matrix, roc_curve, roc_auc_score\n",
        "import seaborn as sns\n",
        "import xgboost as xgb\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Mounting Google Drive:"
      ],
      "metadata": {
        "id": "ueLgxNn08Rgz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# this is where we mount a Google Drive\n",
        "\n",
        "from google.colab import drive\n",
        "import os\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "id": "8d3fs4xP-z-0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "91a35b1b-171b-47da-f425-35f8b2d25153"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Feature extraction\n",
        "There are many softwares to perform this step, each with their strengths and weaknesses. Features extraction step takes the longest time, so we are happy to help you with it. We will supply you with \"something we prepared earlier\", the lists of features already extracted with Radiomics software, and outcomes (benign/malignant). The data is already split into train and validation sets. Therefore, you can skip this step or return to it later. If you want to skip this step, please move to 'Reading the data' step.  \n",
        "\n",
        "If you want to extract the features by yourself anyway, you can adapt the code from the beginners notebook: https://colab.research.google.com/drive/1K-sg8g2Po6Czq5utFS2F6CtetXcrwpIt?usp=sharing. Here we perform features extraction with PyRadiomics software (https://pyradiomics.readthedocs.io/en/latest/) with the precision-medicine-toolbox (https://github.com/primakov/precision-medicine-toolbox) interface. These are the open-source Python packages. The key difference is that the present script is using Pyradiomics for feature extraction whereas in the prepared files, Radiomics software (https://radiomics.bio/) was used."
      ],
      "metadata": {
        "id": "LzuYSO_Nmn43"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Reading the data \n",
        "\n",
        "For this step, you will need to specify the path to the files containing the features and outcomes (in this case 0 or 1 for benign or malignant). It is important that apart from initial dataset cleaning, all the work is only performed in the training dataset and the test set remains untouched.\n",
        "\n",
        "<b>NOTE: the separator character, text quotation, decimal point character change depending on your country settings. If this step fails, add those settings into read_excel function. Also, the outputs from different radiomics softwares can differ wildly, so some thought is necessary when doing this at home.</b>  \n",
        "\n"
      ],
      "metadata": {
        "id": "yja8CdFGvy4b"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data_train = pd.read_excel(\"/content/drive/MyDrive/AI4I_Radiomics_Experts_Data/data_features/WS2022_Experts_training.xlsx\")\n",
        "data_test = pd.read_excel(\"/content/drive/MyDrive/AI4I_Radiomics_Experts_Data/data_features/WS2022_Experts_validation.xlsx\")\n",
        "\n",
        "# let's have a look at our dataframe:\n",
        "data_test"
      ],
      "metadata": {
        "id": "EQ-YoYDCwJXo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 661
        },
        "outputId": "2a593a11-8ac6-4da2-eaa4-f7aa6b35dd1a"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                    ID discretization_method  \\\n",
              "0    B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...                binnum   \n",
              "1    B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...                binnum   \n",
              "2    B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...                binnum   \n",
              "3    B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...                binnum   \n",
              "4    B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...                binnum   \n",
              "..                                                 ...                   ...   \n",
              "336  B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...                binnum   \n",
              "337  B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...                binnum   \n",
              "338  B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...                binnum   \n",
              "339  B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...                binnum   \n",
              "340  B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...                binnum   \n",
              "\n",
              "     discretization_value modality  pixel_spacing_1  pixel_spacing_2  \\\n",
              "0                     100     CESM             0.01             0.01   \n",
              "1                     100     CESM             0.01             0.01   \n",
              "2                     100     CESM             0.01             0.01   \n",
              "3                     100     CESM             0.01             0.01   \n",
              "4                     100     CESM             0.01             0.01   \n",
              "..                    ...      ...              ...              ...   \n",
              "336                   100     CESM             0.01             0.01   \n",
              "337                   100     CESM             0.01             0.01   \n",
              "338                   100     CESM             0.01             0.01   \n",
              "339                   100     CESM             0.01             0.01   \n",
              "340                   100     CESM             0.01             0.01   \n",
              "\n",
              "    resample_method resample_voxel_size structure_name  voxel_distance  ...  \\\n",
              "0     no resampling       no resampling     tumor_mask               1  ...   \n",
              "1     no resampling       no resampling     tumor_mask               1  ...   \n",
              "2     no resampling       no resampling     tumor_mask               1  ...   \n",
              "3     no resampling       no resampling     tumor_mask               1  ...   \n",
              "4     no resampling       no resampling     tumor_mask               1  ...   \n",
              "..              ...                 ...            ...             ...  ...   \n",
              "336   no resampling       no resampling     tumor_mask               1  ...   \n",
              "337   no resampling       no resampling     tumor_mask               1  ...   \n",
              "338   no resampling       no resampling     tumor_mask               1  ...   \n",
              "339   no resampling       no resampling     tumor_mask               1  ...   \n",
              "340   no resampling       no resampling     tumor_mask               1  ...   \n",
              "\n",
              "     Stats_p90  Stats_qcod  Stats_rMeanD  Stats_range    Stats_rms  \\\n",
              "0       2072.0    0.008593     14.702153          197  2037.443275   \n",
              "1       2045.0    0.005435      9.501333          153  2024.170365   \n",
              "2       2024.0    0.005241      9.146604          141  2003.600820   \n",
              "3       2034.0    0.005467      9.466079          126  2011.960535   \n",
              "4       2106.0    0.009161     15.768983          179  2073.695786   \n",
              "..         ...         ...           ...          ...          ...   \n",
              "336     2064.0    0.005138      8.544133          110  2043.806439   \n",
              "337     2084.0    0.007782     13.345616          155  2055.414637   \n",
              "338     2017.0    0.004754      8.203036          124  1998.365225   \n",
              "339     2037.0    0.004704      8.076171          121  2019.393275   \n",
              "340     2031.0    0.004970      8.419451          103  2012.063099   \n",
              "\n",
              "     Stats_skewness  Stats_std   Stats_var  Outcome  outcome  \n",
              "0          0.380419  26.447025  699.445119        0        0  \n",
              "1         -0.192629  16.779550  281.553291        0        0  \n",
              "2          0.061260  16.197340  262.353813        0        0  \n",
              "3          0.022882  17.024644  289.838490        0        0  \n",
              "4         -0.295346  26.140395  683.320268        1        1  \n",
              "..              ...        ...         ...      ...      ...  \n",
              "336        0.330860  15.341589  235.364338        1        1  \n",
              "337       -0.163142  22.641186  512.623307        1        1  \n",
              "338       -0.042320  14.446493  208.701167        0        0  \n",
              "339       -0.106028  14.609857  213.447917        0        0  \n",
              "340       -0.039442  14.921726  222.657920        0        0  \n",
              "\n",
              "[341 rows x 490 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-e52b585d-5719-41ad-bbb9-9d033d99737c\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ID</th>\n",
              "      <th>discretization_method</th>\n",
              "      <th>discretization_value</th>\n",
              "      <th>modality</th>\n",
              "      <th>pixel_spacing_1</th>\n",
              "      <th>pixel_spacing_2</th>\n",
              "      <th>resample_method</th>\n",
              "      <th>resample_voxel_size</th>\n",
              "      <th>structure_name</th>\n",
              "      <th>voxel_distance</th>\n",
              "      <th>...</th>\n",
              "      <th>Stats_p90</th>\n",
              "      <th>Stats_qcod</th>\n",
              "      <th>Stats_rMeanD</th>\n",
              "      <th>Stats_range</th>\n",
              "      <th>Stats_rms</th>\n",
              "      <th>Stats_skewness</th>\n",
              "      <th>Stats_std</th>\n",
              "      <th>Stats_var</th>\n",
              "      <th>Outcome</th>\n",
              "      <th>outcome</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...</td>\n",
              "      <td>binnum</td>\n",
              "      <td>100</td>\n",
              "      <td>CESM</td>\n",
              "      <td>0.01</td>\n",
              "      <td>0.01</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>tumor_mask</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>2072.0</td>\n",
              "      <td>0.008593</td>\n",
              "      <td>14.702153</td>\n",
              "      <td>197</td>\n",
              "      <td>2037.443275</td>\n",
              "      <td>0.380419</td>\n",
              "      <td>26.447025</td>\n",
              "      <td>699.445119</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...</td>\n",
              "      <td>binnum</td>\n",
              "      <td>100</td>\n",
              "      <td>CESM</td>\n",
              "      <td>0.01</td>\n",
              "      <td>0.01</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>tumor_mask</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>2045.0</td>\n",
              "      <td>0.005435</td>\n",
              "      <td>9.501333</td>\n",
              "      <td>153</td>\n",
              "      <td>2024.170365</td>\n",
              "      <td>-0.192629</td>\n",
              "      <td>16.779550</td>\n",
              "      <td>281.553291</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...</td>\n",
              "      <td>binnum</td>\n",
              "      <td>100</td>\n",
              "      <td>CESM</td>\n",
              "      <td>0.01</td>\n",
              "      <td>0.01</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>tumor_mask</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>2024.0</td>\n",
              "      <td>0.005241</td>\n",
              "      <td>9.146604</td>\n",
              "      <td>141</td>\n",
              "      <td>2003.600820</td>\n",
              "      <td>0.061260</td>\n",
              "      <td>16.197340</td>\n",
              "      <td>262.353813</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...</td>\n",
              "      <td>binnum</td>\n",
              "      <td>100</td>\n",
              "      <td>CESM</td>\n",
              "      <td>0.01</td>\n",
              "      <td>0.01</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>tumor_mask</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>2034.0</td>\n",
              "      <td>0.005467</td>\n",
              "      <td>9.466079</td>\n",
              "      <td>126</td>\n",
              "      <td>2011.960535</td>\n",
              "      <td>0.022882</td>\n",
              "      <td>17.024644</td>\n",
              "      <td>289.838490</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...</td>\n",
              "      <td>binnum</td>\n",
              "      <td>100</td>\n",
              "      <td>CESM</td>\n",
              "      <td>0.01</td>\n",
              "      <td>0.01</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>tumor_mask</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>2106.0</td>\n",
              "      <td>0.009161</td>\n",
              "      <td>15.768983</td>\n",
              "      <td>179</td>\n",
              "      <td>2073.695786</td>\n",
              "      <td>-0.295346</td>\n",
              "      <td>26.140395</td>\n",
              "      <td>683.320268</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>336</th>\n",
              "      <td>B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...</td>\n",
              "      <td>binnum</td>\n",
              "      <td>100</td>\n",
              "      <td>CESM</td>\n",
              "      <td>0.01</td>\n",
              "      <td>0.01</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>tumor_mask</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>2064.0</td>\n",
              "      <td>0.005138</td>\n",
              "      <td>8.544133</td>\n",
              "      <td>110</td>\n",
              "      <td>2043.806439</td>\n",
              "      <td>0.330860</td>\n",
              "      <td>15.341589</td>\n",
              "      <td>235.364338</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>337</th>\n",
              "      <td>B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...</td>\n",
              "      <td>binnum</td>\n",
              "      <td>100</td>\n",
              "      <td>CESM</td>\n",
              "      <td>0.01</td>\n",
              "      <td>0.01</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>tumor_mask</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>2084.0</td>\n",
              "      <td>0.007782</td>\n",
              "      <td>13.345616</td>\n",
              "      <td>155</td>\n",
              "      <td>2055.414637</td>\n",
              "      <td>-0.163142</td>\n",
              "      <td>22.641186</td>\n",
              "      <td>512.623307</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>338</th>\n",
              "      <td>B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...</td>\n",
              "      <td>binnum</td>\n",
              "      <td>100</td>\n",
              "      <td>CESM</td>\n",
              "      <td>0.01</td>\n",
              "      <td>0.01</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>tumor_mask</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>2017.0</td>\n",
              "      <td>0.004754</td>\n",
              "      <td>8.203036</td>\n",
              "      <td>124</td>\n",
              "      <td>1998.365225</td>\n",
              "      <td>-0.042320</td>\n",
              "      <td>14.446493</td>\n",
              "      <td>208.701167</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>339</th>\n",
              "      <td>B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...</td>\n",
              "      <td>binnum</td>\n",
              "      <td>100</td>\n",
              "      <td>CESM</td>\n",
              "      <td>0.01</td>\n",
              "      <td>0.01</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>tumor_mask</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>2037.0</td>\n",
              "      <td>0.004704</td>\n",
              "      <td>8.076171</td>\n",
              "      <td>121</td>\n",
              "      <td>2019.393275</td>\n",
              "      <td>-0.106028</td>\n",
              "      <td>14.609857</td>\n",
              "      <td>213.447917</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>340</th>\n",
              "      <td>B:\\CESM Radiomics\\Radiomics\\Radiomics_Data_CES...</td>\n",
              "      <td>binnum</td>\n",
              "      <td>100</td>\n",
              "      <td>CESM</td>\n",
              "      <td>0.01</td>\n",
              "      <td>0.01</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>no resampling</td>\n",
              "      <td>tumor_mask</td>\n",
              "      <td>1</td>\n",
              "      <td>...</td>\n",
              "      <td>2031.0</td>\n",
              "      <td>0.004970</td>\n",
              "      <td>8.419451</td>\n",
              "      <td>103</td>\n",
              "      <td>2012.063099</td>\n",
              "      <td>-0.039442</td>\n",
              "      <td>14.921726</td>\n",
              "      <td>222.657920</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>341 rows × 490 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e52b585d-5719-41ad-bbb9-9d033d99737c')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-e52b585d-5719-41ad-bbb9-9d033d99737c button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-e52b585d-5719-41ad-bbb9-9d033d99737c');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<b>Getting the features and defining the outcome</b>\n",
        "\n",
        "It's a good time to look at the .csv files using Google Tables or similar software. Ask a helper if you need assistance with this. Let's have a look at the columns and get a lis of the features from there:"
      ],
      "metadata": {
        "id": "_FoSexoRwOm8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "counter = 0\n",
        "for item in data_train.columns:\n",
        "  print (counter, item)\n",
        "  counter += 1"
      ],
      "metadata": {
        "id": "-NbbS-JbwePK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aac4ee94-84ee-45b0-f5ee-a436dcf93872"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0 ID\n",
            "1 LesionID\n",
            "2 discretization_method\n",
            "3 discretization_value\n",
            "4 modality\n",
            "5 pixel_spacing_1\n",
            "6 pixel_spacing_2\n",
            "7 resample_method\n",
            "8 resample_voxel_size\n",
            "9 structure_name\n",
            "10 voxel_distance\n",
            "11 Fractal_average\n",
            "12 Fractal_lacunarity\n",
            "13 Fractal_sd\n",
            "14 GLCM_autocorr\n",
            "15 GLCM_average\n",
            "16 GLCM_clusProm\n",
            "17 GLCM_clusShade\n",
            "18 GLCM_clusTend\n",
            "19 GLCM_contrast\n",
            "20 GLCM_correl1\n",
            "21 GLCM_diffAvrg\n",
            "22 GLCM_diffEntro\n",
            "23 GLCM_diffVar\n",
            "24 GLCM_dissimilar\n",
            "25 GLCM_energy\n",
            "26 GLCM_entrop2\n",
            "27 GLCM_homogeneity1\n",
            "28 GLCM_homogeneity2\n",
            "29 GLCM_infoCorr1\n",
            "30 GLCM_infoCorr2\n",
            "31 GLCM_invDiffMomNor\n",
            "32 GLCM_invDiffNorm\n",
            "33 GLCM_inverseVar\n",
            "34 GLCM_maxCorr\n",
            "35 GLCM_maxProb\n",
            "36 GLCM_sumAvg\n",
            "37 GLCM_sumEntro\n",
            "38 GLCM_sumSquares\n",
            "39 GLCM_sumVar\n",
            "40 GLDZM_DZE\n",
            "41 GLDZM_DZN\n",
            "42 GLDZM_DZNN\n",
            "43 GLDZM_DZV\n",
            "44 GLDZM_HIE\n",
            "45 GLDZM_HILDE\n",
            "46 GLDZM_HISDE\n",
            "47 GLDZM_IN\n",
            "48 GLDZM_INN\n",
            "49 GLDZM_IV\n",
            "50 GLDZM_LDE\n",
            "51 GLDZM_LIE\n",
            "52 GLDZM_LILDE\n",
            "53 GLDZM_LISDE\n",
            "54 GLDZM_SDE\n",
            "55 GLDZM_ZP\n",
            "56 GLRLM_GLN\n",
            "57 GLRLM_GLNN\n",
            "58 GLRLM_GLV\n",
            "59 GLRLM_HGRE\n",
            "60 GLRLM_LGRE\n",
            "61 GLRLM_LRE\n",
            "62 GLRLM_LRHGE\n",
            "63 GLRLM_LRLGE\n",
            "64 GLRLM_RE\n",
            "65 GLRLM_RLN\n",
            "66 GLRLM_RLNN\n",
            "67 GLRLM_RLV\n",
            "68 GLRLM_RP\n",
            "69 GLRLM_SRE\n",
            "70 GLRLM_SRHGE\n",
            "71 GLRLM_SRLGE\n",
            "72 GLSZM_HIE\n",
            "73 GLSZM_HILAE\n",
            "74 GLSZM_HISAE\n",
            "75 GLSZM_IN\n",
            "76 GLSZM_INN\n",
            "77 GLSZM_IV\n",
            "78 GLSZM_LAE\n",
            "79 GLSZM_LIE\n",
            "80 GLSZM_LILAE\n",
            "81 GLSZM_LISAE\n",
            "82 GLSZM_SAE\n",
            "83 GLSZM_SZN\n",
            "84 GLSZM_SZNN\n",
            "85 GLSZM_SZV\n",
            "86 GLSZM_ZE\n",
            "87 GLSZM_ZP\n",
            "88 LoG_sigma_0_1_mm_2D_IH_cov\n",
            "89 LoG_sigma_0_1_mm_2D_IH_energy\n",
            "90 LoG_sigma_0_1_mm_2D_IH_entropy\n",
            "91 LoG_sigma_0_1_mm_2D_IH_iqr\n",
            "92 LoG_sigma_0_1_mm_2D_IH_kurtosis\n",
            "93 LoG_sigma_0_1_mm_2D_IH_max\n",
            "94 LoG_sigma_0_1_mm_2D_IH_maxGrad\n",
            "95 LoG_sigma_0_1_mm_2D_IH_maxGradI\n",
            "96 LoG_sigma_0_1_mm_2D_IH_mean\n",
            "97 LoG_sigma_0_1_mm_2D_IH_meanD\n",
            "98 LoG_sigma_0_1_mm_2D_IH_median\n",
            "99 LoG_sigma_0_1_mm_2D_IH_medianD\n",
            "100 LoG_sigma_0_1_mm_2D_IH_min\n",
            "101 LoG_sigma_0_1_mm_2D_IH_minGrad\n",
            "102 LoG_sigma_0_1_mm_2D_IH_minGradI\n",
            "103 LoG_sigma_0_1_mm_2D_IH_mode\n",
            "104 LoG_sigma_0_1_mm_2D_IH_p10\n",
            "105 LoG_sigma_0_1_mm_2D_IH_p90\n",
            "106 LoG_sigma_0_1_mm_2D_IH_qcod\n",
            "107 LoG_sigma_0_1_mm_2D_IH_rMeanD\n",
            "108 LoG_sigma_0_1_mm_2D_IH_range\n",
            "109 LoG_sigma_0_1_mm_2D_IH_rms\n",
            "110 LoG_sigma_0_1_mm_2D_IH_skewness\n",
            "111 LoG_sigma_0_1_mm_2D_IH_std\n",
            "112 LoG_sigma_0_1_mm_2D_IH_uniformity\n",
            "113 LoG_sigma_0_1_mm_2D_IH_var\n",
            "114 LoG_sigma_0_1_mm_2D_Stats_cov\n",
            "115 LoG_sigma_0_1_mm_2D_Stats_energy\n",
            "116 LoG_sigma_0_1_mm_2D_Stats_iqr\n",
            "117 LoG_sigma_0_1_mm_2D_Stats_kurtosis\n",
            "118 LoG_sigma_0_1_mm_2D_Stats_max\n",
            "119 LoG_sigma_0_1_mm_2D_Stats_mean\n",
            "120 LoG_sigma_0_1_mm_2D_Stats_meanD\n",
            "121 LoG_sigma_0_1_mm_2D_Stats_median\n",
            "122 LoG_sigma_0_1_mm_2D_Stats_medianD\n",
            "123 LoG_sigma_0_1_mm_2D_Stats_min\n",
            "124 LoG_sigma_0_1_mm_2D_Stats_p10\n",
            "125 LoG_sigma_0_1_mm_2D_Stats_p90\n",
            "126 LoG_sigma_0_1_mm_2D_Stats_qcod\n",
            "127 LoG_sigma_0_1_mm_2D_Stats_rMeanD\n",
            "128 LoG_sigma_0_1_mm_2D_Stats_range\n",
            "129 LoG_sigma_0_1_mm_2D_Stats_rms\n",
            "130 LoG_sigma_0_1_mm_2D_Stats_skewness\n",
            "131 LoG_sigma_0_1_mm_2D_Stats_std\n",
            "132 LoG_sigma_0_1_mm_2D_Stats_var\n",
            "133 LoG_sigma_0_1_mm_2D_pos_IH_cov\n",
            "134 LoG_sigma_0_1_mm_2D_pos_IH_energy\n",
            "135 LoG_sigma_0_1_mm_2D_pos_IH_entropy\n",
            "136 LoG_sigma_0_1_mm_2D_pos_IH_iqr\n",
            "137 LoG_sigma_0_1_mm_2D_pos_IH_kurtosis\n",
            "138 LoG_sigma_0_1_mm_2D_pos_IH_max\n",
            "139 LoG_sigma_0_1_mm_2D_pos_IH_maxGrad\n",
            "140 LoG_sigma_0_1_mm_2D_pos_IH_maxGradI\n",
            "141 LoG_sigma_0_1_mm_2D_pos_IH_mean\n",
            "142 LoG_sigma_0_1_mm_2D_pos_IH_meanD\n",
            "143 LoG_sigma_0_1_mm_2D_pos_IH_median\n",
            "144 LoG_sigma_0_1_mm_2D_pos_IH_medianD\n",
            "145 LoG_sigma_0_1_mm_2D_pos_IH_min\n",
            "146 LoG_sigma_0_1_mm_2D_pos_IH_minGrad\n",
            "147 LoG_sigma_0_1_mm_2D_pos_IH_minGradI\n",
            "148 LoG_sigma_0_1_mm_2D_pos_IH_mode\n",
            "149 LoG_sigma_0_1_mm_2D_pos_IH_p10\n",
            "150 LoG_sigma_0_1_mm_2D_pos_IH_p90\n",
            "151 LoG_sigma_0_1_mm_2D_pos_IH_qcod\n",
            "152 LoG_sigma_0_1_mm_2D_pos_IH_rMeanD\n",
            "153 LoG_sigma_0_1_mm_2D_pos_IH_range\n",
            "154 LoG_sigma_0_1_mm_2D_pos_IH_rms\n",
            "155 LoG_sigma_0_1_mm_2D_pos_IH_skewness\n",
            "156 LoG_sigma_0_1_mm_2D_pos_IH_std\n",
            "157 LoG_sigma_0_1_mm_2D_pos_IH_uniformity\n",
            "158 LoG_sigma_0_1_mm_2D_pos_IH_var\n",
            "159 LoG_sigma_0_1_mm_2D_pos_Stats_cov\n",
            "160 LoG_sigma_0_1_mm_2D_pos_Stats_energy\n",
            "161 LoG_sigma_0_1_mm_2D_pos_Stats_iqr\n",
            "162 LoG_sigma_0_1_mm_2D_pos_Stats_kurtosis\n",
            "163 LoG_sigma_0_1_mm_2D_pos_Stats_max\n",
            "164 LoG_sigma_0_1_mm_2D_pos_Stats_mean\n",
            "165 LoG_sigma_0_1_mm_2D_pos_Stats_meanD\n",
            "166 LoG_sigma_0_1_mm_2D_pos_Stats_median\n",
            "167 LoG_sigma_0_1_mm_2D_pos_Stats_medianD\n",
            "168 LoG_sigma_0_1_mm_2D_pos_Stats_min\n",
            "169 LoG_sigma_0_1_mm_2D_pos_Stats_p10\n",
            "170 LoG_sigma_0_1_mm_2D_pos_Stats_p90\n",
            "171 LoG_sigma_0_1_mm_2D_pos_Stats_qcod\n",
            "172 LoG_sigma_0_1_mm_2D_pos_Stats_rMeanD\n",
            "173 LoG_sigma_0_1_mm_2D_pos_Stats_range\n",
            "174 LoG_sigma_0_1_mm_2D_pos_Stats_rms\n",
            "175 LoG_sigma_0_1_mm_2D_pos_Stats_skewness\n",
            "176 LoG_sigma_0_1_mm_2D_pos_Stats_std\n",
            "177 LoG_sigma_0_1_mm_2D_pos_Stats_var\n",
            "178 LoG_sigma_0_4_mm_2D_IH_cov\n",
            "179 LoG_sigma_0_4_mm_2D_IH_energy\n",
            "180 LoG_sigma_0_4_mm_2D_IH_entropy\n",
            "181 LoG_sigma_0_4_mm_2D_IH_iqr\n",
            "182 LoG_sigma_0_4_mm_2D_IH_kurtosis\n",
            "183 LoG_sigma_0_4_mm_2D_IH_max\n",
            "184 LoG_sigma_0_4_mm_2D_IH_maxGrad\n",
            "185 LoG_sigma_0_4_mm_2D_IH_maxGradI\n",
            "186 LoG_sigma_0_4_mm_2D_IH_mean\n",
            "187 LoG_sigma_0_4_mm_2D_IH_meanD\n",
            "188 LoG_sigma_0_4_mm_2D_IH_median\n",
            "189 LoG_sigma_0_4_mm_2D_IH_medianD\n",
            "190 LoG_sigma_0_4_mm_2D_IH_min\n",
            "191 LoG_sigma_0_4_mm_2D_IH_minGrad\n",
            "192 LoG_sigma_0_4_mm_2D_IH_minGradI\n",
            "193 LoG_sigma_0_4_mm_2D_IH_mode\n",
            "194 LoG_sigma_0_4_mm_2D_IH_p10\n",
            "195 LoG_sigma_0_4_mm_2D_IH_p90\n",
            "196 LoG_sigma_0_4_mm_2D_IH_qcod\n",
            "197 LoG_sigma_0_4_mm_2D_IH_rMeanD\n",
            "198 LoG_sigma_0_4_mm_2D_IH_range\n",
            "199 LoG_sigma_0_4_mm_2D_IH_rms\n",
            "200 LoG_sigma_0_4_mm_2D_IH_skewness\n",
            "201 LoG_sigma_0_4_mm_2D_IH_std\n",
            "202 LoG_sigma_0_4_mm_2D_IH_uniformity\n",
            "203 LoG_sigma_0_4_mm_2D_IH_var\n",
            "204 LoG_sigma_0_4_mm_2D_Stats_cov\n",
            "205 LoG_sigma_0_4_mm_2D_Stats_energy\n",
            "206 LoG_sigma_0_4_mm_2D_Stats_iqr\n",
            "207 LoG_sigma_0_4_mm_2D_Stats_kurtosis\n",
            "208 LoG_sigma_0_4_mm_2D_Stats_max\n",
            "209 LoG_sigma_0_4_mm_2D_Stats_mean\n",
            "210 LoG_sigma_0_4_mm_2D_Stats_meanD\n",
            "211 LoG_sigma_0_4_mm_2D_Stats_median\n",
            "212 LoG_sigma_0_4_mm_2D_Stats_medianD\n",
            "213 LoG_sigma_0_4_mm_2D_Stats_min\n",
            "214 LoG_sigma_0_4_mm_2D_Stats_p10\n",
            "215 LoG_sigma_0_4_mm_2D_Stats_p90\n",
            "216 LoG_sigma_0_4_mm_2D_Stats_qcod\n",
            "217 LoG_sigma_0_4_mm_2D_Stats_rMeanD\n",
            "218 LoG_sigma_0_4_mm_2D_Stats_range\n",
            "219 LoG_sigma_0_4_mm_2D_Stats_rms\n",
            "220 LoG_sigma_0_4_mm_2D_Stats_skewness\n",
            "221 LoG_sigma_0_4_mm_2D_Stats_std\n",
            "222 LoG_sigma_0_4_mm_2D_Stats_var\n",
            "223 LoG_sigma_0_4_mm_2D_pos_IH_cov\n",
            "224 LoG_sigma_0_4_mm_2D_pos_IH_energy\n",
            "225 LoG_sigma_0_4_mm_2D_pos_IH_entropy\n",
            "226 LoG_sigma_0_4_mm_2D_pos_IH_iqr\n",
            "227 LoG_sigma_0_4_mm_2D_pos_IH_kurtosis\n",
            "228 LoG_sigma_0_4_mm_2D_pos_IH_max\n",
            "229 LoG_sigma_0_4_mm_2D_pos_IH_maxGrad\n",
            "230 LoG_sigma_0_4_mm_2D_pos_IH_maxGradI\n",
            "231 LoG_sigma_0_4_mm_2D_pos_IH_mean\n",
            "232 LoG_sigma_0_4_mm_2D_pos_IH_meanD\n",
            "233 LoG_sigma_0_4_mm_2D_pos_IH_median\n",
            "234 LoG_sigma_0_4_mm_2D_pos_IH_medianD\n",
            "235 LoG_sigma_0_4_mm_2D_pos_IH_min\n",
            "236 LoG_sigma_0_4_mm_2D_pos_IH_minGrad\n",
            "237 LoG_sigma_0_4_mm_2D_pos_IH_minGradI\n",
            "238 LoG_sigma_0_4_mm_2D_pos_IH_mode\n",
            "239 LoG_sigma_0_4_mm_2D_pos_IH_p10\n",
            "240 LoG_sigma_0_4_mm_2D_pos_IH_p90\n",
            "241 LoG_sigma_0_4_mm_2D_pos_IH_qcod\n",
            "242 LoG_sigma_0_4_mm_2D_pos_IH_rMeanD\n",
            "243 LoG_sigma_0_4_mm_2D_pos_IH_range\n",
            "244 LoG_sigma_0_4_mm_2D_pos_IH_rms\n",
            "245 LoG_sigma_0_4_mm_2D_pos_IH_skewness\n",
            "246 LoG_sigma_0_4_mm_2D_pos_IH_std\n",
            "247 LoG_sigma_0_4_mm_2D_pos_IH_uniformity\n",
            "248 LoG_sigma_0_4_mm_2D_pos_IH_var\n",
            "249 LoG_sigma_0_4_mm_2D_pos_Stats_cov\n",
            "250 LoG_sigma_0_4_mm_2D_pos_Stats_energy\n",
            "251 LoG_sigma_0_4_mm_2D_pos_Stats_iqr\n",
            "252 LoG_sigma_0_4_mm_2D_pos_Stats_kurtosis\n",
            "253 LoG_sigma_0_4_mm_2D_pos_Stats_max\n",
            "254 LoG_sigma_0_4_mm_2D_pos_Stats_mean\n",
            "255 LoG_sigma_0_4_mm_2D_pos_Stats_meanD\n",
            "256 LoG_sigma_0_4_mm_2D_pos_Stats_median\n",
            "257 LoG_sigma_0_4_mm_2D_pos_Stats_medianD\n",
            "258 LoG_sigma_0_4_mm_2D_pos_Stats_min\n",
            "259 LoG_sigma_0_4_mm_2D_pos_Stats_p10\n",
            "260 LoG_sigma_0_4_mm_2D_pos_Stats_p90\n",
            "261 LoG_sigma_0_4_mm_2D_pos_Stats_qcod\n",
            "262 LoG_sigma_0_4_mm_2D_pos_Stats_rMeanD\n",
            "263 LoG_sigma_0_4_mm_2D_pos_Stats_range\n",
            "264 LoG_sigma_0_4_mm_2D_pos_Stats_rms\n",
            "265 LoG_sigma_0_4_mm_2D_pos_Stats_skewness\n",
            "266 LoG_sigma_0_4_mm_2D_pos_Stats_std\n",
            "267 LoG_sigma_0_4_mm_2D_pos_Stats_var\n",
            "268 LoG_sigma_0_7_mm_2D_IH_cov\n",
            "269 LoG_sigma_0_7_mm_2D_IH_energy\n",
            "270 LoG_sigma_0_7_mm_2D_IH_entropy\n",
            "271 LoG_sigma_0_7_mm_2D_IH_iqr\n",
            "272 LoG_sigma_0_7_mm_2D_IH_kurtosis\n",
            "273 LoG_sigma_0_7_mm_2D_IH_max\n",
            "274 LoG_sigma_0_7_mm_2D_IH_maxGrad\n",
            "275 LoG_sigma_0_7_mm_2D_IH_maxGradI\n",
            "276 LoG_sigma_0_7_mm_2D_IH_mean\n",
            "277 LoG_sigma_0_7_mm_2D_IH_meanD\n",
            "278 LoG_sigma_0_7_mm_2D_IH_median\n",
            "279 LoG_sigma_0_7_mm_2D_IH_medianD\n",
            "280 LoG_sigma_0_7_mm_2D_IH_min\n",
            "281 LoG_sigma_0_7_mm_2D_IH_minGrad\n",
            "282 LoG_sigma_0_7_mm_2D_IH_minGradI\n",
            "283 LoG_sigma_0_7_mm_2D_IH_mode\n",
            "284 LoG_sigma_0_7_mm_2D_IH_p10\n",
            "285 LoG_sigma_0_7_mm_2D_IH_p90\n",
            "286 LoG_sigma_0_7_mm_2D_IH_qcod\n",
            "287 LoG_sigma_0_7_mm_2D_IH_rMeanD\n",
            "288 LoG_sigma_0_7_mm_2D_IH_range\n",
            "289 LoG_sigma_0_7_mm_2D_IH_rms\n",
            "290 LoG_sigma_0_7_mm_2D_IH_skewness\n",
            "291 LoG_sigma_0_7_mm_2D_IH_std\n",
            "292 LoG_sigma_0_7_mm_2D_IH_uniformity\n",
            "293 LoG_sigma_0_7_mm_2D_IH_var\n",
            "294 LoG_sigma_0_7_mm_2D_Stats_cov\n",
            "295 LoG_sigma_0_7_mm_2D_Stats_energy\n",
            "296 LoG_sigma_0_7_mm_2D_Stats_iqr\n",
            "297 LoG_sigma_0_7_mm_2D_Stats_kurtosis\n",
            "298 LoG_sigma_0_7_mm_2D_Stats_max\n",
            "299 LoG_sigma_0_7_mm_2D_Stats_mean\n",
            "300 LoG_sigma_0_7_mm_2D_Stats_meanD\n",
            "301 LoG_sigma_0_7_mm_2D_Stats_median\n",
            "302 LoG_sigma_0_7_mm_2D_Stats_medianD\n",
            "303 LoG_sigma_0_7_mm_2D_Stats_min\n",
            "304 LoG_sigma_0_7_mm_2D_Stats_p10\n",
            "305 LoG_sigma_0_7_mm_2D_Stats_p90\n",
            "306 LoG_sigma_0_7_mm_2D_Stats_qcod\n",
            "307 LoG_sigma_0_7_mm_2D_Stats_rMeanD\n",
            "308 LoG_sigma_0_7_mm_2D_Stats_range\n",
            "309 LoG_sigma_0_7_mm_2D_Stats_rms\n",
            "310 LoG_sigma_0_7_mm_2D_Stats_skewness\n",
            "311 LoG_sigma_0_7_mm_2D_Stats_std\n",
            "312 LoG_sigma_0_7_mm_2D_Stats_var\n",
            "313 LoG_sigma_0_7_mm_2D_pos_IH_cov\n",
            "314 LoG_sigma_0_7_mm_2D_pos_IH_energy\n",
            "315 LoG_sigma_0_7_mm_2D_pos_IH_entropy\n",
            "316 LoG_sigma_0_7_mm_2D_pos_IH_iqr\n",
            "317 LoG_sigma_0_7_mm_2D_pos_IH_kurtosis\n",
            "318 LoG_sigma_0_7_mm_2D_pos_IH_max\n",
            "319 LoG_sigma_0_7_mm_2D_pos_IH_maxGrad\n",
            "320 LoG_sigma_0_7_mm_2D_pos_IH_maxGradI\n",
            "321 LoG_sigma_0_7_mm_2D_pos_IH_mean\n",
            "322 LoG_sigma_0_7_mm_2D_pos_IH_meanD\n",
            "323 LoG_sigma_0_7_mm_2D_pos_IH_median\n",
            "324 LoG_sigma_0_7_mm_2D_pos_IH_medianD\n",
            "325 LoG_sigma_0_7_mm_2D_pos_IH_min\n",
            "326 LoG_sigma_0_7_mm_2D_pos_IH_minGrad\n",
            "327 LoG_sigma_0_7_mm_2D_pos_IH_minGradI\n",
            "328 LoG_sigma_0_7_mm_2D_pos_IH_mode\n",
            "329 LoG_sigma_0_7_mm_2D_pos_IH_p10\n",
            "330 LoG_sigma_0_7_mm_2D_pos_IH_p90\n",
            "331 LoG_sigma_0_7_mm_2D_pos_IH_qcod\n",
            "332 LoG_sigma_0_7_mm_2D_pos_IH_rMeanD\n",
            "333 LoG_sigma_0_7_mm_2D_pos_IH_range\n",
            "334 LoG_sigma_0_7_mm_2D_pos_IH_rms\n",
            "335 LoG_sigma_0_7_mm_2D_pos_IH_skewness\n",
            "336 LoG_sigma_0_7_mm_2D_pos_IH_std\n",
            "337 LoG_sigma_0_7_mm_2D_pos_IH_uniformity\n",
            "338 LoG_sigma_0_7_mm_2D_pos_IH_var\n",
            "339 LoG_sigma_0_7_mm_2D_pos_Stats_cov\n",
            "340 LoG_sigma_0_7_mm_2D_pos_Stats_energy\n",
            "341 LoG_sigma_0_7_mm_2D_pos_Stats_iqr\n",
            "342 LoG_sigma_0_7_mm_2D_pos_Stats_kurtosis\n",
            "343 LoG_sigma_0_7_mm_2D_pos_Stats_max\n",
            "344 LoG_sigma_0_7_mm_2D_pos_Stats_mean\n",
            "345 LoG_sigma_0_7_mm_2D_pos_Stats_meanD\n",
            "346 LoG_sigma_0_7_mm_2D_pos_Stats_median\n",
            "347 LoG_sigma_0_7_mm_2D_pos_Stats_medianD\n",
            "348 LoG_sigma_0_7_mm_2D_pos_Stats_min\n",
            "349 LoG_sigma_0_7_mm_2D_pos_Stats_p10\n",
            "350 LoG_sigma_0_7_mm_2D_pos_Stats_p90\n",
            "351 LoG_sigma_0_7_mm_2D_pos_Stats_qcod\n",
            "352 LoG_sigma_0_7_mm_2D_pos_Stats_rMeanD\n",
            "353 LoG_sigma_0_7_mm_2D_pos_Stats_range\n",
            "354 LoG_sigma_0_7_mm_2D_pos_Stats_rms\n",
            "355 LoG_sigma_0_7_mm_2D_pos_Stats_skewness\n",
            "356 LoG_sigma_0_7_mm_2D_pos_Stats_std\n",
            "357 LoG_sigma_0_7_mm_2D_pos_Stats_var\n",
            "358 LoG_sigma_1_mm_2D_IH_cov\n",
            "359 LoG_sigma_1_mm_2D_IH_energy\n",
            "360 LoG_sigma_1_mm_2D_IH_entropy\n",
            "361 LoG_sigma_1_mm_2D_IH_iqr\n",
            "362 LoG_sigma_1_mm_2D_IH_kurtosis\n",
            "363 LoG_sigma_1_mm_2D_IH_max\n",
            "364 LoG_sigma_1_mm_2D_IH_maxGrad\n",
            "365 LoG_sigma_1_mm_2D_IH_maxGradI\n",
            "366 LoG_sigma_1_mm_2D_IH_mean\n",
            "367 LoG_sigma_1_mm_2D_IH_meanD\n",
            "368 LoG_sigma_1_mm_2D_IH_median\n",
            "369 LoG_sigma_1_mm_2D_IH_medianD\n",
            "370 LoG_sigma_1_mm_2D_IH_min\n",
            "371 LoG_sigma_1_mm_2D_IH_minGrad\n",
            "372 LoG_sigma_1_mm_2D_IH_minGradI\n",
            "373 LoG_sigma_1_mm_2D_IH_mode\n",
            "374 LoG_sigma_1_mm_2D_IH_p10\n",
            "375 LoG_sigma_1_mm_2D_IH_p90\n",
            "376 LoG_sigma_1_mm_2D_IH_qcod\n",
            "377 LoG_sigma_1_mm_2D_IH_rMeanD\n",
            "378 LoG_sigma_1_mm_2D_IH_range\n",
            "379 LoG_sigma_1_mm_2D_IH_rms\n",
            "380 LoG_sigma_1_mm_2D_IH_skewness\n",
            "381 LoG_sigma_1_mm_2D_IH_std\n",
            "382 LoG_sigma_1_mm_2D_IH_uniformity\n",
            "383 LoG_sigma_1_mm_2D_IH_var\n",
            "384 LoG_sigma_1_mm_2D_Stats_cov\n",
            "385 LoG_sigma_1_mm_2D_Stats_energy\n",
            "386 LoG_sigma_1_mm_2D_Stats_iqr\n",
            "387 LoG_sigma_1_mm_2D_Stats_kurtosis\n",
            "388 LoG_sigma_1_mm_2D_Stats_max\n",
            "389 LoG_sigma_1_mm_2D_Stats_mean\n",
            "390 LoG_sigma_1_mm_2D_Stats_meanD\n",
            "391 LoG_sigma_1_mm_2D_Stats_median\n",
            "392 LoG_sigma_1_mm_2D_Stats_medianD\n",
            "393 LoG_sigma_1_mm_2D_Stats_min\n",
            "394 LoG_sigma_1_mm_2D_Stats_p10\n",
            "395 LoG_sigma_1_mm_2D_Stats_p90\n",
            "396 LoG_sigma_1_mm_2D_Stats_qcod\n",
            "397 LoG_sigma_1_mm_2D_Stats_rMeanD\n",
            "398 LoG_sigma_1_mm_2D_Stats_range\n",
            "399 LoG_sigma_1_mm_2D_Stats_rms\n",
            "400 LoG_sigma_1_mm_2D_Stats_skewness\n",
            "401 LoG_sigma_1_mm_2D_Stats_std\n",
            "402 LoG_sigma_1_mm_2D_Stats_var\n",
            "403 LoG_sigma_1_mm_2D_pos_IH_cov\n",
            "404 LoG_sigma_1_mm_2D_pos_IH_energy\n",
            "405 LoG_sigma_1_mm_2D_pos_IH_entropy\n",
            "406 LoG_sigma_1_mm_2D_pos_IH_iqr\n",
            "407 LoG_sigma_1_mm_2D_pos_IH_kurtosis\n",
            "408 LoG_sigma_1_mm_2D_pos_IH_max\n",
            "409 LoG_sigma_1_mm_2D_pos_IH_maxGrad\n",
            "410 LoG_sigma_1_mm_2D_pos_IH_maxGradI\n",
            "411 LoG_sigma_1_mm_2D_pos_IH_mean\n",
            "412 LoG_sigma_1_mm_2D_pos_IH_meanD\n",
            "413 LoG_sigma_1_mm_2D_pos_IH_median\n",
            "414 LoG_sigma_1_mm_2D_pos_IH_medianD\n",
            "415 LoG_sigma_1_mm_2D_pos_IH_min\n",
            "416 LoG_sigma_1_mm_2D_pos_IH_minGrad\n",
            "417 LoG_sigma_1_mm_2D_pos_IH_minGradI\n",
            "418 LoG_sigma_1_mm_2D_pos_IH_mode\n",
            "419 LoG_sigma_1_mm_2D_pos_IH_p10\n",
            "420 LoG_sigma_1_mm_2D_pos_IH_p90\n",
            "421 LoG_sigma_1_mm_2D_pos_IH_qcod\n",
            "422 LoG_sigma_1_mm_2D_pos_IH_rMeanD\n",
            "423 LoG_sigma_1_mm_2D_pos_IH_range\n",
            "424 LoG_sigma_1_mm_2D_pos_IH_rms\n",
            "425 LoG_sigma_1_mm_2D_pos_IH_skewness\n",
            "426 LoG_sigma_1_mm_2D_pos_IH_std\n",
            "427 LoG_sigma_1_mm_2D_pos_IH_uniformity\n",
            "428 LoG_sigma_1_mm_2D_pos_IH_var\n",
            "429 LoG_sigma_1_mm_2D_pos_Stats_cov\n",
            "430 LoG_sigma_1_mm_2D_pos_Stats_energy\n",
            "431 LoG_sigma_1_mm_2D_pos_Stats_iqr\n",
            "432 LoG_sigma_1_mm_2D_pos_Stats_kurtosis\n",
            "433 LoG_sigma_1_mm_2D_pos_Stats_max\n",
            "434 LoG_sigma_1_mm_2D_pos_Stats_mean\n",
            "435 LoG_sigma_1_mm_2D_pos_Stats_meanD\n",
            "436 LoG_sigma_1_mm_2D_pos_Stats_median\n",
            "437 LoG_sigma_1_mm_2D_pos_Stats_medianD\n",
            "438 LoG_sigma_1_mm_2D_pos_Stats_min\n",
            "439 LoG_sigma_1_mm_2D_pos_Stats_p10\n",
            "440 LoG_sigma_1_mm_2D_pos_Stats_p90\n",
            "441 LoG_sigma_1_mm_2D_pos_Stats_qcod\n",
            "442 LoG_sigma_1_mm_2D_pos_Stats_rMeanD\n",
            "443 LoG_sigma_1_mm_2D_pos_Stats_range\n",
            "444 LoG_sigma_1_mm_2D_pos_Stats_rms\n",
            "445 LoG_sigma_1_mm_2D_pos_Stats_skewness\n",
            "446 LoG_sigma_1_mm_2D_pos_Stats_std\n",
            "447 LoG_sigma_1_mm_2D_pos_Stats_var\n",
            "448 NGLDM_DE\n",
            "449 NGLDM_DN\n",
            "450 NGLDM_DNN\n",
            "451 NGLDM_DV\n",
            "452 NGLDM_GLN\n",
            "453 NGLDM_GLNN\n",
            "454 NGLDM_GLV\n",
            "455 NGLDM_HGE\n",
            "456 NGLDM_HGLDE\n",
            "457 NGLDM_HGSDE\n",
            "458 NGLDM_LDE\n",
            "459 NGLDM_LGE\n",
            "460 NGLDM_LGLDE\n",
            "461 NGLDM_LGSDE\n",
            "462 NGLDM_SDE\n",
            "463 NGLDM_SM\n",
            "464 NGLDM_SM2\n",
            "465 NGTDM_busyness\n",
            "466 NGTDM_coarseness\n",
            "467 NGTDM_complexity\n",
            "468 NGTDM_contrast\n",
            "469 NGTDM_strength\n",
            "470 Stats_cov\n",
            "471 Stats_energy\n",
            "472 Stats_iqr\n",
            "473 Stats_kurtosis\n",
            "474 Stats_max\n",
            "475 Stats_mean\n",
            "476 Stats_meanD\n",
            "477 Stats_median\n",
            "478 Stats_medianD\n",
            "479 Stats_min\n",
            "480 Stats_p10\n",
            "481 Stats_p90\n",
            "482 Stats_qcod\n",
            "483 Stats_rMeanD\n",
            "484 Stats_range\n",
            "485 Stats_rms\n",
            "486 Stats_skewness\n",
            "487 Stats_std\n",
            "488 Stats_var\n",
            "489 Outcome\n",
            "490 outcome\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "And get list of the features:"
      ],
      "metadata": {
        "id": "qMkJDNYKA7rc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "features = list(data_train.columns[11:489])\n",
        "print (len(features))\n",
        "print (features)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dmppgJK-A_g1",
        "outputId": "6c403149-cbe3-471b-fdec-c55b8d24b9eb"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "467\n",
            "['GLCM_diffEntro', 'GLCM_diffVar', 'GLCM_dissimilar', 'GLCM_energy', 'GLCM_entrop2', 'GLCM_homogeneity1', 'GLCM_homogeneity2', 'GLCM_infoCorr1', 'GLCM_infoCorr2', 'GLCM_invDiffMomNor', 'GLCM_invDiffNorm', 'GLCM_inverseVar', 'GLCM_maxCorr', 'GLCM_maxProb', 'GLCM_sumAvg', 'GLCM_sumEntro', 'GLCM_sumSquares', 'GLCM_sumVar', 'GLDZM_DZE', 'GLDZM_DZN', 'GLDZM_DZNN', 'GLDZM_DZV', 'GLDZM_HIE', 'GLDZM_HILDE', 'GLDZM_HISDE', 'GLDZM_IN', 'GLDZM_INN', 'GLDZM_IV', 'GLDZM_LDE', 'GLDZM_LIE', 'GLDZM_LILDE', 'GLDZM_LISDE', 'GLDZM_SDE', 'GLDZM_ZP', 'GLRLM_GLN', 'GLRLM_GLNN', 'GLRLM_GLV', 'GLRLM_HGRE', 'GLRLM_LGRE', 'GLRLM_LRE', 'GLRLM_LRHGE', 'GLRLM_LRLGE', 'GLRLM_RE', 'GLRLM_RLN', 'GLRLM_RLNN', 'GLRLM_RLV', 'GLRLM_RP', 'GLRLM_SRE', 'GLRLM_SRHGE', 'GLRLM_SRLGE', 'GLSZM_HIE', 'GLSZM_HILAE', 'GLSZM_HISAE', 'GLSZM_IN', 'GLSZM_INN', 'GLSZM_IV', 'GLSZM_LAE', 'GLSZM_LIE', 'GLSZM_LILAE', 'GLSZM_LISAE', 'GLSZM_SAE', 'GLSZM_SZN', 'GLSZM_SZNN', 'GLSZM_SZV', 'GLSZM_ZE', 'GLSZM_ZP', 'LoG_sigma_0_1_mm_2D_IH_cov', 'LoG_sigma_0_1_mm_2D_IH_energy', 'LoG_sigma_0_1_mm_2D_IH_entropy', 'LoG_sigma_0_1_mm_2D_IH_iqr', 'LoG_sigma_0_1_mm_2D_IH_kurtosis', 'LoG_sigma_0_1_mm_2D_IH_max', 'LoG_sigma_0_1_mm_2D_IH_maxGrad', 'LoG_sigma_0_1_mm_2D_IH_maxGradI', 'LoG_sigma_0_1_mm_2D_IH_mean', 'LoG_sigma_0_1_mm_2D_IH_meanD', 'LoG_sigma_0_1_mm_2D_IH_median', 'LoG_sigma_0_1_mm_2D_IH_medianD', 'LoG_sigma_0_1_mm_2D_IH_min', 'LoG_sigma_0_1_mm_2D_IH_minGrad', 'LoG_sigma_0_1_mm_2D_IH_minGradI', 'LoG_sigma_0_1_mm_2D_IH_mode', 'LoG_sigma_0_1_mm_2D_IH_p10', 'LoG_sigma_0_1_mm_2D_IH_p90', 'LoG_sigma_0_1_mm_2D_IH_qcod', 'LoG_sigma_0_1_mm_2D_IH_rMeanD', 'LoG_sigma_0_1_mm_2D_IH_range', 'LoG_sigma_0_1_mm_2D_IH_rms', 'LoG_sigma_0_1_mm_2D_IH_skewness', 'LoG_sigma_0_1_mm_2D_IH_std', 'LoG_sigma_0_1_mm_2D_IH_uniformity', 'LoG_sigma_0_1_mm_2D_IH_var', 'LoG_sigma_0_1_mm_2D_Stats_cov', 'LoG_sigma_0_1_mm_2D_Stats_energy', 'LoG_sigma_0_1_mm_2D_Stats_iqr', 'LoG_sigma_0_1_mm_2D_Stats_kurtosis', 'LoG_sigma_0_1_mm_2D_Stats_max', 'LoG_sigma_0_1_mm_2D_Stats_mean', 'LoG_sigma_0_1_mm_2D_Stats_meanD', 'LoG_sigma_0_1_mm_2D_Stats_median', 'LoG_sigma_0_1_mm_2D_Stats_medianD', 'LoG_sigma_0_1_mm_2D_Stats_min', 'LoG_sigma_0_1_mm_2D_Stats_p10', 'LoG_sigma_0_1_mm_2D_Stats_p90', 'LoG_sigma_0_1_mm_2D_Stats_qcod', 'LoG_sigma_0_1_mm_2D_Stats_rMeanD', 'LoG_sigma_0_1_mm_2D_Stats_range', 'LoG_sigma_0_1_mm_2D_Stats_rms', 'LoG_sigma_0_1_mm_2D_Stats_skewness', 'LoG_sigma_0_1_mm_2D_Stats_std', 'LoG_sigma_0_1_mm_2D_Stats_var', 'LoG_sigma_0_1_mm_2D_pos_IH_cov', 'LoG_sigma_0_1_mm_2D_pos_IH_energy', 'LoG_sigma_0_1_mm_2D_pos_IH_entropy', 'LoG_sigma_0_1_mm_2D_pos_IH_iqr', 'LoG_sigma_0_1_mm_2D_pos_IH_kurtosis', 'LoG_sigma_0_1_mm_2D_pos_IH_max', 'LoG_sigma_0_1_mm_2D_pos_IH_maxGrad', 'LoG_sigma_0_1_mm_2D_pos_IH_maxGradI', 'LoG_sigma_0_1_mm_2D_pos_IH_mean', 'LoG_sigma_0_1_mm_2D_pos_IH_meanD', 'LoG_sigma_0_1_mm_2D_pos_IH_median', 'LoG_sigma_0_1_mm_2D_pos_IH_medianD', 'LoG_sigma_0_1_mm_2D_pos_IH_min', 'LoG_sigma_0_1_mm_2D_pos_IH_minGrad', 'LoG_sigma_0_1_mm_2D_pos_IH_minGradI', 'LoG_sigma_0_1_mm_2D_pos_IH_mode', 'LoG_sigma_0_1_mm_2D_pos_IH_p10', 'LoG_sigma_0_1_mm_2D_pos_IH_p90', 'LoG_sigma_0_1_mm_2D_pos_IH_qcod', 'LoG_sigma_0_1_mm_2D_pos_IH_rMeanD', 'LoG_sigma_0_1_mm_2D_pos_IH_range', 'LoG_sigma_0_1_mm_2D_pos_IH_rms', 'LoG_sigma_0_1_mm_2D_pos_IH_skewness', 'LoG_sigma_0_1_mm_2D_pos_IH_std', 'LoG_sigma_0_1_mm_2D_pos_IH_uniformity', 'LoG_sigma_0_1_mm_2D_pos_IH_var', 'LoG_sigma_0_1_mm_2D_pos_Stats_cov', 'LoG_sigma_0_1_mm_2D_pos_Stats_energy', 'LoG_sigma_0_1_mm_2D_pos_Stats_iqr', 'LoG_sigma_0_1_mm_2D_pos_Stats_kurtosis', 'LoG_sigma_0_1_mm_2D_pos_Stats_max', 'LoG_sigma_0_1_mm_2D_pos_Stats_mean', 'LoG_sigma_0_1_mm_2D_pos_Stats_meanD', 'LoG_sigma_0_1_mm_2D_pos_Stats_median', 'LoG_sigma_0_1_mm_2D_pos_Stats_medianD', 'LoG_sigma_0_1_mm_2D_pos_Stats_min', 'LoG_sigma_0_1_mm_2D_pos_Stats_p10', 'LoG_sigma_0_1_mm_2D_pos_Stats_p90', 'LoG_sigma_0_1_mm_2D_pos_Stats_qcod', 'LoG_sigma_0_1_mm_2D_pos_Stats_rMeanD', 'LoG_sigma_0_1_mm_2D_pos_Stats_range', 'LoG_sigma_0_1_mm_2D_pos_Stats_rms', 'LoG_sigma_0_1_mm_2D_pos_Stats_skewness', 'LoG_sigma_0_1_mm_2D_pos_Stats_std', 'LoG_sigma_0_1_mm_2D_pos_Stats_var', 'LoG_sigma_0_4_mm_2D_IH_cov', 'LoG_sigma_0_4_mm_2D_IH_energy', 'LoG_sigma_0_4_mm_2D_IH_entropy', 'LoG_sigma_0_4_mm_2D_IH_iqr', 'LoG_sigma_0_4_mm_2D_IH_kurtosis', 'LoG_sigma_0_4_mm_2D_IH_max', 'LoG_sigma_0_4_mm_2D_IH_maxGrad', 'LoG_sigma_0_4_mm_2D_IH_maxGradI', 'LoG_sigma_0_4_mm_2D_IH_mean', 'LoG_sigma_0_4_mm_2D_IH_meanD', 'LoG_sigma_0_4_mm_2D_IH_median', 'LoG_sigma_0_4_mm_2D_IH_medianD', 'LoG_sigma_0_4_mm_2D_IH_min', 'LoG_sigma_0_4_mm_2D_IH_minGrad', 'LoG_sigma_0_4_mm_2D_IH_minGradI', 'LoG_sigma_0_4_mm_2D_IH_mode', 'LoG_sigma_0_4_mm_2D_IH_p10', 'LoG_sigma_0_4_mm_2D_IH_p90', 'LoG_sigma_0_4_mm_2D_IH_qcod', 'LoG_sigma_0_4_mm_2D_IH_rMeanD', 'LoG_sigma_0_4_mm_2D_IH_range', 'LoG_sigma_0_4_mm_2D_IH_rms', 'LoG_sigma_0_4_mm_2D_IH_skewness', 'LoG_sigma_0_4_mm_2D_IH_std', 'LoG_sigma_0_4_mm_2D_IH_uniformity', 'LoG_sigma_0_4_mm_2D_IH_var', 'LoG_sigma_0_4_mm_2D_Stats_cov', 'LoG_sigma_0_4_mm_2D_Stats_energy', 'LoG_sigma_0_4_mm_2D_Stats_iqr', 'LoG_sigma_0_4_mm_2D_Stats_kurtosis', 'LoG_sigma_0_4_mm_2D_Stats_max', 'LoG_sigma_0_4_mm_2D_Stats_mean', 'LoG_sigma_0_4_mm_2D_Stats_meanD', 'LoG_sigma_0_4_mm_2D_Stats_median', 'LoG_sigma_0_4_mm_2D_Stats_medianD', 'LoG_sigma_0_4_mm_2D_Stats_min', 'LoG_sigma_0_4_mm_2D_Stats_p10', 'LoG_sigma_0_4_mm_2D_Stats_p90', 'LoG_sigma_0_4_mm_2D_Stats_qcod', 'LoG_sigma_0_4_mm_2D_Stats_rMeanD', 'LoG_sigma_0_4_mm_2D_Stats_range', 'LoG_sigma_0_4_mm_2D_Stats_rms', 'LoG_sigma_0_4_mm_2D_Stats_skewness', 'LoG_sigma_0_4_mm_2D_Stats_std', 'LoG_sigma_0_4_mm_2D_Stats_var', 'LoG_sigma_0_4_mm_2D_pos_IH_cov', 'LoG_sigma_0_4_mm_2D_pos_IH_energy', 'LoG_sigma_0_4_mm_2D_pos_IH_entropy', 'LoG_sigma_0_4_mm_2D_pos_IH_iqr', 'LoG_sigma_0_4_mm_2D_pos_IH_kurtosis', 'LoG_sigma_0_4_mm_2D_pos_IH_max', 'LoG_sigma_0_4_mm_2D_pos_IH_maxGrad', 'LoG_sigma_0_4_mm_2D_pos_IH_maxGradI', 'LoG_sigma_0_4_mm_2D_pos_IH_mean', 'LoG_sigma_0_4_mm_2D_pos_IH_meanD', 'LoG_sigma_0_4_mm_2D_pos_IH_median', 'LoG_sigma_0_4_mm_2D_pos_IH_medianD', 'LoG_sigma_0_4_mm_2D_pos_IH_min', 'LoG_sigma_0_4_mm_2D_pos_IH_minGrad', 'LoG_sigma_0_4_mm_2D_pos_IH_minGradI', 'LoG_sigma_0_4_mm_2D_pos_IH_mode', 'LoG_sigma_0_4_mm_2D_pos_IH_p10', 'LoG_sigma_0_4_mm_2D_pos_IH_p90', 'LoG_sigma_0_4_mm_2D_pos_IH_qcod', 'LoG_sigma_0_4_mm_2D_pos_IH_rMeanD', 'LoG_sigma_0_4_mm_2D_pos_IH_range', 'LoG_sigma_0_4_mm_2D_pos_IH_rms', 'LoG_sigma_0_4_mm_2D_pos_IH_skewness', 'LoG_sigma_0_4_mm_2D_pos_IH_std', 'LoG_sigma_0_4_mm_2D_pos_IH_uniformity', 'LoG_sigma_0_4_mm_2D_pos_IH_var', 'LoG_sigma_0_4_mm_2D_pos_Stats_cov', 'LoG_sigma_0_4_mm_2D_pos_Stats_energy', 'LoG_sigma_0_4_mm_2D_pos_Stats_iqr', 'LoG_sigma_0_4_mm_2D_pos_Stats_kurtosis', 'LoG_sigma_0_4_mm_2D_pos_Stats_max', 'LoG_sigma_0_4_mm_2D_pos_Stats_mean', 'LoG_sigma_0_4_mm_2D_pos_Stats_meanD', 'LoG_sigma_0_4_mm_2D_pos_Stats_median', 'LoG_sigma_0_4_mm_2D_pos_Stats_medianD', 'LoG_sigma_0_4_mm_2D_pos_Stats_min', 'LoG_sigma_0_4_mm_2D_pos_Stats_p10', 'LoG_sigma_0_4_mm_2D_pos_Stats_p90', 'LoG_sigma_0_4_mm_2D_pos_Stats_qcod', 'LoG_sigma_0_4_mm_2D_pos_Stats_rMeanD', 'LoG_sigma_0_4_mm_2D_pos_Stats_range', 'LoG_sigma_0_4_mm_2D_pos_Stats_rms', 'LoG_sigma_0_4_mm_2D_pos_Stats_skewness', 'LoG_sigma_0_4_mm_2D_pos_Stats_std', 'LoG_sigma_0_4_mm_2D_pos_Stats_var', 'LoG_sigma_0_7_mm_2D_IH_cov', 'LoG_sigma_0_7_mm_2D_IH_energy', 'LoG_sigma_0_7_mm_2D_IH_entropy', 'LoG_sigma_0_7_mm_2D_IH_iqr', 'LoG_sigma_0_7_mm_2D_IH_kurtosis', 'LoG_sigma_0_7_mm_2D_IH_max', 'LoG_sigma_0_7_mm_2D_IH_maxGrad', 'LoG_sigma_0_7_mm_2D_IH_maxGradI', 'LoG_sigma_0_7_mm_2D_IH_mean', 'LoG_sigma_0_7_mm_2D_IH_meanD', 'LoG_sigma_0_7_mm_2D_IH_median', 'LoG_sigma_0_7_mm_2D_IH_medianD', 'LoG_sigma_0_7_mm_2D_IH_min', 'LoG_sigma_0_7_mm_2D_IH_minGrad', 'LoG_sigma_0_7_mm_2D_IH_minGradI', 'LoG_sigma_0_7_mm_2D_IH_mode', 'LoG_sigma_0_7_mm_2D_IH_p10', 'LoG_sigma_0_7_mm_2D_IH_p90', 'LoG_sigma_0_7_mm_2D_IH_qcod', 'LoG_sigma_0_7_mm_2D_IH_rMeanD', 'LoG_sigma_0_7_mm_2D_IH_range', 'LoG_sigma_0_7_mm_2D_IH_rms', 'LoG_sigma_0_7_mm_2D_IH_skewness', 'LoG_sigma_0_7_mm_2D_IH_std', 'LoG_sigma_0_7_mm_2D_IH_uniformity', 'LoG_sigma_0_7_mm_2D_IH_var', 'LoG_sigma_0_7_mm_2D_Stats_cov', 'LoG_sigma_0_7_mm_2D_Stats_energy', 'LoG_sigma_0_7_mm_2D_Stats_iqr', 'LoG_sigma_0_7_mm_2D_Stats_kurtosis', 'LoG_sigma_0_7_mm_2D_Stats_max', 'LoG_sigma_0_7_mm_2D_Stats_mean', 'LoG_sigma_0_7_mm_2D_Stats_meanD', 'LoG_sigma_0_7_mm_2D_Stats_median', 'LoG_sigma_0_7_mm_2D_Stats_medianD', 'LoG_sigma_0_7_mm_2D_Stats_min', 'LoG_sigma_0_7_mm_2D_Stats_p10', 'LoG_sigma_0_7_mm_2D_Stats_p90', 'LoG_sigma_0_7_mm_2D_Stats_qcod', 'LoG_sigma_0_7_mm_2D_Stats_rMeanD', 'LoG_sigma_0_7_mm_2D_Stats_range', 'LoG_sigma_0_7_mm_2D_Stats_rms', 'LoG_sigma_0_7_mm_2D_Stats_skewness', 'LoG_sigma_0_7_mm_2D_Stats_std', 'LoG_sigma_0_7_mm_2D_Stats_var', 'LoG_sigma_0_7_mm_2D_pos_IH_cov', 'LoG_sigma_0_7_mm_2D_pos_IH_energy', 'LoG_sigma_0_7_mm_2D_pos_IH_entropy', 'LoG_sigma_0_7_mm_2D_pos_IH_iqr', 'LoG_sigma_0_7_mm_2D_pos_IH_kurtosis', 'LoG_sigma_0_7_mm_2D_pos_IH_max', 'LoG_sigma_0_7_mm_2D_pos_IH_maxGrad', 'LoG_sigma_0_7_mm_2D_pos_IH_maxGradI', 'LoG_sigma_0_7_mm_2D_pos_IH_mean', 'LoG_sigma_0_7_mm_2D_pos_IH_meanD', 'LoG_sigma_0_7_mm_2D_pos_IH_median', 'LoG_sigma_0_7_mm_2D_pos_IH_medianD', 'LoG_sigma_0_7_mm_2D_pos_IH_min', 'LoG_sigma_0_7_mm_2D_pos_IH_minGrad', 'LoG_sigma_0_7_mm_2D_pos_IH_minGradI', 'LoG_sigma_0_7_mm_2D_pos_IH_mode', 'LoG_sigma_0_7_mm_2D_pos_IH_p10', 'LoG_sigma_0_7_mm_2D_pos_IH_p90', 'LoG_sigma_0_7_mm_2D_pos_IH_qcod', 'LoG_sigma_0_7_mm_2D_pos_IH_rMeanD', 'LoG_sigma_0_7_mm_2D_pos_IH_range', 'LoG_sigma_0_7_mm_2D_pos_IH_rms', 'LoG_sigma_0_7_mm_2D_pos_IH_skewness', 'LoG_sigma_0_7_mm_2D_pos_IH_std', 'LoG_sigma_0_7_mm_2D_pos_IH_uniformity', 'LoG_sigma_0_7_mm_2D_pos_IH_var', 'LoG_sigma_0_7_mm_2D_pos_Stats_cov', 'LoG_sigma_0_7_mm_2D_pos_Stats_energy', 'LoG_sigma_0_7_mm_2D_pos_Stats_iqr', 'LoG_sigma_0_7_mm_2D_pos_Stats_kurtosis', 'LoG_sigma_0_7_mm_2D_pos_Stats_max', 'LoG_sigma_0_7_mm_2D_pos_Stats_mean', 'LoG_sigma_0_7_mm_2D_pos_Stats_meanD', 'LoG_sigma_0_7_mm_2D_pos_Stats_median', 'LoG_sigma_0_7_mm_2D_pos_Stats_medianD', 'LoG_sigma_0_7_mm_2D_pos_Stats_min', 'LoG_sigma_0_7_mm_2D_pos_Stats_p10', 'LoG_sigma_0_7_mm_2D_pos_Stats_p90', 'LoG_sigma_0_7_mm_2D_pos_Stats_qcod', 'LoG_sigma_0_7_mm_2D_pos_Stats_rMeanD', 'LoG_sigma_0_7_mm_2D_pos_Stats_range', 'LoG_sigma_0_7_mm_2D_pos_Stats_rms', 'LoG_sigma_0_7_mm_2D_pos_Stats_skewness', 'LoG_sigma_0_7_mm_2D_pos_Stats_std', 'LoG_sigma_0_7_mm_2D_pos_Stats_var', 'LoG_sigma_1_mm_2D_IH_cov', 'LoG_sigma_1_mm_2D_IH_energy', 'LoG_sigma_1_mm_2D_IH_entropy', 'LoG_sigma_1_mm_2D_IH_iqr', 'LoG_sigma_1_mm_2D_IH_kurtosis', 'LoG_sigma_1_mm_2D_IH_max', 'LoG_sigma_1_mm_2D_IH_maxGrad', 'LoG_sigma_1_mm_2D_IH_maxGradI', 'LoG_sigma_1_mm_2D_IH_mean', 'LoG_sigma_1_mm_2D_IH_meanD', 'LoG_sigma_1_mm_2D_IH_median', 'LoG_sigma_1_mm_2D_IH_medianD', 'LoG_sigma_1_mm_2D_IH_min', 'LoG_sigma_1_mm_2D_IH_minGrad', 'LoG_sigma_1_mm_2D_IH_minGradI', 'LoG_sigma_1_mm_2D_IH_mode', 'LoG_sigma_1_mm_2D_IH_p10', 'LoG_sigma_1_mm_2D_IH_p90', 'LoG_sigma_1_mm_2D_IH_qcod', 'LoG_sigma_1_mm_2D_IH_rMeanD', 'LoG_sigma_1_mm_2D_IH_range', 'LoG_sigma_1_mm_2D_IH_rms', 'LoG_sigma_1_mm_2D_IH_skewness', 'LoG_sigma_1_mm_2D_IH_std', 'LoG_sigma_1_mm_2D_IH_uniformity', 'LoG_sigma_1_mm_2D_IH_var', 'LoG_sigma_1_mm_2D_Stats_cov', 'LoG_sigma_1_mm_2D_Stats_energy', 'LoG_sigma_1_mm_2D_Stats_iqr', 'LoG_sigma_1_mm_2D_Stats_kurtosis', 'LoG_sigma_1_mm_2D_Stats_max', 'LoG_sigma_1_mm_2D_Stats_mean', 'LoG_sigma_1_mm_2D_Stats_meanD', 'LoG_sigma_1_mm_2D_Stats_median', 'LoG_sigma_1_mm_2D_Stats_medianD', 'LoG_sigma_1_mm_2D_Stats_min', 'LoG_sigma_1_mm_2D_Stats_p10', 'LoG_sigma_1_mm_2D_Stats_p90', 'LoG_sigma_1_mm_2D_Stats_qcod', 'LoG_sigma_1_mm_2D_Stats_rMeanD', 'LoG_sigma_1_mm_2D_Stats_range', 'LoG_sigma_1_mm_2D_Stats_rms', 'LoG_sigma_1_mm_2D_Stats_skewness', 'LoG_sigma_1_mm_2D_Stats_std', 'LoG_sigma_1_mm_2D_Stats_var', 'LoG_sigma_1_mm_2D_pos_IH_cov', 'LoG_sigma_1_mm_2D_pos_IH_energy', 'LoG_sigma_1_mm_2D_pos_IH_entropy', 'LoG_sigma_1_mm_2D_pos_IH_iqr', 'LoG_sigma_1_mm_2D_pos_IH_kurtosis', 'LoG_sigma_1_mm_2D_pos_IH_max', 'LoG_sigma_1_mm_2D_pos_IH_maxGrad', 'LoG_sigma_1_mm_2D_pos_IH_maxGradI', 'LoG_sigma_1_mm_2D_pos_IH_mean', 'LoG_sigma_1_mm_2D_pos_IH_meanD', 'LoG_sigma_1_mm_2D_pos_IH_median', 'LoG_sigma_1_mm_2D_pos_IH_medianD', 'LoG_sigma_1_mm_2D_pos_IH_min', 'LoG_sigma_1_mm_2D_pos_IH_minGrad', 'LoG_sigma_1_mm_2D_pos_IH_minGradI', 'LoG_sigma_1_mm_2D_pos_IH_mode', 'LoG_sigma_1_mm_2D_pos_IH_p10', 'LoG_sigma_1_mm_2D_pos_IH_p90', 'LoG_sigma_1_mm_2D_pos_IH_qcod', 'LoG_sigma_1_mm_2D_pos_IH_rMeanD', 'LoG_sigma_1_mm_2D_pos_IH_range', 'LoG_sigma_1_mm_2D_pos_IH_rms', 'LoG_sigma_1_mm_2D_pos_IH_skewness', 'LoG_sigma_1_mm_2D_pos_IH_std', 'LoG_sigma_1_mm_2D_pos_IH_uniformity', 'LoG_sigma_1_mm_2D_pos_IH_var', 'LoG_sigma_1_mm_2D_pos_Stats_cov', 'LoG_sigma_1_mm_2D_pos_Stats_energy', 'LoG_sigma_1_mm_2D_pos_Stats_iqr', 'LoG_sigma_1_mm_2D_pos_Stats_kurtosis', 'LoG_sigma_1_mm_2D_pos_Stats_max', 'LoG_sigma_1_mm_2D_pos_Stats_mean', 'LoG_sigma_1_mm_2D_pos_Stats_meanD', 'LoG_sigma_1_mm_2D_pos_Stats_median', 'LoG_sigma_1_mm_2D_pos_Stats_medianD', 'LoG_sigma_1_mm_2D_pos_Stats_min', 'LoG_sigma_1_mm_2D_pos_Stats_p10', 'LoG_sigma_1_mm_2D_pos_Stats_p90', 'LoG_sigma_1_mm_2D_pos_Stats_qcod', 'LoG_sigma_1_mm_2D_pos_Stats_rMeanD', 'LoG_sigma_1_mm_2D_pos_Stats_range', 'LoG_sigma_1_mm_2D_pos_Stats_rms', 'LoG_sigma_1_mm_2D_pos_Stats_skewness', 'LoG_sigma_1_mm_2D_pos_Stats_std', 'LoG_sigma_1_mm_2D_pos_Stats_var', 'NGLDM_DE', 'NGLDM_DN', 'NGLDM_DNN', 'NGLDM_DV', 'NGLDM_GLN', 'NGLDM_GLNN', 'NGLDM_GLV', 'NGLDM_HGE', 'NGLDM_HGLDE', 'NGLDM_HGSDE', 'NGLDM_LDE', 'NGLDM_LGE', 'NGLDM_LGLDE', 'NGLDM_LGSDE', 'NGLDM_SDE', 'NGLDM_SM', 'NGLDM_SM2', 'NGTDM_busyness', 'NGTDM_coarseness', 'NGTDM_complexity', 'NGTDM_contrast', 'NGTDM_strength', 'Stats_cov', 'Stats_energy', 'Stats_iqr', 'Stats_kurtosis', 'Stats_max', 'Stats_mean', 'Stats_meanD', 'Stats_median', 'Stats_medianD', 'Stats_min', 'Stats_p10', 'Stats_p90', 'Stats_qcod', 'Stats_rMeanD', 'Stats_range', 'Stats_rms', 'Stats_skewness', 'Stats_std', 'Stats_var']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Separate the features and the outcomes:"
      ],
      "metadata": {
        "id": "LLgipVwkwlj5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "outcome_train = data_train['Outcome']\n",
        "outcome_test = data_test['Outcome']\n",
        "data_train = data_train[features]\n",
        "data_test = data_test[features]\n",
        "\n",
        "# checking how the features dataframe looks like\n",
        "data_test"
      ],
      "metadata": {
        "id": "eRAc1kzqwoQ-",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 488
        },
        "outputId": "60ed46a3-be5c-4f9b-b2fd-2bc260146ccd"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     Fractal_average  Fractal_lacunarity  Fractal_sd  GLCM_autocorr  \\\n",
              "0           2.109282            0.010447    0.215554    2067.797029   \n",
              "1           2.092324            0.013189    0.240349    3058.445449   \n",
              "2           2.150168            0.010916    0.224781    1635.345051   \n",
              "3           2.063521            0.014229    0.246219    2826.252979   \n",
              "4           2.036999            0.012901    0.231412    3746.556382   \n",
              "..               ...                 ...         ...            ...   \n",
              "336         2.100960            0.012393    0.233880    2221.622474   \n",
              "337         2.127763            0.011310    0.226349    3035.236201   \n",
              "338         2.140162            0.011835    0.232799    3505.057172   \n",
              "339         2.140218            0.011237    0.226827    2604.640174   \n",
              "340         2.139643            0.011849    0.232850    2874.124298   \n",
              "\n",
              "     GLCM_average  GLCM_clusProm  GLCM_clusShade  GLCM_clusTend  \\\n",
              "0       43.798278   1.407646e+06     7159.400380     659.846733   \n",
              "1       54.750456   4.812933e+05    -1520.370488     363.746544   \n",
              "2       39.827543   3.878116e+05      550.338057     361.060041   \n",
              "3       52.173545   1.058322e+06      456.179418     570.204419   \n",
              "4       59.727643   1.595368e+06    -7379.912876     777.424703   \n",
              "..            ...            ...             ...            ...   \n",
              "336     45.785092   1.268940e+06     6745.077260     637.488194   \n",
              "337     53.758531   1.362019e+06    -4222.473364     713.990413   \n",
              "338     58.783398   4.152072e+05     -375.705536     370.632807   \n",
              "339     50.430707   5.529747e+05    -1726.052467     413.059270   \n",
              "340     52.977933   8.940524e+05     -667.955293     556.381266   \n",
              "\n",
              "     GLCM_contrast  GLCM_correl1  ...  Stats_min  Stats_p10  Stats_p90  \\\n",
              "0        61.815929      0.828684  ...       1952     2005.0     2072.0   \n",
              "1       120.416026      0.502632  ...       1941     2003.0     2045.0   \n",
              "2       164.613393      0.373679  ...       1948     1983.0     2024.0   \n",
              "3       153.516593      0.575638  ...       1947     1991.0     2034.0   \n",
              "4        60.774660      0.854920  ...       1968     2038.0     2106.0   \n",
              "..             ...           ...  ...        ...        ...        ...   \n",
              "336     136.100367      0.648064  ...       1994     2025.0     2064.0   \n",
              "337     132.968675      0.685966  ...       1973     2025.0     2084.0   \n",
              "338     172.356402      0.365138  ...       1926     1980.0     2017.0   \n",
              "339     167.524630      0.422830  ...       1959     2001.0     2037.0   \n",
              "340     286.530433      0.320196  ...       1958     1993.0     2031.0   \n",
              "\n",
              "     Stats_qcod  Stats_rMeanD  Stats_range    Stats_rms  Stats_skewness  \\\n",
              "0      0.008593     14.702153          197  2037.443275        0.380419   \n",
              "1      0.005435      9.501333          153  2024.170365       -0.192629   \n",
              "2      0.005241      9.146604          141  2003.600820        0.061260   \n",
              "3      0.005467      9.466079          126  2011.960535        0.022882   \n",
              "4      0.009161     15.768983          179  2073.695786       -0.295346   \n",
              "..          ...           ...          ...          ...             ...   \n",
              "336    0.005138      8.544133          110  2043.806439        0.330860   \n",
              "337    0.007782     13.345616          155  2055.414637       -0.163142   \n",
              "338    0.004754      8.203036          124  1998.365225       -0.042320   \n",
              "339    0.004704      8.076171          121  2019.393275       -0.106028   \n",
              "340    0.004970      8.419451          103  2012.063099       -0.039442   \n",
              "\n",
              "     Stats_std   Stats_var  \n",
              "0    26.447025  699.445119  \n",
              "1    16.779550  281.553291  \n",
              "2    16.197340  262.353813  \n",
              "3    17.024644  289.838490  \n",
              "4    26.140395  683.320268  \n",
              "..         ...         ...  \n",
              "336  15.341589  235.364338  \n",
              "337  22.641186  512.623307  \n",
              "338  14.446493  208.701167  \n",
              "339  14.609857  213.447917  \n",
              "340  14.921726  222.657920  \n",
              "\n",
              "[341 rows x 478 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-bf340ece-e9c6-4a66-bd53-e3fb5dec6b7b\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Fractal_average</th>\n",
              "      <th>Fractal_lacunarity</th>\n",
              "      <th>Fractal_sd</th>\n",
              "      <th>GLCM_autocorr</th>\n",
              "      <th>GLCM_average</th>\n",
              "      <th>GLCM_clusProm</th>\n",
              "      <th>GLCM_clusShade</th>\n",
              "      <th>GLCM_clusTend</th>\n",
              "      <th>GLCM_contrast</th>\n",
              "      <th>GLCM_correl1</th>\n",
              "      <th>...</th>\n",
              "      <th>Stats_min</th>\n",
              "      <th>Stats_p10</th>\n",
              "      <th>Stats_p90</th>\n",
              "      <th>Stats_qcod</th>\n",
              "      <th>Stats_rMeanD</th>\n",
              "      <th>Stats_range</th>\n",
              "      <th>Stats_rms</th>\n",
              "      <th>Stats_skewness</th>\n",
              "      <th>Stats_std</th>\n",
              "      <th>Stats_var</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2.109282</td>\n",
              "      <td>0.010447</td>\n",
              "      <td>0.215554</td>\n",
              "      <td>2067.797029</td>\n",
              "      <td>43.798278</td>\n",
              "      <td>1.407646e+06</td>\n",
              "      <td>7159.400380</td>\n",
              "      <td>659.846733</td>\n",
              "      <td>61.815929</td>\n",
              "      <td>0.828684</td>\n",
              "      <td>...</td>\n",
              "      <td>1952</td>\n",
              "      <td>2005.0</td>\n",
              "      <td>2072.0</td>\n",
              "      <td>0.008593</td>\n",
              "      <td>14.702153</td>\n",
              "      <td>197</td>\n",
              "      <td>2037.443275</td>\n",
              "      <td>0.380419</td>\n",
              "      <td>26.447025</td>\n",
              "      <td>699.445119</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2.092324</td>\n",
              "      <td>0.013189</td>\n",
              "      <td>0.240349</td>\n",
              "      <td>3058.445449</td>\n",
              "      <td>54.750456</td>\n",
              "      <td>4.812933e+05</td>\n",
              "      <td>-1520.370488</td>\n",
              "      <td>363.746544</td>\n",
              "      <td>120.416026</td>\n",
              "      <td>0.502632</td>\n",
              "      <td>...</td>\n",
              "      <td>1941</td>\n",
              "      <td>2003.0</td>\n",
              "      <td>2045.0</td>\n",
              "      <td>0.005435</td>\n",
              "      <td>9.501333</td>\n",
              "      <td>153</td>\n",
              "      <td>2024.170365</td>\n",
              "      <td>-0.192629</td>\n",
              "      <td>16.779550</td>\n",
              "      <td>281.553291</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.150168</td>\n",
              "      <td>0.010916</td>\n",
              "      <td>0.224781</td>\n",
              "      <td>1635.345051</td>\n",
              "      <td>39.827543</td>\n",
              "      <td>3.878116e+05</td>\n",
              "      <td>550.338057</td>\n",
              "      <td>361.060041</td>\n",
              "      <td>164.613393</td>\n",
              "      <td>0.373679</td>\n",
              "      <td>...</td>\n",
              "      <td>1948</td>\n",
              "      <td>1983.0</td>\n",
              "      <td>2024.0</td>\n",
              "      <td>0.005241</td>\n",
              "      <td>9.146604</td>\n",
              "      <td>141</td>\n",
              "      <td>2003.600820</td>\n",
              "      <td>0.061260</td>\n",
              "      <td>16.197340</td>\n",
              "      <td>262.353813</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2.063521</td>\n",
              "      <td>0.014229</td>\n",
              "      <td>0.246219</td>\n",
              "      <td>2826.252979</td>\n",
              "      <td>52.173545</td>\n",
              "      <td>1.058322e+06</td>\n",
              "      <td>456.179418</td>\n",
              "      <td>570.204419</td>\n",
              "      <td>153.516593</td>\n",
              "      <td>0.575638</td>\n",
              "      <td>...</td>\n",
              "      <td>1947</td>\n",
              "      <td>1991.0</td>\n",
              "      <td>2034.0</td>\n",
              "      <td>0.005467</td>\n",
              "      <td>9.466079</td>\n",
              "      <td>126</td>\n",
              "      <td>2011.960535</td>\n",
              "      <td>0.022882</td>\n",
              "      <td>17.024644</td>\n",
              "      <td>289.838490</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2.036999</td>\n",
              "      <td>0.012901</td>\n",
              "      <td>0.231412</td>\n",
              "      <td>3746.556382</td>\n",
              "      <td>59.727643</td>\n",
              "      <td>1.595368e+06</td>\n",
              "      <td>-7379.912876</td>\n",
              "      <td>777.424703</td>\n",
              "      <td>60.774660</td>\n",
              "      <td>0.854920</td>\n",
              "      <td>...</td>\n",
              "      <td>1968</td>\n",
              "      <td>2038.0</td>\n",
              "      <td>2106.0</td>\n",
              "      <td>0.009161</td>\n",
              "      <td>15.768983</td>\n",
              "      <td>179</td>\n",
              "      <td>2073.695786</td>\n",
              "      <td>-0.295346</td>\n",
              "      <td>26.140395</td>\n",
              "      <td>683.320268</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>336</th>\n",
              "      <td>2.100960</td>\n",
              "      <td>0.012393</td>\n",
              "      <td>0.233880</td>\n",
              "      <td>2221.622474</td>\n",
              "      <td>45.785092</td>\n",
              "      <td>1.268940e+06</td>\n",
              "      <td>6745.077260</td>\n",
              "      <td>637.488194</td>\n",
              "      <td>136.100367</td>\n",
              "      <td>0.648064</td>\n",
              "      <td>...</td>\n",
              "      <td>1994</td>\n",
              "      <td>2025.0</td>\n",
              "      <td>2064.0</td>\n",
              "      <td>0.005138</td>\n",
              "      <td>8.544133</td>\n",
              "      <td>110</td>\n",
              "      <td>2043.806439</td>\n",
              "      <td>0.330860</td>\n",
              "      <td>15.341589</td>\n",
              "      <td>235.364338</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>337</th>\n",
              "      <td>2.127763</td>\n",
              "      <td>0.011310</td>\n",
              "      <td>0.226349</td>\n",
              "      <td>3035.236201</td>\n",
              "      <td>53.758531</td>\n",
              "      <td>1.362019e+06</td>\n",
              "      <td>-4222.473364</td>\n",
              "      <td>713.990413</td>\n",
              "      <td>132.968675</td>\n",
              "      <td>0.685966</td>\n",
              "      <td>...</td>\n",
              "      <td>1973</td>\n",
              "      <td>2025.0</td>\n",
              "      <td>2084.0</td>\n",
              "      <td>0.007782</td>\n",
              "      <td>13.345616</td>\n",
              "      <td>155</td>\n",
              "      <td>2055.414637</td>\n",
              "      <td>-0.163142</td>\n",
              "      <td>22.641186</td>\n",
              "      <td>512.623307</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>338</th>\n",
              "      <td>2.140162</td>\n",
              "      <td>0.011835</td>\n",
              "      <td>0.232799</td>\n",
              "      <td>3505.057172</td>\n",
              "      <td>58.783398</td>\n",
              "      <td>4.152072e+05</td>\n",
              "      <td>-375.705536</td>\n",
              "      <td>370.632807</td>\n",
              "      <td>172.356402</td>\n",
              "      <td>0.365138</td>\n",
              "      <td>...</td>\n",
              "      <td>1926</td>\n",
              "      <td>1980.0</td>\n",
              "      <td>2017.0</td>\n",
              "      <td>0.004754</td>\n",
              "      <td>8.203036</td>\n",
              "      <td>124</td>\n",
              "      <td>1998.365225</td>\n",
              "      <td>-0.042320</td>\n",
              "      <td>14.446493</td>\n",
              "      <td>208.701167</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>339</th>\n",
              "      <td>2.140218</td>\n",
              "      <td>0.011237</td>\n",
              "      <td>0.226827</td>\n",
              "      <td>2604.640174</td>\n",
              "      <td>50.430707</td>\n",
              "      <td>5.529747e+05</td>\n",
              "      <td>-1726.052467</td>\n",
              "      <td>413.059270</td>\n",
              "      <td>167.524630</td>\n",
              "      <td>0.422830</td>\n",
              "      <td>...</td>\n",
              "      <td>1959</td>\n",
              "      <td>2001.0</td>\n",
              "      <td>2037.0</td>\n",
              "      <td>0.004704</td>\n",
              "      <td>8.076171</td>\n",
              "      <td>121</td>\n",
              "      <td>2019.393275</td>\n",
              "      <td>-0.106028</td>\n",
              "      <td>14.609857</td>\n",
              "      <td>213.447917</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>340</th>\n",
              "      <td>2.139643</td>\n",
              "      <td>0.011849</td>\n",
              "      <td>0.232850</td>\n",
              "      <td>2874.124298</td>\n",
              "      <td>52.977933</td>\n",
              "      <td>8.940524e+05</td>\n",
              "      <td>-667.955293</td>\n",
              "      <td>556.381266</td>\n",
              "      <td>286.530433</td>\n",
              "      <td>0.320196</td>\n",
              "      <td>...</td>\n",
              "      <td>1958</td>\n",
              "      <td>1993.0</td>\n",
              "      <td>2031.0</td>\n",
              "      <td>0.004970</td>\n",
              "      <td>8.419451</td>\n",
              "      <td>103</td>\n",
              "      <td>2012.063099</td>\n",
              "      <td>-0.039442</td>\n",
              "      <td>14.921726</td>\n",
              "      <td>222.657920</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>341 rows × 478 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-bf340ece-e9c6-4a66-bd53-e3fb5dec6b7b')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-bf340ece-e9c6-4a66-bd53-e3fb5dec6b7b button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-bf340ece-e9c6-4a66-bd53-e3fb5dec6b7b');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Exploratory data analysis\n",
        "\n",
        "Before building the models, it is always useful to perform the exploratory data analysis to have a look at the data, understand the distribution of the features, and notice some data errors. To perform these steps, we will be using precision-medicine-toolbox (https://github.com/primakov/precision-medicine-toolbox), an open-source Python package, developed in the D-Lab, Maastricht University. If something does not work, look at the error and at the already produced reatures reports for troubleshooting. Don't hesitate to look into the toolbox documentation to fix the problem: https://precision-medicine-toolbox.readthedocs.io/en/latest/?badge=latest. You can search for available functions to deal with NaN values + the ways to read the data, excluding some features.  \n",
        "  \n",
        "Remember that for now, everything happens in the training data. Time to discuss why this is so important!  "
      ],
      "metadata": {
        "id": "aBm8KYFpyZTW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# let's list the parameters of our features dataset\n",
        "parameters = {\n",
        "    'feature_path': \"/content/drive/MyDrive/AI4I_Radiomics_Experts_Data/data_features/WS2022_Experts_training.xlsx\", # path to csv/xls file with features\n",
        "    'outcome_path': \"/content/drive/MyDrive/AI4I_Radiomics_Experts_Data/data_features/WS2022_Experts_training.xlsx\", #path to csv/xls file with outcome (in our case - the same file)\n",
        "    'patient_column': 'LesionID', # name of column with patient ID\n",
        "    'patient_in_outcome_column': 'LesionID', # name of column with patient ID in clinical data file\n",
        "    'outcome_column': 'Outcome', # name of outcome column\n",
        "    'feature_column': features,\n",
        "    'feature_column_to_drop': ['LoG_sigma_0_4_mm_2D_IH_max', \n",
        "                               'LoG_sigma_0_7_mm_2D_IH_max', \n",
        "                               'LoG_sigma_0_4_mm_2D_pos_IH_range', \n",
        "                               'LoG_sigma_0_4_mm_2D_pos_IH_min', \n",
        "                               'LoG_sigma_0_1_mm_2D_pos_IH_max', \n",
        "                               'LoG_sigma_1_mm_2D_IH_max', \n",
        "                               'LoG_sigma_0_7_mm_2D_IH_range', \n",
        "                               'LoG_sigma_1_mm_2D_pos_IH_max', \n",
        "                               'LoG_sigma_0_4_mm_2D_IH_min', \n",
        "                               'LoG_sigma_0_1_mm_2D_pos_IH_range', \n",
        "                               'LoG_sigma_0_1_mm_2D_IH_range',\n",
        "                               'LoG_sigma_1_mm_2D_IH_min',\n",
        "                               'LoG_sigma_0_4_mm_2D_IH_range',\n",
        "                               'LoG_sigma_1_mm_2D_pos_IH_range',\n",
        "                               'voxel_distance', \n",
        "                               'LoG_sigma_0_1_mm_2D_IH_max',\n",
        "                               'LoG_sigma_0_7_mm_2D_pos_IH_max', \n",
        "                               'LoG_sigma_1_mm_2D_IH_range', \n",
        "                               'LoG_sigma_0_4_mm_2D_pos_IH_max',                                \n",
        "                               'LoG_sigma_0_7_mm_2D_pos_IH_min', \n",
        "                               'LoG_sigma_1_mm_2D_pos_IH_min', \n",
        "                               'LoG_sigma_0_1_mm_2D_pos_IH_min', \n",
        "                               'LoG_sigma_0_7_mm_2D_IH_min', \n",
        "                               'LoG_sigma_0_7_mm_2D_pos_IH_range', \n",
        "                               'LoG_sigma_0_1_mm_2D_IH_min']\n",
        "}\n",
        "\n",
        "# initializing the dataset, containing features\n",
        "fs = AnalysisBox(**parameters)"
      ],
      "metadata": {
        "id": "paW5Ay4lyhi4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5103aff0-b203-4cb1-a0da-5fcd44889eb5"
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of observations: 805\n",
            "Class labels: ['0' '1']\n",
            "Classes balance: [0.484472049689441, 0.515527950310559]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print ('Initial amount of features: ', len(fs._feature_column))\n",
        "fs.handle_nan(axis=1)\n",
        "print ('Amount of features after exclusion of NaN values: ', len(fs._feature_column))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x_pzrlmcIIIY",
        "outputId": "56b37917-376d-4289-a64d-4f349234b644"
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Initial amount of features:  443\n",
            "Number of observations: 805\n",
            "Class labels: ['0' '1']\n",
            "Classes balance: [0.484472049689441, 0.515527950310559]\n",
            "Amount of features after exclusion of NaN values:  443\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Plotting feature distributions to have a visual understanding about feature distributions in classes."
      ],
      "metadata": {
        "id": "U_EgaYg4ykq-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fs.plot_distribution(fs._feature_column)"
      ],
      "metadata": {
        "id": "G37us_8FymlQ"
      },
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "What do you think about the report?\n",
        "\n",
        "Plotting the correlation matrix to have an idea about how many features are inter-correlated. Why is it important to know about the mutual feature correlation?"
      ],
      "metadata": {
        "id": "DvfwDvlnyplK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fs.plot_correlation_matrix(fs._feature_column)"
      ],
      "metadata": {
        "id": "igN2PiRyyrNL"
      },
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Performing Mann-Whitney U-test (with False Discovery Rate correction) to see if the feature distributions in classes are statistically different."
      ],
      "metadata": {
        "id": "RzLIHLwNytEG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fs.plot_MW_p(fs._feature_column)"
      ],
      "metadata": {
        "id": "wbmFZWfTywa5"
      },
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Plotting univariate ROC curves and calculating AUC scores to have some estimation of the predictive power of every separate feature. Can we just build a classifer based on the best feature?"
      ],
      "metadata": {
        "id": "idLMYzbVyybP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fs.plot_univariate_roc(fs._feature_column, auc_threshold=0.70)"
      ],
      "metadata": {
        "id": "QRfw0MIay2XS"
      },
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Writing the basic statistics for every feature into .csv file and having a look at the resulting dataframe."
      ],
      "metadata": {
        "id": "ce691Exmy8oy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fs.calculate_basic_stats()\n",
        "\n",
        "print('Basic statistics for each feature')\n",
        "pd.read_excel('/content/drive/MyDrive/AI4I_Radiomics_Experts_Data/data_features/WS2022_Experts_training_basic_stats.xlsx')"
      ],
      "metadata": {
        "id": "rEjH5fW9zDaV",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 505
        },
        "outputId": "78afed1b-28e2-4807-8014-7fea809ccdf2"
      },
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Basic statistics for each feature\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                 Unnamed: 0  NaN          Mean           Std  \\\n",
              "0         LoG_sigma_0_1_mm_2D_pos_Stats_p10    0  1.253476e+00  1.813572e-01   \n",
              "1         LoG_sigma_0_1_mm_2D_Stats_medianD    0  6.486141e+00  8.536889e-01   \n",
              "2    LoG_sigma_0_4_mm_2D_pos_Stats_kurtosis    0  6.914483e+00  3.654692e+01   \n",
              "3            LoG_sigma_0_7_mm_2D_pos_IH_p90    0  6.683640e+01  1.261423e+01   \n",
              "4            LoG_sigma_1_mm_2D_Stats_energy    0  2.016365e+07  3.470636e+07   \n",
              "..                                      ...  ...           ...           ...   \n",
              "438           LoG_sigma_0_1_mm_2D_Stats_p90    0  1.282868e+01  1.745996e+00   \n",
              "439      LoG_sigma_0_7_mm_2D_pos_IH_maxGrad    0  2.788776e+02  3.422635e+03   \n",
              "440       LoG_sigma_0_7_mm_2D_pos_Stats_max    0  5.081649e+01  6.392501e+01   \n",
              "441              LoG_sigma_0_7_mm_2D_IH_p10    0  2.784472e+01  8.101060e+00   \n",
              "442        LoG_sigma_0_7_mm_2D_Stats_median    0  2.189304e+01  2.055229e+00   \n",
              "\n",
              "               Min           Max  p_MW_corrected  univar_auc  \n",
              "0         0.850453  2.671720e+00    9.386097e-07    0.621909  \n",
              "1         4.149067  1.158367e+01    2.084310e-07    0.624174  \n",
              "2         2.085928  6.058021e+02    1.000000e+00    0.520232  \n",
              "3         4.000000  9.500000e+01    1.000000e+00    0.502539  \n",
              "4    234453.608400  4.507153e+08    6.671890e-06    0.622002  \n",
              "..             ...           ...             ...         ...  \n",
              "438       8.544767  2.892491e+01    8.822910e-09    0.633324  \n",
              "439       4.000000  7.022950e+04    1.000000e+00    0.507130  \n",
              "440      25.419774  1.021395e+03    3.028440e-18    0.683692  \n",
              "441       7.000000  7.300000e+01    1.000000e+00    0.533763  \n",
              "442       9.453333  3.602479e+01    8.529017e-55    0.824594  \n",
              "\n",
              "[443 rows x 8 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-315c2e2e-7c7e-48a9-a1ce-4e245677dc7d\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>NaN</th>\n",
              "      <th>Mean</th>\n",
              "      <th>Std</th>\n",
              "      <th>Min</th>\n",
              "      <th>Max</th>\n",
              "      <th>p_MW_corrected</th>\n",
              "      <th>univar_auc</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>LoG_sigma_0_1_mm_2D_pos_Stats_p10</td>\n",
              "      <td>0</td>\n",
              "      <td>1.253476e+00</td>\n",
              "      <td>1.813572e-01</td>\n",
              "      <td>0.850453</td>\n",
              "      <td>2.671720e+00</td>\n",
              "      <td>9.386097e-07</td>\n",
              "      <td>0.621909</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>LoG_sigma_0_1_mm_2D_Stats_medianD</td>\n",
              "      <td>0</td>\n",
              "      <td>6.486141e+00</td>\n",
              "      <td>8.536889e-01</td>\n",
              "      <td>4.149067</td>\n",
              "      <td>1.158367e+01</td>\n",
              "      <td>2.084310e-07</td>\n",
              "      <td>0.624174</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>LoG_sigma_0_4_mm_2D_pos_Stats_kurtosis</td>\n",
              "      <td>0</td>\n",
              "      <td>6.914483e+00</td>\n",
              "      <td>3.654692e+01</td>\n",
              "      <td>2.085928</td>\n",
              "      <td>6.058021e+02</td>\n",
              "      <td>1.000000e+00</td>\n",
              "      <td>0.520232</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>LoG_sigma_0_7_mm_2D_pos_IH_p90</td>\n",
              "      <td>0</td>\n",
              "      <td>6.683640e+01</td>\n",
              "      <td>1.261423e+01</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>9.500000e+01</td>\n",
              "      <td>1.000000e+00</td>\n",
              "      <td>0.502539</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>LoG_sigma_1_mm_2D_Stats_energy</td>\n",
              "      <td>0</td>\n",
              "      <td>2.016365e+07</td>\n",
              "      <td>3.470636e+07</td>\n",
              "      <td>234453.608400</td>\n",
              "      <td>4.507153e+08</td>\n",
              "      <td>6.671890e-06</td>\n",
              "      <td>0.622002</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>438</th>\n",
              "      <td>LoG_sigma_0_1_mm_2D_Stats_p90</td>\n",
              "      <td>0</td>\n",
              "      <td>1.282868e+01</td>\n",
              "      <td>1.745996e+00</td>\n",
              "      <td>8.544767</td>\n",
              "      <td>2.892491e+01</td>\n",
              "      <td>8.822910e-09</td>\n",
              "      <td>0.633324</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>439</th>\n",
              "      <td>LoG_sigma_0_7_mm_2D_pos_IH_maxGrad</td>\n",
              "      <td>0</td>\n",
              "      <td>2.788776e+02</td>\n",
              "      <td>3.422635e+03</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>7.022950e+04</td>\n",
              "      <td>1.000000e+00</td>\n",
              "      <td>0.507130</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>440</th>\n",
              "      <td>LoG_sigma_0_7_mm_2D_pos_Stats_max</td>\n",
              "      <td>0</td>\n",
              "      <td>5.081649e+01</td>\n",
              "      <td>6.392501e+01</td>\n",
              "      <td>25.419774</td>\n",
              "      <td>1.021395e+03</td>\n",
              "      <td>3.028440e-18</td>\n",
              "      <td>0.683692</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>441</th>\n",
              "      <td>LoG_sigma_0_7_mm_2D_IH_p10</td>\n",
              "      <td>0</td>\n",
              "      <td>2.784472e+01</td>\n",
              "      <td>8.101060e+00</td>\n",
              "      <td>7.000000</td>\n",
              "      <td>7.300000e+01</td>\n",
              "      <td>1.000000e+00</td>\n",
              "      <td>0.533763</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>442</th>\n",
              "      <td>LoG_sigma_0_7_mm_2D_Stats_median</td>\n",
              "      <td>0</td>\n",
              "      <td>2.189304e+01</td>\n",
              "      <td>2.055229e+00</td>\n",
              "      <td>9.453333</td>\n",
              "      <td>3.602479e+01</td>\n",
              "      <td>8.529017e-55</td>\n",
              "      <td>0.824594</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>443 rows × 8 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-315c2e2e-7c7e-48a9-a1ce-4e245677dc7d')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-315c2e2e-7c7e-48a9-a1ce-4e245677dc7d button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-315c2e2e-7c7e-48a9-a1ce-4e245677dc7d');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Features reduction and selection\n",
        "\n",
        "Do you think we need to change somehow features list, based on the exploratory analysis?"
      ],
      "metadata": {
        "id": "mjxiXEuFzENo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "features_non_nan = data_train[features].dropna(axis=1).columns\n",
        "print ('Number of features without missing values: ', len(features_non_nan))\n",
        "\n",
        "features_non_zero_var = data_train[features_non_nan].loc[:,data_train[features_non_nan].std() > 0.3].columns\n",
        "print ('Number of features with non-zero variance: ', len(features_non_zero_var))"
      ],
      "metadata": {
        "id": "8kX0JElHzXn3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a0951a35-2343-4df1-eab4-43b2a3689cee"
      },
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of features without missing values:  467\n",
            "Number of features with non-zero variance:  359\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Removing highly correlated features is a controversial step aimed to reduce the dimensionality of the feature space. Highly correlated features needlessly inflate the dimensionality of feature space. The idea is that highly correlated features can be grouped together and represented by one representative feature.  For features pairs with a high Spearman correlation (r > 0.9) the feature with the highest mean correlation with all remaining features is removed.\n",
        "The opponents to this step argue that just because features are correlated doesn't mean that they don't individually  increase the model's performance. \n",
        "Time to discuss the pro's and con's of this step in more detail!\n",
        "\n",
        "<b>If you have time at the end of today's workshop, the cutoff is certaily a variable that can be played with.</b>"
      ],
      "metadata": {
        "id": "U5IUCHeFzak0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def selectNonIntercorrelated(df_in, ftrs, corr_th):\n",
        "    \n",
        "    # selection of the features, which are not 'highly intercorrelated' (correlation is defined by Spearman coefficient);\n",
        "    # pairwise correlation between all the features is calculated, \n",
        "    # from each pair of features, considered as intercorrelated, \n",
        "    # feature with maximum sum of all the pairwise Spearman correlation coefficients is a 'candidate to be dropped'\n",
        "    # for stability of the selected features, bootstrapping approach is used: \n",
        "    # in each bootstrap split, the random subsample, stratified in relation to outcome, \n",
        "    # is formed, based on original observations from input dataset;\n",
        "    # in each bootstrap split, 'candidates to be dropped' are detected;\n",
        "    # for each input feature, its frequency to appear as 'candidate to be dropped' is calculated,\n",
        "    # features, appeared in 50 % of splits as 'candidate to be dropped', are excluded from feature set\n",
        "    \n",
        "    # input:\n",
        "    # df_in - input dataframe, containing feature values (dataframe, columns = features, rows = observations),\n",
        "    # ftrs - list of dataframe features, used in analysis (list of feature names - string variables),\n",
        "    # corr_th - threshold for Spearman correlation coefficient, defining each pair of features as intercorrelated (float)\n",
        "    \n",
        "    # output:\n",
        "    # non_intercorrelated_features - list of names of features, which did not appear as inter-correlated\n",
        "    \n",
        "    corr_matrix = df_in.corr(method='spearman').abs()\n",
        "    mean_absolute_corr = corr_matrix.mean()\n",
        "    intercorrelated_features_set = []\n",
        "    upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))\n",
        "    high_corrs = upper.where(upper > corr_th).dropna(how='all', axis=1).dropna(how='all', axis=0)\n",
        "\n",
        "    for feature in high_corrs.columns:\n",
        "        mean_absolute_main = mean_absolute_corr[feature]\n",
        "        correlated_with_feature = high_corrs[feature].index[pd.notnull(high_corrs[feature])]\n",
        "        for each_correlated_feature in correlated_with_feature:\n",
        "            mean_absolute = mean_absolute_corr[each_correlated_feature]\n",
        "            if mean_absolute_main > mean_absolute:\n",
        "                if feature not in intercorrelated_features_set:\n",
        "                    intercorrelated_features_set.append(feature)\n",
        "            else:\n",
        "                if each_correlated_feature not in intercorrelated_features_set:\n",
        "                    intercorrelated_features_set.append(each_correlated_feature)\n",
        "\n",
        "    non_intercorrelated_features_set = [e for e in ftrs if e not in intercorrelated_features_set] \n",
        "    \n",
        "    print ('Non intercorrelated features: ', non_intercorrelated_features_set)\n",
        "    \n",
        "    return non_intercorrelated_features_set"
      ],
      "metadata": {
        "id": "n6b4jVyizlb4"
      },
      "execution_count": 41,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will use a threshold of 0.9 for absolute value of Spearmann's correlation."
      ],
      "metadata": {
        "id": "4knIjgI6zpgf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "features_non_intercorrelated = selectNonIntercorrelated(data_train, features_non_zero_var, 0.9)\n",
        "print ('Number of non-intercorrelated features: ', len(features_non_intercorrelated))"
      ],
      "metadata": {
        "id": "sRitM94Czsek",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "439b3b36-f0c3-446c-e491-2f06a25d9214"
      },
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Non intercorrelated features:  ['GLCM_entrop2', 'GLDZM_DZE', 'GLDZM_HILDE', 'GLDZM_HISDE', 'GLDZM_LILDE', 'GLRLM_LRHGE', 'GLSZM_HILAE', 'LoG_sigma_0_1_mm_2D_IH_maxGradI', 'LoG_sigma_0_1_mm_2D_IH_medianD', 'LoG_sigma_0_1_mm_2D_IH_minGradI', 'LoG_sigma_0_1_mm_2D_IH_mode', 'LoG_sigma_0_1_mm_2D_IH_p10', 'LoG_sigma_0_1_mm_2D_IH_p90', 'LoG_sigma_0_1_mm_2D_IH_rms', 'LoG_sigma_0_1_mm_2D_IH_skewness', 'LoG_sigma_0_1_mm_2D_Stats_cov', 'LoG_sigma_0_1_mm_2D_Stats_kurtosis', 'LoG_sigma_0_1_mm_2D_Stats_mean', 'LoG_sigma_0_1_mm_2D_Stats_min', 'LoG_sigma_0_1_mm_2D_Stats_qcod', 'LoG_sigma_0_1_mm_2D_pos_IH_maxGrad', 'LoG_sigma_0_1_mm_2D_pos_IH_maxGradI', 'LoG_sigma_0_1_mm_2D_pos_IH_minGrad', 'LoG_sigma_0_1_mm_2D_pos_IH_minGradI', 'LoG_sigma_0_1_mm_2D_pos_IH_mode', 'LoG_sigma_0_1_mm_2D_pos_IH_p10', 'LoG_sigma_0_1_mm_2D_pos_Stats_max', 'LoG_sigma_0_1_mm_2D_pos_Stats_skewness', 'LoG_sigma_0_4_mm_2D_IH_maxGradI', 'LoG_sigma_0_4_mm_2D_IH_mean', 'LoG_sigma_0_4_mm_2D_IH_medianD', 'LoG_sigma_0_4_mm_2D_IH_minGradI', 'LoG_sigma_0_4_mm_2D_IH_mode', 'LoG_sigma_0_4_mm_2D_IH_p10', 'LoG_sigma_0_4_mm_2D_IH_p90', 'LoG_sigma_0_4_mm_2D_Stats_median', 'LoG_sigma_0_4_mm_2D_Stats_skewness', 'LoG_sigma_0_4_mm_2D_pos_IH_energy', 'LoG_sigma_0_4_mm_2D_pos_IH_maxGradI', 'LoG_sigma_0_4_mm_2D_pos_IH_medianD', 'LoG_sigma_0_4_mm_2D_pos_IH_minGradI', 'LoG_sigma_0_4_mm_2D_pos_IH_mode', 'LoG_sigma_0_4_mm_2D_pos_IH_p10', 'LoG_sigma_0_4_mm_2D_pos_IH_skewness', 'LoG_sigma_0_4_mm_2D_pos_Stats_kurtosis', 'LoG_sigma_0_4_mm_2D_pos_Stats_max', 'LoG_sigma_0_4_mm_2D_pos_Stats_min', 'LoG_sigma_0_4_mm_2D_pos_Stats_p10', 'LoG_sigma_0_7_mm_2D_IH_maxGradI', 'LoG_sigma_0_7_mm_2D_IH_median', 'LoG_sigma_0_7_mm_2D_IH_minGradI', 'LoG_sigma_0_7_mm_2D_IH_mode', 'LoG_sigma_0_7_mm_2D_IH_p10', 'LoG_sigma_0_7_mm_2D_IH_p90', 'LoG_sigma_0_7_mm_2D_IH_skewness', 'LoG_sigma_0_7_mm_2D_pos_IH_maxGradI', 'LoG_sigma_0_7_mm_2D_pos_IH_median', 'LoG_sigma_0_7_mm_2D_pos_IH_minGradI', 'LoG_sigma_0_7_mm_2D_pos_IH_mode', 'LoG_sigma_0_7_mm_2D_pos_IH_p10', 'LoG_sigma_0_7_mm_2D_pos_IH_skewness', 'LoG_sigma_0_7_mm_2D_pos_Stats_kurtosis', 'LoG_sigma_1_mm_2D_IH_maxGradI', 'LoG_sigma_1_mm_2D_IH_median', 'LoG_sigma_1_mm_2D_IH_minGradI', 'LoG_sigma_1_mm_2D_IH_mode', 'LoG_sigma_1_mm_2D_IH_p90', 'LoG_sigma_1_mm_2D_IH_skewness', 'LoG_sigma_1_mm_2D_Stats_iqr', 'LoG_sigma_1_mm_2D_Stats_min', 'LoG_sigma_1_mm_2D_pos_IH_minGrad', 'LoG_sigma_1_mm_2D_pos_IH_p10', 'LoG_sigma_1_mm_2D_pos_Stats_kurtosis', 'LoG_sigma_1_mm_2D_pos_Stats_p10', 'NGTDM_complexity', 'Stats_kurtosis', 'Stats_max', 'Stats_medianD', 'Stats_min', 'Stats_p10', 'Stats_range']\n",
            "Number of non-intercorrelated features:  81\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:25: DeprecationWarning:\n",
            "\n",
            "`np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.\n",
            "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will perform feature selection using Recursive Feature Elimination (RFE). In this step, feature selection is based on the outcome, so simple models are built and those features that contribute the least to the model are removed recursively. Here you can edit the parameters.\n",
        "\n",
        "<b>You might have heard of variable normalization. Why are we not normalizing the variables (e.g. Z-score normalization)?</b>\n",
        "Hint: e.g. https://stackoverflow.com/questions/57339104/is-normalization-necessary-for-randomforest\n",
        "\n",
        "<b>How many features do we need to select with RFE?\n",
        "Time to discuss pros and cons of using many and a little features.</b>  \n",
        "  \n",
        "There are some rules of thumb on how many features we need in the end:  \n",
        "* $int(\\frac{N_{samples}}{10})$ (Abu-Mostafa, Y. S., Magdon-Ismail, M., & Lin, H. T. (2012). Learning from data (Vol. 4, p. 4). New York: AMLBook.)  \n",
        "* $\\sqrt{N_{samples}}$ (Hua, J., Xiong, Z., Lowey, J., Suh, E., & Dougherty, E. R. (2005). Optimal number of features as a function of sample size for various classification rules. Bioinformatics, 21(8), 1509-1515.)"
      ],
      "metadata": {
        "id": "lQ-vBI2QzuxC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print ('Nmber of samples in training dataset: ', np.sum(outcome_train))\n",
        "print ('Number of features to select according to Abu-Mostafa: ', int(np.sum(outcome_train)/10))\n",
        "print ('Number of features to select according to Hua: ', int(np.sum(outcome_train)**0.5))"
      ],
      "metadata": {
        "id": "8nUlfh620JIt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1f3d7691-a481-4f73-efe3-8e06c16a088a"
      },
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Nmber of samples in training dataset:  415\n",
            "Number of features to select according to Abu-Mostafa:  41\n",
            "Number of features to select according to Hua:  20\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's go for the lower number of features since our dataset is not large. Below we implement Recursive Feature Elimination, RFE (https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html), based on Random Forest Classifier, RFC (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html).\n",
        "\n",
        "Which number of features will you chose?"
      ],
      "metadata": {
        "id": "ggNVDDIS0j3K"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "estimator = RandomForestClassifier(n_estimators=100, random_state=np.random.seed(0))\n",
        "selector = RFE(estimator, n_features_to_select=20, step=1)\n",
        "selector = selector.fit(data_train[features_non_intercorrelated], outcome_train)"
      ],
      "metadata": {
        "id": "san1UFHv0k8b"
      },
      "execution_count": 46,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "And then select the features 'supported' by the algorithm."
      ],
      "metadata": {
        "id": "8eQ0lfWX0pdA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "support = selector.get_support()\n",
        "selected_features_set = data_train[features_non_intercorrelated].loc[:, support].columns.tolist()\n",
        "\n",
        "print (selected_features_set)"
      ],
      "metadata": {
        "id": "v_XsyJFs0vKe",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "956a33a3-c88c-41ff-bbf3-6815696cb298"
      },
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['GLDZM_HILDE', 'GLDZM_HISDE', 'GLDZM_LILDE', 'GLSZM_HILAE', 'LoG_sigma_0_1_mm_2D_IH_rms', 'LoG_sigma_0_1_mm_2D_Stats_cov', 'LoG_sigma_0_1_mm_2D_Stats_kurtosis', 'LoG_sigma_0_1_mm_2D_Stats_mean', 'LoG_sigma_0_1_mm_2D_Stats_min', 'LoG_sigma_0_4_mm_2D_Stats_median', 'LoG_sigma_0_4_mm_2D_pos_IH_skewness', 'LoG_sigma_0_4_mm_2D_pos_Stats_max', 'LoG_sigma_1_mm_2D_Stats_iqr', 'LoG_sigma_1_mm_2D_Stats_min', 'LoG_sigma_1_mm_2D_pos_Stats_p10', 'NGTDM_complexity', 'Stats_max', 'Stats_medianD', 'Stats_p10', 'Stats_range']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<b>How many features give the best performance?</b>\n",
        "\n",
        "Time to discuss why we don't just use 100 features!  \n",
        "  \n",
        "The other option to select a number of features is data-driven and is based on ranking the features and then building classifier adding +1 feature at every step and checking its performance. We can evaluate the performance with some score and plot the dependence of this score from the number of the selected features."
      ],
      "metadata": {
        "id": "9sZe7WXF0yuI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# ranking the features based on RFE results\n",
        "\n",
        "features_ranks = pd.DataFrame({'Features': features_non_intercorrelated, 'Ranks': selector.ranking_})\n",
        "features_ranks.sort_values(by='Ranks', inplace = True)\n",
        "\n",
        "# taking one best feature first, building the RFC, estimating the performance;\n",
        "# adding +1 next feature, repeating the steps to estimate the performance\n",
        "\n",
        "ftrs_number_tuning = []\n",
        "acc_tuning = []\n",
        "\n",
        "for i in range (1, len(features_ranks)):\n",
        "    \n",
        "    ftrs_number_tuning.append(i)\n",
        "    estimator_tuning = RandomForestClassifier(n_estimators=100, random_state=np.random.seed(0))\n",
        "    estimator_tuning.fit(data_train[features_ranks['Features'][:i]], outcome_train)\n",
        "    outcome_pred_tuning = estimator_tuning.predict(data_test[features_ranks['Features'][:i]])\n",
        "    acc_tuning.append(balanced_accuracy_score(outcome_test, outcome_pred_tuning))\n",
        "    \n",
        "# plotting the results\n",
        "plt.plot(ftrs_number_tuning, acc_tuning)\n",
        "plt.xlabel('Number of features')\n",
        "plt.ylabel('RFC performance')\n",
        "plt.title('RFC performance depending on number of selected features')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "q5Z6lqOJ2Ii8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "outputId": "6ad0668a-5837-40bd-c408-0701d68f8560"
      },
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAEWCAYAAACqitpwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeXhU5dn48e+djSxASCDsJGxhR3ZQEXeFqi3W19atfbV1aetSq/1Za2utVdvavq120S5qrdaNulVpRdG6gQtCWJSdhEAgIZCEkEAWst6/P86ZMEkmyQlmMpPk/lzXXMmcbe6ZOXPuc57nOc8jqooxxhjTkSJCHYAxxpjux5KLMcaYDmfJxRhjTIez5GKMMabDWXIxxhjT4Sy5GGOM6XCWXDqQiMSJyL9FpFREXgh1PKEgIneLyNOhjqMpEdktIme7//9IRB4LdUzhQkROF5HcEL7+l0Vkr4iUiciMDtpmSN+TXxxPiMh9rcy/T0SKRGR/Z8bVGcIuubgHgUp3R9vvfjm9/eY/ISLV7nzf4xK/+ZeLSIY7PV9EXheRUzop/IuBQUB/Vf1KJ72maSdV/YWqXhPqOEyD3wA3qmpvVV0f6mD8iYiKyNggbTsV+D4wSVUHf85thUUy9Rd2ycX1RVXtDUwHZgB3NJn/a3dH9D3+CSAitwK/A36Bc5BPBf4ELA52wCISCaQBO1S19jjWj+r4qIzpXMe5H6cBmzs6li4gFTioqgWhDiQoxx9VDasHsBs42+/5r4HX/J4/AdwXYL1EoAz4Sjte6wngL8BbwBHgfSDNb/4Ed14xsB34apN1/wwsA8qBD4FqoMaN42qc5H0nkAMUAP8AEt31RwLqLrcHWAFc5W7nQaAEyAZOdqfvdbdxpV8M5wPrgcPu/Lv95vm2f6W7/SLgx37zI4EfATvd974WGNHW+w7wGY5yP7cj7joPAU/7zT8R+Mh9P58Cp/vNew/4JbDafQ+vAsntWPde9/M6ArwJDPCb/3X3cz8I/Bi//Qq42xejh88pDngSOARsBX4A5LbyeZwMrAFK3b8ne425yXZOB3JxzmwLgHzgG022dY3f86uAD/yeK3A9kOm+1r3AGPfzPAw8D8Q0ea0fue9/N3CF37Z64Vxd7AEO4Pxm4pqsezuwH3gqwHsJ+Dtwt1vmxloO7AywruD8HgrcuDcCU7zG5bedocBLQCGwC/huW78FnN+kL7Yy4BJ3+QuADTj75UfACX7bmgGsc7fzT2AJgY9XZwOVQL277Sc87PPfwNkHj+AcG77lTk9osq0y9/0+4f/aAT6T3e739hlQBUS18fpXua97xP0Mr2j6vhq9x8+TCILxoPFBYLi7M/3eb36jD8xv+iKgFohqx2s94X5Qp7o76u9xf6DuF7bX/UKj3J2mCOcS1rduKTAf58cTi99By13mm0AWMBroDbyM++Pj2EHtH+5rxblfXq37mpHAfTg/nIfd+M514+3tt7NMdV//BJwf2IVNtv+ou+1p7g400Z1/m/vZjsf5AU8D+rf1vgN8hh8DD7jxnerG5ztwD8M5uJ/nxniO+zzF7wCZB0xxX/eldq67Exjnvr/3gPvdeZNwfmC+7/UB93NtLbm09Dndj5M8k3D2x89oIbkAyThJ6OvuZ3eZ+7x/WzEH2Nbpbsz3ANHu51ABJPltq63k8irQF5jsvqe3cfbFRGAL7omK32v5vsfTcA6o4935DwJL3ffXB/g38Msm6/7KXTcuwHtp8XfgF+vYFj6HhTgH+344++lEYIjHuHLd/yPcbdwFxLhxZAMLW/stBIoN5/dQAMzD+Y1eiXPM6uVuOwe4xf3OLsY52Wx2vGrhYN/WPn8+zgmCuN9RBTAz0LYCHSsDvN5unCQ5Amd/bPH1cX6fh/32iSHA5FaPr52dPDwc8HfjHBiOuF/s20C/Jh/YUZzMWgIUudOvAPa387WeAJb4Pe8N1Lkf9iXAyibL/xX4qd+6/2gy/24aJ5e3gev9no93d7Yojh3URjc5QGT6PZ/qLjPIb9pBYHoL7+d3wIPu/77tD/ebvxq41P1/O7A4wDZafd9NpqfiHFgS/KY9y7ED9+00OZMFlnPsoPYefgdXnKRQjfOj9bLunX7zrgfecP+/q8n3muBut7Xk0tLn1HAQcp9fQ8vJ5evA6ibTPgauaivmANs6HedsNMpvWgFwot+22kou8/2erwVu93v+W+B3fq/V9Ht8HvgJzoGsHBjjN+8kYJffutVAbCu/sxZ/B36xtpRczgR24JxRR/hN9xKXL7nMA/Y02e4dwN9b+y0Eig2ntOLeJstsxznYnwrsA8Rv3kd4Ty6t7vMB1n8FuDnQttxpT9B2cvmml9fH+Q2VAP9DgBOIQI9wrXO5UFX74HwYE4ABTeb/RlX7uQ/fvIPAgOMoO9zr+0dVy3CKgobilAPPE5ES3wMngQ0OtG4LhuKcyfjk4CSWQa1s44Df/5VuXE2n9QYQkXki8q6IFIpIKfBtmn9W/q1QKnzr4iTQnQFi9vK+/d/fIVUtb/Ie/bf1lSbbOgXnrMdnb5N1o9334GXdlt7bUBp/r+U4+0drPG2L1r/zpt837vNhHl4nkIPauP6ureWbarrfBNyPXIG+x6E4Z63xwFq/7+ENd7pPoaoebSUOL7+DgFT1HZyi1oeBAhF5RET6eozLJw0Y2mRf+pHf67f0WwgkDfh+k22NcN/jUCBP3aOy33v1qtV9XkS+ICKrRKTYnXcezX/v7eW/P7f4+u6+cQnOMSZfRF4TkQmtbThckwsAqvo+Tvb9jYfFP8a59L+wnS8zwveP2yotGefsYy/wvl8S66dO44Hv+IfYxrb34XxhPr4zff8feVvbaM2zOMUCI1Q1EafMWTyuuxfnEjvQ9Lbet08+kCQiCX7TUpts66km20pQ1fv9lhnRZN0anGI4L+u2JJ/G32s8TpHf8cjHKQ4LFG9TTb9vcN5T3nG+dmvKcQ6uPp+rtRGBv8d9ON9FJU4RiO97SFSnwY1PR/wOWqSqf1DVWThXtuNwirG8xOWzF+eKxn9f6qOq5/nND/RbCGQv8PMm24pX1edw9pVhIuL/G0wNvJkWtx1wnxeRXjjFxr/BKcnoh1Pf63utQN+Bl33Ef71Wf3OqulxVz8FJdttwipJbFNbJxfU74BwRmdbaQqpailMc8rCIXCgi8SIS7Wb7X7ey6nkicoqIxOBUeq5S1b3Af4BxIvJ1dzvRIjJHRCa2I/bngFtEZJSbuH4B/FOPozVZC/oAxap6VETmApe3Y93HgHtFJF0cJ4hIf9rxvlU1B8gAfiYiMW6T7y/6LfI08EURWSgikSIS6zaZ9D9Yf01EJrkJ4B7gRVWt87huS14ELvD7Xu/h+Pf154E7RCRJRIYBN7ay7DKcz+5yEYlym8hPwvlMO9oG4CJ3Px+L0zDk8/J9jwtwKq1fUNV6nIPIgyIyEEBEhonIwnZs97h/B+6+N09EonEOlkeB+nbGtRo4IiK3i3MvWqSITBGROe78ln4L4CTA0X7behT4thuTiEiCiJwvIn1wTnBrge+6v5uLgLnt+Jxa2+djcOp1CoFaEfkCTh2szwGgv4gk+k3bgHN8SxaRwcD3jvf1RWSQiCx2T0CqcKou6lvbWNgnF1UtxKn0vsvDsr8FbsVpmVKIk4lvxCmbbMmzwE9xisNmAV9zt3UE58u7FOfMaz/HKi29ehx4CqfVyS6cH8ZN7Vi/LdcD94jIEZzP5/l2rPuAu/ybOBV1f8MpS23v+74cp0y7GOdz/IdvhpukF+MUQfi+j9tovN89hXN1uh+nUcR327FuQKq6GbgB57vNx6lUP957AO5x190F/BcncVW18LoHcQ7K38cphvsBcIGqFh3na7fmQZy6jgM4rdme+Zzb24/zOe1zt/VtVd3mzrsdp0J+lYgcxvkcxrdj25/nd9AX54B+iGOt//6vPXG5JysX4NzasAvnqucxnIYN0MJvwZ13N/CkW0z0VVXNAK7FKao75L7+Ve7rVAMXuc+LcYqRXvb4Plvd593f5XfdOA/h/O6W+q27DSeJZ7uxDsX5zD/FqVt5E6f12nG9vvu4FWf/KMapYwpUmtFAGhcP9iwi8gROBdedoY6lJxKR93Aq1rvM3fIi8h2cyv7TQh2LMeEs7K9cjAklERkiIvNFJEJExuNclfwr1HEZE+7srnBjWheD0xR7FE5TzCU4vT4YY1rRo4vFjDHGBIcVixljjOlw3aZYbMCAATpy5MhQh2GMMV3K2rVri1Q10M2nn0u3SS4jR44kIyMj1GEYY0yXIiLt6UXAMysWM8YY0+GCmlxEZJGIbBeRLBH5YYD5qeL0jbVeRD4TkfPc6SPFGTBsg/v4SzDjNMYY07GCViwmzuBZD+N025wLrBGRpaq6xW+xO4HnVfXPIjIJp/uMke68nao6PVjxGWOMCZ5gXrnMBbJUNdvtFmEJzUeEVJzuHcDpimFfEOMxxhjTSYKZXIbRuDvnXBp3PQ5Ovz1fE2fs52U07m9olFtc9r7bkZ4xxpguItQV+pfhDO85HGdsgqdEJAKns8FUVZ2B01nas+KM4dCIiFwnIhkiklFYWNipgRtjjGlZMJNLHo3HvhhO83EtrsbtyVdVP8bpFXeAqla5Pcyiqms5NjRsI6r6iKrOVtXZKSkd3kzbGGPMcQpmclkDpLtjOMTgdOG+tMkye4CzANzxQmKBQhFJcRsEICKjgXSc4WaNCZq6emXJ6j0cLAvYo74xph2C1lpMVWtF5EacMZgjgcdVdbOI3ANkqOpSnB5mHxWRW3Aq969SVRWRU3HGKanBGZDm26paHKxYjQG4//WtPLpyFyWVNXz7NK8DExpjAgnqHfqqugynot5/2l1+/28B5gdY7yWcIT2N6RTPfJLDoyt3AZBVUBbiaNrvrS0HGNS3FycM7xfqUIwBQl+hb0zIrcws5K5XN3PG+BTmjkomswsml9tf+oxfvbGt7QWN6SSWXEyPtuPAEa5/eh3pA3vzx8tnMn5QH3YWlNGVhqI4WFZFcXk1G/aUUFffdeI23ZslF9NjqSrffmotsTGRPH7VHHr3imLswN6UVdVy4HDzSv2sgjJufX4DJRXVIYi2Zb5ivPLqOrbvPxLiaIxxWHIxPVZmQRnZReV8/5xxDO0XB0D6wN5A4HqXf3+6j5fX5XHtPzI4WlPXqbG2JqvwWKxr9xwKYSTGHGPJpYeoravnrS0HqLdikwYrM4sAWDDu2D1SYxuSS/MrgI15pfTpFcWa3Yf4/gufhs1nmVVQRlx0JCl9erEux5KLCQ+WXHqId7YVcO0/Mli2KT/UoYSND7OKGDUggWHuVQtASp9e9ImNanQ14LMxr5RzJg/iR+dN4LXP8rk/TCrQswrKGDMwgdlpSay15GLChCWXHmJXUTkAT68KyrhAIVNbV0/G7vbfAlVTV8+q7IPMH9u/0XQRYezA3s2KxQ4cPkrhkSqmDkvk2gWj+d+T0nhkRTZPfrT7uGOvqas/7nX97SwoY2xKb2alJbGnuIKCI0c7ZLvGfB6WXHqInOIKAFZlF3fJ+zha8sRHu7n4Lx/z6d6Sdq23fk8JFdV1nDK2ebdBY1N6k1VQ3mjaZ7mlAEwdloiI8NMvTubsiQO577Utx3VHf3F5NdN+9ibPZ+xte+FWlFXVsq/0KGMH9mZmWhIA63La91kYEwyWXHqIvcUVpPWPJzpSeOaT7nH1oqo8u3oPAK9v2t+udT/IKiJC4KTR/ZvNGzuwN0VlVY1ahW3MKyVCYNJQp//UyAjhB4smUFOnvLKh/SNFbMs/TEV1Hfe/vo3Sypp2r++z0z1RGDuwD5OH9iUmKoJ1VqlvwoAllx4i52AFJwzvxxemDOGltblUVh9/a6e6euW2Fz5lzXEUR3WkNbsPkV1YTlx0JMs372/XvSkfZhUxdXg/EuOjm81LH9S8xdimvFLGDuxNfMyxTi3GDerD9BH9eH7N3nbfF5PtFlMWl1fzx7czW112X0kl3/j7ahY/9EGz18lqSC696RUVyQnDEq3exaOf/Xszf3ovK9RhfG5/fX8nD761I+zuzbLk0gPU1tWTV1JJanIcV8xL5fDRWv796fGPy5axu5gX1uby+/+2flAMtiVr9tCnVxS3nJPOrqJyz3fWHz5aw4a9JSwYOyDg/LEpfYBjB25V5bPcUqYMS2y27Fdnj2D7gSMNxWZe7SoqJzY6gktmj+CJj3aTHaABgaryzzV7WPjgCt7dXsinuaXN3mNWYRlREUJa/3gAZqUlsTG3lKra8GkqHY4qq+t4ZtUelqz+fMWSoXa0po6/vL+T7fuPICKhDqcRSy49wL6So9TVK2nJCcwdlcy4Qb3bLBrLL60kr6Qy4LxlG50WZx9kFbHXrctpSX298tjKbK76++o2l22P0soalm3M50vTh7J4ujMG3XKPRWOfZBdTV6/MbyG5DEuKo1dURENyOXC4iqKyKk4IkFwumDaE2OiIdted7CoqZ2T/BP7fwvHERkfy89e2Npq/t7iCq/6+httf2sjEoX159tp5AHzgNp/2ySooY+SABKIjnZ/yzLQkquvq2ZR3uF3xhErB4aO8vjG/0aOl/a4te4sruPOVjZ7uQcrIKaa6rr7LN4BYumEfhypquPLkkaEOpRlLLj3AHvegPiI5HhHhinlpfJpbysZWzrZvenY9X3/sk2b3ctTXK69v2s+M1H6IwAtrc1vcxr6SSq547BPue20rH2QWceHDH3ZYUdqrG/I4WlPPZXNTGdQ3lpmp/Vi+xVty+TCriLjoSGamBe7kMTJCGJ3Su6E58me5TgX51OHNk0vf2GjOmzKEpRv2tauocVdROaNTEkjp04ubzhzL29sKWLGjkKKyKu5eupkzf/seq3cVc/cXJ7Hk2hM5ecwARg1I4IOsxsnF11LMZ2aqr1I//IvGXt2Qx1kPvM93nlnX6HHzc+uPa3t/eX8nT6/aw3vb2x440D9Jd4XPKhBV5YmPdjN+UB9OHJ0c6nCaseTSA/iSi6/o5MszhxEXHdlis+TDR2tYt+cQ2UXlrGxyMMvIOUTBkSq+MX8UC9JTeCFjb8D+rF7dkMfC363g09wSfvU/U1l+y6n0jYvm8kdX8cLnbCGlqjy3ei+Th/ZtKKpaOHkwm/IOe7o6WplZyNxRyfSKimxxGf/myJt8lflDmicXgK/MHsGRqlqWb/aW3GrcM+ZRAxIAuGr+SNL6x3Pbi59y2q/f5alVOVw8awTv3XY6V80fRUSEU9xxytgBrMo+2NCEubq2npziioYbP8G5Tyetf3xY17uUVtRw03PruXnJBsYP6sNL3zmZN763gDe+t4ArT0pj3Z5D7W7kUFldx1K3YcWbHr6HD7KKmJnaj5ioiLD+rFqTkXOILfmHufLkkWFXJAaWXHqEnOJyYiIjGNQ3FnDOthdPH8qrn+ZRVlXbbPlVOw9SrxAVIfyjyX0cr322j15REZw1YSCXzhlBfulRVmY2PlN8alUONy/ZQPrA3rx+8wIumZPKmJTevHL9fOaN6s9tL37G1x77hIv//BGn/vpdJvzkdRY/9AGb8ppfSW3KK+X7z3/K06tyqHUPqhvzStmaf5hL56Y2LLdw8mAA3txyoNXPIr+0kp2F5ZzSQpGYz9iU3uSVVFJZXcfGvFLSB/YhLiZwMjpxdDJp/eP55xpvSXNvcQV19cqoAU5S6BUVyV0XTKLwSBWnjkvhzVtO5ZcXTW34vnzmjx1ARXUdG9xm17sPllNXrw0NEHxmpSaxds+hsKvgBdi8r5RFv1/B6xvzuW3heP75rZOYlZbEhMF9mTC4LxdMG0q9wsc7i9remJ83NudzpKqW0SkJvL2toNV7iIrLq9m87zBnThjICcMSyeiE5HLNk2u49Z8bOvQ7eeKj3fSNjeLCGUM7bJsdyZJLD7C3uILhSXFERhw7u/nyjGEcrann/QBFCL5io2tPHc072wsargZ8RWKnj08hoVcUZ08cRHJCTKOD6pZ9h7n3P1s4fXwKz3/rJNL6JzTMS4yP5u/fmMO1C0ax//BRoiKF6SP6cemcVPaVHuVLD33Az1/bQkV1LQWHj3LbC5/yxYc+4N+f7ePOVzZx/h8+YGVmIc+t3ktcdCSLpx/7UY0ckMCEwX3arHf5MOsgQIv1LT7pg3qjCjsLy9iYF7gy30dE+Mqs4XycfZA9B9u+cvLd0Oq7cgE4a+IgNv1sIX/+2izGpPQOuN5JY/oTIce6rfFdWTVdfmZaEoVHqsg9dHx1F8FSX6/88KWN1NYrL19/MjecMbbRPgkwfUQ/eveKYkVm+5LL82tySU2O5wcLJ1BaWcPqXS0Xv37oXo3PHzuAWSOT2JRXGtS+4sqqanlnWwEvr8/j8Q93d8g295ce5Y1N+7l0bmqjFozhJDyjMh0q52AFqW6RmM/skckkJ8SwfPN+zj9hSKN5H2QVMXdUMleeNJJHVmTz9Koc7jhvYkOR2HlTneVjoiK4aMYwnvx4N0VlVcTHRHLTc+tIjIvmN1+ZRlRk83OX6MgIfnz+JH58fuPpt5w9jvvf2MajK3fx2mf5lFTWUFNXz3ULRnPDmWP5MLOIX7y+la//bTURAhfNHE7f2MbNiBdOHswf3smk8EgVKX16BfwsVmYWMqB3DBMG92n1M/MVNX2QVURRWTVTh/Vtdfn/mTWc3761g9+9vYPTxqVwtKaOqtp65o8d0Ozg70suY1ISGk1v6yCRGBfNCcP78WFWEbeeM47MA2WINE8us9ybKe96dRMpfXpxtKYeBW5fNJ7hSfEBttw5/rU+j415pTx4ybQWBzWLjozg5DH9WbGjEFVtVNxTVVvHHS9t5OJZwznZ7+Rgz8EKPs4+yPfPGcdp41KIjY5g+eb9LZ5AfJhVRJ/YKKYOS6TwSBV/rctmU14ps0cGp95iw54S6hVG9o/nl8u2Mn1EIrPSjr1W7qEKHl2RzdWnjG72O23JM5/kUK/K109MC0rMHcGuXLo5VWXPwQpSkxvvtJERwtkTB/LutgKqa48VIewvPdpQbDQ4MZZFkwezZM1eKqvrWLYxn5ioCM6aOKhh+UvmjKCmTvnXujx+tnQL2UXl/O6S6QzoHfjg3pLE+Gh+edFUnv/WSfSLj+HU9BTeuuU07jhvIn1jo/nC1CG8dctp3L5oAiP7J/CN+SObbWPh5MGown+3Bi4ae2vLAZZ+uo9FUwY31GO0ZGT/BCIjhH+tywNgahsjPA5JjOPM8QN5eV0eNy/ZwO0vbeSuVzfz01c3N1s2u6icpPho+sXHtLrNQE4ZO4ANe0s4fLSGrMIyhvWLa1ZcN25QH6YOS2TD3hJWZhbxWW4Jr322j2c/2dPu1+soFdW1/N/y7UwbnsjiacNaXXbBuBRyD1Wyu8lV4Bub9vPy+jyuf3Yd+/xalL24di8icPHs4cTFRHJqegpvbj4QsAhKVVmZWcTJY/oTFRnRkIiDWTS2ZncxEQLPXnsiw5LiuOGZ9RSVVaGqvLwuly/8biVPfpzDz/7dfF8JpKq2jmc/2cNZEwYxIjl0JwttsSuXbq6kooYjVbXNkgs4B+PnM3L5OPsgp7k9A/sXGQD870lpvLYxn1c35PH6pnxOH5dC717Hdpv0QX2YkdqPP7yTyZGjtVx/+pg2i5xaM3dUMstuXhBwXmx0JN85fQzfOT3w+PYTh/QhNTme5Zv3c5lffQw4dTfffW49JwxL5MfnTWozjpioCNKS49l+4Ihbmd/6lQvAHy6bQc7BCmKjI+gVHclD72Tyyvp91NTVNzQVBsguLGtUJNYe88cO4KF3s/jE7cbHvzLfJzJC+PdNpzSa9rXHPuGNTfu5beH4kFT+PrpiF/sPH+WPl89oM7Gfmu7sPyszCxt9Ts+s2sOQxFgOVzoNApZcdyIRIrywNpdT01MYkuh0QLpw8mDe3HKAz3JLmTai8UlBzsEK8koq+fZpowHo37sXowYkBLVSPyOnmAmD+zK0Xxx/umImX/7TR3z3ufUkxcfw2sZ85oxMYsqwRP7+4W7W7C5mThtXUM+s2sPB8mquCsPmx/7syqWb87UUC5Rc5o8dQHxMZKNWTh9mFZGccKzYaO6oZMYP6sOv3tjGgcNVzYrQAC6dM4IjR2uZmdqPW84ZF6R30jYR4bypQ3hveyG3/nMD+aXO2W1+aSVXP7mG5IQYHr1ydosV802NcQ/crVXm+0voFcWkoX0ZndKbYf3iOGVsCpU1dWze1/iek11F5Q2V+e01M60fcdGRrNhRSHZh42bIrVk4ZTDZ7bjRtCMdOHyUv7y/k/OmDm7zwAmQ1j+B1OR4Vuw4Vu+SeeAIq3cXc9XJI/nl/5zA2pxD/PbNHXyQVUR+6VG+OntEw7JnTRxIZIQEbL33QZOTJ3Cab6/LCU4DiNq6etbvKWHOSOcKafLQRO5dPJmPdh7kzS37+cGi8Sy57iR+sHACg/r24v7Xt7UYx6Hyam5esp57/rOFuaOSm3W6Gm4suXRzOQ3NkJufKcdGR3L6+JSGcV5UlQ+ynCID39mliPC/J6dxqKKmWZGYz+Lpw7j1nHE8fMXMRmfooXDzWenccMYY/rMxnzN+8x4PvrWDq5/IoLyqjr9dNZuBfWLb3ojLN3BYoPtbvPAdUPx7bS53R7kcnXJ8Vy69oiKZOyqZVzfkUVVbH/DKJZCFkwYh4hQtdbb/W76dunrl9kUTPK+zIH0AH+8samj19cwne4iJjODiWcP50rShXD4vlb+8v5P7/rOFpPhozp40sGHdfvExzBuVHLDl4IdZRQxNjG10RTR7ZBIHy6ubFcN1hK35R6iormtUn/PV2SN44KvTeOWG+Vx/utOoIS4mkpvPGsfanEO8vbWg2Xbe3Lyfcx5cwWuf5XPL2eN45pp5Ydn82J8ll25ub8MNlHEB5y+cPJjCI1Ws31tCVkEZBUeqmjXTvXD6MPrGRnHG+MZFYj6x0ZF896z0hmKJUIqLieS2hRN4+9bTOGviIH7/dibb9h/moctnMGFw20Vb/nwH7qmttBRrzcC+saT1j2/UcilQS7H2OmXsAA4fdZqQN22G3Foss1KTAiaXNbuL+fZTa9myr/ld/aUVNdy9dDN/eX+np9fJPVTBlY+vZvHDHzqPhz7gpXW57r083t/zgvQUyqvrWCEuIuQAACAASURBVL+nhMrqOl5el8uiKYPp79bl3XXBJCYM7kNmQRkXzhjW7J6lcycNIqugjJ1+3erU1Ssf7TzIKekDGh2YffUuwSga8900PNs90QDnhO2imcOZPLTxfvWV2cMZNSCBXy/f1nDvWElFNd9bsp7rnlrLwD69WHrjKdx8dnrIT+K8sDqXbi7nYDkpfXq12BLp9PEDiYoQ3ty8nyGJzll90zqThF5RvHz9fPoF6OQxXI1Ijufhy2fyzfmHOFpTd1z1QHNHJTNqQAKnjmveLb9Xc0Ym8862goaWTx2SXNKPvRdfP2heLJoymPte2+o08HBbJdXW1fOjlzeSWVDGW1sPcO2C0Xzv7HR6RUXw2sZ87l66haKyKiIEzp44kLEDW369w0dr+OYTa8gvOdrQ/T/AxTOHc+OZY9v1Hk8a05/ICGFlZiE5B8s5fLSWy+cdq0eLjY7kT1fM5BfLtvKNk0c1W//cyYO5+99beHPzAb5zupOAN+WVUlpZ02xfGJvSm76xUazNKebiWcPbFWdbMnKKGdYvztOJV3RkBP/v3PHc8Ow6/rU+j8S4aH70r40cKq/me2enc8MZY7tEUvGx5NLN7Slu3lLMX2JcNCeN6c/yzfsZO7A3qcnxAVugeC1+CTez/A5y7TU8KZ53/9/pn+v154xM4sW1uewsLGfswN4NyWVkO87imxo/qA8DescAErBX55YsnOwklzc253PdqU6jiJfW5ZJZUMb9F01l3Z5D/OX9nby+KZ+R/RN4f0chU4b15cFLpvGdp9fxf8u389evzw647dq6em58dj3ZheU8+c25n6tRBzj75fQR/Vixo5CICGFMSgLzRjWurxmd0pvHrpwTcP2h/eI4YXgiL6w9dg+WbyiCprFFRAgzgzCKp6qSsfsQJ4/xXjfyhSmDmTos0e0jrZ6JQ/ryxDfmNLvK6Qq6Tho0x2VvcSVpbTRXXDh5MLsPVvD+jsLPfVAwjfkqsH3FI7uKyhmaGOu5UUEgERHC108cyZfbeWf2iOR4pgzr21A0VlldxwNv7WBmaj8umTOCX188jWevnYcAq3cVc+f5E3nl+vksSE/h2gWjWb75AOsDjBWjqtz9782s2FHIfRdO6bB96NT0FD7NLWX9nhIum5va7jqG/5k5nOzCcn71xjZ+9cY23tpygBNHJwdsJj8rNYkdB8oorTj+sXWa2ltcScGRqnbdPxMRIfz4/IlER0Zw81npvHrD/C6ZWMCuXLq1qto69pVWttkW/pxJg7jzlU3U1Gmb3aKY9hk1IIEBvWNYs6uYy+amkl1UzqjjrMz3d/PZ6ce13qLJg/nNmzs4cPgoL67N5cDhKh66fGbDgfvkMQN485bTqKyua3RVdM2CUfzj49386o1tPHftiY0O9I9/uJunV+3hW6eNbtQlz+e1YNwAHvzvDmKiIo6ruOrKk0dyyZwRjabFtFCs5LvCXbf3EGeMHxhwmfbynVB4aSHn78TR/fnsp+eGfYV9W+zKpRvLO1SJ6rEOK1syqG8sM1Kd+wFOasclvGmbiDA7LZk1OcWoKrs+xz0uHWHRFKcPtudW7+Ev7+3k7ImDmh38YqIimhW3JfSK4qYzx7Iqu5j3dzhdBlVW13HnKxu59z9bWDR5MLcv9N4azIsThiUyoHcvFk8belw3nIJTN+P/aOkem+mp/YiMEP787k5yD3VMq7GMnGL6xkY1tDpsj66eWMCSS7eW08o9Lk1996x0bj4rneSE4/sRm5bNGZXM3uJKtuQf5vDR2uO+x6UjjB3YhzEpCfz+7UzKq2u5fdF4z+tePi+NEclx/PqN7WzYW8L5f1jJ06v2cO2CUfz+sult3hzZXlGRESy7+RTuvXBKh243kPiYKH7x5Sls2lfKOQ+s4LGV2QF7+26PjN2HmJWW1OGfS1dhyaUb8zVD9tJf0RnjB4b0BsjubK57ZfC828Hn6BBeuQB8YcoQVJ37LdIHeW9tFhMVwffPGc+W/MNc+PCHVNbU8ew18/jx+ZNaHb7g8xjYJ5bY6OBsu6lL5qTy5i2ncuLoZO57bStf/tOHLQ7hUFpZwzvbAncxA84Nj5kFZUHrr6wrsOTSjeUcrCAuOpKUdvbzZTrWxCF9SIiJ5BV3vJFQFouB0x/cmRMGcutxnEx8adpQzpowkItmDOONm09t1IFkdzA8KZ7Hr5rDHy+bwe6icq55MoPyJsNSVNfWc+2TGXzziQz+G+CGRzh2z0x761u6E0su3ZivGXJ3KL/tyqIiI5iZlkRpZQ1REcLwpNDebDoi2TmADuzrvbcCn4gI4W9XzeGBS6a3qxl0VyIifHHaUP50xSwyC47w/ec/bRiRVVW585WNrN5dTFJ8NL9Zvr3ZaK0Aa3KKiY4UTjjO3h26A0su3URldR0/fOkz/vr+TiqqnTOtPQcrwrrX1J7Edwab2j8+4FAEJvyckj6AH503kTc27+ehd7MA+NsHu3g+I5ebzhzLzxZPYfuBIyz9dF+j9faVVLJk9V5OHN2/04r0wlFQmyKLyCLg90Ak8Jiq3t9kfirwJNDPXeaHqrrMnXcHcDVQB3xXVZcHM9aurLaunpueW9dwif7oymy+deoY9hRXNLqb24SOL7mEur7FtM/Vp4xi877DPPDWDsqqanlsZTaLJg/mlrOdIsU/v7eTB97awfknDCE6MoK6euXW5zdQU1fPvYuD3xAhnAUtuYhIJPAwcA6QC6wRkaWqusVvsTuB51X1zyIyCVgGjHT/vxSYDAwF/isi41Q1eMPFhaGM3cU8ujKbypp6qtzBp8YO7M3/O3c8g92uWlSVH/9rE//dWsC9iyczaWhfHnwrk58v2wp4aylmgm/6CKc34/FtDFJmwouI8MuLprKzsIxHVmQzcUhfHrhkWkMLsNsWjuObT2TwfMZerpiXxiMrslmVXcyvLz6BkT38RCKYVy5zgSxVzQYQkSXAYsA/uSjg600wEfBdXy4GlqhqFbBLRLLc7X0cxHjDzlOrcnhveyEThvQlNiqCPrFRLP10H69vzOeWc8Zx5ckj+cPbmfwzYy83nTmWr580EoCnr5nHmt3FvLwut2FseRNacTGR/Pum+QwOg849TfvERkfy16/P4k/v7uQ7p49p1E/fGeMHMistiT+8nUn6wD789s3tnD91CF/p4D7KuiIJxhgGACJyMbBIVa9xn38dmKeqN/otMwR4E0gCEoCzVXWtiDwErFLVp93l/ga8rqovNnmN64DrAFJTU2fl5OQE5b2EysIHVzA8KY6/XXWs/6Q9Byu4+9+beWdbAcP6xZFXUsmlc0bwy4umWsW9MSHwSfZBLnlkFTGREfTvHcMbN5/apRo7iMhaVQ3cadznEOqaxcuAJ1R1OHAe8JSIeI5JVR9R1dmqOjsl5fh7rg1HVbV17CwsY8KQxsUoqf3j+duVs3n0f2cTFSl8Ycpg7rtwiiUWY0Jk3uj+nDouhZr6eh7sxq3o2iuYxWJ5gH/HPsPdaf6uBhYBqOrHIhILDPC4brdw5GgNfWKb74w7C8qprdeAY5CICOdMGsTZEwc2PDfGhM4fL53BzqIyZqYefy/c3U0wr1zWAOkiMkpEYnAq6Jc2WWYPcBaAiEwEYoFCd7lLRaSXiIwC0oHVQYy10+0truDaf2Rwws/eZFNeabP5W/OdgZsmDmm5AlhELLEYEwYS46MtsTQRtCsXVa0VkRuB5TjNjB9X1c0icg+QoapLge8Dj4rILTiV+1epUwm0WUSex6n8rwVu6C4txapq63h0RTYPvZuFIKjCyswipjQZ7XDb/sP0ior4XON+GGNMqAT1Phf3npVlTabd5ff/FmB+C+v+HPh5MOPrbBXVtSx+6EMyC8o4b+pg7jx/El977BPW5hQDYxotu23/EcYN6mM33BljuiQbz6UTfZR1kMyCMn7zlWkN41PMTEvi7a0HGobB9dmaf4QzxnevRgrGmJ7DTos70YrMQuKiI/nitCEN02anJXGooqZh+FuAwiNVFJVVMWFI88p8Y4zpCiy5dKKVmUWcODq5UffkvhHwMvzG796+/wgAE+1ubmNMF2XJpZPsLa5gV1E5C9IbF3WNSelN39go1vkll237nZZi1lWIMaarsuTSSVZmFgFw6rjGySUiQpiVltQw/gM49S0D+/Siv43DYozpoiy5dJKVmYUMTYxlTErzpsWz0pLILCijpKIacO5xsfoWY0xXZsmlE9TW1fNhVhEL0lMC3vQ4K83pjn39nhJq6urJKiiz+hZjTJdmTZE7wWd5pRw+WsuCcYHHVpk2IpHICGFtziGGJcVRXVffrE8xY4zpSiy5dIIVOwoRgfljAieX+JgoJg3pS0ZOMemDegME7FPMGGO6CisW6wQrM4s4YVgiSQkxLS4zKy2JT/eWsimvlKgIYUxK706M0BhjOpYllyArraxhw96SZq3EmpqVlkRlTR1LP93H2IG9iYmyr8YY03XZESzIPt55kLp6bXZ/S1OzRzo3Ux44XMUEq8w3xnRxllyCbGVmIQkxkcxI7dfqckMS4xiaGAtgzZCNMV2eJZcgW5lZxEljBhDtoXfjmW5XMHblYozp6iy5BFFldR17iiuYPiKx7YWBU8YOICYygslDvS1vjDHhqs3kIiLjRORtEdnkPj9BRO4Mfmhd377SSgCGJcV5Wv6rs0fw/g9OJ6WPdftijOnavFy5PArcAdQAqOpnOEMWmzbsK3GSy9BEb8klIkIY4nFZY4wJZ16SS7yqNh2/vjYYwXQ3DcmlnyUMY0zP4iW5FInIGJwx7hGRi4H8oEbVTeSVHEUEBrutwIwxpqfw0v3LDcAjwAQRyQN2AV8LalTdxL6SSgb1ifXUUswYY7qTNpOLqmYDZ4tIAhChqkeCH1b3sK+kkqH97KrFGNPzeGkt9gsR6aeq5ap6RESSROS+zgiuq3OSi9W3GGN6Hi/lNV9Q1RLfE1U9BJwXvJC6h/p6ZV/pUYZZcjHG9EBekkukiDTceCEicYDdiNGGg+XVVNfW25WLMaZH8lKh/wzwtoj83X3+DeDJ4IXUPVgzZGNMT+alQv9XIvIZcJY76V5VXR7csLq+Y8nFKvSNMT2Pp5EoVfV14PUgx9Kt5LnJxepcjDE9kZfWYheJSKaIlIrIYRE5IiKHOyO4rmxfyVHiYyJJjIsOdSjGGNPpvFy5/Br4oqpuDXYw3YmvGbKIhDoUY4zpdF5aix2wxNJ++0rtHhdjTM/l5colQ0T+CbwCVPkmqurLQYuqG9hXUsnkoTaipDGmZ/KSXPoCFcC5ftMUsOTSgqM1dRSVVXvuat8YY7obL02Rv3G8GxeRRcDvgUjgMVW9v8n8B4Ez3KfxwEBV7efOqwM2uvP2qOqXjjeOzpZfehSwe1yMMT1Xm8lFRGKBq4HJQMNNG6r6zTbWiwQeBs4BcoE1IrJUVbf4beMWv+VvAmb4baJSVad7fB9hxW6gNMb0dF4q9J8CBgMLgfeB4YCXnpHnAlmqmq2q1cASYHEry18GPOdhu2HP7nExxvR0XpLLWFX9CVCuqk8C5wPzPKw3DNjr9zzXndaMiKQBo4B3/CbHikiGiKwSkQtbWO86d5mMwsJCDyF1jn0llYjAoETrgs0Y0zN5SS417t8SEZkCJAIDOziOS4EXVbXOb1qaqs4GLgd+546G2YiqPqKqs1V1dkpKSgeHdPz2lVSS0rsXvaIiQx2KMcaEhJfk8oiIJAE/AZYCW3BurGxLHjDC7/lwd1ogl9KkSExV89y/2cB7NK6PCWv7So5afYsxpkfz0lrsMfff94HR7dj2GiBdREbhJJVLca5CGhGRCUAS8LHftCSgQlWrRGQAMB9vCS0s7CupZOIQu8fFGNNzeWkt1g/4X2Ck//Kq+t3W1lPVWhG5EViO0xT5cVXdLCL3ABmqutRd9FJgiaqq3+oTgb+KSD3O1dX9/q3MwpmqkldSyVkTO7rk0Bhjug4vN1EuA1bh3HNS356Nq+oyd33/aXc1eX53gPU+Aqa257XCRXF5NVU2SJgxpofzklxiVfXWoEfSTewrsRsojTHG030uInKtiAwRkWTfI+iRdVF2j4sxxni7cqkG/g/4MU6fYrh/21O532PY3fnGGOMtuXwf50bKomAH0x3sK6kkNjqCpHgbJMwY03N5KRbLwukV2XjgG8fFBgkzxvRkXq5cyoENIvIujcdzabUpck+1q6jCuto3xvR4XpLLK+7DtOGDzCK25h/mJxdMCnUoxhgTUq0mF7fb/KtU9YzWljNQX6/86o1tDOsXxxXzUkMdjjHGhFSrdS5uR5L1IpLYSfF0Wa9v2s/GvFJuOWccsdHWYaUxpmfzUixWBmwUkbdw6l8Aq3PxV1NXz2/e3M74QX348oyAowoYY0yP4iW5vOw+TAuez9jLrqJyHvvf2URGWCsxY4zx0ivykyISA4xzJ21X1ZrW1ulJKqvr+P1/M5mdlmSdVRpjjMtLr8inA08CuwEBRojIlaq6IrihdQ1PfrybgiNVPHzFTLu3xRhjXF6KxX4LnKuq2wFEZBzOwF6zghlYV6CqLFm9h5NG92fOSOtuzRhjfLzcoR/tSywAqroDsL5NgC35h9l9sIIvTR8a6lCMMSaseLlyyRCRx4Cn3edXABnBC6nrWLYxn8gIYeHkwaEOxRhjwoqX5PId4AbA1/R4JfCnoEXURagqyzbu56TR/UlOiAl1OMYYE1ZaTC4i8raqngXco6q3Aw90Xljhb2v+EXYVlXPtAht5wBhjmmrtymWIiJwMfElEluC0FGugquuCGlmYW7YxnwiBcycPCnUoxhgTdlpLLncBPwGG0/yqRYEzgxVUuHOKxPI5cXR/BvTuFepwjDEm7LSYXFT1RRF5CfiJqt7TiTGFvW37j5BdVM43TxkV6lCMMSYstdVxpQJf6aRYugxfkdiiKdZKzBhjAvFyn8s6EZkT9Ei6CFXltY35zBtlRWLGGNMSL02R5wFXiEgOTq/IgnNRc0JQIwtT2w8cIbuwnG/MtyIxY4xpiZfksjDoUXQhb28tAGCR3ThpjDEtarNYTFVzgBHAme7/FV7W666Kyqro0yuKlD5WJGaMMS1pM0mIyE+B24E73EnRHOsKpseprK4jLsZGmjTGmNZ4uQL5MvAl3FEoVXUf0CeYQYWziuo64i25GGNMq7wkl2q3SbICiEhCcEMKbxXVtcTFeKmqMsaYnstLcnleRP4K9BORa4H/Ao8GN6zwZVcuxhjTNi/DHP9GRM4BDuMMdXyXqr4V9MjCVEV1HX1i7crFGGNa47XV10acrvZXuP97IiKLRGS7iGSJyA8DzH9QRDa4jx0iUuI370oRyXQfV3p9zWCrrK4jLtquXIwxpjVtnoKLyDU4nVi+g3MD5R9F5B5VfbyN9SKBh4FzgFxgjYgsVdUtvmVU9Ra/5W8CZrj/JwM/BWbj1PWsddc91M731+EqamqtWMwYY9rgpXznNmCGqh4EEJH+wEdAq8kFmAtkqWq2u94SYDGwpYXlL8NJKODcuPmWqha7674FLAKe8xBvUDlNka1YzBhjWuOlWOwgcMTv+RF3WluGAXv9nue605oRkTRgFM7VUbvW7WxWoW+MMW3zcgqeBXwiIq/iFFEtBj4TkVsBVLUjRqi8FHhRVevas5KIXAdcB5CamtoBYbROVamsseRijDFt8XLlshN4Bfc+F+BVYBfOjZSt3UyZh9NtjM9wd1ogl9K4yMvTuqr6iKrOVtXZKSkprb2HDnG0ph5V7A59Y4xpg5emyD87zm2vAdJFZBROYrgUuLzpQiIyAUgCPvabvBz4hYgkuc/P5Vj3MyFTUV0LQLy1FjPGmFYFrWZaVWtF5EacRBEJPK6qm0XkHiBDVZe6i14KLHF7AfCtWywi9+IkKIB7fJX7oVRR7ZTaxVuFvjHGtCqoR0lVXQYsazLtribP725h3cdpu0Vap6qscZKLFYsZY0zremzX+cfj2JWLJRdjjGlNi8lFRP5PRL4VYPq3ROT+4IYVnhrqXKxYzBhjWtXalcuZwCMBpj8KXBCccMJbpV25GGOMJ60ll17+lew+qlqP0w1Mj2PFYsYY401ryaVSRNKbTnSnVQYvpPDlu3KxCn1jjGlda5UHdwGvi8h9wFp32myc+02+F+zAwpHVuRhjjDctHiVV9XURuRCn48qb3MmbgP9RVc/d7ncnFTVWLGaMMV60mFxEJEpVNwFhM5ZKqFVW1yECvaKsBbcxxrSmtaPkat8/IvLHTogl7FVU1xEfHYlIj2zPYIwxnrWWXPyPoPODHUhXUFFda2O5GGOMB60ll2bNkHs6G8vFGGO8ae00fIKIfIZzBTPG/R/3uarqCUGPLsxYcjHGGG9aSy4TOy2KLsIZ4tiSizHGtKW1psg5gaaLSATOePcB53dnFdW1duVijDEetNZxZV8RuUNEHhKRc8VxE5ANfLXzQgwfFdV1xEVbhb4xxrSltSPlU8AhnBEirwF+hFPfcqGqbuiE2MJOZY3VuRhjjBetJZfRqjoVQEQeA/KBVFU92imRhSGr0DfGGG9aa4pc4/tHVeuA3J6cWMCp0Ld+xYwxpm2tHSmnichh938B4tznvqbIfYMeXRhRVavQN8YYj1prLWZHUT9VtfXUq3W3b4wxXlgPjB7ZKJTGGOOdJRePrLt9Y4zxzpKLR5XuQGHWcaUxxrTNkotHFb5isWi7cjHGmLZYcvGowupcjDHGM0suHvkq9K21mDHGtM2Si0fHrlyszsUYY9piycWjCrdC34rFjDGmbZZcPKqssWIxY4zxypKLR+VVVqFvjDFeWXLxyHefS2yUJRdjjGmLJRePnIHCIomIkFCHYowxYS+oyUVEFonIdhHJEpEftrDMV0Vki4hsFpFn/abXicgG97E0mHF6UWEDhRljjGdBa1crIpHAw8A5QC6wRkSWquoWv2XSgTuA+ap6SEQG+m2iUlWnByu+9qqsriO+lyUXY4zxIphXLnOBLFXNVtVqYAmwuMky1wIPq+ohAFUtCGI8n0tFdS3x0XaPizHGeBHM5DIM2Ov3PNed5m8cME5EPhSRVSKyyG9erIhkuNMvDPQCInKdu0xGYWFhx0bfREV1nTVDNsYYj0J9Kh4FpAOnA8OBFSIyVVVLgDRVzROR0cA7IrJRVXf6r6yqjwCPAMyePVuDGagzxLElF2OM8SKYVy55wAi/58Pdaf5ygaWqWqOqu4AdOMkGVc1z/2YD7wEzghhrmyosuRhjjGfBTC5rgHQRGSUiMcClQNNWX6/gXLUgIgNwismyRSRJRHr5TZ8PbCGEKmvqbCwXY4zxKGhHS1WtFZEbgeVAJPC4qm4WkXuADFVd6s47V0S2AHXAbap6UEROBv4qIvU4CfB+/1ZmoeBU6NuVizHGeBHUU3FVXQYsazLtLr//FbjVffgv8xEwNZixtZdV6BtjjHd2h75HVqFvjDHeWXLxoLq2ntp6teRijDEeWXLx4NgolFahb4wxXlhy8aCixgYKM8aY9rDk4sGxIY4tuRhjjBeWXDxoKBazpsjGGOOJJRcPjl25WJ2LMcZ4YcnFgwp3FEq7z8UYY7yx5OKBr1gswcZzMcYYTyy5eFDuKxaz8VyMMcYTSy4eVFqxmDHGtIslFw+sKbIxxrSPJRcPKqwpsjHGtIslFw8qa+qIjY4gIkJCHYoxxnQJllw8qKiutXtcjDGmHSy5eFBRXWdFYsYY0w6WXDywsVyMMaZ9LLl4UGHJxRhj2sWSiweVNsSxMca0iyUXDypqrELfGGPaw5KLBxV25WKMMe1iycWDyuo64q21mDHGeGbJxQOr0DfGmPax5OKBU6FvdS7GGOOVJZc21NbVU11XT4JduRhjjGeWXNpQUeN2WmnJxRhjPLPk0obKhu72rVjMGGO8suTSBhvLxRhj2s+SSwBH3aIwcHpEBisWM8aY9rDk4mdtziGueXINk+56g9/9dweq6lcsZsnFGGO86vEVCarK+zsK+fN7O/lkVzH94qM5ecwAfvffTLILy7nghCGAJRdjjGmPHp9cdh+s4Kq/r2FIYiw/uWASl84ZQXxMJH9+fye/fmM7KzILAYiL7vEflTHGeBbUYjERWSQi20UkS0R+2MIyXxWRLSKyWUSe9Zt+pYhkuo8rgxXjqAEJPHX1XN6/7QyuPmUUCb2iEBGuP30sf/nazIb6F7tyMcYY74J2Oi4ikcDDwDlALrBGRJaq6ha/ZdKBO4D5qnpIRAa605OBnwKzAQXWuuseCkasC9JTAk5fNGUIL/SL59UNeYxIjg/GSxtjTLcUzLKeuUCWqmYDiMgSYDGwxW+Za4GHfUlDVQvc6QuBt1S12F33LWAR8FwQ4w1o6vBEpg5P7OyXNcaYLi2YxWLDgL1+z3Pdaf7GAeNE5EMRWSUii9qxLiJynYhkiEhGYWFhB4ZujDHm8wh1U+QoIB04HbgMeFRE+nldWVUfUdXZqjo7JSVw0ZYxxpjOF8zkkgeM8Hs+3J3mLxdYqqo1qroL2IGTbLysa4wxJkwFM7msAdJFZJSIxACXAkubLPMKzlULIjIAp5gsG1gOnCsiSSKSBJzrTjPGGNMFBK1CX1VrReRGnKQQCTyuqptF5B4gQ1WXciyJbAHqgNtU9SCAiNyLk6AA7vFV7htjjAl/oqqhjqFDzJ49WzMyMkIdhjHGdCkislZVZ3f0dkNdoW+MMaYbsuRijDGmw3WbYjERKQRy2rHKAKAoSOF8HuEaF4RvbOEaF4RvbOEaF1hsx+PzxJWmqh1+L0e3SS7tJSIZwShn/LzCNS4I39jCNS4I39jCNS6w2I5HOMZlxWLGGGM6nCUXY4wxHa4nJ5dHQh1AC8I1Lgjf2MI1Lgjf2MI1LrDYjkfYxdVj61yMMcYET0++cjHGGBMkllyMMcZ0uB6XXLwMvdyJsTwuIgUisslvWrKIvOUO7/yW23FnZ8c1QkTe9Rt++uYwii1WRFaLyKdubD9zp48SkU/c7/Wfbmep1bTppQAAB9RJREFUnU5EIkVkvYj8J8zi2i0iG0Vkg4hkuNPC4fvsJyIvisg2EdkqIieFSVzj3c/K9zgsIt8Lk9hucff9TSLynPubCIv9zF+PSi5+Qy9/AZgEXCYik0IY0hM4I2z6+yHwtqqmA2+7zztbLfB9VZ0EnAjc4H5O4RBbFXCmqk4DpgOLRORE4FfAg6o6FjgEXB2C2ABuBrb6PQ+XuADOUNXpfvdDhMP3+XvgDVWdAEzD+exCHpeqbnc/q+nALKAC+FeoYxORYcB3gdmqOgWnU+BLCa/9zKGqPeYBnAQs93t+B3BHiGMaCWzye74dGOL+PwTYHgaf26vAOeEWGxAPrAPm4dydHBXoe+7EeIbjHHDOBP4DSDjE5b72bmBAk2kh/T6BRGAXbsOicIkrQJznAh+GQ2wcG6U3GadX+//gDAsfFvuZ/6NHXbngcfjkEBukqvnu//uBQaEMRkRGAjOATwiT2Nyipw1AAfAWsBMoUdVad5FQfa+/A34A1LvP+4dJXAAKvCkia0XkOndaqL/PUUAh8He3KPExEUkIg7iauhR4zv0/pLGpah7wG2APkA+UAmsJn/2sQU9LLl2KOqchIWsrLiK9gZeA76nqYf95oYxNVevUKa4YDswFJoQiDn8icgFQoKprQx1LC05R1Zk4RcI3iMip/jND9H1GATOBP6vqDKCcJsVMYfAbiAG+BLzQdF4oYnPreBbjJOahQALNi9bDQk9LLl1h+OQDIjIEwP1bEIogRCQaJ7E8o6ovh1NsPqpaAryLUwzQT0R8g9+F4nudD3xJRHYDS3CKxn4fBnEBDWe8qGoBTt3BXEL/feYCuar6ifv8RZxkE+q4/H0BWKeqB9znoY7tbGCXqhaqag3wMs6+Fxb7mb+elly8DL0cakuBK93/r8Sp7+hUIiLA34CtqvpAmMWWIiL93P/jcOqCtuIkmYtDFZuq3qGqw1V1JM5+9Y6qXhHquABEJEFE+vj+x6lD2ESIv09V3Q/sFZHx7qSzgC2hjquJyzhWJAahj20PcKKIxLu/U99nFvL9rJlQV/p09gM4D9iBU07/4xDH8hxOuWkNzlnc1Tjl9G8DmcB/geQQxHUKzuX+Z8AG93FemMR2ArDejW0TcJc7fTSwGsjCKcLoFcLv9XTgP+ESlxvDp+5js2+/D5PvczqQ4X6frwBJ4RCXG1sCcBBI9JsW8tiAnwHb/n97dxciVR3Gcfz7E8ogbSGsi6KwzKAoE9MliMpi2YuC6E0WCSqKykClYIWualkky2Whi7qQjN69ELxIitiFWMVK8SXT3aIX2qyLbgKhkspY9uni+W8dB192tmM27O8Dw86ZOed//jN7dp75/8+e5ynH/1vAzP/DcdZ4c/oXMzOr3XSbFjMzs/+Ag4uZmdXOwcXMzGrn4GJmZrVzcDEzs9o5uFhLkBSS+ivL3ZJ6amr7dUn3nXrNf72fZSXz79BxnusrmW77ptDuQkm319NLs3o4uFirOArcI2nOme5IVeWq6Ml4BHg0Im49znOPAQsiYs0UurGQvA5p0pT892+njQ8uaxVjZJ3wpxqfaBx5SDpSfi6VtF3Su5JGJT0v6X5lPZhhSfMqzXRI2ivp65InbCJBZp+kPZIOSnq80u4OSVvJq6Mb+7O8tD8i6YXy2DPkxamvNo5OSjuzgH2SukoWgi1lv3sk3VjWa5e0syR5/ERZc+RsoBfoUtYd6ZLUI6m70v6IpLnl9pWkN8kL8C6RtKby+iZq45wr6X1lzZwRSV3N/rLMmvnWZXamvQwclLS+iW2uA64CDgOjwMaIaFcWQFsFPFnWm0vm25oHDEm6AngA+DkilkiaCXwsabCsvwi4JiK+q+5M0kVkbY3ryboag5LuioheSbcB3RGxt7pNRNwp6UhkMk4kbSJrc3wk6VJgoLyGL4GbImJMUgfwXETcWwLX4ohYWbbvOcn7MR94MCJ2Seosy+1keYCtyoSWFwA/RsQdpb22SbzPZsdwcLGWERG/lG/dq4HfJ7nZnigp0iV9C0wEh2GgOj21OSLGgW8kjZKZljuBBZVRURv5YfwnsLsxsBRLgG0R8VPZ5zvAzWRqk8nqAK7O1FEAnKfMUN0GvCFpPpme56wm2pzwfUTsKvc7y21/WZ5Fvr4dQH8Zdb0XETumsB+b5hxcrNW8SBYIe63y2BhlirecR6iWeD1auT9eWR7n2OO/MQ9SkN/mV0XEQPUJSUvJ9PCnywzghoj4o2G/LwFDEXG3ss7OthNs//f7UZxTuV/tt4B1EbGhsQFJi8jzOGslfRgRvc2+CJvefM7FWkpEHAY2c2wZ10PkNBRk7Y2pfKNfJmlGOQ9zOVlxcAB4Qll+AElXlqzCJ7MbuEXSHGVZ7eXA9ib7MkhO2VH2u7DcbeOfVOoPVdb/FZhdWT5ETttNBInLTrCfAeDhMipC0sWSLixTe79FxNtA30RbZs1wcLFW1A9U/2vsFfID/QBZ22Uqo4ofyMDwAbCijBo2kifsP5U0AmzgFKP9MgX3NJkC/QCwLyKaTX++GlhcTrJ/Aawoj68H1kna39CPIXIa7bNy8n0LcL6kz4GVZBbw4/V1ENgE7JQ0TNZTmQ1cC+xWVvt8FljbZP/NnBXZzMzq55GLmZnVzsHFzMxq5+BiZma1c3AxM7PaObiYmVntHFzMzKx2Di5mZla7vwAmhZVrinWf8gAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "What can we conclude from the plot? What are the downsides of the presented implementation? Is it correct to train and evaluate the model on the same samples? Is the selected performance metric correct? What other metrics can we use?\n",
        "\n",
        "## Modeling\n",
        "\n"
      ],
      "metadata": {
        "id": "u-7G-Ks52JVd"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### MODEL 1: RFC\n",
        "\n",
        "We started with RFC while performing RFE, so let us train this model with the selected features and evaluate in on test data.  \n",
        "\n",
        "Training the model (it's possible to vary the parameters):"
      ],
      "metadata": {
        "id": "dw1FY2Z7i5Mc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "rfc = RandomForestClassifier(n_estimators=100, random_state=np.random.seed(0))\n",
        "rfc.fit(data_train[selected_features_set], outcome_train)"
      ],
      "metadata": {
        "id": "26P08OHd2Tl_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "66fdfc20-a0ca-4387-9e5a-397cc0c29ee1"
      },
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "RandomForestClassifier()"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Prediction for the testing set:"
      ],
      "metadata": {
        "id": "RKz5JUeC2YAK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "outcome_pred_rfc = rfc.predict(data_test[selected_features_set])"
      ],
      "metadata": {
        "id": "Tvi88TlB2aR6"
      },
      "execution_count": 50,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Performance reporting on some key classification scores:"
      ],
      "metadata": {
        "id": "UFGcuGAi2c5_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print (classification_report(outcome_test, outcome_pred_rfc))"
      ],
      "metadata": {
        "id": "US3QUiD_2l5j",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6d8ee0c1-bf7d-4ebf-f407-67c8350d38d2"
      },
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.86      0.78      0.82       166\n",
            "           1       0.81      0.88      0.84       175\n",
            "\n",
            "    accuracy                           0.83       341\n",
            "   macro avg       0.83      0.83      0.83       341\n",
            "weighted avg       0.83      0.83      0.83       341\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Are precision and recall informative metrics? Why don't we report accuracy as a key metric? In which cases accuracy is not suitable scores?  \n",
        "  \n",
        "The other popular metric for classification is Receiver Operating Characteristic (ROC) curve and area under the curve (AUC). We will calculate true positive rates (TPR) and false positive rates (FPR) while varying classification threshold and plot the curve."
      ],
      "metadata": {
        "id": "FxY5UdRS2oA7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fpr, tpr, _ = roc_curve(outcome_test, rfc.predict_proba(data_test[selected_features_set])[:, 1])\n",
        "roc_auc = roc_auc_score(outcome_test, rfc.predict_proba(data_test[selected_features_set])[:, 1])\n",
        "\n",
        "plt.plot(fpr, tpr)\n",
        "plt.title('RFC ROC curve (AUC = {})'.format(roc_auc))\n",
        "plt.xlabel('FPR')\n",
        "plt.ylabel('TPR')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "yZdZn37z2qPQ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "outputId": "30bd37b6-c674-4ad6-f002-680630195499"
      },
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZwcdZ3/8ddnJplM7nNykMlpEkIurhjACIIghLibeCByreCyou7iva54rCK4v/VA96e7eOCCoHIFXDBqEF0FuQxkQkJIAoEh5JiQY5JMZibJTOb67B9VgU6ne6bDTHVNT72fj0c/pqu+36n6VFd1faq+3+oqc3dERCS5iuIOQERE4qVEICKScEoEIiIJp0QgIpJwSgQiIgmnRCAiknBKBNKtmVkfM1tvZmPijkWks8zsu2b28bjjSKdEkAMz22RmDWa238x2mNntZjYgpfx2M2sKyw+/PphSfpmZVYTjt5vZQ2b29izzSp3WXjP7o5lNT6tTbmZ3mtkeMztgZs+Y2d+k1TEz+6SZrQ3rVJnZfWY2u6s/n4hdAzzm7ttTR5rZ9WbmZnZahvG/TJ9IWHdKyvAFZvaYmdWbWbWZ/cXMFnVl4OE6+Fa4nvaE762d+p8ws1fNrC7cXt6eUjbEzO4ws13h6/q0/30kXI46M3vOzBanlH0pbdtsMLM2MxsRlmfafovDstPDbXBvOP37UpOymZ0TzrvWzDZlWKbU785+M/tDStksM3vYzHabWdYfNJnZVDNrTF+vZlZmZneF864xsztTym4ys5fD9fuimX0o7X//Nvxu7Dezp8xsRkpZHzP7DzN7LZzuD82s9zHElXU9AjcBXzKzkmzLGwt316uDF7AJOC98Pxp4Dvi3lPLbgW9k+d/PAruA9wH9gd7A3wLfyVL/9WkBfcPhJ1PKh4Xx/CyMpS9wKVAHXJRS7wfAK8A7gT5AP+By4LqIP6viLp7eOmB+2jgDNgJ7gJvTyq4HfplhOg5MCd9fFH5e/wAMJjggegfw0y6O/aPABqAcGAusBz6Wpe5pwAHg1HD5Pg5UH/48w/V9X7geJ4br9sMp/z8H6JUyrXpgTJZ5XQ/8Ocft90LgA8CgcN63Ab9PKZ8H/B1Bwt7U3ncnQ9nxwNXAYsDb+Rz/ADyevl7Dcd8L12Fv4OSUsq8D08N1expQA7wtLJsarv+3A72ALwKVKZ/f18JpDwPKgOXA13OJq6P1GNb5Iynf1e7wij2AQnilb8zAt4HfpQxn/CKFG+h+4APHMK8jpgUsBA6kDN8IrAWK0v7vC8DmcOObCrQC845hvsPCnc1r4ZfmwXD8VcATaXVTd6q3Az8CloVfgC8AO9I2/PcCa8L3RcB1BDuyPcASYFiWmMYDDYe/oCnjzwrHXx5OoySl7Pr0HUZqzOHnswX4fB62m6eAa1KGrwaWZ6n7QeCZlOH+YcxjwuHdwFtTyr8EPJ5lWvOAxkzrnzeS6JUdbb9Zpn0KUJ9h/HkcYyJIqTOFLIkAuCTcRo5Yr8D54bRzOvAAlgKfC99fy5Hf36Jwezo3HK4g5TsLXAZszTGudtdjOO7LwM+i3v6O5aWmoWNkZuUER0mVOVQ/AygFHniT8+pPcLSfOq93Ab9y97a06ksIdpzTgHOBKnd/5hhm9wuCI76ZwEjgP47hfy8D/g0YCHyfICG8M638rvD9J4D3EByBH0eQdG7OMt3ZwEZ3b0kbfyXwG4JlhuAMK1fHA+OA+3P9Bwua9va18xqf5V9nEpw9HvZcOC6Th4BiMzstbJb5e2A1QVJ9PZS097PS4vytmTUCTwOPEuzQ0p1JsH5/lTb+H8Pmn5Vm9v4sMUKQhNe1U57JnWGz0h/M7MRc/8nMBgE3EJxVpzud4GzrjrDZbYWZvSPLdPoCb02LO/2zTP8808vLzWxwDnHlsh5fAHL+HPIi7kxUCC+CI4/9BKfbDvwJGJJSfjvBEdi+8LU7HH85sOMY55U6rTbgVWBOSnklGZoXCBKOA/MJjjgyHnlmmeeYcF5DM5RdRcdnBD9PK/8GcFv4fiBBYpgQDr9AeOSVMu9m0o76Uz6/5Wnj+hGc1r8nHP4J8OuU8utp/4xgfvi+NA/bTSswPWV4ajhvy1DXCI7ym4EWjj4D+CXwP+HnOYXgjOpQhun0JjhQ+WyWmG4Fbk8bdwownKCZZGG4nc/P8L9zgL3AmRnKsp0RzCdovuxH0ASzI/W7E9bJeEZAcFDxhUzrFbgl/CyvDpf5kvA7MyLDdO4Afn/4cydoMjoAnA2UAP8abv9fTNl+nyRoFhpNkFhTz87ai6vd9RjWeRfBAU7e9mEdvXRGkLv3uPtAgo1nOjAirfwmdx8Svg6X7QFGmFmvY5zXTe4+hKAtuIHgKPaw3QQ7z3RjUsr3ZKmTzThgr7vXHGOch21NG74LeJ+Z9SHoG3nW3TeHZROABw4fTRMkhlZgVIbp1hDs+FK9l+ALtiwcvhO40MzKwuEWgh3D61I6+poJPhs4ts/nzdpP0LZ+2CBgv4d7gzRXAx8mOGMoAa4Afmtmx4XlnyTYFl4Gfg3cDVSlT8Tdm939IeB8S+v8NrN+BO39d6T9z7PuvsfdW9x9GcFn+r60/51CcLT7KXd/PJeFD6f9pLs3uPtBd/93gp31mR39n5mdRJBcsp2ZNhAknlvDZb6HYDucnzad7xAc6V98+HN39xcJzir/C9hO8F1ezxuf578BqwiO5J8CHiTYdnbmEFdH6xGCbXpfR59BPikRHCN3/wvBUfBNOVT/K3CIoCnkzcxrC/Ap4Pvh6S3A/xLsZNPX3cUEX4SXCM5Yys1sbo6z2goMM7MhGcoOEBzNAWBmozOFmhb3eoL+igs5slno8LwuTEmaQ9y91N23ZZjuGmBSWiK9EhgAbDGzHQQdqL3D+UDQ/j8xbTqTCBLENoLmhK1Ae80fRzCzy9OuqEl/ZWsaWseRTQAnkr1Z5STgt+7+kru3ufvvCXZSbwNw973ufrm7j3b3mQTf3faa/noBb0kb916CI/pH21tewrOWwwNmNoFgu7vR3X/Rwf925Ihpt+NsgvV4eD3/M/B+M3s2LF9D2naXPmxmXyfYBs9397ojKrrf7+6z3H04QefwRGBFWNbg7te6+1h3n0xw8LDSg+bYjuJqdz2GTuDIJsP4xX1KUggvju4sLiPYQZ4YDt9O9qsuPgfsJEgG/Xjj1P3bWeofNS2Ctt5Phe+HE+zsfkZw2lrKG1cNfTDlf/6T4OjxbIIjk1KC0+eMVw0BvyPYYQ8NYzwrHD+NIJmdFE7jxxzdNJSpo/wLwCMER24jUsZ/hmBHNCHls1zczme/hjeu9hhLcPZwfrjsh1/fJPiiQtD+XUtwJUtvgk7w+4F7UqZ5UVjnwwRH6UUEV5Dc0sXbzccIznjGEvSHrCP7VUNXEiTxyQQ7yncBBwmblgh26sOB4nD72Q3MDMumh+P6hst8BdAEnJI2jz8AN2SY90UEybUo/GzrgbNTPvNXgH/OEndRuF1cSJD8Swk77wn6rOanbH+fJ7iCZnhYbuH4GeE2VQr0Ccv6pa3jm8L1WBaWDyM4Y7wy/EwuIkhyI8LyLxJs/6OzxH1q+H9lBH1Nd6WUHV5fRtAXsZUgmeQSV7vrMWU9XBzHvizrthp3AIXwIsOVDwRXyvwqfH877Vx1QdDWXUGQPHYQ7HTflqXuUdMiuBJhW8qXZDxB08DecJorSNuZhhvhpwh2PgfD/7+XcOeRYb7DCJoMdoZfsP9JKfsywY5nK8FOJpdEMJ6g3fV3aeOLCDrZNhDscF4B/l87n90/AT8K319HuMNPq3Mcwan7rHD4bcAT4XK8Bvw3af0fwAKCS//2E+ycHgXe3cXbjRFcYbY3fH2blP6BcN5nptS9gSDJ1xMkkL9LqXtxuCwHCZosLkgpO4GgHbueoMlhBfDetFjGEpwVTckQ5+MEibGO4Ej1kpSyr4Xre3/qK6X87LA89fVoWDaTIJEfIDiq/hMwN+V/J2b4301ZPsvrOfry0TOB58OYKkjpuwindSgt7i+llD8Rfl57CfqZ+qeUnUXwnT8YbqeXt7OOj4grh/U4hqAJqiTbNON4He48EemWwn6GVQQdzNs7qi/SnZnZd4FX3P2HcceSSolARCTh1FksIpJwSgQiIgmnRCAiknDH+kOn2I0YMcInTpwYdxgiIgVl5cqVu929LFNZwSWCiRMnUlGR6RYqIiKSjZltzlampiERkYRTIhARSTglAhGRhFMiEBFJOCUCEZGEiywRmNltFjxke22WcjOzH5hZpZmtMbNToopFRESyi/KM4HaCOzxmcyHBE5umEjz4+kcRxiIiIllE9jsCd3/MzCa2U2UxwSMOHVhuZkPMbIzuMCkiSdfW5uw92MTOukZ21R1iR10jO+saeef0kcwpz/T8qM6J8wdlYznyEYdV4bijEoGZXUNw1sD48dkeBiUi0r00tbTR0NTKweYWDja1cvBQKwebWjjY3EpDUyv7D7Wwe/8hdtY2srPuEDvrgx3/rvpGmluPvjP08AF9elwiyJm730LwsGrmzp2r+2aL9DDuTlVNA9v2NcQdCgDu0NzaxsGmVhqaWzhwKNhxH2wKd+Th+yPK0nb2Dc2tGXfmmQws7cXoQaWMGlTKaZP7M2pQaTjch5Hh+LIBfSjpFU1rfpyJYBvBQ9MPKw/HiSTGzrpGWtqSd2zT0trGhh31rKmq5bmqfTy/rZZ9B5vjDisnJb2K6F9STL+SXvQtKaZ/STF9S4oZObD09eEjy3rRr6Q4fAXv+4bD/Ut6MXxACf1K4j0mj3PuS4Frzewe4DSgVv0DkiS/Xr2NT92zOu4wYlVcZEwbNZAFM0czu3wwE4f3x3J5tH0e9C4uen3nfXhn37d3Mb2Ke95V95ElAjO7m+B5piPMrIrg2ae9Adz9x8AyYCFQSfBs0A9HFYtIZ1TXH+L5bft4bmstz2+rZU1VLXsPHOr0dA+fCNz4nln06YE7l/aYweSyAcw8bhClvYvjDifxorxq6NIOyp3gweQisWlqCZoonqvax/NVtax9rZb9h1peL29oamVXfbDTN4OpIwfwjmlljBlc2iVHrmOH9OWSeboAQuJVEJ3FIl3J3fmfZ7fx8+WbeeG1Oppa2wAY0q83s8cOZurIAa/X7VVcxPTRA5lTPoSZxw2ifx99ZaTn0VYtPVJLaxt3r9jK+tfqjip7eWc9FZtrOGHMIK6aP5E55YM5sXwI5UP7Yt2lgVokj5QIpMd5dksNX35gLS9sr2N4/xKKio7cufcrKebf3zebD84dd1SZSBIpEUiP0BxejnjXM1u4+5ktjBpYyo+vOIULZo7WUb5IB5QIJDa76hp5rbbxTf1va5vz6u4DrKnax5qqWtZvr6OppY3iIuPq+ZP49LumMUDt+SI50TdF8qLmQBNrttWyZuu+4G/VPnbWdf4SzP4lxcwaO5grz5jAnPIhnDphKMcN6dsFEYskhxKBdLn6xmae31bL81XBNfdrtu1j6943bh0wuaw/Z0wezpzyIUwY3o+iN9l0M25YXyaNGECx2vlFOkWJQLrUrU+8yjd+tx4PfyxVPrQvc8oHc/lpE5gzdjCzygczqLR3vEGKyBGUCCRnr1Tv54/rd/LIi7uobch8X5jX9jUwsE8vvn/pycwZO5jhA/rkOUoROVZKBNKhRzbs4sbfrmdj9QEAZowZxITh/TLWnTC8H/MmDeec40fmM0QR6QQlAunQsjXb2VHbyA2LZ3LuCaMYq85YkR5FiUA61OrOkL69+dAZE+MORUQikKxbHsox2VnXyNd/s44HV22jfFjmpiARKXw6I5AjVO6q5/drd/DH9Tt5rqoWM7hs3nj+5YLpcYcmIhFRIhAg+MHXNx96kXsrgsdInzhuCJ+/4HgumDmaKSl34xSRnkeJIOHcnftWVvHvy16gvrGFj541mb9/+yRGDSqNOzQRyRMlggR7aWc9X3lgLc9s2svcCUP5xntnMX30oLjDEpE8UyJIoIamVn7w55f56WMbGVDai2+/fw4XnVquWzKLJJQSQcI8X1XLx365km37GvjAqeV8ceEJDOtfEndYIhIjJYIE2VnXyNV3rKB3cRFLPnoG8yYNizskEekGlAh6uF31jTxVuYeDTa3cu2IL+w+18MA/zuf40QPjDk1Eugklgh5oe20DD6zaxh/X72T11n2v3wm0d7Hxn5eeoiQgIkdQIuiBPnPvapZv3Muc8sF89rxpvPOEkYwY0Ie+JcW6BbSIHEWJoAdqaG7jrGll/Pzv58UdiogUAN1rqIdpbG5lz/5D9NKloCKSIyWCHsTd+cqDa6mqaeCSt46LOxwRKRBqGipwbW3OfSu38uvVr3GwqZXVW/fxyXOncv7M0XGHJiIFQomggO2qa+Tjdz7Lys01TBs1gKH9SrjmrMl8+typcYcmIgVEiaBAHWpp5aO/XMmL2+v5zkXBLSLM1C8gIsdOiaAAuTv/+uBaVm3Zx48uP4ULZ4+JOyQRKWDqLC5Aj75UzZKKKq49Z4qSgIh0WqSJwMwWmNkGM6s0s+sylI83s0fMbJWZrTGzhVHG0xO0tjnfeuhFJgzvxyfVFyAiXSCyRGBmxcDNwIXADOBSM5uRVu0rwBJ3Pxm4BPhhVPH0FA+u2saLO+r55/OPp6SXTuhEpPOi3JPMAyrdfaO7NwH3AIvT6jhw+Ekog4HXIoynR/ivRyqZPXYw71aTkIh0kSgTwVhga8pwVTgu1fXAFWZWBSwDPpFpQmZ2jZlVmFlFdXV1FLEWjJ11jZw+eZgeIiMiXSbutoVLgdvdvRxYCPzCzI6Kyd1vcfe57j63rKws70GKiPRkUSaCbUDqfQ7Kw3GprgaWALj7X4FSYESEMYmISJooE8EKYKqZTTKzEoLO4KVpdbYA5wKY2QkEiSDZbT8iInkWWSJw9xbgWuBh4AWCq4PWmdkNZrYorPY54CNm9hxwN3CV++HHqIiISD5E+stid19G0AmcOu6rKe/XA/OjjEFERNoXd2exHIMXd9TR2NxKae/iuEMRkR5EiaBA1Bxo4iM/r2DEgD5ccfqEuMMRkR5EN53rRppa2vjp4xv52ZObaGppPbKstY22Nrj3o6czalBpTBGKSE+kRNBNLN+4h688uJbKXfs55/gyJgzvf1Sd804Yxcnjh8YQnYj0ZEoEMXiycje/+Ovm14f3H2rhicrdjBvWl59d9VbOmT4yxuhEJGmUCGLwwKpt/O8LO3lL2QAAzODac6bwT+dMoW+JOoJFJL+UCGIyalApD3/mrLjDEBHRVUMiIkmnRCAiknBKBCIiCadEICKScEoEIiIJp0QQg8bm1o4riYjkiRJBnv3lpWqWPb+d0yYPizsUERFAiSCvKnfV84m7nmXaqIHcuHhW3OGIiABKBHnz2EvVvPeHT9G7uIiffmgu/fvot3wi0j0oEUTM3bn1iVe56mfPMHZIXx78p/mMG9Yv7rBERF6nw9IIHWpp5SsPrOW+lVVcMHMU37v4JJ0JiEi3o71SRKrrD/GxX65k5eYaPnnuVD597lSKiizusEREjqJEEJErb3uGjbv3c/Nlp/DuOWPiDkdEJCv1EURk054DXH7aBCUBEen2lAgi4g5qCRKRQqBEEIHq+kM0NLdSNrBP3KGIiHRIiSACKzfvBeDUCfr1sIh0f0oEEajYVEOfXkXMGjso7lBERDqkRBCBFZtrOLF8CH166fnDItL9KRF0sYamVtZtq2XuxKFxhyIikhMlgi72XNU+WtpciUBECoYSQRer2BR0FJ8yXolARAqDEkEXe2ZTDdNGDWBIv5K4QxERyUmkicDMFpjZBjOrNLPrstS52MzWm9k6M7sryniitrOukScrd3PO9JFxhyIikrPI7jVkZsXAzcC7gCpghZktdff1KXWmAl8E5rt7jZkV9B703hVbaW1zLps3Pu5QRERyFuUZwTyg0t03unsTcA+wOK3OR4Cb3b0GwN13RRhPpFpa27j7mS2cOXUEE4b3jzscEZGcRZkIxgJbU4arwnGppgHTzOxJM1tuZgsyTcjMrjGzCjOrqK6ujijcznl0QzXbaxu5/LQJcYciInJM4u4s7gVMBc4GLgV+amZD0iu5+y3uPtfd55aVleU5xNzct3IrZQP7cO4JBd26JSIJFGUi2AaMSxkuD8elqgKWunuzu78KvESQGArKgUMtPLqhmoWzRtO7OO7cKiJybKLca60ApprZJDMrAS4BlqbVeZDgbAAzG0HQVLQxwpgi8eiGag61tLFglp49ICKFJ7JE4O4twLXAw8ALwBJ3X2dmN5jZorDaw8AeM1sPPAJ83t33RBVTVB5au53h/UuYN0l3GxWRwhPpoyrdfRmwLG3cV1PeO/DZ8FWQGptbeeTFXSw66TiK9SQaESlAatDupCde3s2BplY1C4lIwVIi6KSNu/cDcMr4oy52EhEpCEoEXaTI1CwkIoVJiUBEJOGUCEREEk6JQEQk4Y45EZhZkZldHkUwhai+sQVQH4GIFK6sicDMBpnZF83sv8zsfAt8guCXvxfnL8Tuq76xmTuf3sL8KcPpW6IH1YtIYWrvB2W/AGqAvwL/AHwJMOA97r46D7F1ez/5y0b2HmjiugUnxB2KiMib1l4imOzuswHM7L+B7cB4d2/MS2Td3K76Rv77iY0sOvE4ZpcPjjscEZE3rb0+gubDb9y9FahSEnjDr1e9RmNzG58+r+BulioicoT2zghONLM6guYggL4pw+7ugyKPrht7aO12Zo0dxOSyAXGHIiLSKVkTgbur9zOL7bUNPLtlH5+/4Pi4QxER6bSsicDMSoGPAVOANcBt4a2lE+/htTsAWDBrdMyRiIh0Xnt9BHcAc4HngYXAd/MSUQF4aO0Opo0awFvULCQiPUB7fQQzUq4auhV4Jj8hdW+79x9ixaa9XPtOdRKLSM+Q61VDahIK/WHdTtocLlSzkIj0EO2dEZwUXiUEwZVCumqI4GqhicP7MX30wLhDERHpEu0lgufc/eS8RVIAag8289dX9vAPZ07GdG8hEekh2msa8rxFUSD++MJOWtqchbPVLCQiPUd7ZwQjzSzrQ+Xd/XsRxNOt/X7tdsYO6cvssbqlhIj0HO0lgmJgAG/8sjjxnt64l0UnHadmIRHpUdpLBNvd/Ya8RVIAWtqc/n3a+8hERApPe30EOuwVEUmA9hLBuXmLQkREYpM1Ebj73nwGIiIi8dDD60VEEk6JQEQk4ZQIREQSTolARCThlAhERBIu0kRgZgvMbIOZVZrZde3Ue7+ZuZnNjTIeERE5WmSJwMyKgZuBC4EZwKVmNiNDvYHAp4Cno4pFRESyi/KMYB5Q6e4b3b0JuAdYnKHejcC3gMYIY+m0xuZWWtt0Q1YR6XmiTARjga0pw1XhuNeZ2SnAOHf/XXsTMrNrzKzCzCqqq6u7PtIcfP9PL9PU2sbZ08pimb+ISFRi6yw2syLge8DnOqrr7re4+1x3n1tWlv8d8dpttdzy2EY+cGo5b5syIu/zFxGJUpSJYBswLmW4PBx32EBgFvComW0CTgeWdrcO4+bWNv7l/jUM61/CV959VBeHiEjBizIRrACmmtkkMysBLgGWHi5091p3H+HuE919IrAcWOTuFRHGdMxueWwj67fXcePiWQzu1zvucEREulxkicDdW4BrgYeBF4Al7r7OzG4ws0VRzbcrVe7az/f/9DILZ49mwSw9nlJEeqZIn7Li7suAZWnjvpql7tlRxvJm3P7UqxSbcf2imXGHIiISGf2yuB2NzW0M61/CyIGlcYciIhIZJYJ2tLS2xR2CiEjklAja8VxVLcePHhh3GCIikVIiyGLr3oO8uvsAZ07V7wZEpGdTIsjiicrdAEoEItLjKRFk8cTLuxk9qJS3lA2IOxQRkUgpEWTQ2uY8Ubmbt08dgZnFHY6ISKSUCDJYu62W2oZmNQuJSCIoEWSwcnMNAGdMHh5zJCIi0VMiyKA5/P3AgNJIf3gtItItKBGIiCScEoGISMIpEYiIJJwSQZqaA01U7tofdxgiInmj3lDA3VlSsZUHVm1jxaYaWtucKSMH0KdXcdyhiYhETokAeOqVPXzhV88zdeQAPv6Ot/CuGaOYPXYwRUX6MZmI9HxKBMCdT29maL/e/OYTb6e0t84CRCRZEt9HsKuukT+s28lFp5YrCYhIIiU+ESyp2EpLm3PpvPFxhyIiEotEJwJ3554VW5k/ZTiTdZdREUmoRCeCLXsPUlXTwIJZY+IORUQkNolOBBWbgpvLvXXi0JgjERGJT7ITweYaBpb2YtpIPZdYRJIr2Ylg015OnTBUvxcQkURLbCLYd7CJl3ftZ+4ENQuJSLIlNhE8uyXoHzh1wrCYIxERiVdiE8GKTTX0KjJOGjck7lBERGKV2ESwess+Zh43iL4l+jWxiCRbYhPBwaYWhvYviTsMEZHYJTYRiIhIINJEYGYLzGyDmVWa2XUZyj9rZuvNbI2Z/cnMJkQZj4iIHC2yRGBmxcDNwIXADOBSM5uRVm0VMNfd5wD3A9+OKh4REcksyjOCeUClu2909ybgHmBxagV3f8TdD4aDy4HyCON5XWNzK5v3HmRoP/URiIhEmQjGAltThqvCcdlcDTyUqcDMrjGzCjOrqK6u7nRgv1+7g30Hm7no1LzkHRGRbq1bdBab2RXAXOA7mcrd/RZ3n+vuc8vKyjo9vzuf3sykEf05Y/LwTk9LRKTQRZkItgHjUobLw3FHMLPzgC8Di9z9UITxALBhRz0rNtVw2bzxuseQiAjRJoIVwFQzm2RmJcAlwNLUCmZ2MvATgiSwK8JYXrekYislxUW8X81CIiJAhInA3VuAa4GHgReAJe6+zsxuMLNFYbXvAAOA+8xstZktzTK5LrN66z5OGjeEYfoxmYgIAL2inLi7LwOWpY37asr786Kcf7q2NufF7XXqJBYRSdEtOovzpaqmgQNNrUwfMyjuUEREuo1EJYIXdtQBMH20nkgmInJYohLBi9vrMYNpo5QIREQOS1Yi2FHHhGH96N8n0q4REZGCkrBEUM8J6h8QETlCYhLBwaYWNu05wPTRSgQiIqkSkwhe2rkfd5g+Rv0DIiKpEpMIXtweXDF0gs4IRESOkJhEUNvQDMCIgfpFsa5tNZQAAAahSURBVIhIqsQkAhERyUyJQEQk4ZQIREQSTolARCThlAhERBJOiUBEJOGUCEREEk6JQEQk4ZQIREQSTolARCThlAhERBJOiUBEJOGUCEREEk6JQEQk4ZQIREQSTolARCThlAhERBJOiUBEJOGUCEREEk6JQEQk4ZQIREQSTolARCThIk0EZrbAzDaYWaWZXZehvI+Z3RuWP21mE6OMR0REjhZZIjCzYuBm4EJgBnCpmc1Iq3Y1UOPuU4D/AL4VVTwiIpJZlGcE84BKd9/o7k3APcDitDqLgTvC9/cD55qZRRiTiIikiTIRjAW2pgxXheMy1nH3FqAWGJ4+ITO7xswqzKyiurr6TQUzaUR/Fs4eTZHyjIjIEXrFHUAu3P0W4BaAuXPn+puZxvkzR3P+zNFdGpeISE8Q5RnBNmBcynB5OC5jHTPrBQwG9kQYk4iIpIkyEawApprZJDMrAS4BlqbVWQpcGb6/CPizu7+pI34REXlzImsacvcWM7sWeBgoBm5z93VmdgNQ4e5LgVuBX5hZJbCXIFmIiEgeRdpH4O7LgGVp476a8r4R+ECUMYiISPv0y2IRkYRTIhARSTglAhGRhFMiEBFJOCu0qzXNrBrY/Cb/fQSwuwvDKQRa5mTQMidDZ5Z5gruXZSoouETQGWZW4e5z444jn7TMyaBlToaolllNQyIiCadEICKScElLBLfEHUAMtMzJoGVOhkiWOVF9BCIicrSknRGIiEgaJQIRkYTrkYnAzBaY2QYzqzSz6zKU9zGze8Pyp81sYv6j7Fo5LPNnzWy9ma0xsz+Z2YQ44uxKHS1zSr33m5mbWcFfapjLMpvZxeG6Xmdmd+U7xq6Ww7Y93sweMbNV4fa9MI44u4qZ3WZmu8xsbZZyM7MfhJ/HGjM7pdMzdfce9SK45fUrwGSgBHgOmJFW5x+BH4fvLwHujTvuPCzzOUC/8P3Hk7DMYb2BwGPAcmBu3HHnYT1PBVYBQ8PhkXHHnYdlvgX4ePh+BrAp7rg7ucxnAacAa7OULwQeAgw4HXi6s/PsiWcE84BKd9/o7k3APcDitDqLgTvC9/cD55oV9MOMO1xmd3/E3Q+Gg8sJnhhXyHJZzwA3At8CGvMZXERyWeaPADe7ew2Au+/Kc4xdLZdldmBQ+H4w8Foe4+ty7v4YwfNZslkM/NwDy4EhZjamM/PsiYlgLLA1ZbgqHJexjru3ALXA8LxEF41cljnV1QRHFIWsw2UOT5nHufvv8hlYhHJZz9OAaWb2pJktN7MFeYsuGrks8/XAFWZWRfD8k0/kJ7TYHOv3vUMF8fB66TpmdgUwF3hH3LFEycyKgO8BV8UcSr71ImgeOpvgrO8xM5vt7vtijSpalwK3u/t3zewMgqceznL3trgDKxQ98YxgGzAuZbg8HJexjpn1Ijid3JOX6KKRyzJjZucBXwYWufuhPMUWlY6WeSAwC3jUzDYRtKUuLfAO41zWcxWw1N2b3f1V4CWCxFCoclnmq4ElAO7+V6CU4OZsPVVO3/dj0RMTwQpgqplNMrMSgs7gpWl1lgJXhu8vAv7sYS9Mgepwmc3sZOAnBEmg0NuNoYNldvdadx/h7hPdfSJBv8gid6+IJ9wukcu2/SDB2QBmNoKgqWhjPoPsYrks8xbgXAAzO4EgEVTnNcr8Wgp8KLx66HSg1t23d2aCPa5pyN1bzOxa4GGCKw5uc/d1ZnYDUOHuS4FbCU4fKwk6ZS6JL+LOy3GZvwMMAO4L+8W3uPui2ILupByXuUfJcZkfBs43s/VAK/B5dy/Ys90cl/lzwE/N7DMEHcdXFfKBnZndTZDMR4T9Hl8DegO4+48J+kEWApXAQeDDnZ5nAX9eIiLSBXpi05CIiBwDJQIRkYRTIhARSTglAhGRhFMiEBFJOCUCkRyZWauZrU55TTSzs82sNhx+wcy+FtZNHf+imd0Ud/wi2fS43xGIRKjB3U9KHRHewvxxd/8bM+sPrDaz34TFh8f3BVaZ2QPu/mR+QxbpmM4IRLqIux8AVgJT0sY3AKvp5I3BRKKiRCCSu74pzUIPpBea2XCCexqtSxs/lOB+P4/lJ0yRY6OmIZHcHdU0FDrTzFYBbcA3w1sgnB2Of44gCfx/d9+Rx1hFcqZEINJ5j7v732Qbb2aTgOVmtsTdV+c7OJGOqGlIJGLh7aC/CXwh7lhEMlEiEMmPHwNnhVcZiXQruvuoiEjC6YxARCThlAhERBJOiUBEJOGUCEREEk6JQEQk4ZQIREQSTolARCTh/g9CaKF8sQ3PBwAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "In which cases this metric does not give a correct representation of the model performance? Is AUC always the best metric?\n",
        "Most cetainly not! Especially for unbalanced datasets, AUC can be meaningless. \n",
        "http://www.davidsbatista.net/blog/2018/08/19/NLP_Metrics/\n",
        "\"With imbalanced classes, it’s easy to get a high accuracy without actually making useful predictions. So, accuracy as an evaluation metrics makes sense only if the class labels are uniformly distributed\"    \n",
        "\n",
        "To have a better understanding of model behaviour, we will plot a confusion matrix."
      ],
      "metadata": {
        "id": "QneA8cJZ2sP1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "cm = confusion_matrix(outcome_test, outcome_pred_rfc)\n",
        "f = sns.heatmap(cm, annot=True)\n",
        "plt.title('Confusion matrix for RFC')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "HzW4Fgn02xoP",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        },
        "outputId": "05b9345d-3aa8-4664-b370-90181823646f"
      },
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAEICAYAAABhxi57AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAbu0lEQVR4nO3deXxU9dXH8c8JIcgeIYosorigVas+yoOK1gWtFZeCdV+qIm1ebrVWW5S64IaPrXVfi7K5IahFcXtcEAoqoriBIGAeFGTfBAEpIcl5/piLjoFkJpNJfszl+/Z1X2Tu/c3vnknimZNzfzNj7o6IiNS/vNABiIhsrZSARUQCUQIWEQlECVhEJBAlYBGRQJSARUQCUQLOIWbW2MxeMrNVZvZsLeY5x8zeyGZsoZjZL8xsZob33cPMPjWz1WZ2ebZjE0lFCbgOmNnZZjbZzNaY2UIze83MDsvC1KcCbYDW7n5appO4+1PufmwW4qlTZuZmtlt1Y9x9grvvkeEp+gJj3b25u9+X4Rw/MLMbzWxD9HNfaWbvmdkhScePNLOK6PjG7aWk453N7FkzWxY9yU4xsyvNrEFtY5MtkxJwlpnZlcA9wG0kkmVH4CGgZxam3wmY5e5lWZgr55lZfi2n2AmYluVzj3D3ZkARMBao/JfKAndvlrSdFM23KzAJ+Ab4ubu3BE4DugDNM4lRcoC7a8vSBrQE1gCnVTOmEYkEvSDa7gEaRceOBOYBVwFLgIVA7+jYTUApsCE6Rx/gRuDJpLl3BhzIj25fAMwGVgNfAeck7X8n6X7dgA+BVdG/3ZKOjQNuAd6N5nkDKKrisW2Mv29S/L2A44FZwArgr0njuwITgZXR2AeAgujY+OixrI0e7xlJ818NLAKe2Lgvus+u0TkOiG63A5YCR24m1reBcuA/0fydo5/f49F95gDXAXlJ37N3gbuB5cCtm5mz8s9jr+gxbJf8/anie/ck8Ero32Ft9bupAs6uQ4BtgFHVjLkWOBjYH9iPRBK6Lun4DiQSQXsSSfZBM9vW3fuTqKpHeKJyGlRdIGbWFLgP6OHuzUkk2U83M64V8Eo0tjVwF/CKmbVOGnY20BvYHigA/lzNqXcg8T1oD9wAPAqcCxwI/AK43sw6RWPLgT+RqBYPAY4GLgFw98OjMftFj3dE0vytSFSvxckndvf/I5GcnzSzJsAQYJi7j6scpLt3ByYAl0XzzwLuJ/G93wU4AjgvetwbHUTiCa0NMKCa7wFmVhDdfznwbXVjI8cAz6UxTmJECTi7WgPLvPoWwTnAze6+xN2Xkqhsf5t0fEN0fIO7v0qiOsu0x1kB7GNmjd19obtv7s/tE4Av3f0Jdy9z9+HADOCkpDFD3H2Wu68DRpJ48qjKBmCAu28AniGRXO9199XR+aeTeOLB3T9y9/ej834N/JNE4kv1mPq7+/oonp9w90eBEhJ/zrcl8YSXUtRnPRPoF8X6NXAnP/3ZLHD3+6N4Nzl35HQzWwmsA34PnFrp96Fd1B/euJ0e7W9N4q8A2YooAWfXcqAoRW+yHYk/bzeaE+37YY5K/8N+DzSraSDuvpbEn+0XAQvN7BUz2zONeDbG1D7p9qIaxLPc3cujrzcmqcVJx9dtvH900ellM1tkZt+RqPCLqpkbYKm7/yfFmEeBfYD73X19irEbFQEN2fRnk/x9+CaNeUa6eyGJKvlzEpV/sgXuXpi0jYz2LyfxhCFbESXg7JoIrCfR96zKAhJ/Pm/UMdqXibVAk6TbOyQfdPfX3f2XJP7HnkEiMaWKZ2NM8zOMqSYeJhHX7u7eAvgrYCnuU+3b95lZMxJ99UHAjVGLJR3LSFTvlX82yd+HtN860N2XkWiR3Ghm6STWt4BT0p1f4kEJOIvcfRWJvueDZtbLzJqYWUMz62Fmf4+GDQeuM7PtzKwoGv9khqf8FDjczDqaWUug38YDZtbGzHpGveD1JFoZFZuZ41Wgc7R0Lt/MziBx8ejlDGOqiebAd8CaqDq/uNLxxST6sTVxLzDZ3X9Horf9SDp3iqr2kcAAM2tuZjsBV5L5zwZ3nwm8TuKiZCr9gW5mdoeZ7QBgZruZ2ZNmVphpDLJlUwLOMne/k8T/uNeRuJr+DXAZ8EI05FZgMjAFmAp8HO3L5FxvAiOiuT7ip0kzL4pjAYmVAUewaYLD3ZcDJ5JYebGcRLI4Marg6tqfSVzgW02iOh9R6fiNwLBKvdIqmVlP4Dh+fJxXAgeY2TlpxvMHEn9VzAbeAZ4GBqd536rcARSb2fbVDYouIB5CYiXLNDNbBTxP4ndldS1jkC2UuesN2UVEQlAFLCISiBKwiEggSsAiIoEoAYuIBFLbNzNJad3j/XSVTzZxUN9xoUOQLdCURRNTrQNPacOy2WnnnIZFu9T6fLWhClhEJJA6r4BFROpVRXnqMVsIJWARiZfy3Hm7bCVgEYkV98294n7LpAQsIvFSoQQsIhKGKmARkUB0EU5EJBBVwCIiYbhWQYiIBKKLcCIigagFISISiC7CiYgEogpYRCQQXYQTEQlEF+FERMJwVw9YRCQM9YBFRAJRC0JEJJAcqoD1kUQiEi/lG9LfUjCzwWa2xMw+38yxq8zMzawoum1mdp+ZlZjZFDM7INX8SsAiEi8VFelvqQ0Fjqu808x2BI4F5ibt7gHsHm3FwMOpJlcCFpF48Yr0t1RTuY8HVmzm0N1AXyD5E5h7Ao97wvtAoZm1rW5+JWARiZcaVMBmVmxmk5O24lTTm1lPYL67f1bpUHvgm6Tb86J9VdJFOBGJlxqsgnD3gcDAdMebWRPgryTaD7WmBCwiseJpXFyrhV2BTsBnZgbQAfjYzLoC84Edk8Z2iPZVSS0IEYmXLPaAN5nafaq7b+/uO7v7ziTaDAe4+yJgNHBetBriYGCVuy+sbj4lYBGJlyyugjCz4cBEYA8zm2dmfaoZ/iowGygBHgUuSTW/WhAiEi9ZfCGGu5+V4vjOSV87cGlN5lcCFpF40UuRRUQCyaGXIisBi0i8lOkN2UVEwlAFLCISiHrAIiKBqAIWEQlEFbCISCCqgEVEAtEqCBGRQNxTj9lCKAGLSLyoBywiEogSsIhIILoIJyISSHl56AjSpgQsIvGiFoSISCBKwCIigagHLCIShldoHbCISBhqQYiIBKJVECIigagCFhEJRAk4Hvq/9BHjSxbRqmkjni8+ZpPjY2cu4KHx0zGM/DzjL8fuy3/tWFSrc65aV0rfUR+wYOVa2hU25Y6Tu9KicQGvfD6XoRNn4Q5NCvK5tsf+7NGmsFbnkvpX0KiAIS88TEFBQxrkN+Ctl8fy0B2PMfSFh2nSrAkArYq25fNPpnNF72sCR5uj9GY88fDr/XbizC67cN1LH232+EGdtufIzm0xM2YtXkXfUZN44aJj05r7wzlLGT1lDrec1OUn+we/N5ODdt6OC7sdxuD3ZjJ44iyu6L4P7QubMujcw2nRuIB3ShZxy6uf8GTvo2r9GKV+la4v5XenXMa679eRn9+AYaP/yTtjJnJBr4t/GHPXY7cx9vUJAaPMcTlUAeeFDmBLdmDHIlo0LqjyeJOCfMwMgHUbyjDsh2NDJ87i7MFvc9qjb/HQv6enfc5xsxZy0s87AnDSzzsyduYCAPbv0PqHWPZt34rF362r8eORLcO67xM/u/yG+eTn5+NJFVvTZk3oetiBvP3av0OFl/sqPP0tBTMbbGZLzOzzpH13mNkMM5tiZqPMrDDpWD8zKzGzmWb2q1Tzp6yAzWxPoCfQPto1Hxjt7l+kjH4r8PaM+dw3bhor1q7n/jO6AfDe7MXMXbGGp3ofhQN/HDmRj+Yu48COqdsTy9euZ7vmjQEoarYNy9eu32TMqM++5rBd22T1cUj9ycvL45k3htCxUweeGfI8Uz/58Qm6e48jmPTOZNau+T5ghDkuu6sghgIPAI8n7XsT6OfuZWb2N6AfcLWZ7QWcCewNtAPeMrPO7l5lQNUmYDO7GjgLeAb4INrdARhuZs+4++1V3K8YKAa4/4Lj6HPU/ikfZa7qvmd7uu/Zno/mLuOhf0/nn+f8gvdnL2HiV0s447G3gUR1PHfFGg7sWMS5Q8ZSWlbBug1lrFpXyumPjgHgiu770K1SUjUzzH56vg+/XsoLn85hyHmH18vjk+yrqKjg9GPOp3mLZtw95HZ223MXSmbMBqDHyb/kX0+NDhxhbvMstiDcfbyZ7Vxp3xtJN98HTo2+7gk84+7rga/MrAToCkysav5UFXAfYG9335C808zuAqYBm03A7j4QGAiw7vF+udMRr4UDOxYxb+Vavv1+PY7Tp1tnTj1gl03GbezbVtUDbt20EUtXr2O75o1ZunodrZo0+uHYrMWruOmVj3nwzG4UJu2X3LT6uzV8+O7HHHrUwZTMmE1hq5bss/9euvhWWzV4JVxysRgZGOWvdF0IjIi+bk8iIW80jx87B5uVqgdcQaKUrqxtdGyrNnfFmh/6d18s/JbSsgoKGxdwyC5teOGzOXxfmvhsqsXfrWPF2v+kNecRndvy0tS5ALw0dS5Hdm4LwMJV33PV8+9za88u7NS6eR08GqkP27YupHmLZgA02qYRhxz+33xVMgeAX57YnfFvvUvp+tKQIeY+r0h7c/eB7t4laUs7+ZrZtUAZ8FSmoaaqgK8AxpjZl8A30b6OwG7AZZmeNFdcM+oDJs9Zysp1pRx736tcfPhelJUnnndOO3AXxsyYz0tT55Kfl8c2DRvw9990xczotksbvlq2mvOGjgMSF+sG9OxCq6apz3nhIZ3pO+oDRn36Ne1aNuHvvzkIgIETvmDlulJue+1TAPLzjKf7dK+Txy11p2j71tx63w00aJBHXp7x+ui3Gf/muwAc1+sYBt//ROAIY6Ae3gvCzC4ATgSO9h+vos4Hdkwa1iHaV/U8nmLNnJnlkehjJF+E+7C6xnKyraUFITVzUN9xoUOQLdCURRMt9ajqrb3hzLRzTtObn0l5vqgH/LK77xPdPg64CzjC3ZcmjdsbeJpEvmwHjAF2z/giHIC7V/DTvoaIyJYri29HaWbDgSOBIjObB/QnseqhEfBmtAz1fXe/yN2nmdlIYDqJ1sSlqQpVvRBDROIliy0Idz9rM7sHVTN+ADAg3fmVgEUkVrK5DK2uKQGLSLzoDdlFRAJRAhYRCURvyC4iEoY+E05EJBQlYBGRQLQKQkQkEFXAIiKBKAGLiITh5WpBiIiEoQpYRCQMLUMTEQlFCVhEJJDcaQErAYtIvHhZ7mRgJWARiZfcyb9KwCISL7oIJyISiipgEZEwVAGLiISiClhEJAwvCx1B+pSARSRWsvip9HVOCVhE4iWHEnBe6ABERLLJK9LfUjGzwWa2xMw+T9rXyszeNLMvo3+3jfabmd1nZiVmNsXMDkg1vxKwiMRKNhMwMBQ4rtK+a4Ax7r47MCa6DdAD2D3aioGHU02uBCwiseLllvaWci738cCKSrt7AsOir4cBvZL2P+4J7wOFZta2uvnVAxaRWKmHi3Bt3H1h9PUioE30dXvgm6Rx86J9C6mCKmARiRWvsLQ3Mys2s8lJW3GNzuXuQMav/FAFLCKxUpMK2N0HAgNreIrFZtbW3RdGLYYl0f75wI5J4zpE+6qkClhEYsXd0t4yNBo4P/r6fODFpP3nRashDgZWJbUqNksVsIjESjZ7wGY2HDgSKDKzeUB/4HZgpJn1AeYAp0fDXwWOB0qA74HeqeZXAhaRWKlIY3VDutz9rCoOHb2ZsQ5cWpP5lYBFJFa8InsJuK4pAYtIrCgBi4gE4rnzdsBKwCISL6qARUQCqcXysnqnBCwisVKexVUQdU0JWERiRRWwiEgg6gGLiASiVRAiIoGoAhYRCaS8InfeY0wJWERiRS0IEZFAKrQKQkQkDC1DExEJRC2IJM1/93hdn0Jy0LoFE0KHIDGlFoSISCBaBSEiEkgOdSCUgEUkXtSCEBEJRKsgREQCyeKHItc5JWARiRVHFbCISBBlakGIiISRSxVw7iyYExFJQ0UNtlTM7E9mNs3MPjez4Wa2jZl1MrNJZlZiZiPMrCDTWJWARSRWHEt7q46ZtQcuB7q4+z5AA+BM4G/A3e6+G/At0CfTWJWARSRWslkBk2jTNjazfKAJsBDoDjwXHR8G9Mo0ViVgEYmVciztzcyKzWxy0la8cR53nw/8A5hLIvGuAj4CVrp7WTRsHtA+01h1EU5EYqUmn0jk7gOBgZs7ZmbbAj2BTsBK4FnguNpH+CMlYBGJlYrsrYI4BvjK3ZcCmNm/gEOBQjPLj6rgDsD8TE+gFoSIxIrXYEthLnCwmTUxMwOOBqYDY4FTozHnAy9mGqsSsIjESrYuwrn7JBIX2z4GppLIlwOBq4ErzawEaA0MyjRWtSBEJFYqLHsvxHD3/kD/SrtnA12zMb8SsIjESnnoAGpACVhEYqUmqyBCUwIWkVjJ4iqIOqcELCKxoo8kEhEJRC0IEZFA9IkYIiKBlKsCFhEJQxWwiEggSsAiIoHk0EfCKQGLSLyoAhYRCUQvRRYRCUTrgEVEAlELQkQkECVgEZFA9F4QIiKBqAcsIhKIVkGIiARSkUNNCCVgEYkVXYQTEQkkd+pfJWARiRlVwCIigZRZ7tTASsAiEiu5k34hL3QAIiLZVFGDLRUzKzSz58xshpl9YWaHmFkrM3vTzL6M/t0201iVgEUkVirwtLc03Av8r7vvCewHfAFcA4xx992BMdHtjCgBi0iseA226phZS+BwYBCAu5e6+0qgJzAsGjYM6JVprErAIhIrNWlBmFmxmU1O2oqTpuoELAWGmNknZvaYmTUF2rj7wmjMIqBNprHqIpyIxEp5DS7DuftAYGAVh/OBA4A/uPskM7uXSu0Gd3ezzJddqAIWkVjJ4kW4ecA8d58U3X6OREJebGZtAaJ/l2QaqxKwiMSK1+C/audxXwR8Y2Z7RLuOBqYDo4Hzo33nAy9mGqtaECISK1l+JdwfgKfMrACYDfQmUbiONLM+wBzg9EwnVwKuIx06tGPo4HvZvk0R7s5jjz3F/Q8M4pRTTuSG66/kZ3vuziHdTuCjj6eEDlVq6Lrb7mL8ux/QattCXnjykU2Of/DxFC6/5ibat90BgGOO6MbFF55Tq3OWlpbS75Y7mT7zSwpbtuAfN/ejfds2vPfBx9zzyBA2bCijYcN8rrq0DwcduH+tzpXrsvluaO7+KdBlM4eOzsb8akHUkbKyMv7S9yb23e8oDj3sJC6++AJ+9rPdmTZtBqed/nsmTHg/dIiSoV7H/5JH7rq12jEH7LcPzw97kOeHPVij5Dt/4WIuuKzvJvv/9fIbtGjejNdGDua3Z/TirocGA7BtYQse+NuNjHriYQZcdxX9bv5HzR5MDGVrGVp9UAVcRxYtWsKiRYne/Jo1a5kx40vat9uBt8ZMCByZ1FaX/X/O/IWLM7rvS6+/zVPPvsiGDWXsu/ceXHfVpTRo0CDl/d6eMJFL+pwLwLFH/oLb7noYd+dnnXf7YcxunXbiP+vXU1paSkFBQUbxxUHZFpFa06MKuB7stFMH9t9vHyZ98EnoUKSefPb5F/zm/Eu46KrrKZk9B4D/+3ou/zvm3zzxyJ08P+xB8vLyePmNsWnNt2TpcnbYvgiA/PwGNGvahJWrvvvJmDfHvcNee+y2VSdfyN5FuPqQcQVsZr3dfUgVx4qBYgBr0JK8vKaZnibnNW3ahJEjHuXKP/dn9eo1ocORerDXHrvy5vPDaNKkMePf+4DL+93MqyMGMWnyp0yfUcKZff4IwPr162m1bSEAl/e7mfkLFrOhbAMLFy/llPMvBeDc03ty8gnHpjxnyew53PXQYAbePaDuHliO2FrejvImYLMJOHlxc35B+/BPM4Hk5+fz7IhHGT58FC+88FrocKSeNGv6Y8FxeLeu3Hrng3y7chXuzq97HMOfLu69yX3u+58bgEQP+NoBdzL0gb//5Pj227Vm0ZJl7LD9dpSVlbNm7fcUtmwBwKIlS/njX2/htuv/TMcO7erwkeWGLaGyTVe1LQgzm1LFNpVavPxua/HowDv5YkYJ99xb1QttJI6WLV+BeyIJTJ0+kwp3Clu24OAu+/PmuHdY/u1KAFZ9t5oFi9LrJR912MG8+OpbALwxbgIHHbgfZsZ3q9dwyV/6c8VFvTlg373r5gHlmGy+G1pdS1UBtwF+BXxbab8B79VJRDFxaLf/5rfnnsqUqdOZ/OEbAFx//e0UNCrg3rtvZbvtWjH6xcf57LNpHH9i7ZYoSf36S//b+fCTKaxc+R1H9zqXS/r8lrKyMgDOOPkE3hj7DiNGvUKD/AZsU1DAHTddg5mxa6ed+MPvz6P4imup8Aoa5udz7ZWX0G6H1LXMb078Ff1uuYMep19IyxbNueOmxCtihz//Et/MW8AjQ57mkSFPAzDwngG0jlobW6Nyz50K2LyaYM1sEDDE3d/ZzLGn3f3sVCfYmlsQUrV1C7QaRDbVsGgXq+0cZ+90cto55+k5o2p9vtqotgJ29z7VHEuZfEVE6lsu9YC1DlhEYmVL6O2mSwlYRGIlmy9FrmtKwCISK2pBiIgEkkurIJSARSRW1IIQEQlEF+FERAJRD1hEJBC1IEREAqnu1b1bGiVgEYmVmnwsfWhKwCISK2pBiIgEohaEiEggqoBFRALRMjQRkUBy6aXI+lRkEYmVCjztLR1m1sDMPjGzl6PbncxskpmVmNkIM8v4Y6iVgEUkVrKdgIE/Al8k3f4bcLe770bi49qq/OCKVJSARSRW3D3tLRUz6wCcADwW3TagO/BcNGQY0CvTWJWARSRWalIBm1mxmU1O2oorTXcP0Jcf3+OnNbDS3cui2/OA9pnGqotwIhIrNVkF4e4DgYGbO2ZmJwJL3P0jMzsyO9H9lBKwiMRKuWftDSkPBX5tZscD2wAtgHuBQjPLj6rgDsD8TE+gFoSIxEq2esDu3s/dO7j7zsCZwNvufg4wFjg1GnY+8GKmsSoBi0is1MEqiMquBq40sxISPeFBmU6kFoSIxEpdvBLO3ccB46KvZwNdszGvErCIxEpFDr0STglYRGJF7wUhIhJIFldB1DklYBGJFbUgREQCUQtCRCQQVcAiIoGoAhYRCaTcy0OHkDYlYBGJFX0op4hIIPpQThGRQFQBi4gEolUQIiKBaBWEiEggeimyiEgg6gGLiASiHrCISCCqgEVEAtE6YBGRQFQBi4gEolUQIiKB6CKciEggakGIiASiV8KJiASiClhEJJBc6gFbLj1b5DozK3b3gaHjkC2Lfi+2XnmhA9jKFIcOQLZI+r3YSikBi4gEogQsIhKIEnD9Up9PNke/F1spXYQTEQlEFbCISCBKwCIigSgB1xMzO87MZppZiZldEzoeCc/MBpvZEjP7PHQsEoYScD0wswbAg0APYC/gLDPbK2xUsgUYChwXOggJRwm4fnQFStx9truXAs8APQPHJIG5+3hgReg4JBwl4PrRHvgm6fa8aJ+IbMWUgEVEAlECrh/zgR2TbneI9onIVkwJuH58COxuZp3MrAA4ExgdOCYRCUwJuB64exlwGfA68AUw0t2nhY1KQjOz4cBEYA8zm2dmfULHJPVLL0UWEQlEFbCISCBKwCIigSgBi4gEogQsIhKIErCISCBKwCIigSgBi4gE8v8g+ERpcM6DjQAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "### MODEL 2: XGBoost\n",
        "\n",
        "The second model we will build is XGBoost because recently the algorithm was successfull in many machine learning competitions.\n",
        "\n",
        "<b>XGBoost - what is it? It's always best to understand your models. </b>\n",
        "\n",
        "https://xgboost.readthedocs.io/en/stable/python/python_intro.html#\n",
        "\n",
        "Train the model: Here you can chose the max number of boosting iterations, a balance between computing time and accuracy.\n",
        "\n",
        "Presenting the data in the appropriate format for the library."
      ],
      "metadata": {
        "id": "YETFriH023pW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "dtrain = xgb.DMatrix(data_train[selected_features_set], label=outcome_train)\n",
        "dtest = xgb.DMatrix(data_test[selected_features_set], label=outcome_test)"
      ],
      "metadata": {
        "id": "zc7V2Uxw3EzF"
      },
      "execution_count": 54,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Defining the parameters: the learning objective is logistic regression for binary classification with probability output, the metric is the area under precision-recall curve (why not ROC AUC?)."
      ],
      "metadata": {
        "id": "O0F-YNfT3G_Q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "param = {'objective': 'binary:logistic', 'eval_metric': 'aucpr'}"
      ],
      "metadata": {
        "id": "Mt119Hy83Lv-"
      },
      "execution_count": 55,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will train the model on the training set and evaluate on test set."
      ],
      "metadata": {
        "id": "JsilJkzb3Ohc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "evallist = [(dtest, 'eval'), (dtrain, 'train')]"
      ],
      "metadata": {
        "id": "l5Vdwp5n3Q3k"
      },
      "execution_count": 56,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Training the model (number of iterations can be changed here!) and calculating outcomes for the test set."
      ],
      "metadata": {
        "id": "jVkWe4n53Sq1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "num_round = 10\n",
        "bst = xgb.train(param, dtrain, num_round, evallist)\n",
        "\n",
        "outcome_pred_xgb = bst.predict(dtest)"
      ],
      "metadata": {
        "id": "JjSYef5k3Vm7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1eef636e-ae46-4003-c282-690473d478be"
      },
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[0]\teval-aucpr:0.825354\ttrain-aucpr:0.929223\n",
            "[1]\teval-aucpr:0.833927\ttrain-aucpr:0.964685\n",
            "[2]\teval-aucpr:0.844201\ttrain-aucpr:0.974634\n",
            "[3]\teval-aucpr:0.858133\ttrain-aucpr:0.983149\n",
            "[4]\teval-aucpr:0.867161\ttrain-aucpr:0.988734\n",
            "[5]\teval-aucpr:0.865829\ttrain-aucpr:0.993923\n",
            "[6]\teval-aucpr:0.868113\ttrain-aucpr:0.99517\n",
            "[7]\teval-aucpr:0.884191\ttrain-aucpr:0.99697\n",
            "[8]\teval-aucpr:0.889242\ttrain-aucpr:0.997305\n",
            "[9]\teval-aucpr:0.881052\ttrain-aucpr:0.997962\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Classification report:"
      ],
      "metadata": {
        "id": "FsKuk8pI3YPf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print (classification_report(outcome_test, outcome_pred_xgb>0.5))"
      ],
      "metadata": {
        "id": "Kk4JUhKk3bKc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bf2c1820-52cc-408d-8000-6261e3c1646b"
      },
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.87      0.75      0.81       166\n",
            "           1       0.79      0.90      0.84       175\n",
            "\n",
            "    accuracy                           0.82       341\n",
            "   macro avg       0.83      0.82      0.82       341\n",
            "weighted avg       0.83      0.82      0.82       341\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "ROC and ROC AUC:"
      ],
      "metadata": {
        "id": "MXlktROy3c7b"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fpr, tpr, _ = roc_curve(outcome_test, outcome_pred_xgb)\n",
        "roc_auc = roc_auc_score(outcome_test, outcome_pred_xgb)\n",
        "\n",
        "plt.plot(fpr, tpr)\n",
        "plt.title('XGBoost ROC curve (AUC = {})'.format(roc_auc))\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "2Kg5dlpk3gEd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        },
        "outputId": "c74f53f4-9e87-49e7-f0eb-d9f0605f575a"
      },
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZxcVZn/8c/T3el0ls5G9rUTSIAEBLRNIKwjiwk6gMtEFpEoGnF+IL9xRZlhGBRndMZxxOGnhhHBjVWBoEGRJYZlAmkFA0lICNkTSDoLSWdPdz+/P85puCl6qSRVXV23v+/Xq15d95xT9z636t7nnjr3dl1zd0REpPiVFDoAERHJDSV0EZGUUEIXEUkJJXQRkZRQQhcRSQkldBGRlFBClw7LzLqa2SIzG1LoWEQOl5ldY2bfzucyiiqhm1lPM1tpZpclyirNbLWZfTRRVm1mvzWzrWb2ZkwKN5tZ31g/3cwazGxHfCw3s8/lOfazzGxtG23uMLN9MaYtZvZHMzsmo81wM/ulmW02s51m9ryZfTCjjZnZ583s5dhmrZndZ2bH52Pd8mgGMNfdX08WmtmNZuZmNqmZ8l9kziS2PSox/X4zm2tmdWZWa2Z/MrMLchl4/Ay+HT+nzfG5tdL+GjNbYWbbzazGzE5L1PUxszvNbGN83Jjx2ioze9LMdpnZK2Z2TqLuYjNbYmbb4mvvNLNeifodGY8GM/tBon6amS2O79UiM7soY9lj4r5WZ2abzOw7ibo5ZrYnMe8libqvZyx3t5k1mln/jPn3i5/R0xnlLcZlZj/KmPdeM6vLmOcDcd9YZWaXZnxu11vIKdvN7O6M92uamT0b3+s5GTGNM7OHYrxbzOwPZnZ0osltwGVmNpB8cfeiegDvB2qBAXH6h8BvEvWTgR3A14BBsWwk8C/AWXF6OvB04jUnAXXASXmM+yxgbRtt7gC+GZ93i9PPJOr7ASuBnwKDY5tLgO3ARxPtbgFeA94HdAW6A5cB1+X5synN8fwWAqdmlBmwHNgM3JpRdyPwi2bm48BR8flH4/v1aaA3oVNzJnBbjmP/LLAEGA4MAxYBV7XQdhKwE3hPXL/PxW28NNb/FLgvfo5V8bP9ZOL1/wv8Z9wePgK8mdg/RgD94/OewC+BW1qIo2fcd86I08OAfcDUGNcHgF3AwFhfHmP5AtADqADelZjfHODTWb5fNwJPNFN+GzA3Y39tNa4W9qvbE9N3AffE9T0N2AZMiHVXAK/E960n8BBwZ+K15wDTgBuAORnLmQhcSdhPuwDfAF5pZn2+lLd9MF8zzucjfkB3EZLkZmBwou5p4AdtvH56cgOJZc8DlyamLyAklDfjhnlsou7YWPZmbHNBou58ws5bB6wDvhQ39t1AY9xhdgBDW1ivb2bMa2di+hvAy0BJxuu+CqyKG/dYoAGYeBDvZz9C0lgPbAUebOV9SibHOwgH1NmEhPRV4A0SiR34ELAgPi8BriMkgc3AvUC/FmIaGd+zsozyM2L5ZXEe5Ym6G2klocf3ZzXw5XbYRp8FZiSmrwTmtdD2Y8DziekeMeYhcXoT8N5E/deBp+LzccBeoDJR/xTNHDwICepnwOwW4riCcLC0OD0J2JjRphY4JT6f0RRHC/ObQxYJnbcP0ldklE8mHKw+yYEJvdW4Msp7EPbFMxPT+4BxiTY/B/4tPr8/uX3EGPYA3TPm+2kyEnoL+5UDRyTKLgOezNd2V1RDLgn/QEjm9xOOdm8AmFkP4BTg1wczMzN7L2HHqInT4wgHjP8LDCAkrIfNrNzMugAPA48CA4FrgF8mvlr9BPisu1cCxxF6HTsJvYn17t4zPta3EVMPQu97WaL4XODX7t6Y0fxeQgIcB5xN+Cbw/EG8BT8n9P4mxHX63kG89lLgZqAS+D4hsb8vo/5X8fk1wEWEHvFQwsHj1hbmezyw3N3rM8qvILz/98bpvz2IWI8m9Lzuz/YFZnaphWG7lh4jW3jpBOCviem/xrLmPAKUmtkkMysFPgW8SDg4vhVKxvPjEstZ7u51ifoDlmVmp5nZNkJi+wjwXy3EcQXwM4+Zh7A/LDazC8ysNA5r7AUWxPqTgZVm9kgcbplj7xzW+9dY94yZndXCck8nbHdv7bfxffhv4GpCUkxqK66kjxCS/dw4PQ6od/eliTaZn03me92V0FE6WGcAb7j75kTZYuCEQ5hXdvJ1pMj3A3iM8DWrd6JsOOHDPyZR9h1CT3on8I+xbDpQH8vr4mt+wNs9k38C7k3Mo4TQ2z6LsPG9QaKXTEj+N8bnqwlft3tlxHsW2Q257IlxNQIrOPAr7DKa73lVxHU4FbieFnqCLSxzSFxW32bqptN2D/1nGfXfJH69JST5ncCoOL0YODtj2fvJ6IX72z2ZeRll3QnDJRfF6R8DDyXqb6T1Hvqp8XlFO2yfDRnb4di4bGumrRF63fvjdpnZI/8F8Jv4fh5F+IazN9Zd3sz7dDNwRzPLGRbfo3HN1I2KMY/OKL+S8I2ynrC/fSBR92iMeSph+OXLhJ52eayfFGPuSjhY1AFHNrPsn2TGS+i0/bCV7bDFuDLaPU7cN+P06YQkm2zzGWJvm9DzXkoY2uoNzIqf2ykZr2m1h07IReuASzLKxwIN+druirKHbmYfJ7zhjwHJs8ZbCcnprasi3P0r7t4HeAAoS7Sd5+59PPSkBxOO0N+KdUMJQxhN82gE1hB2iKHAGj+wl7wq1kHoEZwPrLJwsu2Ug1y9/4jxVhGGFpInVTYl1y1hSKJ+cwttWjIC2OLuWw8yziZrMqZ/BXzYzLoCHwb+4u5N7+Uo4IGm3i0hwTcAg5qZ71ZCMkj6EGEHnh2nfwlMNbMBcbqeMHb5lviNCkLiaeoptcdVMzuAXonpXsAOj3t1hisJwwoTCInx48BvzWxorP88YVt4lTCmexfQdII9czlNy6rLKMPd1wG/B+5uJobLCUlzRVNBPLn6HUJnpJzwzep/zOzE2GR3fM0j7r4P+A/gCMKQJO7+nLvXufted78TeIawb5BYRnfg74A7E2VD4zpf30yc2cTV1G5kbPOzRHFb79fthPd3DmE49clY3uoFDRnLHUA42P0/d78ro7qSMGafF0WX0OMZ4u8RjqqfBaaZ2ekAHoY2niMkkqy5+wbC172mr+/rCcmnaZlGSHzrYt0IM0u+dyNjHe4+390vJHyFfJC3hwaa25Fbi2k1cC3wfTPrFosfIyTLzM9tGiGxLiX0SIabWXWWi1oD9DOzPs3U7ST0igEws8HNhZoR9yLCAW4qBw63NC1rajyQNj0qYqLJtAAYbWbJg/AVhHHg1Wb2BuFEYZe4HAjfjqoy5jOakOjXEU5SriEcdLNiZpfZO68EST5aGnJZyIFfrU+IZc05Efituy9190Z3/z3wOmH8Fnff4u6Xuftgd59A2G+bhtQWAmPMLHnwa21ZZcCRzZR/gkRSTcQ1191rYlzzCftX01U0Czi47do5cDgDwkF6CyGBNplIOOguip/z94GJZvZGHIppK64mlxMuKlieKFsKlJlZcgjlrfcrzu+f3b3K3YfH8nXx0SYLV9I9Csxy95ubaXIsBw7F5Va+uv75ehAS5G2J6U8Tzkp3jdOnERLRdbx9Nn44oXdwozfzFY7Qq3gMuCdOHx3ncTYhYXyJ+FUyPpbH+Xch9ADqgGNi3WXEYSBCz2tVfH4MoUfTu5V1u4PESdFYVgNcm4hzNW9f5VLB21e5fCzxmh8QenNnxZgqgItp4SoX4HeExNs3rlPTVQ5NJ9xOjPP4Ee8ccvlmM/P7KqFns5t4hUUs/wfCjjsqTg8ALmzl/VgATI7PhxF68+fFdW96/Bvw59hmIKH3c3lcj36E8fK7E/P8aGzzSULPrCRuMzNzvJ1eRfgG0vStbiEtX+VyBSHRjCEkvHMJwwjHxPoj42dfSjhQbiJelRHr5xF6xxWEBJm8yuUyYGR8Pgr4E4mrwmL5ZML2XplRfmZc1olx+iTCt5zzEvvJLkIiLY2f72txm+tDuCKtgnAQuSwuY1zGMh4Fbsoo65rxGV9LSNiDs4krMZ8lwKeaeb/vJvTCexCG4ZJXufSL77cB4wkXISRPbpfGdbqKMC5fAXSJdb0IB9r/bmW7mAl8JZfb2gHzz9eM8xJsOKG2HuiTUf4EcHNiehLha/mb8fEyYVzxiFg/nZAcmq442Rg/4IGJeXyIcLXKtrgTJHegCbFsW2zzoVheTvhKu5WQZOcDpyVed3vc8N4ki6tcYtnHCL2DpgPWyBjrlriDzCcjKcaN8VpCEtkVX39Pch0y2vcj9M42xNiTl4FeH3eeNYShgGwS+kjC0NfvMspLCJe4LSEcBF8DvtXK5/1/eHsc9Tpi4s5oM5QwnHJcnJ5MuNJpa9xW/oeM8wPAFMKVIDsIJ8zm0MIY7GFsq0YYFtgSH98hMX4el316ou1NhIN1HeFAcHmi7bS4LrsIJ0vfn7GsqrgOu+N7e06i7mbCcMHO+HcmiasuYpsfAz9vYT2uJpy7qSN0ZL6YUf/hWL89xtCUGAfEbbOOsL3PA87NeO0wwreno9p4L6fzzjH0tuI6hWYOUont/cFYv5oDr24bF9/DXYRvml9oJhbPeNwR666I0zt5O7fs4O0DakX8DAblcltLPppOAop0OHEc/gXCidTX22ov0pGZ2TXACHf/St6WoYQuIpIORXdSVEREmqeELiKSEkroIiIpUdZ2k/zo37+/V1VVFWrxIiJF6c9//vMmdx/QXF3BEnpVVRU1NTWFWryISFEys1Ut1WnIRUQkJZTQRURSQgldRCQllNBFRFJCCV1EJCXaTOhmdruFm8u+3EK9mdktZrbMzBaY2btzH6aIiLQlmx76HYRfp2vJVMJdOMYS7jH4w8MPS0REDlab16G7+1wzq2qlyYW8fR/CeWbWx8yG6NfxRKRY7KtvZPWWnbxWu5PltTvZvS/zVra5dfaxgzhhRHP3lDk8ufjHomEceBuytbHsHQndzGYQevGMHNnSjV5EpBjt2FvPY4s2sL8h8x7mHc++hkZWbd7F8todvFa7k9VbdtHQ+PYvz1rmfZVybGCvig6b0LPm7jMJP7BPdXW1frdXJAu79tXz0tptB3cPwwJ4fPEGbntqRdsNO4jyshLG9O/B+CG9+OC7hjBmQA+OHNCT0f17UFnRpe0ZdEC5SOjrCPfbbNJ0t2sROURbdu5j9/4GAP77iVe56/nMe3F3TF1KjUeuPYOuZR37ArqyUmNgZQWlJXnuirezXCT0WcDVZnY34dZv2zR+LnLoFq7fxgduefqAskG9uvK9j53Ywis6joGVXTlqYM9Ch9FptZnQzewuws2G+5vZWuCfCTfgxd1/RLh35/mE+/vtItx8V6TT+tpvFnDP/EPvUTcNrVzzvqMY0bc7AEcPrszLmKukSzZXuVzSRr0TbuYrUhR+/ee13PLEq3mb/+vb9jCsbzcuOnHYIc+jZ9cyPnXaaLqUduyhC+lYCvbzuSL58It5q/j9y2+02mbphjq279nPlAmD8xLDSSPgA+8ayrnjB+Vl/iItUUKXorZmyy5mzl1Ofbzk7NGFb7CvoZFxgypbfM2Ift2ZNLofX5lyTHuFKdIulNClw/vL6q3MW7652bqalVt54pWN9O9ZjplRUmJ8ZvIYPn/22HaOUqTwlNClw2hodOYs2ciufQ0HlP/XY0t5rXZni68b1KsrT3/1fRpvlk5PCV0KYtvu/azYdGCSXrR+O19/4KVm23/wXUP4j787odm6LqUlqbueWORQKKFL3tXt2c+++gP/Hfyau17g2deaH0a57RPVjO7f/YCyEf2607WsNG8xiqSBErrk1cL12/jbHzxNYzP/t37iiD5cmzHW3bOijOpRfbF8/5iGSAopoUvOXXrbPF5c8yYA9Y1Oo8NnzxjDsL7dDmg3+cgjOGpgy1ejiMjBUUKXnLn1yWU8/Nf1LNlQx4ShvThlzBEA9Ozahc+ddSTlHfz3PUSKnRJ6J3RvzRoeXbgh5/N9fsVmystKmDJhMJ85YwzvHtk358sQkZYpoXdCv3puNa9uqGPUET1yOt/hfbtz8cQRfOKUqpzOV0Syo4TeCWzduY+HF6ynviGcmayt28t7qvrxs09NLHBkIpJLSuidwIMvruNfHl50QNkZ4/oXKBoRyRcl9E6gqWf+zHXvo2d5+Mh7ddNHL5I22qtTwt3fusNNpn3xHo+9u3WhZ1d95CJppb07JT5/94s8/Nf1rbYp1T/riKSaEnqRevrVTXzh3hffulP5m7v3c+SAHkyrHtFs+2F9u9GtXP86L5JmSuhFavHr29lYt5ePVY+gS1noeU89bginHqWTnSKdlRJ6kfunvx2vcXERAZTQi86uffX85KkVLf5SoYh0XkroRWb+yq18949LKSsxxgzoQYV+H0VEIiX0Dqhuz37+umZbs3Uvrwvl9111Cifpt1JEJEEJvQP67qNLuePZla22qazQRyciB1JW6IB27q3niB7l/Ojy9zRbX1lRpt8RF5F3UELvoLqWlfDeqn6FDkNEiogSegfy7GubuO7XL1Fbt5e+3bsUOhwRKTJK6B3Iy+u2sXrLLi46cSgnx7v9iIhkSwm9A9i2ez/ffXTJW/fhvPlDx9ND/ywkIgdJWaOA3J27nl9Dzcot/OaFdfTv2ZVJo/tR0UW/uSIiB08JvYA2bN/L1x94CYAe5aXMuvpUhvbpVuCoRKRYKaEXUIOHX0r8tw8fz7TqEZSU6OdtReTQZfV/42Y2xcyWmNkyM7uumfqRZvakmb1gZgvM7Pzch5oue/Y3sKluLwAlZkrmInLY2uyhm1kpcCtwLrAWmG9ms9w9eZPKfwTudfcfmtl4YDZQlYd4U2Pq959ixaadAG/9/K2IyOHIZshlIrDM3ZcDmNndwIVAMqE70Cs+7w20fuscYcP2PZx2VH8uOmkY758wuNDhiEgKZJPQhwFrEtNrgUkZbW4EHjWza4AewDnNzcjMZgAzAEaOHHmwsabC9/64lIcXrGfXvgaOHVLJR98zvNAhiUhK5Oq3Vy8B7nD34cD5wM/N7B3zdveZ7l7t7tUDBgzI0aKLw5wlG5n+0+e5/ekV1O2p54IThvLBdw0tdFgikiLZ9NDXAckbVQ6PZUlXAlMA3P1/zawC6A9szEWQafC7Ba/zzLJNjB/Si2nvHcFlk0YVOiQRSZlsEvp8YKyZjSYk8ouBSzParAbOBu4ws2OBCqA2l4GmwYCeXXno6tMKHYaIpFSbCd3d683sauAPQClwu7svNLObgBp3nwV8EbjNzP6BcIJ0unu8yLqT272vgaderWXt1t2FDkVEUi6rfyxy99mESxGTZTckni8CTs1taOlw35/XcMNDCwE4dkivNlqLiBw6/adonu3Z3wDAA38/mTEDehY4GhFJMyX0djJuUKV+QVFE8kq3jBcRSQkldBGRlFBCFxFJCQ3q5snOvfX8eO5y5i3fXOhQRKSTUELPk5pVW7nl8VfpWlbC0YMq6VqmL0Mikl9K6HnSGP+v6u4ZJ3PSyL4FjkZEOgN1G/Ogtm4vr7+5p9BhiEgnox56DjU2Opt27mXyvz5BfWPooXct0w2fRaR9KKHn0Ad+8DSLX98OwCdOGcXfHDOQY4dUFjgqEekslNBz4FuzF/PgC+vYWLeXiVX9OG/CIKa9dwS9KroUOjQR6USU0HPguRVbKC0xLp00kstPHqUf4RKRglBCz5Fxgyr51oeOL3QYItKJ6SoXEZGUUEIXEUkJJXQRkZRQQhcRSQkldBGRlFBCFxFJCSV0EZGUUEIXEUkJ/WPRYfjL6q3cV7OGNVt20WdY70KHIyKdnHroh+Gu51Zzz/w1lJUYE0f3K3Q4ItLJqYd+GBwY0rsbz1z3vkKHIiKiHrqISFoooYuIpIQSuohISmgM/RA8+cpGPn/XC+za38CQ3hWFDkdEBFBCPySvbqyjbm890ydXMUlXt4hIB5HVkIuZTTGzJWa2zMyua6HNNDNbZGYLzexXuQ2zY/ry+49m6vFDCh2GiAiQRQ/dzEqBW4FzgbXAfDOb5e6LEm3GAl8DTnX3rWY2MF8Bi4hI87LpoU8Elrn7cnffB9wNXJjR5jPAre6+FcDdN+Y2zI5hb30Ddz2/mudXbCl0KCIi75DNGPowYE1iei0wKaPNOAAzewYoBW50999nzsjMZgAzAEaOHHko8RbU/BVb+dpvXgKgf89yyst0kZCIdBy5OilaBowFzgKGA3PN7Hh3fzPZyN1nAjMBqqurPUfLbhdvbNvDik07APjFlZOYOLofXUqV0EWk48gmoa8DRiSmh8eypLXAc+6+H1hhZksJCX5+TqIsoPqGRjbW7eX07zxJQ2M4BvWvVO9cRDqebBL6fGCsmY0mJPKLgUsz2jwIXAL81Mz6E4Zglucy0EL5wC1Ps2RDHQCfPLWKc44dxNGDKgsclYjIO7WZ0N293syuBv5AGB+/3d0XmtlNQI27z4p155nZIqAB+LK7b85n4O1lzdZdTBzdj/OPG8xH3jOcyoouhQ5JRKRZWY2hu/tsYHZG2Q2J5w58IT5S54ThvZl+6uhChyEi0ir9p2gLHnhhLY8t3sje+sZChyIikhUl9AyLX9/OQy+u59d/WcuOPfWM6d+DiaOPKHRYIiJtUkLP8NNnVnBvzVrKy0q48rTRfHXKMYUOSUQkK0roGRodhvXRXYhEpPjoYmoRkZRQQhcRSQkldBGRlFBCFxFJCSV0EZGU0FUu0cpNO/nhnNeYv1K/dS4ixanTJ/Td+xr4xbxVzH21lqde3cTQ3hWcMW5AocMSETlonT6hP79yCzfPXgzA8L7dmPOlsyjT75yLSBHq9Am9Mf7G+W/+fjInjeiDmRU4IhGRQ9MpE/rufQ1s270fgC079wFQYqZkLiJFrVMm9Cnfn8uqzbsOKOtSqmQuIsWtUyb0jdv3cvrY/px//BAAKivKOHZwrwJHJSJyeDplQgc4dkgvLpk4stBhiIjkjC7nEBFJiU7VQ//jog38aelG9jXoLkQikj6dIqGv2LSTRxe+wZ3PrqR2x176di/n+GG9Cx2WiEhOdYqE/uM/vcbd89cA8IlTRnHThccVOCIRkdzrFAm9vtEZ0ruCx794Jt26lBY6HBGRvOgUCR3CPw51L+80qysinZCuchERSQkldBGRlFBCFxFJCSV0EZGUUEIXEUkJJXQRkZRQQhcRSQkldBGRlMgqoZvZFDNbYmbLzOy6Vtp9xMzczKpzF6KIiGSjzYRuZqXArcBUYDxwiZmNb6ZdJXAt8FyugxQRkbZl00OfCCxz9+Xuvg+4G7iwmXbfAL4N7MlhfCIikqVsEvowYE1iem0se4uZvRsY4e6/a21GZjbDzGrMrKa2tvaggxURkZYd9klRMysB/hP4Yltt3X2mu1e7e/WAAQMOd9EiIpKQTUJfB4xITA+PZU0qgeOAOWa2EjgZmKUToyIi7SubhD4fGGtmo82sHLgYmNVU6e7b3L2/u1e5exUwD7jA3WvyErGIiDSrzYTu7vXA1cAfgMXAve6+0MxuMrML8h2giIhkJ6s7Prj7bGB2RtkNLbQ96/DDEhGRg6X/FBURSQkldBGRlFBCFxFJiVTfNfnlddv49u9f4ZU36igv1bFLRNIt1Ql99kuv89Srm3j3yD6cdlT/QocjIpJXqUzo+xsa+cr9C3jghXWMH9KL+6+aTEmJFTosEZG8SmVCn7d8Mw+8sI7pk6u4buoxSuYi0imkcmC5vsEBuOikYVR0KS1wNCIi7SOVCV1EpDNSQhcRSQkldBGRlEjVSdE9+xv41B3zWbV5V6FDERFpd6nqoW/asZdnX9tMvx7lTKseztGDKgsdkohIu0lVD73J5aeMYlr1iLYbioikSKp66CIinZkSuohISiihi4ikRGoSen1DI7v2NRQ6DBGRgknNSdHzb3mKpRt2AFCm324RkU4oNQl9zZbdTBrdj/OPH8K54wcVOhwRkXaXmoQOcMKIPlwxuarQYYiIFERqxtBFRDo7JXQRkZQo+iGXReu385fVW6lvbCx0KCIiBVX0Cf2Gh16mZtVWAAZWdi1wNCIihVP0CX1/QyOTjzyCWy45iSN6lBc6HBGRginqhF7f0IgD5WUl9O+p3rmIdG5Fe1L01ieXcdT1j7Bg7TZKTf9IJCJStD305bU7qawoY8bpY/ibYwYWOhwRkYIr2oQO0KuiC9ecPbbQYYiIdAhZDbmY2RQzW2Jmy8zsumbqv2Bmi8xsgZk9bmajch+qiIi0ps2EbmalwK3AVGA8cImZjc9o9gJQ7e7vAu4HvpPrQEVEpHXZ9NAnAsvcfbm77wPuBi5MNnD3J9296c7M84DhuQ1TRETakk1CHwasSUyvjWUtuRJ4pLkKM5thZjVmVlNbW5t9lCIi0qacXrZoZh8HqoF/b67e3We6e7W7Vw8YMCCXixYR6fSyucplHTAiMT08lh3AzM4BrgfOdPe9uQlPRESylU0PfT4w1sxGm1k5cDEwK9nAzE4Cfgxc4O4bcx+miIi0pc2E7u71wNXAH4DFwL3uvtDMbjKzC2Kzfwd6AveZ2YtmNquF2YmISJ5k9Y9F7j4bmJ1RdkPi+Tk5jktERA5S0f6Wi4iIHEgJXUQkJZTQRURSQgldRCQllNBFRFJCCV1EJCWU0EVEUkIJXUQkJZTQRURSQgldRCQllNBFRFJCCV1EJCWU0EVEUkIJXUQkJZTQRURSQgldRCQllNBFRFJCCV1EJCWU0EVEUiKre4p2JNv37OfRhRtYsWlHoUMREelQii6h31+zlpt+uwiAE0f0KXA0IiIdR9El9P0NjQA8/sUzGdanW4GjERHpOIp2DH1I7woqupQWOgwRkQ6jaBO6iIgcSAldRCQllNBFRFJCCV1EJCWU0EVEUkIJXUQkJZTQRURSQgldRCQlskroZjbFzJaY2TIzu66Z+q5mdk+sf87MqnIdqIiItK7NhG5mpcCtwFRgPHCJmY3PaHYlsNXdjwK+B3w714GKiEjrsumhTwSWuftyd98H3A1cmNHmQuDO+Px+4Gwzs9yFKSIibckmoQ8D1iSm18ayZtu4ez2wDTgic0ZmNsPMasyspra29pACHt2/B+cfP5gSHS9ERA7Qrr+26GwdHZ8AAAQVSURBVO4zgZkA1dXVfijzOG/CYM6bMDincYmIpEE2PfR1wIjE9PBY1mwbMysDegObcxGgiIhkJ5uEPh8Ya2ajzawcuBiYldFmFnBFfP5R4Al3P6QeuIiIHJo2h1zcvd7Mrgb+AJQCt7v7QjO7Cahx91nAT4Cfm9kyYAsh6YuISDvKagzd3WcDszPKbkg83wP8XW5DExGRg6H/FBURSQkldBGRlFBCFxFJCSV0EZGUsEJdXWhmtcCqQ3x5f2BTDsMpBlrnzkHr3DkczjqPcvcBzVUULKEfDjOrcffqQsfRnrTOnYPWuXPI1zpryEVEJCWU0EVEUqJYE/rMQgdQAFrnzkHr3DnkZZ2LcgxdRETeqVh76CIikkEJXUQkJTp0Qu+MN6fOYp2/YGaLzGyBmT1uZqMKEWcutbXOiXYfMTM3s6K/xC2bdTazafGzXmhmv2rvGHMti217pJk9aWYvxO37/ELEmStmdruZbTSzl1uoNzO7Jb4fC8zs3Ye9UHfvkA/CT/W+BowByoG/AuMz2vw98KP4/GLgnkLH3Q7r/DdA9/j8c51hnWO7SmAuMA+oLnTc7fA5jwVeAPrG6YGFjrsd1nkm8Ln4fDywstBxH+Y6nwG8G3i5hfrzgUcAA04GnjvcZXbkHnpnvDl1m+vs7k+6+644OY9wB6lils3nDPAN4NvAnvYMLk+yWefPALe6+1YAd9/YzjHmWjbr7ECv+Lw3sL4d48s5d59LuD9ESy4EfubBPKCPmQ05nGV25ISes5tTF5Fs1jnpSsIRvpi1uc7xq+gId/9dewaWR9l8zuOAcWb2jJnNM7Mp7RZdfmSzzjcCHzeztYT7L1zTPqEVzMHu721q15tES+6Y2ceBauDMQseST2ZWAvwnML3AobS3MsKwy1mEb2Fzzex4d3+zoFHl1yXAHe7+XTM7hXAXtOPcvbHQgRWLjtxD74w3p85mnTGzc4DrgQvcfW87xZYvba1zJXAcMMfMVhLGGmcV+YnRbD7ntcAsd9/v7iuApYQEX6yyWecrgXsB3P1/gQrCj1ilVVb7+8HoyAm9M96cus11NrOTgB8Tknmxj6tCG+vs7tvcvb+7V7l7FeG8wQXuXlOYcHMim237QULvHDPrTxiCWd6eQeZYNuu8GjgbwMyOJST02naNsn3NAj4Rr3Y5Gdjm7q8f1hwLfSa4jbPE5xN6Jq8B18eymwg7NIQP/D5gGfA8MKbQMbfDOj8GbABejI9ZhY453+uc0XYORX6VS5afsxGGmhYBLwEXFzrmdljn8cAzhCtgXgTOK3TMh7m+dwGvA/sJ37iuBK4Crkp8xrfG9+OlXGzX+td/EZGU6MhDLiIichCU0EVEUkIJXUQkJZTQRURSQgldRCQllNBFRFJCCV1EJCX+P+WthaC7xdS0AAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create and display the confusion matrix and derived values for the Xgboost model."
      ],
      "metadata": {
        "id": "TeaigfZW3iXE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "cm = confusion_matrix(outcome_test, outcome_pred_xgb>0.5)\n",
        "f = sns.heatmap(cm, annot=True)"
      ],
      "metadata": {
        "id": "HtPhR7Jx33CP",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "outputId": "bfa12140-9754-47ff-fa33-0ff0e87fee6e"
      },
      "execution_count": 60,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWAAAAD4CAYAAADSIzzWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAW+klEQVR4nO3deXxU1fnH8c+ThBBRICCKCCpacf9V64qtViuogAtutbiiUlMVd+uC9CeiouCCS3FpKii41rpUa9GqqFWqoqgUFfFH1IphdcMNkzAzz++PGXGEJDOZTHIyl++b13kxc+7l3DMv48PDc869Y+6OiIi0vqLQExARWVMpAIuIBKIALCISiAKwiEggCsAiIoGUtPQFlo8/WdssZDUHjf8o9BSkDZpW/ZQ1d4wVn36Qdcxp122zZl+vOZQBi4gE0uIZsIhIq0rEQ88gawrAIhIt8VjoGWRNAVhEIsU9EXoKWVMAFpFoSSgAi4iEoQxYRCQQLcKJiASiDFhEJAzXLggRkUC0CCciEohKECIigWgRTkQkEGXAIiKBaBFORCQQLcKJiIThrhqwiEgYqgGLiASiEoSISCDKgEVEAomvCD2DrCkAi0i0qAQhIhKIShAiIoEUUAasr6UXkWhJJLJvGZjZJDNbamZv13PsPDNzM+uWem9mdpOZVZnZbDPbMdP4CsAiEikeX5F1y8KdwIBVO81sI2A/YH5a90CgT6pVALdmGlwBWESixRPZt0xDub8AfF7PoeuBCwBP6xsMTPGkV4ByM+vR2PgKwCISLU0oQZhZhZnNTGsVmYY3s8HAAnf/zyqHegIfp72vTvU1SItwIhItTdgF4e6VQGW255tZB+BikuWHZlMAFpFoadldED8BNgX+Y2YAvYA3zGxXYAGwUdq5vVJ9DVIJQkSiJY814NWGdn/L3dd3997u3ptkmWFHd18MPAYcn9oN0Rf40t0XNTaeMmARiZZY/h7Ibmb3AXsD3cysGhjl7hMbOH0qMAioApYDJ2YaXwFYRKIlj3fCuftRGY73TnvtwPCmjK8ALCLRUkB3wikAi0i06FkQIiKBKAMWEQlEGbCISCB53AXR0hSARSRa3DOf00YoAItItKgGLCISiAKwiEggWoQTEQkkHg89g6wpAItItKgEISISiAKwiEggqgGLiIThCe0DFhEJQyUIEZFAtAtCRCQQZcAiIoEoAEdD6X5DKd7sp/jyr6mZculqx4u32o12uwwAA6+roe6Ze/BPq5t30eISSgecRFH3TfDvvqHuH5X4V59RtPHWlO55OBQXQzxO3QsPkvh4bvOuJUEUFRVxy9QJfLb4U0aecAkj/ngRW/60D7EVcebOmsv1F91IPFY4/4xucwroYTz6VuRGxN55iZqHb2zwuH/5KTUPXEPNlNGseOUflO57XNZjW6d1af/r36/WX7LdHnjNcmomjST2xjO02/Pw5LW++4bav/2RmimjqX1yEqUDT2r6B5I24bBhhzK/av7K99MemcYJew3jt/0raF/WnkFHDQw4uwhIJLJvgSkANyKxYB7UfNvw8UXvQ+3y1OsPsI5dVh4r3no32h99MWXHXkK7/seCWVbXLP7JDsTnvARA/P9ep3jjrQDwTz7Gv/0y+fqzhVhJKRTrHzCFpluPbuzWb1em3vvkyr5Xn31t5eu5s95jvR7dQkwtOhKefQss4//BZrYVMBjomepaADzm7u+25MQKTcl2e5D48G0ArOsGlGyxC7X3j4NEnHb7HE3xVn2Jv/tyxnFsnXL86y+SbzyB134HZetAzTcrzynusyOJJR9BvHAePC1Jwy89lcoxt9NhnbVWO1ZcUsy+h/fj5lG3BphZhBTQLohGM2AzuxC4HzDg1VQz4D4zu6iRP1dhZjPNbOakl6NfpyzaaEtKttuDuhcfAqB4462x7ptQdvRIyo69hOKNt6aoPJnVlB58GmXHXkL7Q8+kqPsmlB17SfKcbX+e1bVs3Q1pt+fh1D1zd4t9HmkZffvtxhefLmPeW/PqPX7WlWcwe8ZbvPXq2608s2jxRCLrlomZTTKzpWb2dlrfNWY218xmm9kjZlaedmyEmVWZ2Xtmtn+m8TNlwMOAbd19xSqTGg+8A4yt7w+5eyVQCbB8/Mnh8/wWZN16Urrv8dQ+fFNaucKIz3mJFdMfWe38usduSZ7RaV1K9z+R2r9e+6Pj/s0yrGMX/JsvwIqw9mutzH5tnS60P/g06p6chH/5SYt+Lsm/bXfZlp/v15fd9tmF0valdOjYgRE3XchVZ47juHOOpbxrOaMuHB16moUvv6WFO4EJwJS0vqeBEe4eM7NxwAjgQjPbBhgCbAtsCDxjZlu4e4MpeaYacCI10Kp6pI6t0axj12RAfGISvmzJyv74/Hcp7rMTrNUx2VHWAevYNasx4+/PonibZDZcvMVOxOe/lzzQfi3aH3oGK158iMTC9/P6OaR1TBw7iSG7HMMxux/PFcOvZNa/Z3HVmeMYdNQAdtlrJ644/Uq8gFbw2yxPZN8yDeX+AvD5Kn1Pufv39b9XgF6p14OB+9291t0/BKqAXRsbP1MGfDYwzczmAR+n+jYGNgdOzzj7Alc66GSKe20Ba61D2clXs+Llx7CiYgBis/9Fu74HYmVrU9rvGAA8Eaf23jH454tY8e+/UXb4OcnFt0Scumfvxb/+vLHLJcd9ezqlA4dRdtIYvOZb6v5RCUDJDvtg5evTru9BtOt7EAA1D10P333dQp9eWsvZV53Fkuol/PHR5I6b6U9M564b7gk8qwLWhAzYzCqAirSuytS/4LN1EvCX1OueJAPy96r5Ye2s/utn+hvXzIpIRvH0RbjXGkur00W9BCG5OWj8R6GnIG3QtOqnstsu1IhvLxmSdcxZ+7L7M17PzHoDj7v7dqv0jwR2Bg5zdzezCcAr7n536vhE4Al3f7ChsTPugnD3BD+O6iIibVcrPI7SzE4ADgT6+Q9Z7AJgo7TTeqX6GqR9wCISLS28D9jMBgAXAAe7+/K0Q48BQ8ysvZltCvQhuXOsQdrJLyKRks32smyZ2X3A3kA3M6sGRpHc9dAeeNqSN1i94u6nuPs7ZvYAMAeIAcMzlWoVgEUkWvK4Dc3dj6qne2Ij548BxmQ7vgKwiERLG7jFOFsKwCISLQV0K7ICsIhEir4TTkQkFAVgEZFA2sBzfrOlACwi0aIMWEQkEAVgEZEwPK4ShIhIGMqARUTC0DY0EZFQFIBFRAIpnBKwArCIRIvHCicCKwCLSLQUTvxVABaRaNEinIhIKMqARUTCUAYsIhKKMmARkTA8FnoG2VMAFpFIaYVvpc8bBWARiRYFYBGRMAopAy4KPQERkXzyRPYtEzObZGZLzezttL6uZva0mc1L/d4l1W9mdpOZVZnZbDPbMdP4CsAiEiket6xbFu4EBqzSdxEwzd37ANNS7wEGAn1SrQK4NdPgCsAiEin5zIDd/QXg81W6BwOTU68nA4ek9U/xpFeAcjPr0dj4qgGLSKR4IqvMtjm6u/ui1OvFQPfU657Ax2nnVaf6FtEAZcAiEilNyYDNrMLMZqa1iiZdy92BnG+9UwYsIpHinn0G7O6VQGUTL7HEzHq4+6JUiWFpqn8BsFHaeb1SfQ1SBiwikZLPGnADHgOGpl4PBR5N6z8+tRuiL/BlWqmiXsqARSRSEtntbsiKmd0H7A10M7NqYBQwFnjAzIYBHwFHpk6fCgwCqoDlwImZxlcAFpFIyecinLsf1cChfvWc68DwpoyvACwikdIKuyDyRgFYRCLFC+dxwArAIhItyoBFRAJpyja00BSARSRS4nncBdHSFIBFJFKUAYuIBKIasIhIINoFISISiDJgEZFA4onCecSNArCIRIpKECIigSS0C0JEJAxtQxMRCUQliDSdLpra0peQAvTdwhdDT0EiSiUIEZFAtAtCRCSQAqpAKACLSLSoBCEiEoh2QYiIBJL7lx23PgVgEYkURxmwiEgQMZUgRETCKKQMuHA2zImIZCHRhJaJmZ1jZu+Y2dtmdp+ZlZnZpmY2w8yqzOwvZlaa61wVgEUkUhzLujXGzHoCZwI7u/t2QDEwBBgHXO/umwNfAMNynasCsIhESj4zYJJl2rXMrAToACwC9gEeTB2fDByS61wVgEUkUuJY1s3MKsxsZlqr+H4cd18AXAvMJxl4vwReB5a5eyx1WjXQM9e5ahFORCKlKd9I5O6VQGV9x8ysCzAY2BRYBvwVGND8Gf5AAVhEIiWRv10Q/YEP3f0TADN7GPgFUG5mJaksuBewINcLqAQhIpHiTWgZzAf6mlkHMzOgHzAHeA44InXOUODRXOeqACwikZKvRTh3n0Fyse0N4C2S8bISuBA418yqgHWBibnOVSUIEYmUhOXvRgx3HwWMWqX7A2DXfIyvACwikRIPPYEmUAAWkUhpyi6I0BSARSRS8rgLosUpAItIpOgriUREAlEJQkQkEH0jhohIIHFlwCIiYSgDFhEJRAFYRCSQAvpKOAVgEYkWZcAiIoHoVmQRkUC0D1hEJBCVIEREAlEAFhEJRM+CEBEJRDVgEZFAtAtCRCSQRAEVIRSARSRStAgnIhJI4eS/CsAiEjGFlAEXhZ6AiEg+xcyzbpmYWbmZPWhmc83sXTPb3cy6mtnTZjYv9XuXXOeqACwikeJNaFm4EXjS3bcCtgfeBS4Cprl7H2Ba6n1OFIBFJFISTWiNMbPOwC+BiQDuXufuy4DBwOTUaZOBQ3KdqwKwiERKAs+6mVmFmc1MaxVpQ20KfALcYWZvmtntZrY20N3dF6XOWQx0z3WuWoQTkUhpyi4Id68EKhs4XALsCJzh7jPM7EZWKTe4u5tlUUxugDJgEYmUfJUggGqg2t1npN4/SDIgLzGzHgCp35fmOlcFYBGJlDiedWuMuy8GPjazLVNd/YA5wGPA0FTfUODRXOeqEoSIREqe9wGfAdxjZqXAB8CJJBPXB8xsGPARcGSugysAi0ikeB7vhXP3WcDO9Rzql4/xFYBFJFIK6U44BeAW8ufK6zhgUH+WfvIpO/ws+Zfl9ttvyy0TxtK+rD2xWIwzzriY12bOCjxTaao/XDmeF/79Kl27lPO3u2+r95xX35jNuBv/RCwWo0t5J+68+ZpmXbOuro4Rl1/HnPfmUd65E9deNoKePbrz0qtvcMNtd7BiRYx27Uo4b/gwdttph2Zdq9AV0tPQtAjXQqZMeYADDjzmR31jrxzJ5VeMZ+dd9mP06GsZe9XIQLOT5jhk0L7cNv6KBo9/9fU3XHHdBCaMG8Wj9/yJ667I/r/zgkVLOOH0C1brf/jxp+jUcR2eeGASx/3mEMbfMgmALuWdmDDuUh6561bG/OE8Rlx2bdM/UMTk+U64FqUA3EJenD6Dz79Y9qM+d6djp44AdOrckYWLloSYmjTTzjv8D51T/x3rM/Xp5+m/1y/oscH6AKzbpXzlsb//81mG/PYsDh86nNFX30Q8nt3jw5998WUGD+oPwH5778mM12fh7my9xeasv966AGy+6SbU1NZSV1eX60eLhBiedQtNJYhWdO7vRzH18Xu5euz/UlRk7LnX4NBTkhbw3/nVxOJxTjj9ApYv/45jfj2YwQP78/5/5/PktH9x123X0a6khMuvncDjTz3H4IH9M4659JPP2GD9bgCUlBSzztodWPblV3Qp77zynKefn842W25OaWlpi322QpDPRbiWlnMANrMT3f2OBo5VABUAVtyZoqK1c71MpPyu4njOO/9SHnlkKkcccRB//tN17D9wSOhpSZ7F4wnmzJ3H7TeNpba2lmN+dy7bb7sVM2bOYs7cKoYMOwuA2tpauqay4zNHXMaChUtYEVvBoiWfcPjQ4QAce+RgDj1gv4zXrPrgI8bfMonK68e03AcrEGvKItxooN4AnH57X0lpz8L566iFHX/crznn3EsAePDBv1N5W/MWZqRt6r5+Nzp37kiHtcrosFYZO+2wHe9VfYi7c/DA/pxz6omr/Zmbrkr+XCxYtISRY67jzglX/+j4+uuty+Kln7LB+usRi8X55tvllHfuBMDipZ9w1sWXc+X//p6Ne23Y8h+wjSukDLjRGrCZzW6gvUUzHkCxplq4aAl7/XJ3APb51R7Mq/ow8IykJfxqz768OfsdYrE439XU8NY777FZ743ou/MOPP38dD5LrQ18+dXXLFyc3TrAr/boy6NTnwHgqedfZLedtsfM+Orrbzjt/FGcfcqJ7PjTbVvsMxWSPN6K3OIyZcDdgf2BL1bpN+ClFplRRNx9183s9cvd6datK//9YCajL7uWU045n/HjL6OkpITamhpOPXX11W5p+84fNZbX3pzNsmVf0e+QYzlt2HHEYjEAfnPoAfyk98b8YredOWzoqRRZEYcftD99NusNwBknH0/F2SNJeIJ2JSWMPPc0Ntwgcy5z2IH7M+Lyaxh45El07tSRa0Ynnwlz30N/5+Pqhdx2x73cdse9AFTeMOZHC39rmrgXTgZs3shkzWwicIe7T6/n2L3ufnSmC6gEIfX5buGLoacgbVC7bptZc8c4epNDs4459370SLOv1xyNZsDuPqyRYxmDr4hIayukGrC2oYlIpLSF2m62FIBFJFIK6VZkBWARiRSVIEREAimkXRAKwCISKSpBiIgEokU4EZFAVAMWEQlEJQgRkUAau7u3rVEAFpFIyfR1822JArCIREohlSD0lUQiEinunnXLhpkVm9mbZvZ46v2mZjbDzKrM7C9mlvNXkCgAi0ikJPCsW5bOAt5Nez8OuN7dNyf5qN4GH1qWiQKwiESKN+FXJmbWCzgAuD313oB9gAdTp0wGDsl1rgrAIhIpcfesm5lVmNnMtFaxynA3ABfww/0d6wLL3D2Wel8N9Mx1rlqEE5FIacoiXPr3V67KzA4Elrr762a2d35m92MKwCISKXncBfEL4GAzGwSUAZ2AG4FyMytJZcG9gAW5XkAlCBGJlHztgnD3Ee7ey917A0OAZ939GOA54IjUaUOBR3OdqwKwiERKC+yCWNWFwLlmVkWyJjwx14FUghCRSGmJh/G4+/PA86nXHwC75mNcBWARiZS4F84DKRWARSRS9DAeEZFACulZEArAIhIpeiC7iEggCZUgRETCUAYsIhKIdkGIiASiEoSISCAqQYiIBKIMWEQkEGXAIiKBxD0eegpZUwAWkUjRrcgiIoHoVmQRkUCUAYuIBKJdECIigWgXhIhIILoVWUQkENWARUQCUQ1YRCQQZcAiIoFoH7CISCCFlAEXhZ6AiEg+xT2RdWuMmW1kZs+Z2Rwze8fMzkr1dzWzp81sXur3LrnOVQFYRCIl4Z51yyAGnOfu2wB9geFmtg1wETDN3fsA01Lvc6IALCKR4u5ZtwzjLHL3N1KvvwbeBXoCg4HJqdMmA4fkOlcFYBGJFG/CLzOrMLOZaa2ivjHNrDfwM2AG0N3dF6UOLQa65zpXLcKJSKQ0ZRHO3SuBysbOMbN1gIeAs939KzNL//NuZjmv+ikAi0ik5PNGDDNrRzL43uPuD6e6l5hZD3dfZGY9gKU5j19IWzYKnZlVpP7GFVlJPxdtkyVT3cnA5+5+dlr/NcBn7j7WzC4Curr7BTldQwG49ZjZTHffOfQ8pG3Rz0XbZGZ7AC8CbwHf71m7mGQd+AFgY+Aj4Eh3/zyXa6gEISJSD3efDlgDh/vl4xraBSEiEogCcOtSnU/qo5+LNZRqwCIigSgDFhEJRAFYRCQQBeBWYmYDzOw9M6tK7R2UNZyZTTKzpWb2dui5SBgKwK3AzIqBm4GBwDbAUamnKsma7U5gQOhJSDgKwK1jV6DK3T9w9zrgfpJPVJI1mLu/AOS0gV+iQQG4dfQEPk57X53qE5E1mAKwiEggCsCtYwGwUdr7Xqk+EVmDKQC3jteAPma2qZmVAkOAxwLPSUQCUwBuBe4eA04H/knya00ecPd3ws5KQjOz+4CXgS3NrNrMhoWek7Qu3YosIhKIMmARkUAUgEVEAlEAFhEJRAFYRCQQBWARkUAUgEVEAlEAFhEJ5P8BWfVLIs+qEjAAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "<b>Almost done! You will now compare the performance of the models.</b>\n",
        "  \n",
        "Which classifier is better?  \n",
        "\n",
        "To compare ROC AUC scores, we will perform a permuation test for the probabilities obtained on the test set for the both classifiers."
      ],
      "metadata": {
        "id": "UJpHgmIW36hA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# adapted from https://stackoverflow.com/questions/52373318/how-to-compare-roc-auc-scores-of-different-binary-classifiers\n",
        "#-and-assess-statist\n",
        "\n",
        "def permutation_test_between_clfs(y_test, pred_proba_1, pred_proba_2, nsamples=100):\n",
        "    auc_differences = []\n",
        "    auc1 = roc_auc_score(y_test.ravel(), pred_proba_1.ravel())\n",
        "    auc2 = roc_auc_score(y_test.ravel(), pred_proba_2.ravel())\n",
        "    observed_difference = auc1 - auc2\n",
        "    for _ in range(nsamples):\n",
        "        mask = np.random.randint(2, size=len(pred_proba_1.ravel()))\n",
        "        p1 = np.where(mask, pred_proba_1.ravel(), pred_proba_2.ravel())\n",
        "        p2 = np.where(mask, pred_proba_2.ravel(), pred_proba_1.ravel())\n",
        "        auc1 = roc_auc_score(y_test.ravel(), p1)\n",
        "        auc2 = roc_auc_score(y_test.ravel(), p2)\n",
        "        auc_differences.append(auc1 - auc2)\n",
        "    return observed_difference, np.mean(auc_differences >= observed_difference)\n",
        "\n",
        "print ('Difference, p-value: ', \n",
        "       permutation_test_between_clfs(outcome_test,\n",
        "                                     outcome_pred_xgb,\n",
        "                                     rfc.predict_proba(data_test[selected_features_set])[:,1]))"
      ],
      "metadata": {
        "id": "0fu8MAO44AP8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ca2a4249-3d7a-4fb4-bed1-c46b4d4a1077"
      },
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Difference, p-value:  (-0.0033734939759035854, 0.57)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "After this test, which classifier is better?  \n",
        "  \n",
        "<b>Time for a pre-hackathon!</b> You can go back and \"finetune\" the models (or even implement your models) by changing the parameters you feel comfortable with. See you can increase the model performance. Can you beat the other groups?"
      ],
      "metadata": {
        "id": "xmqNYAaL4J8D"
      }
    }
  ]
}