Hands_On_1.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "aAUE4hqdgV5W"
   },
   "source": [
    "# SPH6004 Hands-On 1\n",
    "## Linear Regression, Logistic Regression, and Gradient Descent\n",
    "\n",
    "Contents:\n",
    "  * Recap of linear and logistic regressions\n",
    "  * Loss functions\n",
    "  * Solving regression models by using gradient descent method\n",
    "  * Examples\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install plotly==5.11.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will be using the following packages\n",
    "\n",
    "# PyTorch package and submodules\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.optim import SGD #gradient descent optimizer\n",
    "\n",
    "# NumPy for math operations, and Pandas for processing tabular data.\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Plotly plotting package\n",
    "import plotly.graph_objects as go\n",
    "import plotly.express as px\n",
    "\n",
    "# We use toy datasets in scikit-learn package\n",
    "from sklearn.datasets import load_diabetes, load_breast_cancer"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "E8m_AfeNYBf6"
   },
   "source": [
    "## Recap of linear regression\n",
    "\n",
    "*Linear Regression* is one of the simplest machine learning model for predicting **continuous outcomes** (e.g. time, length). As shown in the figure below, linear regression model tries to learn the statistics (patterns) that exist in a dataset by trying to draw the best fitted **line** to the data.\n",
    "\n",
    "<p align=\"center\">\n",
    "    <img width=\"75%\" src=\"https://upload.wikimedia.org/wikipedia/commons/3/3a/Linear_regression.svg\"> \n",
    "</p>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model formulation\n",
    "\n",
    "Linear regression is a supervised learning model. We are given a sequence of **independent variables** $(\\mathbf x^{(i)})_{i=1}^N$ and the corresponding **dependent variables/labels** $(y^{(i)})_{i=1}^N$.\n",
    "\n",
    "* Each independent variable $\\mathbf x^{(i)}$ is a $M$-dimensional vector of the form\n",
    "\n",
    "$$\n",
    "\\mathbf x^{(i)} = (x_1^{(i)}, x_2^{(i)}, \\cdots, x_M^{(i)})\n",
    "$$\n",
    "\n",
    "* Each label $y^{(i)}$ is a real number.\n",
    "* Linear regression model has the form (for convenience we define $x^{(i)}_0=1$)\n",
    "\n",
    "\\begin{equation}\n",
    "h_{\\theta}(\\mathbf{x}^{(i)}) = \\theta_{0} + \\theta_{1}x_{1}^{(i)} + ... + \\theta_{M}x_{M}^{(i)} = \\sum_{k = 0}^{M}{ \\theta_{k} x_{k}^{(i)} },\n",
    "\\end{equation}\n",
    "\n",
    "where the $\\theta_i$'s are parameters of the model. The term $\\theta_{0}$ is called the **intercept** or **bias** term of the model.\n",
    "\n",
    "Our goal is to find the 'best' parameters $\\hat{\\theta}_i$ such that $h_{\\hat{\\theta}}(\\mathbf x^{(i)})\\sim y^{(i)}$."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss function\n",
    "\n",
    "We find $\\hat{\\theta}$ by *minimizing* the **Mean Square Error** $J(\\theta)$ between $h_{\\theta}(\\mathbf{x})$ and $y$:\n",
    "\n",
    "\\begin{equation}\n",
    "J_{\\text{MSE}}(\\theta) =  \\frac{1}{2N} \\sum_{i = 1}^{N} \\left( h_\\theta(\\mathbf{x}^{(i)}) - y^{(i)} \\right)^2.\n",
    "\\end{equation}\n",
    "\n",
    "This function $J(\\theta)$ is called a **loss function** for our linear regression model.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recap of Logistic Regression\n",
    "\n",
    "Logistic regression is a binary **classification** model that fits data $\\{(\\mathbf x^{(i)}, y^{(i)})\\}$ with binary labels $y^{(i)}\\in \\{0,1\\}$. Mathematically the model with parameter $\\theta$ has the form\n",
    "\n",
    "$$\n",
    "f_\\theta(\\mathbf x) = \\sigma(h_\\theta(\\mathbf x)) = \\sigma(\\theta_{0} + \\theta_{1}x_{1} + ... + \\theta_{M}x_{M}) = \\sigma\\left (\\sum_{k = 0}^{M}{ \\theta_{k} x_{k} }\\right ),\n",
    "$$\n",
    "\n",
    "where $\\sigma$ is the **sigmoid function** defined by\n",
    "\n",
    "$$\n",
    "\\sigma(\\xi) = \\frac{1}{1+\\exp(-\\xi)}.\n",
    "$$\n",
    "\n",
    "<p align=\"center\">\n",
    "    <img width=\"50%\" src=\"https://miro.medium.com/max/1280/0*gKOV65tvGfY8SMem.png\"> \n",
    "</p>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The sigmoid function $\\sigma$ has range in $(0,1)$.\n",
    "* Usually $\\sigma(h_\\theta(\\mathbf x))$ is interpreted as the probability of $\\mathbf x$ being in class with label $y=1$.\n",
    "    * To make binary predictions, one can set a threshold $T=0.5$: the prediction is $0$ if $\\sigma(h_\\theta(\\mathbf x))\\leq 0.5$, and the prediction is $1$ if $\\sigma(h_\\theta(\\mathbf x))>0.5$."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loss function\n",
    "\n",
    "For the logistic regression model, one usually uses the **binary cross entropy** as the loss function\n",
    "\n",
    "$$\n",
    "J_{\\text{BCE}}(\\theta) = \\frac{1}{N}\\sum_{i=1}^{N}\\left(-  y^{(i)}\\log(f_\\theta(\\mathbf x)) - (1-y^{(i)})\\log (1 - f_\\theta(\\mathbf x)) \\right).\n",
    "$$"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* One *minimizes* $J_{\\text{BCE}}$ to find the best parameter $\\theta$.\n",
    "* For data with class label $y=1$, minimizing $J_{\\text{BCE}}$ is equivalent to maximizing $f_{\\theta}$ (recall this is the probability that $\\mathbf x$ belong to class $y=1$).\n",
    "* For data with class label $y=0$, minimizing $J_{\\text{BCE}}$ is equivalent to maximizing $(1-f_{\\theta})$ (recall this is the probability that $\\mathbf x$ belong to class $y=0$)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "yzmKef0IbBW2"
   },
   "source": [
    "\n",
    "## Gradient descent method for solving optimization problems\n",
    "\n",
    "Given a model with parameter $\\theta$ and loss function $J$, one usually estimates the best parameter $\\hat{\\theta}$ by minimizing $J$:\n",
    "\n",
    "$$\n",
    "\\hat{\\theta} = \\argmin_{\\theta} J(\\theta).\n",
    "$$\n",
    "\n",
    "The image below illustrates the plot of a loss function $J$ with respect to a 2-dimensional parameter $\\theta = (\\theta_0,\\theta_1)$.\n",
    "* Gradient of $J$, denoted by $\\nabla_\\theta J$, gives direction of $\\theta$ that leads to fastest *increase* of $J$.\n",
    "* Gradient descent method minimizes $J$ by following *opposite* direction of gradient of $J$.\n",
    "* We only follow the downward direction a short time, controlled by *learning* rate $\\eta$, and then re-check the gradient direction at the new location.\n",
    "\n",
    "\n",
    "<img src=\"https://hackernoon.com/hn-images/1*f9a162GhpMbiTVTAua_lLQ.png\" alt=\"Drawing\" stylewidth=\"200\" height=\"200\"/>\n",
    "\n",
    "### Steps for gradient descent\n",
    "\n",
    "We start from a randomly drawn parameter $\\theta_0$. For $t=0,1,2,\\cdots$ we repeat the following steps\n",
    "1.  Evaluate gradient of the loss function at time $t$, i.e. $\\nabla_\\theta J(\\theta_t)$.\n",
    "2.  We update the parameter by\n",
    "    $$ \\theta_{t+1} \\leftarrow \\theta_t-\\eta*\\nabla_\\theta J(\\theta_t).$$"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "id": "TmHvNO6Mpe8C"
   },
   "source": [
    "## Examples\n",
    "\n",
    "For this tutorial, we will use Python with the package [PyTorch](https://pytorch.org/) to build linear and logistic regression models.\n",
    "\n",
    "Basic steps:\n",
    "1.  Load and check data. How many observations? What is input shape?\n",
    "2.  Normalize input data.\n",
    "3.  Build the model and solve for best parameters.\n",
    "4.  Test the model with best parameters on test dataset."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example -- linear regression\n",
    "\n",
    "We use the diabetes dataset from `scikit-learn` package to build our linear regression model. Our aim is to predict a quantitative measure of disease progression one year after baseline. There are 10 independent variables which have already been normalized (to have zero mean):\n",
    "* age age in years\n",
    "* sex\n",
    "* bmi body mass index\n",
    "* bp average blood pressure\n",
    "* s1 tc, total serum cholesterol\n",
    "* s2 ldl, low-density lipoproteins\n",
    "* s3 hdl, high-density lipoproteins\n",
    "* s4 tch, total cholesterol / HDL\n",
    "* s5 ltg, possibly log of serum triglycerides level\n",
    "* s6 glu, blood sugar level"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 442 entries, 0 to 441\n",
      "Data columns (total 10 columns):\n",
      " #   Column  Non-Null Count  Dtype  \n",
      "---  ------  --------------  -----  \n",
      " 0   age     442 non-null    float64\n",
      " 1   sex     442 non-null    float64\n",
      " 2   bmi     442 non-null    float64\n",
      " 3   bp      442 non-null    float64\n",
      " 4   s1      442 non-null    float64\n",
      " 5   s2      442 non-null    float64\n",
      " 6   s3      442 non-null    float64\n",
      " 7   s4      442 non-null    float64\n",
      " 8   s5      442 non-null    float64\n",
      " 9   s6      442 non-null    float64\n",
      "dtypes: float64(10)\n",
      "memory usage: 34.7 KB\n"
     ]
    }
   ],
   "source": [
    "X_df, y_df = load_diabetes(return_X_y=True, as_frame=True)\n",
    "\n",
    "# In total we have 442 observations\n",
    "X_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>bmi</th>\n",
       "      <th>bp</th>\n",
       "      <th>s1</th>\n",
       "      <th>s2</th>\n",
       "      <th>s3</th>\n",
       "      <th>s4</th>\n",
       "      <th>s5</th>\n",
       "      <th>s6</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.038076</td>\n",
       "      <td>0.050680</td>\n",
       "      <td>0.061696</td>\n",
       "      <td>0.021872</td>\n",
       "      <td>-0.044223</td>\n",
       "      <td>-0.034821</td>\n",
       "      <td>-0.043401</td>\n",
       "      <td>-0.002592</td>\n",
       "      <td>0.019907</td>\n",
       "      <td>-0.017646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.001882</td>\n",
       "      <td>-0.044642</td>\n",
       "      <td>-0.051474</td>\n",
       "      <td>-0.026328</td>\n",
       "      <td>-0.008449</td>\n",
       "      <td>-0.019163</td>\n",
       "      <td>0.074412</td>\n",
       "      <td>-0.039493</td>\n",
       "      <td>-0.068332</td>\n",
       "      <td>-0.092204</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.085299</td>\n",
       "      <td>0.050680</td>\n",
       "      <td>0.044451</td>\n",
       "      <td>-0.005670</td>\n",
       "      <td>-0.045599</td>\n",
       "      <td>-0.034194</td>\n",
       "      <td>-0.032356</td>\n",
       "      <td>-0.002592</td>\n",
       "      <td>0.002861</td>\n",
       "      <td>-0.025930</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        age       sex       bmi        bp        s1        s2        s3  \\\n",
       "0  0.038076  0.050680  0.061696  0.021872 -0.044223 -0.034821 -0.043401   \n",
       "1 -0.001882 -0.044642 -0.051474 -0.026328 -0.008449 -0.019163  0.074412   \n",
       "2  0.085299  0.050680  0.044451 -0.005670 -0.045599 -0.034194 -0.032356   \n",
       "\n",
       "         s4        s5        s6  \n",
       "0 -0.002592  0.019907 -0.017646  \n",
       "1 -0.039493 -0.068332 -0.092204  \n",
       "2 -0.002592  0.002861 -0.025930  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We convert dataframe to PyTorch tensor datatype,\n",
    "# and then split it into training and testing parts.\n",
    "X = torch.tensor(X_df.to_numpy(),dtype=torch.float32)\n",
    "m,n = X.shape\n",
    "y = torch.tensor(y_df.to_numpy(),dtype=torch.float32).reshape(m,1)\n",
    "\n",
    "# We use an approx 6:4 train test splitting\n",
    "cases = ['train','test']\n",
    "case_list = np.random.choice(cases,size=X.shape[0],replace=True,p=[0.6,0.4])\n",
    "X_train = X[case_list=='train']\n",
    "X_test = X[case_list=='test']\n",
    "y_train = y[case_list=='train']\n",
    "y_test = y[case_list=='test']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training input has size:  torch.Size([269, 10])\n"
     ]
    }
   ],
   "source": [
    "print('Training input has size: ',X_train.shape)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In PyTorch, data are stored as `torch.tensor` datatype. Models in PyTorch expect the following:\n",
    "* the leading dimension of data is the index of this data, and \n",
    "* the rest of the dimensions are dimension of each data point"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the build-in linear model, mean square error loss, as well as gradient descent optimizer in PyTorch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parameter containing:\n",
      "tensor([[-0.0133,  0.2229,  0.1529,  0.2443,  0.1799,  0.2941, -0.0109,  0.2435,\n",
      "         -0.0343, -0.1217]], requires_grad=True)\n",
      "Parameter containing:\n",
      "tensor([-0.0205], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "# Linear regression is a linear model.\n",
    "# We have 10 dimensional inputs and 1 dimensional outputs.\n",
    "# We want to have the bias term in our model.\n",
    "\n",
    "h = torch.nn.Linear(\n",
    "    in_features=10,\n",
    "    out_features=1,\n",
    "    bias=True\n",
    ")\n",
    "\n",
    "# parameters of our model 'h' are \n",
    "# randomly initialized, which are stored in\n",
    "print(h.weight)\n",
    "print(h.bias)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 1 : average rooted training MSE 172.546\n",
      "Iter 500 : average rooted training MSE 67.053\n",
      "Iter 1000 : average rooted training MSE 62.034\n",
      "Iter 1500 : average rooted training MSE 59.273\n",
      "Iter 2000 : average rooted training MSE 57.610\n",
      "Iter 2500 : average rooted training MSE 56.502\n",
      "Iter 3000 : average rooted training MSE 55.703\n",
      "Iter 3500 : average rooted training MSE 55.095\n",
      "Iter 4000 : average rooted training MSE 54.617\n",
      "Iter 4500 : average rooted training MSE 54.234\n",
      "Iter 5000 : average rooted training MSE 53.925\n",
      "Iter 5500 : average rooted training MSE 53.673\n",
      "Iter 6000 : average rooted training MSE 53.468\n",
      "Iter 6500 : average rooted training MSE 53.301\n",
      "Iter 7000 : average rooted training MSE 53.163\n",
      "Iter 7500 : average rooted training MSE 53.050\n",
      "Iter 8000 : average rooted training MSE 52.958\n",
      "Iter 8500 : average rooted training MSE 52.881\n",
      "Iter 9000 : average rooted training MSE 52.818\n",
      "Iter 9500 : average rooted training MSE 52.765\n",
      "Iter 10000 : average rooted training MSE 52.722\n"
     ]
    }
   ],
   "source": [
    "h = torch.nn.Linear(\n",
    "    in_features=10,\n",
    "    out_features=1,\n",
    "    bias=True\n",
    ")\n",
    "\n",
    "# For torch SGD, we need to tell it which parameter we what to optimize.\n",
    "GD_optimizer = torch.optim.SGD(lr=0.05, params=h.parameters())\n",
    "J_MSE = torch.nn.MSELoss()\n",
    "\n",
    "# Apply gradient descent 100 times\n",
    "nIter = 10000\n",
    "printInterval = 500\n",
    "\n",
    "# PyTorch optimization steps:\n",
    "# 1. Clear gradient\n",
    "# 2. Calculate model and loss values (so-called forward path)\n",
    "# 3. Calculate gradient of loss (so-called backward path)\n",
    "# 4. Ask optimizer to update parameters of model\n",
    "for i in range(nIter):\n",
    "    # Step 1\n",
    "    GD_optimizer.zero_grad()\n",
    "    # Step 2\n",
    "    pred = h(X_train)\n",
    "    loss = J_MSE(pred,y_train)\n",
    "    # Step 3\n",
    "    loss.backward()\n",
    "    # Step 4\n",
    "    GD_optimizer.step()\n",
    "    # Print loss value to track optimization progress\n",
    "    if i == 0 or ((i+1) % printInterval) == 0:\n",
    "        # We take square root of MSE (PyTorch internally averaged in J_MSE)\n",
    "        # so that scale of printout is same as scale of y values.\n",
    "        print('Iter {} : average rooted training MSE {:.3f}'.format(i+1,torch.sqrt(loss).item()))\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "mode": "markers",
         "type": "scatter",
         "x": [
          151,
          75,
          97,
          138,
          63,
          310,
          179,
          185,
          118,
          166,
          144,
          68,
          202,
          283,
          129,
          87,
          55,
          92,
          259,
          155,
          225,
          59,
          52,
          170,
          61,
          128,
          163,
          97,
          160,
          178,
          48,
          111,
          85,
          200,
          113,
          65,
          55,
          134,
          164,
          102,
          53,
          81,
          104,
          229,
          179,
          200,
          200,
          173,
          84,
          121,
          109,
          115,
          268,
          158,
          272,
          336,
          281,
          60,
          259,
          178,
          96,
          126,
          292,
          197,
          25,
          84,
          220,
          152,
          47,
          101,
          127,
          151,
          107,
          138,
          101,
          79,
          91,
          86,
          122,
          158,
          222,
          277,
          99,
          196,
          77,
          191,
          70,
          73,
          49,
          248,
          185,
          160,
          220,
          259,
          90,
          246,
          275,
          47,
          78,
          51,
          258,
          150,
          63,
          50,
          39,
          103,
          308,
          145,
          45,
          127,
          182,
          283,
          102,
          200,
          230,
          156,
          233,
          332,
          89,
          65,
          124,
          114,
          142,
          220,
          230,
          242,
          131,
          135,
          199,
          270,
          164,
          95,
          200,
          139,
          88,
          88,
          71,
          77,
          272,
          60,
          54,
          311,
          281,
          321,
          242,
          167,
          197,
          71,
          168,
          217,
          121,
          72,
          201,
          110,
          51,
          69,
          198,
          242,
          175,
          281,
          72,
          140,
          131,
          55,
          42,
          233,
          91,
          111,
          310,
          183,
          49,
          48,
          57
         ],
         "y": [
          194.68418884277344,
          74.93843078613281,
          102.3429183959961,
          81.08935546875,
          131.44615173339844,
          207.2694549560547,
          116.07972717285156,
          158.32394409179688,
          103.08070373535156,
          202.25131225585938,
          183.65670776367188,
          119.60832214355469,
          145.4923553466797,
          171.97406005859375,
          155.6863555908203,
          103.42755126953125,
          75.91481018066406,
          79.86872100830078,
          213.4916534423828,
          155.7664337158203,
          162.87307739257812,
          138.6973419189453,
          199.09402465820312,
          94.02507019042969,
          110.15026092529297,
          107.05424499511719,
          190.64688110351562,
          124.61393737792969,
          112.55926513671875,
          125.7117691040039,
          84.54915618896484,
          128.4434051513672,
          149.39344787597656,
          86.92093658447266,
          104.34371185302734,
          58.640419006347656,
          61.730812072753906,
          148.27557373046875,
          186.91958618164062,
          109.87841796875,
          114.57862091064453,
          151.31106567382812,
          57.82874298095703,
          196.52651977539062,
          168.07577514648438,
          146.19985961914062,
          158.3362579345703,
          199.34078979492188,
          190.53712463378906,
          162.66696166992188,
          102.12654113769531,
          96.64725494384766,
          210.2899627685547,
          76.82437896728516,
          231.0326385498047,
          241.05551147460938,
          242.68618774414062,
          119.16407775878906,
          226.92822265625,
          186.43991088867188,
          118.29127502441406,
          174.4999237060547,
          206.02117919921875,
          147.77610778808594,
          143.26535034179688,
          127.44821166992188,
          262.2231750488281,
          248.106201171875,
          58.666778564453125,
          94.49446868896484,
          121.1739501953125,
          169.37750244140625,
          176.8760528564453,
          177.20489501953125,
          184.95709228515625,
          120.83934020996094,
          94.22277069091797,
          153.57177734375,
          170.6123046875,
          105.15000915527344,
          198.61270141601562,
          176.0907440185547,
          218.7666473388672,
          159.18545532226562,
          172.78526306152344,
          128.68020629882812,
          171.05345153808594,
          151.86375427246094,
          103.74308776855469,
          226.73638916015625,
          138.07113647460938,
          119.66720581054688,
          174.21585083007812,
          160.8661346435547,
          135.55282592773438,
          226.18882751464844,
          226.37368774414062,
          105.06438446044922,
          121.8951416015625,
          82.26077270507812,
          244.6568145751953,
          204.64398193359375,
          101.29948425292969,
          150.0738983154297,
          62.305320739746094,
          124.71607208251953,
          252.87686157226562,
          127.47151184082031,
          44.251792907714844,
          161.98536682128906,
          139.49029541015625,
          192.90281677246094,
          114.1467514038086,
          114.20513916015625,
          149.38832092285156,
          154.4691162109375,
          205.47479248046875,
          214.5709686279297,
          83.01703643798828,
          130.09420776367188,
          135.7840118408203,
          87.92311096191406,
          141.77769470214844,
          196.01858520507812,
          294.60906982421875,
          299.02362060546875,
          154.1107940673828,
          103.56602478027344,
          112.08253479003906,
          283.1229248046875,
          180.4686737060547,
          148.40570068359375,
          149.16810607910156,
          187.62850952148438,
          100.43841552734375,
          102.18509674072266,
          87.28380584716797,
          51.298683166503906,
          198.7804718017578,
          134.2752227783203,
          96.06072998046875,
          173.56468200683594,
          192.13800048828125,
          223.36476135253906,
          244.96969604492188,
          181.60423278808594,
          208.83807373046875,
          110.26298522949219,
          141.6196746826172,
          177.87901306152344,
          217.26025390625,
          111.50021362304688,
          105.78825378417969,
          159.73348999023438,
          72.32943725585938,
          104.44163513183594,
          173.85891723632812,
          169.2871856689453,
          181.16029357910156,
          261.73876953125,
          55.71715545654297,
          175.7486572265625,
          172.9810028076172,
          148.953369140625,
          88.42798614501953,
          195.40609741210938,
          160.58290100097656,
          167.43556213378906,
          255.15719604492188,
          123.90857696533203,
          136.7267608642578,
          57.085670471191406,
          53.81800079345703
         ]
        }
       ],
       "layout": {
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "xaxis": {
         "title": {
          "text": "true value"
         }
        },
        "yaxis": {
         "scaleanchor": "x",
         "scaleratio": 1,
         "title": {
          "text": "predicted value"
         }
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Calculate testing results.\n",
    "# In testing, no need to calculate\n",
    "# gradient, so we use torch.no_grad() to\n",
    "# disable it.\n",
    "with torch.no_grad():\n",
    "    test_pred = h(X_test)\n",
    "\n",
    "fig = go.Figure(\n",
    "    data=go.Scatter(\n",
    "        x=y_test.squeeze(),\n",
    "        y=test_pred.squeeze(),\n",
    "        mode='markers'\n",
    "    )\n",
    ")\n",
    "fig.update_xaxes(\n",
    "    title='true value'\n",
    "  )\n",
    "fig.update_yaxes(\n",
    "    title='predicted value',\n",
    "    scaleanchor='x',\n",
    "    scaleratio=1\n",
    "  )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example -- logistic regression\n",
    "\n",
    "We use the Breast Cancer Wisconsin (Diagnostic) Dataset provided in the Python package `sklearn` to demonstrate training a logistic model. Features are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass to describe charateristics of the cell nulcei present in the image.\n",
    "\n",
    "- 569 instances\n",
    "- dependent variable $\\mathbf x$ has 30 dimensions\n",
    "    - radius\n",
    "    - texture, smoothness\n",
    "    - perimeter, area\n",
    "    - ...\n",
    "- label $y$ has two classes\n",
    "    - $0$ for benign (357 instances)\n",
    "    - $1$ for malignant (212 instances)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 569 entries, 0 to 568\n",
      "Data columns (total 30 columns):\n",
      " #   Column                   Non-Null Count  Dtype  \n",
      "---  ------                   --------------  -----  \n",
      " 0   mean radius              569 non-null    float64\n",
      " 1   mean texture             569 non-null    float64\n",
      " 2   mean perimeter           569 non-null    float64\n",
      " 3   mean area                569 non-null    float64\n",
      " 4   mean smoothness          569 non-null    float64\n",
      " 5   mean compactness         569 non-null    float64\n",
      " 6   mean concavity           569 non-null    float64\n",
      " 7   mean concave points      569 non-null    float64\n",
      " 8   mean symmetry            569 non-null    float64\n",
      " 9   mean fractal dimension   569 non-null    float64\n",
      " 10  radius error             569 non-null    float64\n",
      " 11  texture error            569 non-null    float64\n",
      " 12  perimeter error          569 non-null    float64\n",
      " 13  area error               569 non-null    float64\n",
      " 14  smoothness error         569 non-null    float64\n",
      " 15  compactness error        569 non-null    float64\n",
      " 16  concavity error          569 non-null    float64\n",
      " 17  concave points error     569 non-null    float64\n",
      " 18  symmetry error           569 non-null    float64\n",
      " 19  fractal dimension error  569 non-null    float64\n",
      " 20  worst radius             569 non-null    float64\n",
      " 21  worst texture            569 non-null    float64\n",
      " 22  worst perimeter          569 non-null    float64\n",
      " 23  worst area               569 non-null    float64\n",
      " 24  worst smoothness         569 non-null    float64\n",
      " 25  worst compactness        569 non-null    float64\n",
      " 26  worst concavity          569 non-null    float64\n",
      " 27  worst concave points     569 non-null    float64\n",
      " 28  worst symmetry           569 non-null    float64\n",
      " 29  worst fractal dimension  569 non-null    float64\n",
      "dtypes: float64(30)\n",
      "memory usage: 133.5 KB\n"
     ]
    }
   ],
   "source": [
    "X_raw, y_df = load_breast_cancer(return_X_y=True, as_frame=True)\n",
    "X_raw.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean radius</th>\n",
       "      <th>mean texture</th>\n",
       "      <th>mean perimeter</th>\n",
       "      <th>mean area</th>\n",
       "      <th>mean smoothness</th>\n",
       "      <th>mean compactness</th>\n",
       "      <th>mean concavity</th>\n",
       "      <th>mean concave points</th>\n",
       "      <th>mean symmetry</th>\n",
       "      <th>mean fractal dimension</th>\n",
       "      <th>...</th>\n",
       "      <th>worst radius</th>\n",
       "      <th>worst texture</th>\n",
       "      <th>worst perimeter</th>\n",
       "      <th>worst area</th>\n",
       "      <th>worst smoothness</th>\n",
       "      <th>worst compactness</th>\n",
       "      <th>worst concavity</th>\n",
       "      <th>worst concave points</th>\n",
       "      <th>worst symmetry</th>\n",
       "      <th>worst fractal dimension</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>...</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>569.000000</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>5.690000e+02</td>\n",
       "      <td>5.690000e+02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>-1.311195e-16</td>\n",
       "      <td>6.243785e-17</td>\n",
       "      <td>-1.248757e-16</td>\n",
       "      <td>-2.185325e-16</td>\n",
       "      <td>-8.366672e-16</td>\n",
       "      <td>1.998011e-16</td>\n",
       "      <td>3.746271e-17</td>\n",
       "      <td>-4.995028e-17</td>\n",
       "      <td>1.748260e-16</td>\n",
       "      <td>4.838933e-16</td>\n",
       "      <td>...</td>\n",
       "      <td>-8.241796e-16</td>\n",
       "      <td>1.248757e-17</td>\n",
       "      <td>-3.496520e-16</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-2.122887e-16</td>\n",
       "      <td>-3.621395e-16</td>\n",
       "      <td>8.741299e-17</td>\n",
       "      <td>2.122887e-16</td>\n",
       "      <td>2.622390e-16</td>\n",
       "      <td>-5.744282e-16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>...</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "      <td>1.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-2.027864e+00</td>\n",
       "      <td>-2.227289e+00</td>\n",
       "      <td>-1.982759e+00</td>\n",
       "      <td>-1.453164e+00</td>\n",
       "      <td>-3.109349e+00</td>\n",
       "      <td>-1.608721e+00</td>\n",
       "      <td>-1.113893e+00</td>\n",
       "      <td>-1.260710e+00</td>\n",
       "      <td>-2.741705e+00</td>\n",
       "      <td>-1.818265e+00</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.725382e+00</td>\n",
       "      <td>-2.222039e+00</td>\n",
       "      <td>-1.691872e+00</td>\n",
       "      <td>-1.221348</td>\n",
       "      <td>-2.680337e+00</td>\n",
       "      <td>-1.442609e+00</td>\n",
       "      <td>-1.304683e+00</td>\n",
       "      <td>-1.743529e+00</td>\n",
       "      <td>-2.159060e+00</td>\n",
       "      <td>-1.600431e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-6.887793e-01</td>\n",
       "      <td>-7.253249e-01</td>\n",
       "      <td>-6.913472e-01</td>\n",
       "      <td>-6.666089e-01</td>\n",
       "      <td>-7.103378e-01</td>\n",
       "      <td>-7.464292e-01</td>\n",
       "      <td>-7.430941e-01</td>\n",
       "      <td>-7.372951e-01</td>\n",
       "      <td>-7.026215e-01</td>\n",
       "      <td>-7.220040e-01</td>\n",
       "      <td>...</td>\n",
       "      <td>-6.743279e-01</td>\n",
       "      <td>-7.479711e-01</td>\n",
       "      <td>-6.889721e-01</td>\n",
       "      <td>-0.641571</td>\n",
       "      <td>-6.906227e-01</td>\n",
       "      <td>-6.804845e-01</td>\n",
       "      <td>-7.558491e-01</td>\n",
       "      <td>-7.557349e-01</td>\n",
       "      <td>-6.412994e-01</td>\n",
       "      <td>-6.913035e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>-2.148925e-01</td>\n",
       "      <td>-1.045442e-01</td>\n",
       "      <td>-2.357726e-01</td>\n",
       "      <td>-2.949274e-01</td>\n",
       "      <td>-3.486040e-02</td>\n",
       "      <td>-2.217454e-01</td>\n",
       "      <td>-3.419391e-01</td>\n",
       "      <td>-3.973715e-01</td>\n",
       "      <td>-7.156354e-02</td>\n",
       "      <td>-1.781226e-01</td>\n",
       "      <td>...</td>\n",
       "      <td>-2.688030e-01</td>\n",
       "      <td>-4.347738e-02</td>\n",
       "      <td>-2.857288e-01</td>\n",
       "      <td>-0.340881</td>\n",
       "      <td>-4.680159e-02</td>\n",
       "      <td>-2.692639e-01</td>\n",
       "      <td>-2.180402e-01</td>\n",
       "      <td>-2.232725e-01</td>\n",
       "      <td>-1.272975e-01</td>\n",
       "      <td>-2.162538e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>4.689800e-01</td>\n",
       "      <td>5.836621e-01</td>\n",
       "      <td>4.992377e-01</td>\n",
       "      <td>3.631877e-01</td>\n",
       "      <td>6.356397e-01</td>\n",
       "      <td>4.934227e-01</td>\n",
       "      <td>5.255994e-01</td>\n",
       "      <td>6.463664e-01</td>\n",
       "      <td>5.303125e-01</td>\n",
       "      <td>4.705693e-01</td>\n",
       "      <td>...</td>\n",
       "      <td>5.215568e-01</td>\n",
       "      <td>6.577623e-01</td>\n",
       "      <td>5.398040e-01</td>\n",
       "      <td>0.357275</td>\n",
       "      <td>5.970195e-01</td>\n",
       "      <td>5.391944e-01</td>\n",
       "      <td>5.306742e-01</td>\n",
       "      <td>7.118836e-01</td>\n",
       "      <td>4.497425e-01</td>\n",
       "      <td>4.503661e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>3.967796e+00</td>\n",
       "      <td>4.647799e+00</td>\n",
       "      <td>3.972634e+00</td>\n",
       "      <td>5.245913e+00</td>\n",
       "      <td>4.766717e+00</td>\n",
       "      <td>4.564409e+00</td>\n",
       "      <td>4.239858e+00</td>\n",
       "      <td>3.924477e+00</td>\n",
       "      <td>4.480808e+00</td>\n",
       "      <td>4.906602e+00</td>\n",
       "      <td>...</td>\n",
       "      <td>4.090590e+00</td>\n",
       "      <td>3.882489e+00</td>\n",
       "      <td>4.283568e+00</td>\n",
       "      <td>5.924959</td>\n",
       "      <td>3.951897e+00</td>\n",
       "      <td>5.108382e+00</td>\n",
       "      <td>4.696536e+00</td>\n",
       "      <td>2.683516e+00</td>\n",
       "      <td>6.040726e+00</td>\n",
       "      <td>6.840837e+00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        mean radius  mean texture  mean perimeter     mean area  \\\n",
       "count  5.690000e+02  5.690000e+02    5.690000e+02  5.690000e+02   \n",
       "mean  -1.311195e-16  6.243785e-17   -1.248757e-16 -2.185325e-16   \n",
       "std    1.000000e+00  1.000000e+00    1.000000e+00  1.000000e+00   \n",
       "min   -2.027864e+00 -2.227289e+00   -1.982759e+00 -1.453164e+00   \n",
       "25%   -6.887793e-01 -7.253249e-01   -6.913472e-01 -6.666089e-01   \n",
       "50%   -2.148925e-01 -1.045442e-01   -2.357726e-01 -2.949274e-01   \n",
       "75%    4.689800e-01  5.836621e-01    4.992377e-01  3.631877e-01   \n",
       "max    3.967796e+00  4.647799e+00    3.972634e+00  5.245913e+00   \n",
       "\n",
       "       mean smoothness  mean compactness  mean concavity  mean concave points  \\\n",
       "count     5.690000e+02      5.690000e+02    5.690000e+02         5.690000e+02   \n",
       "mean     -8.366672e-16      1.998011e-16    3.746271e-17        -4.995028e-17   \n",
       "std       1.000000e+00      1.000000e+00    1.000000e+00         1.000000e+00   \n",
       "min      -3.109349e+00     -1.608721e+00   -1.113893e+00        -1.260710e+00   \n",
       "25%      -7.103378e-01     -7.464292e-01   -7.430941e-01        -7.372951e-01   \n",
       "50%      -3.486040e-02     -2.217454e-01   -3.419391e-01        -3.973715e-01   \n",
       "75%       6.356397e-01      4.934227e-01    5.255994e-01         6.463664e-01   \n",
       "max       4.766717e+00      4.564409e+00    4.239858e+00         3.924477e+00   \n",
       "\n",
       "       mean symmetry  mean fractal dimension  ...  worst radius  \\\n",
       "count   5.690000e+02            5.690000e+02  ...  5.690000e+02   \n",
       "mean    1.748260e-16            4.838933e-16  ... -8.241796e-16   \n",
       "std     1.000000e+00            1.000000e+00  ...  1.000000e+00   \n",
       "min    -2.741705e+00           -1.818265e+00  ... -1.725382e+00   \n",
       "25%    -7.026215e-01           -7.220040e-01  ... -6.743279e-01   \n",
       "50%    -7.156354e-02           -1.781226e-01  ... -2.688030e-01   \n",
       "75%     5.303125e-01            4.705693e-01  ...  5.215568e-01   \n",
       "max     4.480808e+00            4.906602e+00  ...  4.090590e+00   \n",
       "\n",
       "       worst texture  worst perimeter  worst area  worst smoothness  \\\n",
       "count   5.690000e+02     5.690000e+02  569.000000      5.690000e+02   \n",
       "mean    1.248757e-17    -3.496520e-16    0.000000     -2.122887e-16   \n",
       "std     1.000000e+00     1.000000e+00    1.000000      1.000000e+00   \n",
       "min    -2.222039e+00    -1.691872e+00   -1.221348     -2.680337e+00   \n",
       "25%    -7.479711e-01    -6.889721e-01   -0.641571     -6.906227e-01   \n",
       "50%    -4.347738e-02    -2.857288e-01   -0.340881     -4.680159e-02   \n",
       "75%     6.577623e-01     5.398040e-01    0.357275      5.970195e-01   \n",
       "max     3.882489e+00     4.283568e+00    5.924959      3.951897e+00   \n",
       "\n",
       "       worst compactness  worst concavity  worst concave points  \\\n",
       "count       5.690000e+02     5.690000e+02          5.690000e+02   \n",
       "mean       -3.621395e-16     8.741299e-17          2.122887e-16   \n",
       "std         1.000000e+00     1.000000e+00          1.000000e+00   \n",
       "min        -1.442609e+00    -1.304683e+00         -1.743529e+00   \n",
       "25%        -6.804845e-01    -7.558491e-01         -7.557349e-01   \n",
       "50%        -2.692639e-01    -2.180402e-01         -2.232725e-01   \n",
       "75%         5.391944e-01     5.306742e-01          7.118836e-01   \n",
       "max         5.108382e+00     4.696536e+00          2.683516e+00   \n",
       "\n",
       "       worst symmetry  worst fractal dimension  \n",
       "count    5.690000e+02             5.690000e+02  \n",
       "mean     2.622390e-16            -5.744282e-16  \n",
       "std      1.000000e+00             1.000000e+00  \n",
       "min     -2.159060e+00            -1.600431e+00  \n",
       "25%     -6.412994e-01            -6.913035e-01  \n",
       "50%     -1.272975e-01            -2.162538e-01  \n",
       "75%      4.497425e-01             4.503661e-01  \n",
       "max      6.040726e+00             6.840837e+00  \n",
       "\n",
       "[8 rows x 30 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Question: What happens if you don't normalize your data? Give it a try.\n",
    "\n",
    "X_df = (X_raw-X_raw.mean())/X_raw.std()\n",
    "X_df.describe()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, we transform data to `torch.tensor` and do a train test split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We convert dataframe to PyTorch tensor datatype,\n",
    "# and then split it into training and testing parts.\n",
    "X = torch.tensor(X_df.to_numpy(),dtype=torch.float32)\n",
    "m,n = X.shape\n",
    "y = torch.tensor(y_df.to_numpy(),dtype=torch.float32).reshape(m,1)\n",
    "\n",
    "# We use an approx 6:4 train test splitting\n",
    "cases = ['train','test']\n",
    "case_list = np.random.choice(cases,size=X.shape[0],replace=True,p=[0.6,0.4])\n",
    "X_train = X[case_list=='train']\n",
    "X_test = X[case_list=='test']\n",
    "y_train = y[case_list=='train']\n",
    "y_test = y[case_list=='test']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 1: average BCE loss is 0.504\n",
      "Iter 1000: average BCE loss is 0.231\n",
      "Iter 2000: average BCE loss is 0.179\n",
      "Iter 3000: average BCE loss is 0.155\n",
      "Iter 4000: average BCE loss is 0.140\n",
      "Iter 5000: average BCE loss is 0.130\n",
      "Iter 6000: average BCE loss is 0.122\n",
      "Iter 7000: average BCE loss is 0.116\n",
      "Iter 8000: average BCE loss is 0.111\n",
      "Iter 9000: average BCE loss is 0.107\n",
      "Iter 10000: average BCE loss is 0.104\n"
     ]
    }
   ],
   "source": [
    "h = torch.nn.Linear(\n",
    "    in_features=n,\n",
    "    out_features=1,\n",
    "    bias=True\n",
    ")\n",
    "sigma = torch.nn.Sigmoid()\n",
    "\n",
    "# Logistic model is linear+sigmoid\n",
    "f = torch.nn.Sequential(\n",
    "    h,\n",
    "    sigma\n",
    ")\n",
    "\n",
    "J_BCE = torch.nn.BCELoss()\n",
    "GD_optimizer = torch.optim.SGD(lr=0.001,params=f.parameters())\n",
    "\n",
    "nIter = 10000\n",
    "printInterval = 1000\n",
    "\n",
    "for i in range(nIter):\n",
    "    GD_optimizer.zero_grad()\n",
    "    pred = f(X_train)\n",
    "    loss = J_BCE(pred,y_train)\n",
    "    loss.backward()\n",
    "    GD_optimizer.step()\n",
    "    if i == 0 or ((i+1)%printInterval) == 0:\n",
    "        print('Iter {}: average BCE loss is {:.3f}'.format(i+1,loss.item()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on test dataset is 98.52%\n"
     ]
    }
   ],
   "source": [
    "# Test on test data\n",
    "\n",
    "threshold = 0.5\n",
    "\n",
    "with torch.no_grad():\n",
    "    pred_test = f(X_test)\n",
    "\n",
    "# The output has shape M*1. Use .squeeze to remove the trailing dimension with size 1.\n",
    "binary_pred = np.where(pred_test.squeeze()>threshold,'Malignant','Benign')\n",
    "label = np.where(y_test.squeeze()>0.5,'Malignant','Benign')\n",
    "acc = (binary_pred==label).sum()/binary_pred.shape[0]\n",
    "print('Accuracy on test dataset is {:.2f}%'.format(acc*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 294,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Pred</th>\n",
       "      <th>Benign</th>\n",
       "      <th>Malignant</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Label</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Benign</th>\n",
       "      <td>79</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Malignant</th>\n",
       "      <td>1</td>\n",
       "      <td>134</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Pred       Benign  Malignant\n",
       "Label                       \n",
       "Benign         79          2\n",
       "Malignant       1        134"
      ]
     },
     "execution_count": 294,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.crosstab(\n",
    "    index=label,\n",
    "    columns=binary_pred,\n",
    "    rownames=['Label'],\n",
    "    colnames=['Pred']\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 297,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "config": {
        "plotlyServerURL": "https://plot.ly"
       },
       "data": [
        {
         "hovertemplate": "label=Benign<br>index=%{x}<br>pred_probability=%{y}<extra></extra>",
         "legendgroup": "Benign",
         "marker": {
          "color": "#636efa",
          "symbol": "circle"
         },
         "mode": "markers",
         "name": "Benign",
         "orientation": "v",
         "showlegend": true,
         "type": "scatter",
         "x": [
          0,
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          13,
          14,
          15,
          16,
          17,
          18,
          20,
          21,
          22,
          24,
          28,
          34,
          37,
          38,
          39,
          49,
          53,
          56,
          57,
          60,
          62,
          66,
          67,
          68,
          70,
          72,
          74,
          77,
          78,
          81,
          83,
          84,
          88,
          91,
          92,
          98,
          99,
          100,
          101,
          102,
          103,
          107,
          122,
          127,
          128,
          129,
          133,
          134,
          141,
          142,
          146,
          148,
          150,
          152,
          156,
          160,
          172,
          174,
          180,
          185,
          190,
          193,
          194,
          196,
          198,
          202,
          212,
          213,
          214,
          215
         ],
         "xaxis": "x",
         "y": [
          0.000034037668228847906,
          0.00028404552722349763,
          0.006090116687119007,
          0.2604905068874359,
          0.006450210697948933,
          0.143690288066864,
          0.041323479264974594,
          0.03337659314274788,
          0.0826970711350441,
          0.007475544698536396,
          0.002544783055782318,
          0.00006442476296797395,
          0.009250358678400517,
          0.0017523791175335646,
          0.00012075794802512974,
          0.08438608795404434,
          0.0011066208826377988,
          0.34198492765426636,
          0.0008688063244335353,
          0.1354045867919922,
          0.012656038627028465,
          0.0490669384598732,
          0.0010256681125611067,
          0.3613421320915222,
          0.04325533285737038,
          0.0007236137753352523,
          0.0031612752936780453,
          0.0006677585188299417,
          0.005010838154703379,
          0.8280677795410156,
          0.016044309362769127,
          0.260835200548172,
          0.007826286368072033,
          0.0000720908137736842,
          0.000049226920964429155,
          0.07713589072227478,
          0.04795929044485092,
          1.0240366776770315e-7,
          0.0027538929134607315,
          0.0012058905558660626,
          1.055116882753282e-8,
          0.00023778757895343006,
          0.0000491089049319271,
          0.11944492906332016,
          0.005605872254818678,
          0.0005410925368778408,
          0.00024995714193210006,
          0.0010205344296991825,
          0.00007296168041648343,
          0.6458888649940491,
          0.009128005243837833,
          6.029573569321656e-7,
          0.000017588110495125875,
          0.01826884038746357,
          0.13434140384197235,
          0.0033648416865617037,
          0.0006697304779663682,
          0.011951961554586887,
          8.91419205117927e-8,
          0.000014827320228505414,
          0.0009792905766516924,
          0.4074940085411072,
          0.0037554174195975065,
          0.004266864620149136,
          0.0009049384971149266,
          0.011628098785877228,
          0.00033238998730666935,
          0.0017465964192524552,
          0.004836997017264366,
          0.0011069366009905934,
          0.005103526171296835,
          0.27656692266464233,
          0.00043665809789672494,
          0.000005212737960391678,
          0.020257648080587387,
          0.003692596685141325,
          0.360197514295578,
          0.0013284721644595265,
          0.0005022069672122598,
          0.07399850338697433,
          0.000001688951556388929
         ],
         "yaxis": "y"
        },
        {
         "hovertemplate": "label=Malignant<br>index=%{x}<br>pred_probability=%{y}<extra></extra>",
         "legendgroup": "Malignant",
         "marker": {
          "color": "#EF553B",
          "symbol": "circle"
         },
         "mode": "markers",
         "name": "Malignant",
         "orientation": "v",
         "showlegend": true,
         "type": "scatter",
         "x": [
          11,
          12,
          19,
          23,
          25,
          26,
          27,
          29,
          30,
          31,
          32,
          33,
          35,
          36,
          40,
          41,
          42,
          43,
          44,
          45,
          46,
          47,
          48,
          50,
          51,
          52,
          54,
          55,
          58,
          59,
          61,
          63,
          64,
          65,
          69,
          71,
          73,
          75,
          76,
          79,
          80,
          82,
          85,
          86,
          87,
          89,
          90,
          93,
          94,
          95,
          96,
          97,
          104,
          105,
          106,
          108,
          109,
          110,
          111,
          112,
          113,
          114,
          115,
          116,
          117,
          118,
          119,
          120,
          121,
          123,
          124,
          125,
          126,
          130,
          131,
          132,
          135,
          136,
          137,
          138,
          139,
          140,
          143,
          144,
          145,
          147,
          149,
          151,
          153,
          154,
          155,
          157,
          158,
          159,
          161,
          162,
          163,
          164,
          165,
          166,
          167,
          168,
          169,
          170,
          171,
          173,
          175,
          176,
          177,
          178,
          179,
          181,
          182,
          183,
          184,
          186,
          187,
          188,
          189,
          191,
          192,
          195,
          197,
          199,
          200,
          201,
          203,
          204,
          205,
          206,
          207,
          208,
          209,
          210,
          211
         ],
         "xaxis": "x",
         "y": [
          0.8442091345787048,
          0.950793981552124,
          0.9934101700782776,
          0.9840652346611023,
          0.9717535376548767,
          0.9917812943458557,
          0.9929794669151306,
          0.9878712892532349,
          0.9844777584075928,
          0.9995207786560059,
          0.9743314385414124,
          0.95577472448349,
          0.8261221647262573,
          0.538157045841217,
          0.9865062832832336,
          0.9882906675338745,
          0.9631986618041992,
          0.9762551188468933,
          0.991051971912384,
          0.8575484156608582,
          0.7170164585113525,
          0.9949245452880859,
          0.9994295239448547,
          0.9901130199432373,
          0.886405348777771,
          0.9718934893608093,
          0.9789069294929504,
          0.6847632527351379,
          0.9846259951591492,
          0.9901912212371826,
          0.9927868247032166,
          0.9771652221679688,
          0.9955675005912781,
          0.5166444182395935,
          0.9365119338035583,
          0.9796035885810852,
          0.9969383478164673,
          0.9759731292724609,
          0.9721340537071228,
          0.9941034913063049,
          0.9166997075080872,
          0.9021464586257935,
          0.9815157651901245,
          0.9972084164619446,
          0.8472604751586914,
          0.9828040599822998,
          0.9109028577804565,
          0.8909381031990051,
          0.9809402823448181,
          0.915195107460022,
          0.9800710082054138,
          0.9764833450317383,
          0.9739437699317932,
          0.9559590816497803,
          0.99244225025177,
          0.9223405718803406,
          0.9969651103019714,
          0.9672375917434692,
          0.9725691080093384,
          0.9934521913528442,
          0.963740885257721,
          0.9995546936988831,
          0.9957861304283142,
          0.993361234664917,
          0.9713521599769592,
          0.992573618888855,
          0.9932936429977417,
          0.998934805393219,
          0.9979401230812073,
          0.9922531247138977,
          0.9853610396385193,
          0.9769901037216187,
          0.9842021465301514,
          0.982585072517395,
          0.9720836877822876,
          0.9982733726501465,
          0.9974581599235535,
          0.8097907900810242,
          0.9976546168327332,
          0.9937373399734497,
          0.9323890209197998,
          0.975526750087738,
          0.8831868171691895,
          0.9528221487998962,
          0.9344586133956909,
          0.9853963255882263,
          0.9980828762054443,
          0.9637821912765503,
          0.9703492522239685,
          0.9778387546539307,
          0.9688276648521423,
          0.9055966734886169,
          0.9823887944221497,
          0.9321572780609131,
          0.9819508790969849,
          0.9591571092605591,
          0.9633579850196838,
          0.9838588833808899,
          0.9398806691169739,
          0.9982737302780151,
          0.9723122119903564,
          0.9241542220115662,
          0.9235132336616516,
          0.9973483085632324,
          0.8710933327674866,
          0.795330286026001,
          0.8890237212181091,
          0.9550360441207886,
          0.746600866317749,
          0.8831323981285095,
          0.9879520535469055,
          0.9787991046905518,
          0.9489423632621765,
          0.7433968186378479,
          0.9943596720695496,
          0.8626119494438171,
          0.9544083476066589,
          0.7132842540740967,
          0.9815945625305176,
          0.7366730570793152,
          0.7863152623176575,
          0.9463046193122864,
          0.9638271331787109,
          0.9021912813186646,
          0.8957096338272095,
          0.9237420558929443,
          0.9969552755355835,
          0.9936023354530334,
          0.34599435329437256,
          0.8208728432655334,
          0.9372915029525757,
          0.8623172044754028,
          0.9755514860153198,
          0.8774149417877197,
          0.854669451713562
         ],
         "yaxis": "y"
        }
       ],
       "layout": {
        "legend": {
         "title": {
          "text": "label"
         },
         "tracegroupgap": 0
        },
        "margin": {
         "t": 60
        },
        "template": {
         "data": {
          "bar": [
           {
            "error_x": {
             "color": "#2a3f5f"
            },
            "error_y": {
             "color": "#2a3f5f"
            },
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "bar"
           }
          ],
          "barpolar": [
           {
            "marker": {
             "line": {
              "color": "#E5ECF6",
              "width": 0.5
             },
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "barpolar"
           }
          ],
          "carpet": [
           {
            "aaxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "baxis": {
             "endlinecolor": "#2a3f5f",
             "gridcolor": "white",
             "linecolor": "white",
             "minorgridcolor": "white",
             "startlinecolor": "#2a3f5f"
            },
            "type": "carpet"
           }
          ],
          "choropleth": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "choropleth"
           }
          ],
          "contour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "contour"
           }
          ],
          "contourcarpet": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "contourcarpet"
           }
          ],
          "heatmap": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmap"
           }
          ],
          "heatmapgl": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "heatmapgl"
           }
          ],
          "histogram": [
           {
            "marker": {
             "pattern": {
              "fillmode": "overlay",
              "size": 10,
              "solidity": 0.2
             }
            },
            "type": "histogram"
           }
          ],
          "histogram2d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2d"
           }
          ],
          "histogram2dcontour": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "histogram2dcontour"
           }
          ],
          "mesh3d": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "type": "mesh3d"
           }
          ],
          "parcoords": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "parcoords"
           }
          ],
          "pie": [
           {
            "automargin": true,
            "type": "pie"
           }
          ],
          "scatter": [
           {
            "fillpattern": {
             "fillmode": "overlay",
             "size": 10,
             "solidity": 0.2
            },
            "type": "scatter"
           }
          ],
          "scatter3d": [
           {
            "line": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatter3d"
           }
          ],
          "scattercarpet": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattercarpet"
           }
          ],
          "scattergeo": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergeo"
           }
          ],
          "scattergl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattergl"
           }
          ],
          "scattermapbox": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scattermapbox"
           }
          ],
          "scatterpolar": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolar"
           }
          ],
          "scatterpolargl": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterpolargl"
           }
          ],
          "scatterternary": [
           {
            "marker": {
             "colorbar": {
              "outlinewidth": 0,
              "ticks": ""
             }
            },
            "type": "scatterternary"
           }
          ],
          "surface": [
           {
            "colorbar": {
             "outlinewidth": 0,
             "ticks": ""
            },
            "colorscale": [
             [
              0,
              "#0d0887"
             ],
             [
              0.1111111111111111,
              "#46039f"
             ],
             [
              0.2222222222222222,
              "#7201a8"
             ],
             [
              0.3333333333333333,
              "#9c179e"
             ],
             [
              0.4444444444444444,
              "#bd3786"
             ],
             [
              0.5555555555555556,
              "#d8576b"
             ],
             [
              0.6666666666666666,
              "#ed7953"
             ],
             [
              0.7777777777777778,
              "#fb9f3a"
             ],
             [
              0.8888888888888888,
              "#fdca26"
             ],
             [
              1,
              "#f0f921"
             ]
            ],
            "type": "surface"
           }
          ],
          "table": [
           {
            "cells": {
             "fill": {
              "color": "#EBF0F8"
             },
             "line": {
              "color": "white"
             }
            },
            "header": {
             "fill": {
              "color": "#C8D4E3"
             },
             "line": {
              "color": "white"
             }
            },
            "type": "table"
           }
          ]
         },
         "layout": {
          "annotationdefaults": {
           "arrowcolor": "#2a3f5f",
           "arrowhead": 0,
           "arrowwidth": 1
          },
          "autotypenumbers": "strict",
          "coloraxis": {
           "colorbar": {
            "outlinewidth": 0,
            "ticks": ""
           }
          },
          "colorscale": {
           "diverging": [
            [
             0,
             "#8e0152"
            ],
            [
             0.1,
             "#c51b7d"
            ],
            [
             0.2,
             "#de77ae"
            ],
            [
             0.3,
             "#f1b6da"
            ],
            [
             0.4,
             "#fde0ef"
            ],
            [
             0.5,
             "#f7f7f7"
            ],
            [
             0.6,
             "#e6f5d0"
            ],
            [
             0.7,
             "#b8e186"
            ],
            [
             0.8,
             "#7fbc41"
            ],
            [
             0.9,
             "#4d9221"
            ],
            [
             1,
             "#276419"
            ]
           ],
           "sequential": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ],
           "sequentialminus": [
            [
             0,
             "#0d0887"
            ],
            [
             0.1111111111111111,
             "#46039f"
            ],
            [
             0.2222222222222222,
             "#7201a8"
            ],
            [
             0.3333333333333333,
             "#9c179e"
            ],
            [
             0.4444444444444444,
             "#bd3786"
            ],
            [
             0.5555555555555556,
             "#d8576b"
            ],
            [
             0.6666666666666666,
             "#ed7953"
            ],
            [
             0.7777777777777778,
             "#fb9f3a"
            ],
            [
             0.8888888888888888,
             "#fdca26"
            ],
            [
             1,
             "#f0f921"
            ]
           ]
          },
          "colorway": [
           "#636efa",
           "#EF553B",
           "#00cc96",
           "#ab63fa",
           "#FFA15A",
           "#19d3f3",
           "#FF6692",
           "#B6E880",
           "#FF97FF",
           "#FECB52"
          ],
          "font": {
           "color": "#2a3f5f"
          },
          "geo": {
           "bgcolor": "white",
           "lakecolor": "white",
           "landcolor": "#E5ECF6",
           "showlakes": true,
           "showland": true,
           "subunitcolor": "white"
          },
          "hoverlabel": {
           "align": "left"
          },
          "hovermode": "closest",
          "mapbox": {
           "style": "light"
          },
          "paper_bgcolor": "white",
          "plot_bgcolor": "#E5ECF6",
          "polar": {
           "angularaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "radialaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "scene": {
           "xaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "yaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           },
           "zaxis": {
            "backgroundcolor": "#E5ECF6",
            "gridcolor": "white",
            "gridwidth": 2,
            "linecolor": "white",
            "showbackground": true,
            "ticks": "",
            "zerolinecolor": "white"
           }
          },
          "shapedefaults": {
           "line": {
            "color": "#2a3f5f"
           }
          },
          "ternary": {
           "aaxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "baxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           },
           "bgcolor": "#E5ECF6",
           "caxis": {
            "gridcolor": "white",
            "linecolor": "white",
            "ticks": ""
           }
          },
          "title": {
           "x": 0.05
          },
          "xaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          },
          "yaxis": {
           "automargin": true,
           "gridcolor": "white",
           "linecolor": "white",
           "ticks": "",
           "title": {
            "standoff": 15
           },
           "zerolinecolor": "white",
           "zerolinewidth": 2
          }
         }
        },
        "xaxis": {
         "anchor": "y",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "index"
         }
        },
        "yaxis": {
         "anchor": "x",
         "domain": [
          0,
          1
         ],
         "title": {
          "text": "pred_probability"
         }
        }
       }
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Use plotly express (px) to visualize test results.\n",
    "# px expects DataFrame input\n",
    "\n",
    "pred_df = pd.DataFrame(\n",
    "    {\n",
    "        'pred_probability':pred_test.squeeze(),\n",
    "        'label':label\n",
    "    }\n",
    ")\n",
    "fig = px.scatter(data_frame=pred_df,y='pred_probability',color='label')\n",
    "fig"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "aAUE4hqdgV5W",
    "yzmKef0IbBW2"
   ],
   "name": "Gradient_Descent.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "env3.10",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "vscode": {
   "interpreter": {
    "hash": "5ec3c21330053bd888ae0423f793f928621505090c66dc4d3b71e8bb37792d66"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}