tutorial_normalisation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Normalization"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The terms normalization and standardization are sometimes used interchangeably, but they usually refer to different things. Normalization usually means to scale a variable to have a values between 0 and 1, while standardization transforms data to have a mean of zero and a standard deviation of 1. \n",
    "\n",
    "The goal of normalization is to change the values of numeric columns in the dataset to a common scale, without distorting differences in the ranges of values. For machine learning, every dataset does not require normalization. It is required only when features have different ranges. In machine learning it is often required to scale features. For example age will have a range of 0-100 while kiloplatelets/mL might have ranges from 25100 till 800000. So, these two features are in very different ranges. In the analysis the feature platelets will intrinsically influence the result more due to its larger value. To avoid such we scale the features. \n",
    "\n",
    "\n",
    "##  Standardization \n",
    "Standardization converts the values of a given independent variable to a normal distribution with mean zero and standard deviation of 1\n",
    "\n",
    "$\\displaystyle x' = \\frac{x - \\bar{x}}{\\sigma}$\n",
    "\n",
    "\n",
    "\n",
    "## Normalization \n",
    "Also known as min-max scaling or min-max normalization. It is the simplest method and consists in rescaling the range of features to scale the range in [0, 1] or [−1, 1]. Selecting the target range depends on the nature of the data. The general formula for a min-max of [0, 1] is given as:\n",
    "\n",
    "${\\displaystyle x'={\\frac {x-{\\text{min}}(x)}{{\\text{max}}(x)-{\\text{min}}(x)}}}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Standardization in python\n",
    "\n",
    "The preprocessing module of `sklean` provides a utility class `StandardScaler` that scales according the standardization. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this dataset contains 299 rows\n",
      "(299, 11)\n",
      "(299, 1)\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "df = pd.read_csv('data/heart_failure_clinical_records_dataset.csv')\n",
    "print(f'this dataset contains {len(df)} rows')\n",
    "y = np.array(df['DEATH_EVENT']).reshape(-1, 1)\n",
    "X = np.array(df.iloc[:,0:11])\n",
    "print(X.shape)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[7.500e+01, 0.000e+00, 5.820e+02, ..., 1.300e+02, 1.000e+00,\n",
       "        0.000e+00],\n",
       "       [5.500e+01, 0.000e+00, 7.861e+03, ..., 1.360e+02, 1.000e+00,\n",
       "              nan],\n",
       "       [6.500e+01, 0.000e+00, 1.460e+02, ..., 1.290e+02, 1.000e+00,\n",
       "        1.000e+00],\n",
       "       ...,\n",
       "       [4.500e+01, 0.000e+00, 2.060e+03, ..., 1.380e+02, 0.000e+00,\n",
       "        0.000e+00],\n",
       "       [4.500e+01, 0.000e+00, 2.413e+03, ..., 1.400e+02, 1.000e+00,\n",
       "        1.000e+00],\n",
       "       [5.000e+01, 0.000e+00, 1.960e+02, ..., 1.360e+02, 1.000e+00,\n",
       "        1.000e+00]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# normaliseer data\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = StandardScaler()\n",
    "scaler = scaler.fit(X)\n",
    "X_scale = scaler.transform(X)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "array([[ 1.18934352, -0.87872174,  0.23365768, ..., -1.74232053,\n",
    "         0.72804852, -0.6770032 ],\n",
    "       [ 0.34147055, -0.87872174, -0.57955124, ..., -1.99760559,\n",
    "         0.72804852,  1.47709789],\n",
    "       [-0.93033892,  1.13801668, -0.64483177, ...,  0.04467489,\n",
    "         0.72804852, -0.6770032 ],\n",
    "       ...,\n",
    "       [-0.50640243, -0.87872174,  2.54272337, ...,  0.555245  ,\n",
    "        -1.37353483, -0.6770032 ],\n",
    "       [-1.35427541, -0.87872174,  3.64876211, ...,  0.81053006,\n",
    "         0.72804852,  1.47709789],\n",
    "       [-0.93033892, -0.87872174, -0.48629334, ..., -0.21061017,\n",
    "         0.72804852,  1.47709789]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If your data contains many outliers, scaling using the mean and variance of the data is likely to not work very well. In these cases, you can use `robust_scale` and `RobustScale`r as drop-in replacements instead. They use more robust estimates for the center and range of your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this dataset contains 299 rows\n",
      "(299, 11)\n",
      "(299, 1)\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "df = pd.read_csv('data/heart_failure_clinical_records_dataset.csv')\n",
    "print(f'this dataset contains {len(df)} rows')\n",
    "y = np.array(df['DEATH_EVENT']).reshape(-1, 1)\n",
    "X = np.array(df.iloc[:,0:11])\n",
    "print(X.shape)\n",
    "print(y.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## min-max normalization in python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.63636364, 0.        , 0.07131921, ..., 0.48571429, 1.        ,\n",
       "        0.        ],\n",
       "       [0.27272727, 0.        , 1.        , ..., 0.65714286, 1.        ,\n",
       "               nan],\n",
       "       [0.45454545, 0.        , 0.01569278, ..., 0.45714286, 1.        ,\n",
       "        1.        ],\n",
       "       ...,\n",
       "       [0.09090909, 0.        , 0.25988773, ..., 0.71428571, 0.        ,\n",
       "        0.        ],\n",
       "       [0.09090909, 0.        , 0.30492473, ..., 0.77142857, 1.        ,\n",
       "        1.        ],\n",
       "       [0.18181818, 0.        , 0.02207196, ..., 0.65714286, 1.        ,\n",
       "        1.        ]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# normaliseer data\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "scaler = MinMaxScaler()\n",
    "scaler = scaler.fit(X)\n",
    "X_scale = scaler.transform(X)\n",
    "X_scale"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More to read\n",
    "\n",
    "Next to the most common methods of normalisation there are a couple more. More to read:  \n",
    "https://scikit-learn.org/stable/modules/preprocessing.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}