Assignment-1/programming/dou-salaries/salary-prediction-wlr.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Передбачення зарплат на IT-ринку України"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "У цьому завданні ви працюватимете з реальними даними з [зарплатного опитування DOU.ua за травень 2016р](https://dou.ua/lenta/articles/salary-report-may-june-2016/). Ви реалізуєте зважену лінійну регресію, яка передбачає зарплати Java-інженерів, та навчите свою модель за допомогою градієнтного спуску.\n",
    "\n",
    "Заповніть пропущений код в розділі «Моделювання» (позначено коментарями) та запустіть розділ «Тестування», щоб перевірити його правильність."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext ipython_unittest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Підготовка даних"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_salaries = pd.read_csv(\"data/2016_may_final.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Оберемо тільки Java-інженерів з-поміж усіх респондентів."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_java = pd.DataFrame(df_salaries[(df_salaries[\"Язык.программирования\"] == \\\n",
    "                                                                    \"Java\") &\n",
    "                                   (df_salaries[\"cls\"] == \"DEV\")])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Перейменуємо деякі колонки:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_java.rename(\n",
    "    columns={\n",
    "        \"exp\": \"TotalExperience\",\n",
    "        \"loc\": \"Location\"\n",
    "    },\n",
    "    inplace=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Закодуємо рівень англійської мови числами від 1 (найнижчий) до 5 (найвищий):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_java[\"EnglishLevel\"] = df_java[\"Уровень.английского\"].map({\n",
    "    \"элементарный\": 1,\n",
    "    \"ниже среднего\": 2,\n",
    "    \"средний\": 3,\n",
    "    \"выше среднего\": 4,\n",
    "    \"продвинутый\": 5\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Закодуємо колонку Location (найбільші IT-міста або \"other\") за допомогою one-hot encoding:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "city_columns = [\n",
    "    \"LocationOther\",\n",
    "    \"LocationDnipro\",\n",
    "    \"LocationKyiv\",\n",
    "    \"LocationLviv\",\n",
    "    \"LocationOdesa\",\n",
    "    \"LocationKharkiv\"\n",
    "]\n",
    "df_java[city_columns] = pd.get_dummies(df_java[\"Location\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Відберемо такі ознаки:\n",
    "\n",
    "* Загальна кількість років досвіду\n",
    "* Рівень англійської мови\n",
    "* Місто"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_columns = [\"TotalExperience\", \"EnglishLevel\"] + city_columns\n",
    "df_X = df_java[feature_columns]\n",
    "df_y = df_java[[\"salary\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X shape: (929, 8)\n"
     ]
    }
   ],
   "source": [
    "print(\"X shape:\", df_X.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TotalExperience</th>\n",
       "      <th>EnglishLevel</th>\n",
       "      <th>LocationOther</th>\n",
       "      <th>LocationDnipro</th>\n",
       "      <th>LocationKyiv</th>\n",
       "      <th>LocationLviv</th>\n",
       "      <th>LocationOdesa</th>\n",
       "      <th>LocationKharkiv</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.5</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>5.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>4.0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>6.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>3.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>3.0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>2.0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    TotalExperience  EnglishLevel  LocationOther  LocationDnipro  \\\n",
       "5               0.5             3              1               0   \n",
       "7               5.0             1              0               0   \n",
       "17              0.0             3              0               0   \n",
       "27              4.0             3              0               0   \n",
       "28              6.0             4              0               0   \n",
       "39              3.0             4              0               1   \n",
       "46              2.0             4              0               0   \n",
       "49              3.0             3              0               0   \n",
       "59              2.0             3              0               0   \n",
       "89              1.0             5              0               0   \n",
       "\n",
       "    LocationKyiv  LocationLviv  LocationOdesa  LocationKharkiv  \n",
       "5              0             0              0                0  \n",
       "7              1             0              0                0  \n",
       "17             1             0              0                0  \n",
       "27             1             0              0                0  \n",
       "28             1             0              0                0  \n",
       "39             0             0              0                0  \n",
       "46             0             0              0                1  \n",
       "49             1             0              0                0  \n",
       "59             0             0              0                1  \n",
       "89             1             0              0                0  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_X.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>3400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>2880</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>1425</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>1700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>1800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>1235</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>1200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    salary\n",
       "5      500\n",
       "7     1600\n",
       "17     600\n",
       "27    3400\n",
       "28    2880\n",
       "39    1425\n",
       "46    1700\n",
       "49    1800\n",
       "59    1235\n",
       "89    1200"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_y.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Розділимо вибірку на навчальну та тестову:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_set_size = 0.8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_assignment = np.random.uniform(size=len(df_X))\n",
    "\n",
    "X_train = df_X[dataset_assignment <= training_set_size].values\n",
    "y_train = df_y[dataset_assignment <= training_set_size].values.flatten()\n",
    "\n",
    "X_test = df_X[dataset_assignment > training_set_size].values\n",
    "y_test = df_y[dataset_assignment > training_set_size].values.flatten()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Щоб градієнтний спуск швидше збігався, нормалізуємо навчальну вибірку так, щоб кожна ознака мала $\\mu = 0, \\sigma = 1$:\n",
    "\n",
    "$ x' = \\frac{x - \\bar{x}}{\\sigma}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_means = np.average(X_train, axis=0)\n",
    "feature_sigmas = np.std(X_train, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = (X_train - feature_means) / feature_sigmas\n",
    "X_test = (X_test - feature_means) / feature_sigmas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Додаємо уявну ознаку $x_0 = 1$ (intercept term)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not np.all(X_train[:, 0] == 1):\n",
    "    X_train = np.insert(X_train, 0, values=1, axis=1)\n",
    "    \n",
    "if not np.all(X_test[:, 0] == 1):\n",
    "    X_test = np.insert(X_test, 0, values=1, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X train:  (741, 9)\n",
      "y train:  (741,)\n",
      "\n",
      "X test:   (188, 9)\n",
      "y test:   (188,)\n"
     ]
    }
   ],
   "source": [
    "print(\"X train: \", X_train.shape)\n",
    "print(\"y train: \", y_train.shape)\n",
    "print()\n",
    "print(\"X test:  \", X_test.shape)\n",
    "print(\"y test:  \", y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Моделювання"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Реалізуйте функцію гіпотези лінійної регресії в матричній формі."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_linear(theta, X):\n",
    "    # =============== TODO: Your code here ===============\n",
    "    # Compute the hypothesis function for linear regression.\n",
    "    y_hat = X.dot(theta)\n",
    "    return y_hat\n",
    "    # ===================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Реалізуйте функцію зважування всіх навчальних прикладів $x^{(i)}$, якщо нам дана точка передбачення $x$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_example_weights(X, x_pred, tau):\n",
    "    # =============== TODO: Your code here ===============\n",
    "    # Compute the weight for each example, given the\n",
    "    # prediction point (x_pred).\n",
    "    W = np.exp(- (X - x_pred).dot((X - x_pred).T) /(2 * tau ** 2))\n",
    "    weights = np.diag(W)\n",
    "    return weights\n",
    "    # ===================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Реалізуйте функцію втрат зваженої лінійної регресії. Подумайте, як обчислити цей вираз відразу в матричному вигляді."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cost_function(theta, X, y, weights):\n",
    "    # =============== TODO: Your code here ===============\n",
    "    # Given the currently learned model weights (theta),\n",
    "    # compute the overall loss on the training set (X),\n",
    "    # taking the weights into account.\n",
    "    W = np.diag(weights) / 2\n",
    "    y_hat = predict_linear(theta, X)\n",
    "    res = y_hat - y\n",
    "    J = res.T.dot(W.dot(res)) / len(y)\n",
    "    return J\n",
    "    # ===================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Реалізуйте обчислення градієнта функції втрат зваженої лінійної регресії. Подумайте, як обчислити цей вираз відразу в матричному вигляді."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cost_function_gradient(theta, X, y, weights):\n",
    "    # =============== TODO: Your code here ===============\n",
    "    # Given the currently learned model weights (theta),\n",
    "    # compute the gradient of the cost function on the\n",
    "    # training set (X), taking the weights into account.\n",
    "    W = np.diag(weights)\n",
    "    y_hat = predict_linear(theta, X)\n",
    "    res = y_hat - y\n",
    "    grad_J = X.T.dot(W.dot(res))\n",
    "    return grad_J \n",
    "    # ===================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Реалізуйте один крок градієнтного спуску."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_model_weights(theta, learning_rate, cost_gradient):\n",
    "    # =============== TODO: Your code here ===============\n",
    "    # Given the learning rate and the gradient of the\n",
    "    # cost function, take one gradient descent step and\n",
    "    # return the updated vector theta.\n",
    "    return theta - learning_rate * cost_gradient\n",
    "    # ===================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Навчаємо модель:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(X, y, weights, loss_fun, grad_fun, learning_rate,\n",
    "                     convergence_threshold, max_iters, verbose=False):\n",
    "    theta = np.zeros(X.shape[1])\n",
    "    losses = []\n",
    "    \n",
    "    for i in range(max_iters):\n",
    "        loss = loss_fun(theta, X, y, weights)\n",
    "        losses.append(loss)\n",
    "        \n",
    "        if verbose:\n",
    "            print(\"Iteration: {0:3} Loss: {1}\".format(i + 1, loss))\n",
    "\n",
    "        if len(losses) > 2 \\\n",
    "        and np.abs(losses[-1] - losses[-2]) <= convergence_threshold:\n",
    "            break\n",
    "        \n",
    "        grad = grad_fun(theta, X, y, weights)\n",
    "        theta = update_model_weights(theta, learning_rate, grad)\n",
    "        \n",
    "    return theta, np.array(losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Передбачення нових даних"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_weighted_linear(X, y, x_pred, verbose=False):\n",
    "    weights = get_example_weights(X, x_pred, tau=0.1)\n",
    "    theta, losses = gradient_descent(\n",
    "        X,\n",
    "        y,\n",
    "        weights,\n",
    "        loss_fun=cost_function,\n",
    "        grad_fun=cost_function_gradient,\n",
    "        learning_rate=0.005,\n",
    "        convergence_threshold=0.0001,\n",
    "        max_iters=500,\n",
    "        verbose=verbose\n",
    "    )\n",
    "    return predict_linear(theta, x_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_pred = X_train[4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration:   1 Loss: 2051.275762424122\n",
      "Iteration:   2 Loss: 1513.80754533939\n",
      "Iteration:   3 Loss: 1121.5661412085203\n",
      "Iteration:   4 Loss: 835.3104813807897\n",
      "Iteration:   5 Loss: 626.4026457393463\n",
      "Iteration:   6 Loss: 473.942834742058\n",
      "Iteration:   7 Loss: 362.6784874795104\n",
      "Iteration:   8 Loss: 281.4783673182373\n",
      "Iteration:   9 Loss: 222.21895783853518\n",
      "Iteration:  10 Loss: 178.97176060000322\n",
      "Iteration:  11 Loss: 147.410189437005\n",
      "Iteration:  12 Loss: 124.37672511648795\n",
      "Iteration:  13 Loss: 107.56702714333422\n",
      "Iteration:  14 Loss: 95.29940026075593\n",
      "Iteration:  15 Loss: 86.3465523464754\n",
      "Iteration:  16 Loss: 79.8128122333628\n",
      "Iteration:  17 Loss: 75.04452393759317\n",
      "Iteration:  18 Loss: 71.56465284995903\n",
      "Iteration:  19 Loss: 69.02506168722135\n",
      "Iteration:  20 Loss: 67.1716817389729\n",
      "Iteration:  21 Loss: 65.8190950315699\n",
      "Iteration:  22 Loss: 64.83198452854961\n",
      "Iteration:  23 Loss: 64.11159658700915\n",
      "Iteration:  24 Loss: 63.5858613313323\n",
      "Iteration:  25 Loss: 63.2021825552563\n",
      "Iteration:  26 Loss: 62.92217583131277\n",
      "Iteration:  27 Loss: 62.71782841149648\n",
      "Iteration:  28 Loss: 62.56869674349827\n",
      "Iteration:  29 Loss: 62.45986123317697\n",
      "Iteration:  30 Loss: 62.3804336412496\n",
      "Iteration:  31 Loss: 62.322467789451316\n",
      "Iteration:  32 Loss: 62.2801645997701\n",
      "Iteration:  33 Loss: 62.24929193637043\n",
      "Iteration:  34 Loss: 62.22676120936045\n",
      "Iteration:  35 Loss: 62.2103183825062\n",
      "Iteration:  36 Loss: 62.19831847232921\n",
      "Iteration:  37 Loss: 62.189560978764874\n",
      "Iteration:  38 Loss: 62.18316978334519\n",
      "Iteration:  39 Loss: 62.178505499540584\n",
      "Iteration:  40 Loss: 62.175101506512576\n",
      "Iteration:  41 Loss: 62.172617266891585\n",
      "Iteration:  42 Loss: 62.17080425834318\n",
      "Iteration:  43 Loss: 62.16948111060899\n",
      "Iteration:  44 Loss: 62.1685154606533\n",
      "Iteration:  45 Loss: 62.16781071064699\n",
      "Iteration:  46 Loss: 62.167296364015726\n",
      "Iteration:  47 Loss: 62.16692097274004\n",
      "Iteration:  48 Loss: 62.166646990332964\n",
      "Iteration:  49 Loss: 62.16644701557097\n",
      "Iteration:  50 Loss: 62.166301051189\n",
      "Iteration:  51 Loss: 62.166194503290946\n",
      "Iteration:  52 Loss: 62.16611672133018\n"
     ]
    }
   ],
   "source": [
    "predicted = predict_weighted_linear(X_train, y_train, x_pred, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Оцінимо, наскільки модель помиляється на тестовій вибірці:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test_pred = np.array([predict_weighted_linear(X_train, y_train, X_test[i]) \\\n",
    "                        for i in range(len(y_test))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_residuals = pd.DataFrame(y_test - y_test_pred, columns=[\"residual\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Text(0.5, 1.0, 'Error Distribution')]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnYAAAFDCAYAAAC3C0N6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAHv9JREFUeJzt3X+8ZXVd7/HX20EMrz8AGRUZhiGZTLyR0oh0K8QfwZAm1EMSbuVcsyZMkltWDGlxFeliqXQxMTGIsashWQQBhaNXxboiDFcCB0QmoJgLBjSAKIqCn/7Y6zzaDvucs8+Zs3+t/Xo+Hutx1v6utfb6rP1j7ff5rrX2TlUhSZKkyfe4URcgSZKkpWGwkyRJagmDnSRJUksY7CRJklrCYCdJktQSBjtJkqSWMNhJ0gIl2ZLk8FmmHZ5k2xKt59NJfnEp7kvSdNhl1AVI0iAluR14BvAo8DXg74ATq+pri73Pqnre0lQnSUvLHjtJ0+Anq+pJwPOBFwCnjLgeSRoIg52kqVFVXwGuoBPwSPKEJO9K8i9J/jXJHyfZrZm2V5JLk9yfZHuSzyZ5XDPt9iQvb8Z3S3J+kvuS3Ai8sHudSSrJAV23z0/yjmZ8j2Yd9zTLX5pkRa/akxyQ5DNJHkhyb5KPDuAhkjThDHaSpkYTmo4CtjZN7wS+j07QOwDYB/jdZtqbgW3AcjqHcn8b6PUbjKcCz26GI4F1CyjpccCfAvsBK4FvAH80y7ynAR8H9gBWAO9dwHokTQmDnaRp8NdJHgTuAO4GTk0S4JeAX6uq7VX1IPB7wHHNMt8G9gb2q6pvV9Vnq/ePa/8McHpzH3cAZ/VbVFX9W1X9ZVU91Kz/dODFs8z+bToB8FlV9c2q+vt+1yNpehjsJE2DY6rqycDhwPcDe9HpiXsicG1zuPV+OhdWLG+W+QM6PXsfT3Jrkg2z3Pez6ATGGf/cb1FJnpjkA0n+OclXgSuB3ZMs6zH7bwEBrm6uyv2FftcjaXoY7CRNjar6DHA+8C7gXjqHPp9XVbs3w1Obiyyoqger6s1V9b3ATwK/nuRlPe72LmDfrtsrd5j+EJ0AOeOZXeNvBp4DvKiqngIc1rSnR+1fqapfqqpnAb8MnN197p4kgcFO0vT5Q+DHgYOADwJnJnk6QJJ9khzZjL+yuWAhwFfpfF3Koz3u70LglOZCiBXAr+4w/TrgvyZZlmQt332o9cl0wuX9Sfakc75eT0mO7bqw4j465/v1qkfSFDPYSZoqVXUP8CHgd4CT6Rxuvao5FPoJOj1oAKub218DPgecXVWf7nGXb6Nz+PU2Ohc3/NkO00+i0+N3P/CzwF93TftDYDc6vYdX0TkUPJsXAp9P8jXgEuCkqrpt/i2WNE3S+1xgSZIkTRp77CRJklrCYCdJktQSBjtJkqSWMNhJkiS1hMFOkiSpJXYZdQGjstdee9WqVatGXYYkSdK8rr322nuravl8801tsFu1ahWbN28edRmSJEnzStLXzxV6KFaSJKklDHaSJEktYbCTJElqCYOdJElSSxjsJEmSWsJgJ0mS1BIGO0mSpJYw2EmSJLWEwU6SJKklDHaSJEktYbCTJE2UVRsuY9WGy0ZdhjSWDHaSJEktYbCTJElqCYOdJElSSxjsJEmSWsJgJ2mqeRK+pDYZWbBL8j1Jrk7yj0m2JHlb075/ks8nuSXJR5Ps2rQ/obm9tZm+quu+Tmnab05y5Gi2SJIkabRG2WP3MPDSqvpB4PnA2iSHAu8Ezqyq1cB9wOub+V8P3FdVBwBnNvOR5EDgOOB5wFrg7CTLhrolkiRJY2Bkwa46vtbcfHwzFPBS4GNN+0bgmGb86OY2zfSXJUnTfkFVPVxVtwFbgUOGsAmSJEljZaTn2CVZluQ64G5gE/BPwP1V9UgzyzZgn2Z8H+AOgGb6A8DTutt7LCNJkjQ1RhrsqurRqno+sIJOL9tze83W/M0s02Zrf4wk65NsTrL5nnvuWUzJkiRJY2ssroqtqvuBTwOHArsn2aWZtAK4sxnfBuwL0Ex/KrC9u73HMjuu55yqWlNVa5YvX77UmyFJkjRSo7wqdnmS3Zvx3YCXAzcBnwJe3cy2Dri4Gb+kuU0z/f9UVTXtxzVXze4PrAauHs5WSJIkjY9d5p9lYPYGNjZXsD4OuLCqLk1yI3BBkncAXwDObeY/F/izJFvp9NQdB1BVW5JcCNwIPAK8saoeHfK2SJIkjdzIgl1VXQ+8oEf7rfS4qrWqvgkcO8t9nQ6cvtQ1SpIkTZKxOMdOkiRJO89gJ0mS1BIGO0mSpJYw2EmSJLWEwU6SJKklDHaSJEktYbCTJElqCYOdJElSSxjsJEmSWsJgJ0mS1BIGO0mSpJYw2EmSJLWEwU6SJKklDHaSJEktYbCTJElqCYOdpJFZteEyVm24bNRlaIF8zqTxZbCTJElqCYOdJElSSxjsJEmSWsJgJ0mS1BIGO0mSpJYw2EmSJLWEwU6SJKklDHaSJEktYbCTJElqCYOdJElSSxjsJEmSWsJgtwj+vqUkSRpHBjtJkqSWMNhJE87eY0nSDIOdJElSS4ws2CXZN8mnktyUZEuSk5r2PZNsSnJL83ePpj1JzkqyNcn1SQ7uuq91zfy3JFk3qm2SJEmTbdKPgoyyx+4R4M1V9VzgUOCNSQ4ENgCfrKrVwCeb2wBHAaubYT3wfugEQeBU4EXAIcCpM2FQkiRpmows2FXVXVX1/5rxB4GbgH2Ao4GNzWwbgWOa8aOBD1XHVcDuSfYGjgQ2VdX2qroP2ASsHeKmSJIkjYWxOMcuySrgBcDngWdU1V3QCX/A05vZ9gHu6FpsW9M2W7skSdJUGXmwS/Ik4C+B/15VX51r1h5tNUd7r3WtT7I5yeZ77rln4cVKU2y272/0ex0laXyMNNgleTydUPfhqvqrpvlfm0OsNH/vbtq3Aft2Lb4CuHOO9seoqnOqak1VrVm+fPnSbYgkSdIYGOVVsQHOBW6qqvd0TboEmLmydR1wcVf7a5urYw8FHmgO1V4BHJFkj+aiiSOaNkmSpKmyywjX/SPAzwM3JLmuaftt4AzgwiSvB/4FOLaZdjnwE8BW4CHgdQBVtT3JacA1zXxvr6rtw9kESZKk8TGyYFdVf0/v8+MAXtZj/gLeOMt9nQect3TVSZIkTZ6RXzwhSZKkpWGwkyRJagmDnSRJUksY7CRJklrCYCdJktQSBjtJ38VfkpCkyWWwkyRJagmDnSRJUksY7CRJklrCYCdJktQSBjtJUqt5QZCmydQGuxv+/wO+0SeQO2hJkmY3tcFOkiSpbQx2kiRJLWGwkyRJagmDnSRJUksY7CRJklrCYCdJktQSBjtJkqSWMNhJmjp+F6KktjLYSbPww3/8+JxI0twMdpIkSS1hsJOkAfEn8CQNm8FOkiSpJQx2kiRJLTFvsEvyvCTLm/GnJfmTJBckOXDw5UmSJKlf/fTY/XHX+OnAV4CLgPMGUpGksee5Y5I0nuYMdklOBQ4A3tCM/xSwDPh+YEWS301y2ODLlKT2MBhLGpRd5ppYVW9LcgzwEeCZwGFVdQpAkpdX1duHUKMkSZL6MGewa7wduBL4NnAcdM67A+4dYF1TYeY/9tvPeMWIK5Gk6eM+WG007zl2VXVRVT2rqvarqs81bVuq6qcGX55GxcNEkiRNnpF+3UmS85LcneSLXW17JtmU5Jbm7x5Ne5KclWRrkuuTHNy1zLpm/luSrBvFtkiSJI3aqL/H7nxg7Q5tG4BPVtVq4JPNbYCjgNXNsB54P3SCIHAq8CLgEODUmTAoSZI0TUYa7KrqSmD7Ds1HAxub8Y3AMV3tH6qOq4Ddk+wNHAlsqqrtVXUfsInHhkVJkqTWG3WPXS/PqKq7AJq/T2/a9wHu6JpvW9M2W/tjJFmfZHOSzY8+9MCSFy5JkjRKiwp2SW5qhhOXuqC5VtujreZof2xj1TlVtaaq1ix74lOXtDhJkqRRW1Swq6rnAj8K3La05QDwr80hVpq/dzft24B9u+ZbAdw5R7skTT2/DFmaLv38VuyyJJ/Ysb2q/q2qBrG3uASYubJ1HXBxV/trm6tjDwUeaA7VXgEckWSP5qKJI5o2SZoaBjhpYdr6fpn3C4qr6tEkDyV5alUt6YlpSf4cOBzYK8k2Ole3ngFcmOT1wL8AxzazXw78BLAVeAh4XVPf9iSnAdc08729qna8IEOSJKn1+vnlCYBvAjck2QR8faaxqt60MyuvquNnmfSyHvMW8MZZ7uc84LydqaXtVm24zG9XlySp5foNdpc1g4bMQCZJkvrVV7Crqo1JdgW+r2m6uaq+PbiyJEmStFB9XRWb5HDgFuB9wNnAl5McNsC6NOU8EVwaHt9rUnv0+3Un7waOqKoXV9VhdH7t4czBlSUN1qQHx0muXdLkmvR95zToN9g9vqpunrlRVV8GHj+YkiaLL/IOH4Px5WtUk8DXqLQ0+r14YnOSc4E/a27/LHDtYEqSJEnSYvTbY/cGYAvwJuAk4EbghEEVpfn53+1g+LhKkibZvD12SZYB51bVzwHvGXxJkiRJWox5e+yq6lFgefN1J5pC9mJJkjQZ+j0UezvwD0l+J8mvzwwDrEsLsNDgNa5BbZpP8p/W7ZYkLa1+g92dwKXN/E/uGiRJLbJU/2T4z4o0Gv2eY/ekqvrNIdQjSZKkRer3HLuDh1CLJEkjN82nhWjy9fs9dtcluQT4C+DrM41V9VcDqUqSpAGYCWy3n/GKEVciDUa/59jtCfwb8FLgJ5vhlYMqSpKkSTdbz5+9gd9tEntIx7nevnrsqup1gy5EkiRJO2fOHrskF3aNv3OHaR8fVFGSpN4msXdD0vDMdyh2ddf4j+8wbfkS1yJJkqSdMF+wq0VOkzRC9upI0nSa7xy7JyZ5AZ0AuFsznmbYbdDFSZIkqX/z9djdBbwHeBfwlWb83V23NUT2wEjz832ydHwspcEaxHtszmBXVS+Za1jyajR07rini4do++fjpFFq4+vP/c9w9Ps9dhqwNr3g27IdkjQN3Ge3i8FuDgsNW20KZ5NmmI+9z7E0/twfa1oZ7IbAHcz08fmWpNn5uTg4fQe7JPsk+S9JDpsZBlmYxp9vSmk8+aEpTa++gl3zqxP/ALwV+M1m+I0B1iVNHD9IpYXxPaNBmeZ/bvr6rVjgGOA5VfXwIIuRpLaa+ZC5/YxXjLgSSW3W76HYW4HHD7IQSZIm0aB7h6a150mL02+wewi4LskHkpw1MwyyMO08dwZqs1G9vn1f7RwfP7XNuL2m+w12lwCnAf8XuLZrGBtJ1ia5OcnWJBtGXY80n3HbGaj9pvm8I2kYxuH91Vewq6qNwJ/zH4HuI03bWEiyDHgfcBRwIHB8kgNHW5WkYRmHnelSM4RpKbT9NdT9PvE909HvVbGHA7fQCU9nA18es687OQTYWlW3VtW3gAuAo0dc06L54pSk/gz7y8kNERp3/R6KfTdwRFW9uKoOA44EzhxcWQu2D3BH1+1tTZskSUMzW+AzBGpoqmreAbi+n7ZRDcCxwJ903f554L095lsPbAY2r1y5smbsd/Kltd/Jlz5mvNts7bNZqvtZqJ25736WXexj0M94P/ez0Palem5nm77Qx2yhz89Clt2Z1+ggXveLeR7mG1+ohd7PQp/nxb6md7yPxT4PO7M/2ZnX4mztO/M+6ec+l/o90M+yw9xn78z+ZKHPw0Lm2Zl97TD3IXPdZ6/x2eZZ6LILXe9iH3tgc/WRifrtsduc5NwkhzfDBxmviye2Aft23V4B3LnjTFV1TlWtqao1y5cvH1px0lLw+88kdXOfMDmG+Vz1G+zeAGwB3gScBNwInDCoohbhGmB1kv2T7AocR+dKXmlJuAOVJE2Cvn55ojq/OPGeZhg7VfVIkhOBK4BlwHlVtWXEZakPBqal42O58xb6GE7TYz5N26rR8XW28+YMdkkurKqfSXIDUDtOr6qDBlbZAlXV5cDlo65D48WdxPgZ1XPia2EwfFy1I18TozVfj91Jzd9XDroQSZK0OIYpzZgz2FXVXc3or1TVyd3TkrwTOPmxS0mSFsMP553j4ze3SXt8BlHvpD0Gi9HvxRM/3qPtqKUsRNJjTcNOqJdJ2+7bz3jFxNUsqZ3mO8fuDcCvAM9Ocn3XpCfT+d1YSZLUYv380+I/No81qsdkvnPsPgL8LfA/gQ1d7Q9W1faBVaVWcwcwOXyupOHx/TY6bXrs5zvH7gHggST/C9heVQ8CJHlykhdV1eeHUaQ0SG16Q0uSpltf32MHvB84uOv213u0SWPBoKal5mtK0qToN9il+Z0yAKrqO0n6XVaaaoaC6eLzvfN8DAfP8+baq99wdmuSN9HppYPOBRW3DqYkSZKk/7AzIXPaAmq/we4E4CzgrXR+geKTwPpBFTVs0/akS5Kkdur3t2LvBo4bcC2SJEnaCfN9j91vVdXvJ3kvvX8r9k0Dq0ytYq/o+PE5kaT2ma/H7qbm7+ZBF6LJYBiQ2mGY72X3G9LwzPc9dn/T/N04nHK0s9yBDpeP93frfjx8bAbPx1hqt8W8x+c7FPs39DgEO6OqXrXgNU6oadiBTsM2jpqP8XD5eEuaNvMdin1X8/engWcC/7u5fTxw+4Bqkvrih3a7+HxK6of7irnNdyj2MwBJTquqw7om/U2SKwdamSaebz5Jk8L9ldricX3OtzzJ987cSLI/sHwwJUmSpMUypE63fr+g+NeATyeZ+bWJVcAvD6QiSdJ38YNaUr/6/YLiv0uyGvj+pulLVfXw4MqSJOm7GXCl+fV1KDbJE4HfBE6sqn8EViZ55UArkyT8MJfULoPep/V7KPZPgWuBH25ubwP+Arh0EEWNkh8i7eVzK2ncuF/SUus32D27ql6T5HiAqvpGkgywLkkt4oeXJA1Hv1fFfivJbjRfVpzk2YDn2EmS1CL+Ezb5+u2xOxX4O2DfJB8GfgT4b4MqSpIkSQs3b7BrDrl+ic6vTxwKBDipqu4dcG2t5H9DGke+LrUUfB1JozdvsKuqSvLXVfVDwGVDqEmSJEmL0O85dlcleeFAK5EkSdJO6fccu5cAJyS5Hfg6ncOxVVUHDaowSZIkLUy/we6ogVYhSZI0xiblHNI5g12S7wFOAA4AbgDOrapHhlHYpJuUF4AkSYvh59x4mu8cu43AGjqh7ijg3Uux0iTHJtmS5DtJ1uww7ZQkW5PcnOTIrva1TdvWJBu62vdP8vkktyT5aJJdl6JGSZKkSTNfsDuwqn6uqj4AvBr4sSVa7xfpfH3Kld2NSQ4EjgOeB6wFzk6yLMky4H10wuWBwPHNvADvBM6sqtXAfcDrl6hGSZKkiTJfsPv2zMhSHoKtqpuq6uYek44GLqiqh6vqNmArcEgzbK2qW6vqW8AFwNHNd+y9FPhYs/xG4JilqlOSJGmSzHfxxA8m+WozHmC35vbMVbFPWeJ69gGu6rq9rWkDuGOH9hcBTwPu7wqd3fM/RpL1wHqAlStXLlHJkiSpjSbxPMI5g11VLVvsHSf5BPDMHpPeUlUXz7ZYrzLo3bNYc8zfU1WdA5wDsGbNmlnnkyRJmkT9ft3JglXVyxex2DZg367bK4A7m/Fe7fcCuyfZpem1655fkiRpqvT7yxPDcglwXJInJNkfWA1cDVwDrG6ugN2VzgUWl1RVAZ+ic2EHwDpgtt5ASZIm2iQeGtRwDazHbi5Jfgp4L7AcuCzJdVV1ZFVtSXIhcCPwCPDGqnq0WeZE4ApgGXBeVW1p7u5k4IIk7wC+AJw75M2RpEXzg1rSUhpJsKuqi4CLZpl2OnB6j/bLgct7tN9K56pZSZKkqTZuh2IlSZK0SAY7SZKkljDYSZLGlucgSgtjsJMkSWqJkVw8IUnSJLDHUJPGHjtJkqSWMNhJkiS1hMFOkiSpJQx2kiRJLWGwkyRJagmDnSRJUksY7CRJklrCYCdJktQSBjtJkqSW8JcnNPb85ndJmkzuv4fPHjtJkqSWMNhJkiS1hMFOkiSpJQx2kiRJLWGwkyRJGrGlutDEYCdJktQSBjtJkqSWMNhJkiS1hMFOkiSpJQx2kiRJLWGwkyRJagmDnSRJUksY7CRJklrCYCdJktQSBjtJkqSWGEmwS/IHSb6U5PokFyXZvWvaKUm2Jrk5yZFd7Wubtq1JNnS175/k80luSfLRJLsOe3skSZLGwah67DYB/7mqDgK+DJwCkORA4DjgecBa4Owky5IsA94HHAUcCBzfzAvwTuDMqloN3Ae8fqhbIkmSNCZGEuyq6uNV9Uhz8ypgRTN+NHBBVT1cVbcBW4FDmmFrVd1aVd8CLgCOThLgpcDHmuU3AscMazskSZLGyTicY/cLwN824/sAd3RN29a0zdb+NOD+rpA40y5JkjR1dhnUHSf5BPDMHpPeUlUXN/O8BXgE+PDMYj3mL3oH0Jpj/tlqWg+sB1i5cuWstUuSJE2igQW7qnr5XNOTrANeCbysqmbC2DZg367ZVgB3NuO92u8Fdk+yS9Nr1z1/r5rOAc4BWLNmzawBUJIkaRKN6qrYtcDJwKuq6qGuSZcAxyV5QpL9gdXA1cA1wOrmCthd6VxgcUkTCD8FvLpZfh1w8bC2Q5IkaZwMrMduHn8EPAHY1Ln+gauq6oSq2pLkQuBGOodo31hVjwIkORG4AlgGnFdVW5r7Ohm4IMk7gC8A5w53UyRJksbDSIJdVR0wx7TTgdN7tF8OXN6j/VY6V81KkiRNtXG4KlaSJElLwGAnSZLUEgY7SZKkljDYSZIktYTBTpIkqSUMdpIkSS1hsJMkSWoJg50kSVJLGOwkSZJawmAnSZLUEgY7SZKkljDYSZIktYTBTpIkqSUMdpIkSS1hsJMkSWoJg50kSVJLGOwkSZJawmAnSZLUEgY7SZKkljDYSZIktYTBTpIkqSUMdpIkSS1hsJMkSWoJg50kSVJLGOwkSZJawmAnSZLUEgY7SZKkljDYSZIktYTBTpIkqSUMdpIkSS1hsJMkSWqJkQS7JKcluT7JdUk+nuRZTXuSnJVkazP94K5l1iW5pRnWdbX/UJIbmmXOSpJRbJMkSdKojarH7g+q6qCqej5wKfC7TftRwOpmWA+8HyDJnsCpwIuAQ4BTk+zRLPP+Zt6Z5dYOayMkSZLGyUiCXVV9tevmfwKqGT8a+FB1XAXsnmRv4EhgU1Vtr6r7gE3A2mbaU6rqc1VVwIeAY4a3JZIkSeNjl1GtOMnpwGuBB4CXNM37AHd0zbataZurfVuP9tnWuZ5O7x4rV67cuQ2QJEkagNvPeMWilx1Yj12STyT5Yo/haICqektV7Qt8GDhxZrEed1WLaO+pqs6pqjVVtWb58uUL2yBJkqQxN7Aeu6p6eZ+zfgS4jM45dNuAfbumrQDubNoP36H90037ih7zS5IkTZ1RXRW7uuvmq4AvNeOXAK9tro49FHigqu4CrgCOSLJHc9HEEcAVzbQHkxzaXA37WuDi4W2JJEnS+BjVOXZnJHkO8B3gn4ETmvbLgZ8AtgIPAa8DqKrtSU4Drmnme3tVbW/G3wCcD+wG/G0zSJIkTZ10LiadPmvWrKnNmzePugxJkqR5Jbm2qtbMN5+/PCFJktQSBjtJkqSWMNhJkiS1hMFOkiSpJQx2kiRJLWGwkyRJagmDnSRJUktM7ffYJbkH+Dpwb9O015SOj3r90zY+6vVP8/io1z9t46Ne/7SNj3r90zw+rPXsV1Xz/9B9VU3tAGye9vFRr3/axke9/mkeH/X6p2181OuftvFRr3+ax4e5zn4GD8VKkiS1hMFOkiSpJaY92J3j+MjXP23jo17/NI+Pev3TNj7q9U/b+KjXP83jw1znvKb24glJkqS2mfYeO0mSpNYw2EmSJLWEwU6SJKklDHaSJEktYbCTJElqCYOdpNZL8miS67qGDSOq4/Ykey1iuSOT/I8keyS5fBC1SWqHXUZdgCQNwTeq6vmjLmIn/BjwKeAw4B9GXIukMWaPnaSplOSpSW5O8pzm9p8n+aVm/P1JNifZkuRtXcvcnuT3knyumX5wkiuS/FOSE5p5Dk9yZZKLktyY5I+TPGZfm+Tnklzd9CB+IMmyHvO8Jsl1wJuAPwQ+CLwuySWDeVQkTTqDnaRpsNsOh2JfU1UPACcC5yc5Dtijqj7YzP+WqloDHAS8OMlBXfd1R1X9MPBZ4Hzg1cChwNu75jkEeDPwA8CzgZ/uLibJc4HXAD/S9CQ+CvzsjkVX1UeBg4EvVtUPAF8EXlBVr9qZB0NSe3koVtI06Hkotqo2JTkWeB/wg12TfibJejr7yL2BA4Hrm2kzvWU3AE+qqgeBB5N8M8nuzbSrq+pW6PQEAj8KfKzr/l8G/BBwTRKA3YC7Z6l9NfBPzfgTm/VJUk8GO0lTqzlE+lzgG8CewLYk+wO/Abywqu5Lcj7wPV2LPdz8/U7X+MztmX3qjr/VuOPtABur6pR56tsM7AXskuRGYO/m0OyvVtVn+9hESVPGQ7GSptmvATcBxwPnJXk88BTg68ADSZ4BHLWI+z0kyf5NcHwN8Pc7TP8k8OokTwdIsmeS/Xa8k+Zw8GXA0cDv0zlE/HxDnaTZGOwkTYMdz7E7I8n3Ab8IvLkJSlcCb62qfwS+AGwBzmNxV6F+DjiDzjlxtwEXdU+sqhuBtwIfT3I9sInOId9eDgauo3Nl7GcWUYukKZKqHY8QSJIWK8nhwG9U1StHXYuk6WOPnSRJUkvYYydJktQS9thJkiS1hMFOkiSpJQx2kiRJLWGwkyRJagmDnSRJUksY7CRJklri3wEVCdOcf/p9DwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAFNCAYAAABIc7ibAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XuYJHV97/H3V1bDZbgGHWEFB4xikDUIE48+mjgrXhBUNInnyCFGFN2T4C0Rj66Xo+QkJqghGjE5BpUgSFiViBA2RtAwEI2oCyKLAgFxuSxkN2bjwuIKLn7PH10Tm3H6MrNV/euefb+ep5+t+lV1/T5dXV18qaqpisxEkiRJg/Ww0gEkSZJ2RBZhkiRJBViESZIkFWARJkmSVIBFmCRJUgEWYZIkSQVYhElatCJiS0QcXNOy3hERH6+GJyIiI2JJTcs+sMq6Ux3LkzQaLMIkzUtErIuIrVXRMPP6yIAzTEXET9v6vzMiPhMRv9o+X2aOZeatfSzrzl59ZuafZOZrtjd71ee6iHhO27Jvr7I+WMfyJY0GizBJC/GiqmiYeb1+rpnmOlI036NHXea/KzPHgN2BpwE3Av8cEUfNZ/nbmUGSFswiTFJtIuLEiPhqRHwwIjYBp3Zoe1hEvCsibouIjRFxTkTsWS1j5lTfSRFxO/BP3frMljsz893Ax4H3teXJiPilaviYiPhuRNwbEesj4i0RsRvwBWD/tqNq+0fEqRFxQUR8KiLuAU6s2j41q/tXR8RdEXF3RJzS1u/ZEfHHbeP/dbQtIs4FDgT+vurvrbNPb1YZLo6ITRFxS0S8tm1Zp1ZH/c6pPst3ImJy3l+WpOIswiTV7b8BtwKPAt7boe3E6rUcOBgYA2af0nwW8MvA8+fR9+eAI6riarZPAP8rM3cHDgP+KTPvA15AdVStet1VzX8ccAGwF3Beh/6WA48HngesbD/F2ElmvgK4nZ8dTXz/HLOdD9wJ7A/8FvAns47wvRhYVWW7mJ9fd5JGgEWYpIX4fET8sO312rZpd2XmGZm5LTO3dmg7AfjzzLw1M7cAbwdePuu036mZeV/bMvpxFxC0ipPZfgIcGhF7ZOZ/ZuY1PZb1tcz8fGb+tEuGP6wyrgX+Bjh+HlnnFBEHAM8E3paZP87Ma2kd4XtF22xfycx/qK4hOxf4le3tV9LgWYRJWoiXZOZeba+PtU27Y475Z7ftD9zWNn4bsAQY77GcXpYCCfxwjmm/CRwD3BYRV0TE03ssq5/+2+e5jdbn2l77A5sy895Zy17aNv5vbcM/Anb2ujVp9FiESapb9tF2F/DYtvEDgW3Ahh7L6eWlwDXVacaHBsj8ZmYeR+uU6OeBz/Top5/+D2gbPpDW5wK4D9i1bdqj57Hsu4B9ImL3Wcte30ceSSPEIkxSCecDfxARB0XEGPAnwKczc9t8FxQtSyPiPcBrgHfMMc8jIuKEiNgzM38C3APM3A5iA/CLM38YME//JyJ2jYgnAa8CPl21XwscExH7RMSjgd+f9b4NtK6F+zmZeQfwL8CfRsTOEfFk4CQ6X5cmaURZhElaiJm/7Jt5XTjP959F61qmK4HvAz8G3jDPZewfEVuALcA3gWXAVGZe2mH+VwDrqr92/F3gtwEy80ZaReGt1fVt8zmleAVwC/Bl4M/a+j4X+DawDriUnxVnM/4UeFfV31vmWO7xwASto2IXAu/JzMvmkUvSCIjMhRzxlyRJ0vbwSJgkSVIBFmGSJEkFWIRJkiQVYBEmSZJUgEWYJElSASNxh+V99903JyYmGu3jvvvuY7fd5nrc3PAZlazmrNeo5ITRyWrOeo1KThidrOas3yCyXn311T/IzEf2nDEzh/515JFHZtMuv/zyxvuoy6hkNWe9RiVn5uhkNWe9RiVn5uhkNWf9BpEVWJN91DeejpQkSSrAIkySJKkAizBJkqQCLMIkSZIKsAiTJEkqwCJMkiSpAIswSZKkAizCJEmSCrAIkyRJKsAiTJIkqQCLMEmSpAJG4gHekkbL2vWbOXHl6o7T15127Hb3MdFl+XX1IUlN8kiYJElSARZhkiRJBViESZIkFWARJkmSVIBFmCRJUgEWYZIkSQVYhEmSJBVgESZJklSARZgkSVIBFmGSJEkFWIRJkiQVYBEmSZJUQGNFWEScFREbI+L6Oaa9JSIyIvZtqn9JkqRh1uSRsLOBo2c3RsQBwHOB2xvsW5Ikaag1VoRl5pXApjkmfRB4K5BN9S1JkjTsBnpNWES8GFifmd8eZL+SJEnDJjKbOyAVERPAJZl5WETsClwOPC8zN0fEOmAyM3/Q4b0rgBUA4+PjR65ataqxnABbtmxhbGys0T7qMipZzVmvUckJsHHTZjZs7Tx92dI9t7uPtes3d53eTx+jsk7NWb9RyWrO+g0i6/Lly6/OzMle8y1pNMVDPQ44CPh2RAA8BrgmIp6amf82e+bMPBM4E2BycjKnpqYaDTc9PU3TfdRlVLKas16jkhPgjPMu4vS1nXcv606Y2u4+Tly5uuv0fvoYlXVqzvqNSlZz1m+Ysg6sCMvMtcCjZsZ7HQmTJElazJq8RcX5wNeAQyLizog4qam+JEmSRk1jR8Iy8/ge0yea6luSJGnYecd8SZKkAizCJEmSCrAIkyRJKsAiTJIkqQCLMEmSpAIswiRJkgqwCJMkSSrAIkySJKkAizBJkqQCLMIkSZIKsAiTJEkqwCJMkiSpgMYe4C1JO4KJlau7Tl932rEDSiJp1HgkTJIkqQCLMEmSpAIswiRJkgqwCJMkSSrAIkySJKkAizBJkqQCLMIkSZIKsAiTJEkqwCJMkiSpAIswSZKkAizCJEmSCrAIkyRJKsAiTJIkqYDGirCIOCsiNkbE9W1tH4iIGyPiuoi4MCL2aqp/SZKkYdbkkbCzgaNntV0GHJaZTwb+FXh7g/1LkiQNrcaKsMy8Etg0q+3SzNxWjV4FPKap/iVJkoZZyWvCXg18oWD/kiRJxURmNrfwiAngksw8bFb7O4FJ4DeyQ4CIWAGsABgfHz9y1apVjeUE2LJlC2NjY432UZdRyWrOeo1KToCNmzazYWvn6cuW7rndfaxdv7nr9H766JWzDnV81lH57kclJ4xOVnPWbxBZly9ffnVmTvaab0mjKeYQEa8EXggc1akAA8jMM4EzASYnJ3NqaqrRXNPT0zTdR11GJas56zUqOQHOOO8iTl/befey7oSp7e7jxJWru07vp49eOetQx2cdle9+VHLC6GQ1Z/2GKetAi7CIOBp4G/CszPzRIPuWJEkaJk3eouJ84GvAIRFxZ0ScBHwE2B24LCKujYiPNtW/JEnSMGvsSFhmHj9H8yea6k+SJGmUeMd8SZKkAizCJEmSCrAIkyRJKsAiTJIkqQCLMEmSpAIswiRJkgqwCJMkSSrAIkySJKkAizBJkqQCLMIkSZIKsAiTJEkqwCJMkiSpgMYe4C1JJU2sXN1znlOWDSCIJHXgkTBJkqQCLMIkSZIKsAiTJEkqwCJMkiSpAIswSZKkAizCJEmSCrAIkyRJKsAiTJIkqQCLMEmSpAIswiRJkgqwCJMkSSrAIkySJKmAxoqwiDgrIjZGxPVtbftExGURcXP1795N9S9JkjTMmjwSdjZw9Ky2lcCXM/PxwJercUmSpB1OY0VYZl4JbJrVfBzwyWr4k8BLmupfkiRpmA36mrDxzLwboPr3UQPuX5IkaShEZja38IgJ4JLMPKwa/2Fm7tU2/T8zc87rwiJiBbACYHx8/MhVq1Y1lhNgy5YtjI2NNdpHXUYlqznrNSo5ATZu2syGrZ2nL1u6Z89lrF2/ucZEcxvfha45B6XX+hiV735UcsLoZDVn/QaRdfny5Vdn5mSv+ZY0muLnbYiI/TLz7ojYD9jYacbMPBM4E2BycjKnpqYaDTY9PU3TfdRlVLKas16jkhPgjPMu4vS1nXcv606Y6rmME1eurjHR3E5Ztq1rzkHptT5G5bsflZwwOlnNWb9hyjro05EXA6+shl8JXDTg/iVJkoZCk7eoOB/4GnBIRNwZEScBpwHPjYibgedW45IkSTucxo7DZ+bxHSYd1VSfkiRJo8I75kuSJBVgESZJklSARZgkSVIBFmGSJEkFWIRJkiQVYBEmSZJUgEWYJElSAX0VYRFxWNNBJEmSdiT9Hgn7aER8IyJOjoi9es8uSZKkbvoqwjLzmcAJwAHAmoj424h4bqPJJEmSFrG+rwnLzJuBdwFvA54FfDgiboyI32gqnCRJ0mLV7zVhT46IDwI3AM8GXpSZv1wNf7DBfJIkSYtSvw/w/gjwMeAdmbl1pjEz74qIdzWSTJJUm4mVq3vOs+60YweQRNKMfouwY4CtmfkgQEQ8DNg5M3+Umec2lk6SJGmR6veasC8Bu7SN71q1SZIkaQH6LcJ2zswtMyPV8K7NRJIkSVr8+i3C7ouII2ZGIuJIYGuX+SVJktRFv9eE/T7w2Yi4qxrfD/gfzUSSJEla/PoqwjLzmxHxROAQIIAbM/MnjSaTJElaxPo9Egbwq8BE9Z6nRASZeU4jqSRJkha5voqwiDgXeBxwLfBg1ZyARZgkSdIC9HskbBI4NDOzyTCSJEk7in7/OvJ64NFNBpEkSdqR9HskbF/guxHxDeD+mcbMfHEjqSRJkha5fouwU5sMIUmStKPp9xYVV0TEY4HHZ+aXImJXYKdmo0mSJC1efV0TFhGvBS4A/rpqWgp8fqGdRsQfRMR3IuL6iDg/InZe6LIkSZJGUb8X5r8OeAZwD0Bm3gw8aiEdRsRS4I3AZGYeRuuI2ssXsixJkqRR1W8Rdn9mPjAzEhFLaN0nbKGWALtUy9kVuKvH/JIkSYtKv0XYFRHxDlqF03OBzwJ/v5AOM3M98GfA7cDdwObMvHQhy5IkSRpV0c/9VyPiYcBJwPNoPTvyi8DHF3Lz1ojYG/g7Wg8A/yGtgu6CzPzUrPlWACsAxsfHj1y1atV8u5qXLVu2MDY21mgfdRmVrOas16jkBNi4aTMbtpZO0dv4LgxFzmVL9+w6vY7vfu36zdudo5dR2kZHJas56zeIrMuXL786Myd7zddXEVaniHgZcHRmnlSN/w7wtMw8udN7Jicnc82aNY3mmp6eZmpqqtE+6jIqWc1Zr1HJCXDGeRdx+tr5PJq2jFOWbRuKnOtOO7br9Dq++4mVq7c7Ry+jtI2OSlZz1m8QWSOiryKs32dHfp85rgHLzIMXkO124GnVbS62AkcBzVZYkiRJQ2Y+z46csTPwMmCfhXSYmV+PiAuAa4BtwLeAMxeyLEmSpFHV781a/2NW04ci4ivAuxfSaWa+B3jPQt4rSZK0GPR7OvKIttGH0ToytnsjiSRJknYA/Z6OPL1teBuwDvjvtaeRJEnaQfR7OnJ500EkSZJ2JP2ejnxzt+mZ+ef1xJEkSdoxzOevI38VuLgafxFwJXBHE6EkSZIWu36LsH2BIzLzXoCIOBX4bGa+pqlgkiRJi1m/z448EHigbfwBYKL2NJIkSTuIfo+EnQt8IyIupHXn/JcC5zSWSpIkaZHr968j3xsRXwB+rWp6VWZ+q7lYkiRJi1u/pyMBdgXuycy/AO6MiIMayiRJkrTo9VWERcR7gLcBb6+aHg58qqlQkiRJi12/R8JeCrwYuA8gM+/CxxZJkiQtWL8X5j+QmRkRCRARuzWYSVJBEytXd52+7rRjB5REM9au38yJfi/SotPvkbDPRMRfA3tFxGuBLwEfay6WJEnS4tbvX0f+WUQ8F7gHOAR4d2Ze1mgySZKkRaxnERYROwFfzMznABZekiRJNeh5OjIzHwR+FBF7DiCPJEnSDqHfC/N/DKyNiMuo/kISIDPf2EgqSZKkRa7fImx19ZIkSVINuhZhEXFgZt6emZ8cVCBJkqQdQa9rwj4/MxARf9dwFkmSpB1GryIs2oYPbjKIJEnSjqRXEZYdhiVJkrQdel2Y/ysRcQ+tI2K7VMNU45mZezSaTpIkaZHqWoRl5k6DCiJJkrQj6ffZkbWKiL0i4oKIuDEiboiIp5fIIUmSVEq/9wmr218A/5iZvxURjwB2LZRDkiSpiIEXYRGxB/DrwIkAmfkA8MCgc0iSJJVU4nTkwcC/A38TEd+KiI9HxG4FckiSJBUTmYO980RETAJXAc/IzK9HxF8A92Tm/5k13wpgBcD4+PiRq1atajTXli1bGBsba7SPuoxKVnPWa1A5167fvN3LGN8FNmytIUzDhiXnsqV7dp2+cdPmnjl7LaOf77XXMnoZld8SjE5Wc9ZvEFmXL19+dWZO9pqvRBH2aOCqzJyoxn8NWJmZx3Z6z+TkZK5Zs6bRXNPT00xNTTXaR11GJas56zWonBMrt/8xsacs28bpa0tdctq/Ycm57rSOuz8Azjjvop45ey2jn++11zJ6GZXfEoxOVnPWbxBZI6KvImzgpyMz89+AOyLikKrpKOC7g84hSZJUUqn/BXwDcF71l5G3Aq8qlEOSJKmIIkVYZl4L9DxMJ0mStFgVuVmrJEnSjs4iTJIkqQCLMEmSpAIswiRJkgqwCJMkSSrAIkySJKkAizBJkqQCLMIkSZIKsAiTJEkqwCJMkiSpAIswSZKkAizCJEmSCrAIkyRJKmBJ6QCSpO03sXJ16QiS5skjYZIkSQVYhEmSJBVgESZJklSARZgkSVIBFmGSJEkFWIRJkiQVYBEmSZJUgEWYJElSARZhkiRJBViESZIkFWARJkmSVIBFmCRJUgHFirCI2CkivhURl5TKIEmSVErJI2FvAm4o2L8kSVIxRYqwiHgMcCzw8RL9S5IklVbqSNiHgLcCPy3UvyRJUlGRmYPtMOKFwDGZeXJETAFvycwXzjHfCmAFwPj4+JGrVq1qNNeWLVsYGxtrtI+6jEpWc9arjpxr12+uKU1347vAhq0D6Wq7DEvOZUv37Dp946bNI5Gzn2201zbYq4+6lrEj/e4HYVRywmCyLl++/OrMnOw1X4ki7E+BVwDbgJ2BPYDPZeZvd3rP5ORkrlmzptFc09PTTE1NNdpHXUYlqznrVUfOiZWr6wnTwynLtnH62iUD6Wt7DEvOdacd23X6GeddNBI5+9lGe22Dvfqoaxk70u9+EEYlJwwma0T0VYQN/HRkZr49Mx+TmRPAy4F/6laASZIkLUbeJ0ySJKmAose3M3MamC6ZQZIkqQSPhEmSJBVgESZJklSARZgkSVIBFmGSJEkFWIRJkiQVYBEmSZJUgEWYJElSARZhkiRJBViESZIkFWARJkmSVIBFmCRJUgEWYZIkSQUUfYC3pHpNrFxdOoIWoNf3dsqyAQXpoVfOs4/ebUBJNKOf3/y6044dQBIthEfCJEmSCrAIkyRJKsAiTJIkqQCLMEmSpAIswiRJkgqwCJMkSSrAIkySJKkAizBJkqQCLMIkSZIKsAiTJEkqwCJMkiSpAIswSZKkAgZehEXEARFxeUTcEBHfiYg3DTqDJElSaUsK9LkNOCUzr4mI3YGrI+KyzPxugSySJElFDPxIWGbenZnXVMP3AjcASwedQ5IkqaSi14RFxATwFODrJXNIkiQNWmRmmY4jxoArgPdm5ufmmL4CWAEwPj5+5KpVqxrNs2XLFsbGxhrtoy6jknVUcm7ctJkNW7vPs2zpno3nWLt+c9fpB+25U8/12WsZgzK+Cz3X6TAwZ73q2Eb7+a3VsYxe+6d+fkuD2C+MSs5h2Y/2YxD/bVq+fPnVmTnZa74iRVhEPBy4BPhiZv55r/knJydzzZo1jWaanp5mamqq0T7qMipZRyXnGeddxOlru18eue60YxvPMbFyddfpZx+9W8/12WsZg3LKsm091+kwMGe96thG+/mt1bGMXvunfn5Lg9gvjErOYdmP9mMQ/22KiL6KsBJ/HRnAJ4Ab+inAJEmSFqMS14Q9A3gF8OyIuLZ6HVMghyRJUjEDP76dmV8BYtD9SpIkDRPvmC9JklSARZgkSVIBFmGSJEkFWIRJkiQVYBEmSZJUgEWYJElSARZhkiRJBViESZIkFWARJkmSVIBFmCRJUgEWYZIkSQVYhEmSJBVgESZJklTAktIBhsXa9Zs5ceXqrvOsO+3YAaVRvyZ6fGcwmO+tnxzbq59tVBp1g/gtQT2/p15Z+9n39FrGKcu2+btfxDwSJkmSVIBFmCRJUgEWYZIkSQVYhEmSJBVgESZJklSARZgkSVIBFmGSJEkFWIRJkiQVYBEmSZJUgEWYJElSARZhkiRJBViESZIkFVCkCIuIoyPipoi4JSJWlsggSZJU0sCLsIjYCfhL4AXAocDxEXHooHNIkiSVVOJI2FOBWzLz1sx8AFgFHFcghyRJUjElirClwB1t43dWbZIkSTuMyMzBdhjxMuD5mfmaavwVwFMz8w2z5lsBrKhGDwFuajjavsAPGu6jLqOS1Zz1GpWcMDpZzVmvUckJo5PVnPUbRNbHZuYje820pOEQc7kTOKBt/DHAXbNnyswzgTMHFSoi1mTm5KD62x6jktWc9RqVnDA6Wc1Zr1HJCaOT1Zz1G6asJU5HfhN4fEQcFBGPAF4OXFwghyRJUjEDPxKWmdsi4vXAF4GdgLMy8zuDziFJklRSidORZOY/AP9Qou8uBnbqswajktWc9RqVnDA6Wc1Zr1HJCaOT1Zz1G5qsA78wX5IkST62SJIkqYgdpgiLiD+KiOsi4tqIuDQi9q/aIyI+XD1C6bqIOKLtPa+MiJur1yvb2o+MiLXVez4cEVFjzg9ExI1VlgsjYq+qfSIitlb5r42Ij/bKExH7RMRlVf7LImLvpnNW095eZbkpIp7f1j7n46qqP9L4epXz09UfbNSV82UR8Z2I+GlETLa1D9X67Ja1mjY063RWrlMjYn3bejxmoZkHaRgyzBYR66rt7tqIWFO1zbnNddtvNZDrrIjYGBHXt7XNO1d02J82nHPots+IOCAiLo+IG6rf+5uq9qFap11yDuM63TkivhER366y/mHVPud+MCJ+oRq/pZo+0eszNCYzd4gXsEfb8BuBj1bDxwBfAAJ4GvD1qn0f4Nbq372r4b2rad8Anl695wvAC2rM+TxgSTX8PuB91fAEcH2H98yZB3g/sLIaXjmzrIZzHgp8G/gF4CDge7T+AGOnavhg4BHVPIdW7/kM8PJq+KPA79WY85dp3WduGphsax+q9dkj61Ct01mZTwXeMkf7vDMP6jUMGTrkWgfsO6ttzm2ODvuthnL9OnBE++9lvrnosj9tOOfQbZ/AfsAR1fDuwL9WeYZqnXbJOYzrNICxavjhwNerdTXnfhA4mZ/VAC8HPt3tMzT128rMHedIWGbe0za6GzBzMdxxwDnZchWwV0TsBzwfuCwzN2XmfwKXAUdX0/bIzK9l61s7B3hJjTkvzcxt1ehVtO6j1lGPPMcBn6yGPzmgnMcBqzLz/sz8PnALrUdVzfm4qogI4NnABQ3lvCEz+77Rb6n12SPrUK3TPs0r84CzDUOGfnXa5jrtt2qXmVcCm7Yz15z70wHk7KTY9pmZd2fmNdXwvcANtJ4aM1TrtEvOTkqu08zMLdXow6tX0nk/2L6uLwCOqvabnT5DY3aYIgwgIt4bEXcAJwDvrpo7PUapW/udc7Q34dW0/g9oxkER8a2IuCIifq1q65ZnPDPvhtYPCnjUAHLOd33+IvDDtoJukI+xGtb1Oduwr9PXV6dJzoqfnaKdb+ZBGoYMc0ng0oi4OlpPDIHO21zpzzDfXCXzDu32WZ0GewqtIzdDu05n5YQhXKcRsVNEXAtspFWQfo/O+8H/ylRN30xrvznw7XRRFWER8aWIuH6O13EAmfnOzDwAOA94/czb5lhULqC9tpzVPO8EtlVZAe4GDszMpwBvBv42IvaoI0/NOYdyfc5h4OtzO7IOfJ0+pPPumf8f8DjgcFrr9PQFZh6kYcgwl2dk5hHAC4DXRcSvd5l3WD/DsH3vQ7t9RsQY8HfA7886U/Nzs3bINJCsc+QcynWamQ9m5uG0zso8ldblHZ36Lf79zyhyn7CmZOZz+pz1b4HVwHvo/BilO4GpWe3TVftj5pi/tpzRusDyhcBR1SkxMvN+4P5q+OqI+B7whB55NkTEfpl5d3X4emPTOen+WKq52n9A6/D6kur/SGpfnx3eM/D1udCsFFin7frNHBEfAy5ZYOZB6uvRaYOWmXdV/26MiAtp/Yek0zZX+jPMN1en/WmjMnPDzPAwbZ8R8XBahc15mfm5qnno1ulcOYd1nc7IzB9GxDSta8I67Qdnst4ZEUuAPWmdyh787yobvOBsmF7A49uG3wBcUA0fy0MvevxG1b4P8H1aFzzuXQ3vU037ZjXvzIXbx9SY82jgu8AjZ7U/kuoCQVoXOK7vlQf4AA+90PP9A8j5JB56YeOttC7MXFINH8TPLs58UvWez/LQiydPbuD7n+ahF7sP1frskXUo12m17P3ahv+A1vUUC8o8qNcwZJgj027A7m3D/1L9xubc5uiw32ow3wQPveB9Xrnosj9tOOfQbZ/VujkH+NCs9qFap11yDuM6fSSwVzW8C/DPtA4QzLkfBF7HQy/M/0y3z9Dob6vJhQ/Ti1Y1fz1wHfD3wNK2De0vaZ0/XstD/+P3aloX5t0CvKqtfbJa1veAj1Dd9LamnLfQOid9bfWa2VB+E/hOtYFcA7yoVx5a57i/DNxc/VvnD3TOnNW0d1ZZbqLtL0dp/ZXPv1bT3tnWfjCtv0i8pfrR/EKNOV9K6/9u7gc2AF8cxvXZLeuwrdNZmc+tfjfX0XoG7H4LzTzI1zBkmJXn4Gpb/Ha1Xb6z2zZHl/1WA9nOp3Xa6SfV9nnSQnLRYX/acM6h2z6BZ9I6xXUdP9t/HjNs67RLzmFcp08GvlVluh54d9vv6uf2g8DO1fgt1fSDe32Gpl7eMV+SJKmARXVhviRJ0qiwCJMkSSrAIkySJKkAizBJkqQCLMIkSZIKsAiTJEkqwCJMUm0i4sGIuLZ6rNFnI2LX7VjWVERcUg2/OCJWdpl3r4g4uW18/4i4oNP888wxHRE3VZ/r2rqWO88Mpw66T0nNswiTVKetmXl4Zh4GPAD8bvvEaJn3ficzL87M07rMshdwctv8d2Xmb823ny5OqD7X4XMtt3r0ScfxTnrNFxEvrR66SM/IAAADG0lEQVRK/HsR8dWIWDav1JKG2qJ6dqSkofLPwJMjYoLWY1cuB54OvCQiDgH+kNbjQb5H627fWyLiaOBDtJ5/ec3MgiLiRFp3Cn99RIzTegTJwdXk3wPeCDyuKlguo3WH8Usy87CI2JnWQ4cnaT1s/s2ZeXm1zBcDu9J6IPGFmfnWfj9cRJxN63lzTwGuiYh7gf1pPTrnBxHx6i79Hkvrrt27Ac/u0s1fAc8C/me1LEmLiEWYpNpVR3heAPxj1XQIrULr5IjYF3gX8JzMvC8i3ga8OSLeD3yMVlFyC/DpDov/MHBFZr40InYCxmg9a++wzDy86n+ibf7XAWTmsoh4InBpRDyhmnY4rSLqfuCmiDgjM++Yo8/zImJrNXxZZv7vavgJ1ed4sDpleCTwzMzcGhGndOn36cCTM3NT57UItIq38Wo5G3rMK2nEWIRJqtMu1dEoaB0J+wSto0O3ZeZVVfvTgEOBr0YEtB7q+zXgicD3M/NmgIj4FLBijj6eDfwOQGY+CGyOiL27ZHomcEY1/40RcRut4gngy5m5uervu8BjaT0TdbYTMnPNHO2frTLMuDgzZ4q1bv1e1kcBBq2HC/8RsCwi9gfekZk/6ON9kkaARZikOm2dORo1oyq07mtvolWEHD9rvsNpPTC4btFl2v1tww8y/33ifV3Gu/U7+31zysyvAs+OiPfRyvc+Wg+mlrQIeGG+pEG7CnhGRPwSQETsWp2muxE4KCIeV813fIf3f5nWdWBExE4RsQdwL7B7h/mvBE6o5n8CcCBwUx0fpIe++42IGzu0H1YNbgWuo/NnlDSCLMIkDVRm/jtwInB+RFxHqyh7Ymb+mNbpx9UR8RXgtg6LeBOwPCLWAlcDT8rM/6B1evP6iPjArPn/Ctipmv/TwImZeT/zc17bLSq+1Od7+uq3ukau01GzP67WxWuBNwP/d565JQ2xyGzi6L8kqR8R8ULg4Mz8cJd5Ts3MUweXStIgWIRJ0pCLiKnMnC6dQ1K9LMIkSZIK8JowSZKkAizCJEmSCrAIkyRJKsAiTJIkqQCLMEmSpAL+P4BOjrEtEcq1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = df_residuals.plot.bar(figsize=(10, 5))\n",
    "ax.legend().remove()\n",
    "ax.get_xaxis().set(ticklabels=[])\n",
    "ax.set(xlabel=\"Example #\")\n",
    "ax.set(ylabel=\"Prediction Error, $\")\n",
    "ax.set(title=\"Residuals\")\n",
    "\n",
    "bins = np.arange(-3000, 3001, 100)\n",
    "ticks = np.arange(-3000, 3001, 500)\n",
    "\n",
    "ax = df_residuals.hist(bins=bins, figsize=(10, 5))[0][0]\n",
    "ax.set(xticks=ticks)\n",
    "ax.set(xlabel=\"Prediction Error, $\")\n",
    "ax.set(ylabel=\"Frequency\")\n",
    "ax.set(title=\"Error Distribution\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Тестування"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Запустіть комірку нижче, щоб перевірити правильність вашого коду:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/unittest.status+json": {
       "color": "yellow",
       "message": "",
       "previous": 0
      },
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/unittest.status+json": {
       "color": "lightgreen",
       "message": "............\n----------------------------------------------------------------------\nRan 12 tests in 0.002s\n\nOK\n",
       "previous": 0
      },
      "text/plain": [
       "Success"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "............\n",
      "----------------------------------------------------------------------\n",
      "Ran 12 tests in 0.002s\n",
      "\n",
      "OK\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<unittest.runner.TextTestResult run=12 errors=0 failures=0>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%unittest_main\n",
    "\n",
    "class WLRTests(unittest.TestCase):\n",
    "\n",
    "    x = np.array([1, -0.5, 3, 1])\n",
    "    X = np.array([\n",
    "        [1, -0.5, 3, 1],\n",
    "        [2, 8, -0.33, 5],\n",
    "        [0, 0, 0, 0]\n",
    "    ])\n",
    "    y = np.array([40, 100, 12])\n",
    "    theta = np.array([2, 5, 7, 9])\n",
    "    eps = 0.001\n",
    "\n",
    "    def assertFloatEquals(self, a, b):\n",
    "        self.assertTrue(np.abs(a - b) < self.eps)\n",
    "    \n",
    "    def assertArrayEquals(self, a, b):\n",
    "        a = np.array(a)\n",
    "        b = np.array(b)\n",
    "        self.assertEqual(a.shape, b.shape)\n",
    "        self.assertTrue(np.all(np.abs(a - b) < self.eps))\n",
    "    \n",
    "    def test_predict_linear_should_compute_correct_prediction_for_1_example(self):\n",
    "        expected = 29.5\n",
    "        actual = predict_linear(self.theta, self.x)\n",
    "        self.assertEqual(actual, expected)\n",
    "    \n",
    "    def test_predict_linear_should_compute_correct_predictions_for_multiple_examples(self):\n",
    "        expected = [29.5, 86.69, 0]\n",
    "        actual = (predict_linear(self.theta, self.X))\n",
    "        self.assertArrayEquals(actual, expected)\n",
    "    \n",
    "    def test_get_example_weights_should_return_properly_shaped_vector(self):\n",
    "        weights = get_example_weights(self.X, self.x, tau=5)\n",
    "        self.assertTrue(weights.shape[0] == self.X.shape[0])\n",
    "    \n",
    "    def test_get_example_weights_should_compute_correct_weights(self):\n",
    "        expected = [1.000, 0.134, 0.798]\n",
    "        actual = get_example_weights(self.X, self.x, tau=5)\n",
    "        self.assertArrayEquals(actual, expected)\n",
    "    \n",
    "    def test_cost_function_should_compute_correct_cost_unweighted(self):\n",
    "        weights = np.ones(self.X.shape[0])\n",
    "        expected = 71.901\n",
    "        actual = cost_function(self.theta, self.X, self.y, weights)\n",
    "        self.assertFloatEquals(actual, expected)\n",
    "    \n",
    "    def test_cost_function_should_compute_correct_cost_weighted(self):\n",
    "        weights = np.array([0.5, 0.1, 0.28])\n",
    "        expected = 18.860\n",
    "        actual = cost_function(self.theta, self.X, self.y, weights)\n",
    "        self.assertFloatEquals(actual, expected)\n",
    "\n",
    "    def test_cost_gradient_should_return_properly_shaped_vector(self):\n",
    "        weights = np.ones(self.X.shape[0])\n",
    "        grad = cost_function_gradient(self.theta, self.X, self.y, weights)\n",
    "        self.assertTrue(grad.shape == self.theta.shape)\n",
    "        \n",
    "    def test_cost_gradient_should_compute_correct_gradient_unweighted(self):\n",
    "        weights = np.ones(self.X.shape[0])\n",
    "        expected = [-37.12, -101.23, -27.108, -77.05]\n",
    "        actual = cost_function_gradient(self.theta, self.X, self.y, weights)\n",
    "        self.assertArrayEquals(actual, expected)\n",
    "    \n",
    "    def test_cost_gradient_should_compute_correct_gradient_weighted(self):\n",
    "        weights = np.array([0.5, 0.1, 0.28])\n",
    "        expected = [-7.912, -8.023, -15.311, -11.905]\n",
    "        actual = cost_function_gradient(self.theta, self.X, self.y, weights)\n",
    "        self.assertArrayEquals(actual, expected)\n",
    "    \n",
    "    def test_update_model_weights_should_not_update_when_gradient_is_zero(self):\n",
    "        grad = np.zeros(self.theta.shape[0])\n",
    "        theta_new = update_model_weights(self.theta, learning_rate=1, cost_gradient=grad)\n",
    "        self.assertArrayEquals(theta_new, self.theta)\n",
    "    \n",
    "    def test_update_model_weights_should_update_with_complete_gradient_if_learning_rate_is_one(self):\n",
    "        grad = np.array([1.35, -0.89, 0.16, 0.98])\n",
    "        expected = [0.65, 5.89, 6.84, 8.02]\n",
    "        actual = update_model_weights(self.theta, learning_rate=1, cost_gradient=grad)\n",
    "        self.assertArrayEquals(actual, expected)\n",
    "    \n",
    "    def test_update_model_weights_should_take_learning_rate_into_account(self):\n",
    "        grad = np.array([1.35, -0.89, 0.16, 0.98])\n",
    "        expected = [1.730, 5.178, 6.968, 8.804]\n",
    "        actual = update_model_weights(self.theta, learning_rate=0.2, cost_gradient=grad)\n",
    "        self.assertArrayEquals(actual, expected)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}