lab-07-3-linear_regression_eager (preprocess, decay, l2_loss).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 07-3 linear regression - Eager (normalization, decay, l2_loss)\n",
    "\n",
    "* linear regression에 Normalization, Learning Decay, L2_loss를 적용\n",
    "\n",
    "### 기본 Library 선언 및 Tensorflow 버전 확인"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.12.0\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import tensorflow as tf\n",
    "import tensorflow.contrib.eager as tfe\n",
    "\n",
    "tf.enable_eager_execution()\n",
    "tf.set_random_seed(777)  # for reproducibility\n",
    "\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 정규화를 위한 함수 (최대 최소값이 1과 0이되도록 Scaling한다)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalization(data):\n",
    "    numerator = data - np.min(data, 0)\n",
    "    denominator = np.max(data, 0) - np.min(data, 0)\n",
    "    return numerator / denominator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 강의에 설명할 Data입니다\n",
    "* X Data(feature)의 값은 해당 배열의 첫번째 값부터 4번째 값까지로 정의 되고 Y Data(label)는 해당 배열의 마지막 값을 정의(5번째 값)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAD8CAYAAACyyUlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGU1JREFUeJzt3X+MVed95/H3xxCnwY0DjqcI8WtoQi3hakXMFaZKE6XBwYMbBWdlZbHYeDalmUQxq2SzUoPLSmSdWpt0lXplbUJFAmvoTo1dEtdo5YQgx6p3peJwx6b4dxkTMIMwEEPsZKnsxf7uH/eZ+DCeuffOPHfmzB0+L+nqnvs9zznnOwjNZ845z71XEYGZmVmOy8puwMzM2p/DxMzMsjlMzMwsm8PEzMyyOUzMzCybw8TMzLI5TMzMLJvDxMzMsjlMzMws2/SyG5goV199dXR2dpbdhplZW+nr6/tFRHQ0GnfJhElnZyfVarXsNszM2oqkY82M82UuMzPL5jAxM7NsDhMzM8vmMDEzs2wOEzMzy+YwmUp6e6GzEy67rPbc21t2R2Z2ibhkpgZPeb290NMD58/XXh87VnsNsG5deX2Z2SXBZyZTxaZNbwfJoPPna3Uzs3HmMJkqXnppdHUzsxZymEwVCxaMrm5m1kIOk6nirrtgxoyLazNm1OpmZuPMYTJVrFsHW7fCwoUg1Z63bvXNdzObEJ7NNZWsW+fwMLNSNDwzkbRd0mlJTxdq90s6mB5HJR1M9U5J/1JY99eFbZZJekpSv6R7JCnVr5K0T9Lh9Dwr1ZXG9Us6JOm6wr660/jDkrpb+Q9iZmaj18xlrnuBrmIhIv5NRCyNiKXAD4AfFla/OLguIr5YqG8BPg8sTo/BfW4EHomIxcAj6TXA6sLYnrQ9kq4CNgPXA8uBzYMBZGZm5WgYJhHxGHB2uHXp7OIzwH319iFpDnBlROyPiAB2Ajen1WuAHWl5x5D6zqjZD8xM+7kR2BcRZyPiHLCPIWFnZmYTK/cG/EeAUxFxuFBbJOlJSf8g6SOpNhcYKIwZSDWA2RFxMi2/DMwubHN8mG1GqpuZWUlyb8DfysVnJSeBBRHxiqRlwN9LurbZnUVESIrMnn5DUg+1S2Qs8PstzMzGzZjPTCRNB/41cP9gLSJej4hX0nIf8CLwe8AJYF5h83mpBnAqXb4avBx2OtVPAPOH2Wak+jtExNaIqEREpaOj4VcYm5nZGOVc5roBeD4ifnP5SlKHpGlp+Xep3Tw/ki5jvSZpRbrPchvwUNpsDzA4I6t7SP22NKtrBfBq2s9eYJWkWenG+6pUMzOzkjS8zCXpPuBjwNWSBoDNEbENWMs7b7x/FLhT0v8D3gK+GBGDN++/RG1m2HuAH6UHwDeBByStB45Ru6EP8DBwE9APnAc+BxARZyV9AziQxt1ZOIaZmZVAtclVU1+lUolqtVp2G2ZmbUVSX0RUGo3zx6mYmVk2h4mZmWVzmJiZWTaHiZmZZXOYmJlZNoeJmZllc5iYmVk2h4mZmWVzmJiZWTaHiZmZZXOYmJlZNoeJmZllc5iYmVk2h4mZmWVzmJiZWTaHiZmZZXOYmJlZNoeJmZllaxgmkrZLOi3p6ULt65JOSDqYHjcV1t0hqV/SC5JuLNS7Uq1f0sZCfZGkx1P9fkmXp/q70+v+tL6z0THMzKwczZyZ3At0DVO/OyKWpsfDAJKWAGuBa9M235U0TdI04DvAamAJcGsaC/CttK8PAueA9am+HjiX6nencSMeY3Q/tpmZtVLDMImIx4CzTe5vDbArIl6PiJ8D/cDy9OiPiCMR8QawC1gjScDHgd1p+x3AzYV97UjLu4GVafxIxzAzs5Lk3DPZIOlQugw2K9XmAscLYwZSbaT6+4FfRsSFIfWL9pXWv5rGj7QvMzMryVjDZAvwAWApcBL4dss6aiFJPZKqkqpnzpwpux0zsylrTGESEaci4s2IeAv4Hm9fZjoBzC8MnZdqI9VfAWZKmj6kftG+0vr3pfEj7Wu4PrdGRCUiKh0dHWP5Uc3MrAljChNJcwovPw0MzvTaA6xNM7EWAYuBnwEHgMVp5tbl1G6g74mIAB4FbknbdwMPFfbVnZZvAX6axo90DDMzK0kzU4PvA/4RuEbSgKT1wF9KekrSIeCPgP8AEBHPAA8AzwI/Bm5PZzAXgA3AXuA54IE0FuBrwFcl9VO7J7It1bcB70/1rwIb6x0j89/BytDbC52dcNlltefe3rI7MrMxUu2P/amvUqlEtVotuw0b1NsLPT1w/vzbtRkzYOtWWLeuvL7M7CKS+iKi0mic3wFv5di06eIggdrrTZvK6cfMsjhMrBwvvTS6uplNag4TK8eCBaOrm9mk5jCxctx1V+0eSdGMGbW6mbXGBE5ycZhYOdatq91sX7gQpNqzb76btc7gJJdjxyCi9tzTM26B4tlcZmZTUWdnLUCGWrgQjh5tejeezWVmdimb4EkuDhMzs6logie5OEzMzKaiCZ7k4jAxM5uKJniSy/TGQ8zMrC2tWzdhMyR9ZmJmZtkcJmZmls1hYmZm2RwmZmaWzWFiZmbZHCZmZpbNYWJmZtma+Q747ZJOS3q6UPuvkp6XdEjSg5JmpnqnpH+RdDA9/rqwzbL0vfH9ku6RpFS/StI+SYfT86xUVxrXn45zXWFf3Wn8YUndrfwHMTOz0WvmzOReoGtIbR/w+xHxr4B/Bu4orHsxIpamxxcL9S3A54HF6TG4z43AIxGxGHgkvQZYXRjbk7ZH0lXAZuB6YDmweTCAzMysHA3DJCIeA84Oqf0kIi6kl/uBefX2IWkOcGVE7I/aZ97vBG5Oq9cAO9LyjiH1nVGzH5iZ9nMjsC8izkbEOWrBNjTszMxsArXinsmfAD8qvF4k6UlJ/yDpI6k2FxgojBlINYDZEXEyLb8MzC5sc3yYbUaqm5lZSbI+m0vSJuACMPjVXSeBBRHxiqRlwN9LurbZ/UVESGrZt3VJ6qF2iYwF/m5xM7NxM+YzE0n/DvgksC5duiIiXo+IV9JyH/Ai8HvACS6+FDYv1QBOpctXg5fDTqf6CWD+MNuMVH+HiNgaEZWIqHR0dIzxJzUzs0bGFCaSuoA/Az4VEecL9Q5J09Ly71K7eX4kXcZ6TdKKNIvrNuChtNkeYHBGVveQ+m1pVtcK4NW0n73AKkmz0o33ValmZu2ot7f2FbOXXVZ7HqfvKLfx1czU4PuAfwSukTQgaT3w34H3AvuGTAH+KHBI0kFgN/DFiBi8ef8l4PtAP7UzlsH7LN8EPiHpMHBDeg3wMHAkjf9e2p60v28AB9LjzsIxzKydfjn39kJPT+27yiNqzz09k7tnG5bSFaopr1KpRLVaLbsNs/E1+Mv5/Pm3azNmjOuXImXp7KwFyFALF8LRoxPdjQ1DUl9EVBqN8zvgzaaSTZsuDhKovd60qZx+GnnppdHVbdJymJhNJe32y3mkWZaefdl2HCZmU0m7/XK+667aZbiiGTNqdWsrDhOzqaTdfjmvW1e7n7NwIUi158l6f8fqynrToplNMoO/hDdtql3aWrCgFiST+ZfzunWTuz9risPEbKrxL2crgS9zmZlZNoeJmZllc5iYmTWrnT5dYIL5nomZWTOGfrrA4Ee/gO9R4TMTM7PmtNunC0wwh4mZWTPa7dMFJpjDxMysGe326QITzGFiZtaMdvt0gQnmMDEza4Y/+qUuz+YyM2uWP11gRD4zMTOzbA4TMzPL1lSYSNou6bSkpwu1qyTtk3Q4Pc9KdUm6R1K/pEOSrits053GH5bUXagvk/RU2uYeSRrrMczGhd/5bFZXs2cm9wJdQ2obgUciYjHwSHoNsBpYnB49wBaoBQOwGbgeWA5sHgyHNObzhe26xnIMs3Ex+M7nY8cg4u13PjtQzH6jqTCJiMeAs0PKa4AdaXkHcHOhvjNq9gMzJc0BbgT2RcTZiDgH7AO60rorI2J/RASwc8i+RnMMs9bzO5/NGsq5ZzI7Ik6m5ZeB2Wl5LnC8MG4g1erVB4apj+UYZq3ndz6bNdSSG/DpjCJasa9WHkNSj6SqpOqZM2fGqTOb8vzOZ7OGcsLk1OClpfR8OtVPAPML4+alWr36vGHqYznGRSJia0RUIqLS0dEx6h/QDPA7n82akBMme4DBGVndwEOF+m1pxtUK4NV0qWovsErSrHTjfRWwN617TdKKNIvrtiH7Gs0xzFrP73w2a6ipd8BLug/4GHC1pAFqs7K+CTwgaT1wDPhMGv4wcBPQD5wHPgcQEWclfQM4kMbdGRGDN/W/RG3G2HuAH6UHoz2G2bjxO5/N6lLtVsTUV6lUolqtlt2GmVlbkdQXEZVG4/wOeDMzy+YwMTOzbA6TRvwxGmZmDfkj6OsZ/BiNwXc/D36MBvhmrJlZgc9M6vHHaJiZNcVhUo8/RsPMrCkOk3r8MRpmZk1xmNTjj9EwM2uKw6Qef4yGmVlTPJurEX+MhplZQz4zMTOzbA4TMzPL5jAxM7NsDhMzM8vmMDEzs2wOEzMzy+YwMTOzbA4TMzPLNuYwkXSNpIOFx2uSviLp65JOFOo3Fba5Q1K/pBck3Viod6Vav6SNhfoiSY+n+v2SLk/1d6fX/Wl951h/DjMzyzfmMImIFyJiaUQsBZYB54EH0+q7B9dFxMMAkpYAa4FrgS7gu5KmSZoGfAdYDSwBbk1jAb6V9vVB4BywPtXXA+dS/e40zszMStKqy1wrgRcj4lidMWuAXRHxekT8HOgHlqdHf0QciYg3gF3AGkkCPg7sTtvvAG4u7GtHWt4NrEzjzcysBK0Kk7XAfYXXGyQdkrRd0qxUmwscL4wZSLWR6u8HfhkRF4bUL9pXWv9qGm9mZiXIDpN0H+NTwN+l0hbgA8BS4CTw7dxjjJWkHklVSdUzZ86U1YaZ2ZTXijOT1cATEXEKICJORcSbEfEW8D1ql7EATgDzC9vNS7WR6q8AMyVNH1K/aF9p/fvS+ItExNaIqEREpaOjI/sHNTOz4bUiTG6lcIlL0pzCuk8DT6flPcDaNBNrEbAY+BlwAFicZm5dTu2S2Z6ICOBR4Ja0fTfwUGFf3Wn5FuCnabyZmZUg6/tMJF0BfAL4QqH8l5KWAgEcHVwXEc9IegB4FrgA3B4Rb6b9bAD2AtOA7RHxTNrX14Bdkv4CeBLYlurbgL+R1A+cpRZAZmZWEl0qf9BXKpWoVqtlt2Fm1lYk9UVEpdE4vwPezMyyOUzMzCybw8TMzLI5TMzMLJvDxMzMsjlMzMwsm8PEzMyyOUzMzCybw8TMzLI5TMzMLJvDxMzMsjlMzMwsm8PEzMyyOUzMzCybw8TMzLI5TMzMLJvDxMzMsjlMzMwsW3aYSDoq6SlJByVVU+0qSfskHU7Ps1Jdku6R1C/pkKTrCvvpTuMPS+ou1Jel/fenbVXvGGZmNvFadWbyRxGxtPA9wRuBRyJiMfBIeg2wGlicHj3AFqgFA7AZuB5YDmwuhMMW4POF7boaHMPMzCbYeF3mWgPsSMs7gJsL9Z1Rsx+YKWkOcCOwLyLORsQ5YB/QldZdGRH7IyKAnUP2NdwxzMxsgrUiTAL4iaQ+ST2pNjsiTqbll4HZaXkucLyw7UCq1asPDFOvdwwzM5tg01uwjz+MiBOSfgfYJ+n54sqICEnRguOMaKRjpHDrAViwYMF4tmBmdknLPjOJiBPp+TTwILV7HqfSJSrS8+k0/AQwv7D5vFSrV583TJ06xyj2tjUiKhFR6ejoyPkxzcysjqwwkXSFpPcOLgOrgKeBPcDgjKxu4KG0vAe4Lc3qWgG8mi5V7QVWSZqVbryvAvamda9JWpFmcd02ZF/DHcPMzCZY7mWu2cCDabbudOBvI+LHkg4AD0haDxwDPpPGPwzcBPQD54HPAUTEWUnfAA6kcXdGxNm0/CXgXuA9wI/SA+CbIxzDzMwmmGqTpKa+SqUS1Wq17DbMzNqKpL7C2z5G5HfAm5lZNoeJmZllc5iYmVk2h4mZmWVzmJiZWTaHiZmZZXOYmJlZNoeJmZllc5iYmVk2h4mZmWVzmJiZWTaHiZmZZXOYmJlZNoeJmZllc5iYmVk2h4mZmWVzmJiZWTaHiZmZZRtzmEiaL+lRSc9KekbSl1P965JOSDqYHjcVtrlDUr+kFyTdWKh3pVq/pI2F+iJJj6f6/ZIuT/V3p9f9aX3nWH8OMzPLl3NmcgH4jxGxBFgB3C5pSVp3d0QsTY+HAdK6tcC1QBfwXUnTJE0DvgOsBpYAtxb28620rw8C54D1qb4eOJfqd6dxZmZWkjGHSUScjIgn0vKvgOeAuXU2WQPsiojXI+LnQD+wPD36I+JIRLwB7ALWSBLwcWB32n4HcHNhXzvS8m5gZRpvZmYlaMk9k3SZ6UPA46m0QdIhSdslzUq1ucDxwmYDqTZS/f3ALyPiwpD6RftK619N483MrATZYSLpt4EfAF+JiNeALcAHgKXASeDbucfI6K1HUlVS9cyZM2W1YWY25WWFiaR3UQuS3oj4IUBEnIqINyPiLeB71C5jAZwA5hc2n5dqI9VfAWZKmj6kftG+0vr3pfEXiYitEVGJiEpHR0fOj2pmZnXkzOYSsA14LiL+qlCfUxj2aeDptLwHWJtmYi0CFgM/Aw4Ai9PMrcup3aTfExEBPArckrbvBh4q7Ks7Ld8C/DSNNzOzEkxvPGREHwY+Czwl6WCq/Tm12VhLgQCOAl8AiIhnJD0APEttJtjtEfEmgKQNwF5gGrA9Ip5J+/sasEvSXwBPUgsv0vPfSOoHzlILIDMzK4kulT/oK5VKVKvVstswM2srkvoiotJonN8Bb2Zm2RwmZmaWzWFiZmbZHCZmZpbNYWJmZtkcJmZmls1hYmZm2RwmZmaWzWFiZmbZHCZmZpbNYWJmZtkcJmZmls1hYmZm2RwmZmaWzWFiZmbZHCZmZpbNYWJmZtkcJmZmlq2tw0RSl6QXJPVL2lh2P2Zml6q2DRNJ04DvAKuBJcCtkpa0/EA33ADS248bbmj5IVqmnXqF9urXvY6fduq3nXqFie03ItryAfwBsLfw+g7gjpHGL1u2LEZt5coIeOdj5crR72u8tVOvEe3Vr3sdP+3Ubzv1GtGyfoFqNPE7WbWx7UfSLUBXRPxpev1Z4PqI2DDc+EqlEtVqdbQH4fmrF7JhzdfeuW5J60+Csjz77MjrJluv0F79utfx0079tlOvcFG/HzvSx396dNvb60bxe19SX0RUGo2bPrru2oukHqAHYMGCBWPax29deINrfnHsnStmX5/TWus9NkyPgyZbr9Be/brX8dNO/bZTr3BRv3N+dWbcD9fOZyZ/AHw9Im5Mr+8AiIj/Mtz4sZ6ZjGiy/bu1U6/QXv261/HTTv22U6/Qsn6bPTNp2xvwwAFgsaRFki4H1gJ7WnqElStHVy9TO/UK7dWvex0/7dRvO/UKE99vMzdWJusDuAn4Z+BFYFO9sWO6AR/xzptYk/VmW0R79RrRXv261/HTTv22U68RLemXqX4DfrTGdJnLzOwSdylc5jIzs0nCYWJmZtkcJmZmls1hYmZm2RwmZmaW7ZKZzSXpDFDnLawNXQ38okXtjLd26hXaq1/3On7aqd926hXy+l0YER2NBl0yYZJLUrWZ6XGTQTv1Cu3Vr3sdP+3Ubzv1ChPTry9zmZlZNoeJmZllc5g0b2vZDYxCO/UK7dWvex0/7dRvO/UKE9Cv75mYmVk2n5mYmVk2h0kDkrokvSCpX9LGsvupR9J2SaclPV12L41Imi/pUUnPSnpG0pfL7qkeSb8l6WeS/in1+5/L7qkRSdMkPSnpf5XdSyOSjkp6StJBSZP6E1klzZS0W9Lzkp5L3600KUm6Jv2bDj5ek/SVcTmWL3ONTNI0ah9x/wlggNp3qNwaEXW+v7M8kj4K/BrYGRG/X3Y/9UiaA8yJiCckvRfoA26exP+2Aq6IiF9Lehfwf4AvR8T+klsbkaSvAhXgyoj4ZNn91CPpKFCJiEn/3g1JO4D/HRHfT9+lNCMifll2X42k32cnqH29ec577oblM5P6lgP9EXEkIt4AdgFrSu5pRBHxGHC27D6aEREnI+KJtPwr4DlgbrldjSx9tcOv08t3pcek/UtM0jzgj4Hvl93LVCLpfcBHgW0AEfFGOwRJshJ4cTyCBBwmjcwFjhdeDzCJf+G1K0mdwIeAx8vtpL502eggcBrYFxGTud//BvwZ8FbZjTQpgJ9I6pPUU3YzdSwCzgD/I11C/L6kK8puqklrgfvGa+cOEyuVpN8GfgB8JSJeK7ufeiLizYhYCswDlkualJcSJX0SOB0RfWX3Mgp/GBHXAauB29Ml28loOnAdsCUiPgT8X2BS30sFSJfjPgX83Xgdw2FS3wlgfuH1vFSzFkj3Hn4A9EbED8vup1npssajQFfZvYzgw8Cn0n2IXcDHJf3PcluqLyJOpOfTwIPULjFPRgPAQOGsdDe1cJnsVgNPRMSp8TqAw6S+A8BiSYtSsq8F9pTc05SQbmhvA56LiL8qu59GJHVImpmW30NtUsbz5XY1vIi4IyLmRUQntf+zP42If1tyWyOSdEWahEG6ZLQKmJQzEiPiZeC4pGtSaSUwKSeNDHEr43iJC2qnbDaCiLggaQOwF5gGbI+IZ0pua0SS7gM+BlwtaQDYHBHbyu1qRB8GPgs8le5DAPx5RDxcYk/1zAF2pBkxlwEPRMSkn3LbJmYDD9b+vmA68LcR8eNyW6rr3wO96Q/MI8DnSu6nrhTQnwC+MK7H8dRgMzPL5ctcZmaWzWFiZmbZHCZmZpbNYWJmZtkcJmZmls1hYmZm2RwmZmaWzWFiZmbZ/j+rTL55JRrcaQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xy = np.array([[828.659973, 833.450012, 908100, 828.349976, 831.659973],\n",
    "               [823.02002, 828.070007, 1828100, 821.655029, 828.070007],\n",
    "               [819.929993, 824.400024, 1438100, 818.97998, 824.159973],\n",
    "               [816, 820.958984, 1008100, 815.48999, 819.23999],\n",
    "               [819.359985, 823, 1188100, 818.469971, 818.97998],\n",
    "               [819, 823, 1198100, 816, 820.450012],\n",
    "               [811.700012, 815.25, 1098100, 809.780029, 813.669983],\n",
    "               [809.51001, 816.659973, 1398100, 804.539978, 809.559998]])\n",
    "\n",
    "x_train = xy[:, 0:-1]\n",
    "y_train = xy[:, [-1]]\n",
    "\n",
    "plt.plot(x_train, 'ro')\n",
    "plt.plot(y_train)\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Data에 표준화를 적용하여 실행한다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.         1.         0.         1.         1.        ]\n",
      " [0.70548491 0.70439552 1.         0.71881783 0.83755792]\n",
      " [0.54412549 0.50274824 0.57608696 0.60646801 0.6606331 ]\n",
      " [0.33890353 0.31368023 0.10869565 0.45989134 0.43800918]\n",
      " [0.51436    0.4258239  0.30434783 0.58504805 0.42624401]\n",
      " [0.49556179 0.4258239  0.31521739 0.48131134 0.49276137]\n",
      " [0.11436064 0.         0.20652174 0.22007776 0.18597238]\n",
      " [0.         0.07747099 0.5326087  0.         0.        ]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VNX9//HXJ5MFkrAnLLIkgIBQZDOyCKgV7BfUgrtg3NFQFcWv/mxtqd+2tnxba2u/WEFFxAWjgDvuVdQqyBYEQXZkBwmryCKQ5fz+mMEGTMgQZubOTN7Px4PH5N57MvdjHuadO+eee4455xARkfiS4HUBIiISegp3EZE4pHAXEYlDCncRkTikcBcRiUMKdxGROKRwFxGJQwp3EZE4pHAXEYlDiV6dOCMjw2VnZ3t1ehGRmDR//vwdzrnMytp5Fu7Z2dkUFBR4dXoRkZhkZuuDaaduGRGROKRwFxGJQwp3EZE4pHAXEYlDCncRkThUabib2UQz22ZmX1Vw3MzsETNbbWaLzKxb6MsM6N8fzP7zr3//sJ0qJPLzITsbEhL8r/n5XlckItVEMFfuzwADjnN8INAm8C8PeOzkyypH//4wffrR+6ZPj96Az8+HvDxYvx6c87/m5SngRSQiKg1359ynwK7jNBkMPOf8ZgN1zaxJqAr8QSDY19Y7hYf6XktRgu+o/VFn1Cg4cODofQcO+PeLiIRZKPrcmwIby2xvCuz7ETPLM7MCMyvYvn17lU72QZsejD3rKq7IfZCNtRtW6T0iYsOGE9svIhJCEb2h6pwb75zLcc7lZGZW+vRsufLmvsbY1//M1w2ac8GNj/Bu27NCXGWItGhxYvtFREIoFOG+GWheZrtZYF9o9ev3w5cXrpjJ20/fSatdW7j1kt/w29cXc7CoJOSnPCmjR0Nq6tH7UlP9+0VEwiwU4T4NuC4waqYnsMc5900I3vdoH354VMC32FPIS9+8S97ZrXh+9gYuGfc5X2/fF/LTVlluLowfD1lZ/pE9WVn+7dxcrysTkWrAnHPHb2D2InAukAEUAr8DkgCcc4+bmQGP4h9RcwC40TlX6YxgOTk5LlQTh328fBt3T13IoeJS/ji4I5ed0Swk7ysiEm3MbL5zLqfSdpWFe7iEMtwBtu45yMjJC5izdheXdmvKHwd3JC3Fs0kvRUTCIthwj5snVBvXqcELt/RkZL82vL5gMz9/dAZLt3zndVkiIp6Im3AH8CUY/31+W/Jv7sm+g8VcPG4mk2atw6tPJyIiXomrcD+iV+sGvDuyL2e1bsD9byzhtvwv2PN9kddliYhETFyGO0CD9BQmXn8mv7ngND5YWsgFYz7jiw27vS5LRCQi4jbcARISjLyzW/PSL3phBlc+PovH//01paXqphGR+BbX4X5E1xb1ePvOvvzsJ434y7vLufGZeezYd8jrskREwqZahDtAnZpJjL26G3+6uCOz1uzkgjGf8fnXO7wuS0QkLKpNuAOYGdf0zOL123qTXiOR3AlzePiDlRSXlHpdmohISFWrcD+iwym1eeuOPlzWrRmPTF/F1RPm8M2e770uS0QkZKpluAOkJifytys68/CVnflq8x4uGPMZHy0v9LosEZGQqLbhfsSl3Zrx5h19aFynJjc9U8Cf3lrK4WJ104hIbKv24Q7QOjOd1247i+t7ZTFhxlquePxzNuw8UPk3iohEKYV7QI0kH38Y3JHHr+nG2h37ufCRz3hr0RavyxIRqRKF+zEGdGzC23f25dRG6Yx4YQG/fjUKFwIREamEwr0czeunMnV4L35xTmtenLuBwY/OZFXhXq/LEhEJmsK9Akm+BO4beBrP3tSdHfsOMejRmUydt1EzTIpITFC4V+Kctpm8O7IvXVvU5ZevLOKuKQvZd6jY67JERI5L4R6EhrVrMGlYD+45vy1vfrmFix75jK827/G6LBGRCincg+RLMO7o14bJeb04WFTKpeM+55mZa9VNIyJRSeF+grq3rM+7I/vSt00Gv39zKcMnzefbA4e9LktE5CgK9yqol5bMhOtzuP+iDny8YhsXPjKD+et3eV2WiMgPFO5VZGYM69OSV249C1+CceUTsxn78WotBCIiUUHhfpI6NavLW3f2YWDHxjz0/gquf3ou2/dqIRAR8ZbCPQRq10jin0O78pdLT2fu2l0MHPMZM1ZpIRAR8Y7CPUTMjCHdWzBtRB/qpSZx7cQ5/O39FVoIREQ8oXAPsXaNazFtRB+uPKM5j368miHjZ7PlWy0EIiKRpXAPg5rJPh68vBNjhnRh+da9DHp0BnPW7PS6LBGpRhTuYTS4S1Nev703tWskkTthDpNmrdNDTyISEQr3MDu1YTqvj+jN2W0zuf+NJdz3ymIOFWsKYREJL4V7BNSukcSE63K447xTmVKwkSHjZ1P43UGvyxKROBZUuJvZADNbYWarzey+co63MLOPzWyBmS0yswtCX2psS0gw7vlZOx6/phsrtu7lon/OYP763V6XJSJxqtJwNzMfMBYYCHQAhppZh2Oa/RaY6pzrCgwBxoW60HgxoGMTXrutN6nJPoaMn8XkuRu8LklE4lAwV+7dgdXOuTXOucPAZGDwMW0cUDvwdR1Ai48eR7vGtZh2ex96tc7gvlcX89vXF3O4WOPhRSR0ggn3psDGMtubAvvK+j1wjZltAt4B7ghJdXGsTmoST99wJr84pzXPz95A7oTZbNurfngRCY1Q3VAdCjzjnGsGXABMMrMfvbeZ5ZlZgZkVbN++PUSnjl2+BOO+gafxyNCuLN68h0H/nMmXG7/1uiwRiQPBhPtmoHmZ7WaBfWUNA6YCOOdmATWAjGPfyDk33jmX45zLyczMrFrFcWhQ51N49dbeJPqMK56YxcvzN3ldkojEuGDCfR7Qxsxamlky/hum045pswHoB2Bm7fGHuy7NT0CHU2ozbUQfcrLq8f9e+pLfT1tCkealEZEqqjTcnXPFwAjgfWAZ/lExS8zsATMbFGh2D3CLmX0JvAjc4PQo5gmrn5bMczd1Z1ifljzz+TqufWoOO/dp+mAROXHmVQbn5OS4goICT84dC179YhO/fnUxGekpPHHtGXRsWsfrkkQkCpjZfOdcTmXt9IRqlLq0WzNe/sVZOOe4/PHPeWPhsbc5REQqpnCPYqc3q8O0O/rQqVldRk5eyOi3l2p+eBEJisI9ymWkp5B/cw+u65XFk5+t5Yan57F7/2GvyxKRKKdwjwFJvgQeGNyRv17WiblrdzFo7AyWffOd12WJSBRTuMeQK89szpThPTlcXMql4z7n7UXfeF2SiEQphXuM6dqiHm+O6EP7JrW4/YUv+Ot7yykp1ahTETmawj0GNaxdgxfzejK0e3PGffI1w56dx57vi7wuS0SiiMI9RqUk+vjzpZ0YfUlHZq7ewcVjZ7KqcK/XZYlIlFC4h1N+PmRnQ0KC/zU/P+SnyO2RxQu39GTvwWIuHjuT95dsDfk5RCT2KNzDJT8f8vJg/Xpwzv+alxeWgD8zuz5v3tGbUxumM3zSfB7+YCWl6ocXqdYU7uEyahQcOHD0vgMH/PvDoEmdmkwZ3ovLz2jGI9NXkTdpPnsPqh9epLpSuIfL+vUntj8EaiT5eOjyTvxh0E/4eMU2Lh47k6+37wvb+UQkeincwyWhgh9tRftDxMy4/qxsnh/Wg90Hirj40Zl8tLwwrOcUkeijcA+X0grmgKlof4j1at2AaSN606JBKsOeLeDRj1ahWZhFqg+FexxrVi+Vl39xFoM7n8Lf/rWS2/K/YP+hYq/LEpEIULiHS4MGJ7Y/TGom+/jHVV347YXteX/JVi4ZN5P1O/dHtAYRiTyFe7iMGQPJyUfvS072748wM+Pmvq147qYebNt7iJ//cwb/XqlVEEXimcI9XHJzYeJEyMoCM//rxIn+/R7p0yaDabf34ZS6Nbnx6bk88e+v1Q8vEqe0zF41dOBwMfe+tIi3F3/DzzufwoOXnU5qcqLXZYlIELTMnlQoNTmRR6/uyq8GnMZbi7Zw2WOz2LjrQOXfKCIxQ+FeTZkZt57bmok3nMmm3QcY9OgMPl+9w+uyRCREFO7V3E/bNWTaiD40SE/h2olzmThjrdcliUgIKNyFlhlpvH57b/qd1pAH3lrKox+t8rokETlJCncBID0lkceuOYNLujblb/9aybhPVntdkoicBA2RkB/4Eoy/XdGZklLHX99bgc+M4ee09rosEakChbscxZdgPHxlZ0qd48/vLseX4H8ASkRii8JdfiTRl8D/XdWFUuf409vLSDDjpj4tvS5LRE6Awl3KlehLYMyQrpSWLuCBt5biS/BPJSwisUE3VKVCSb4EHhnalfM7NOJ305YwaXb4FhoRkdBSuMtxJScmMPbqbvRv35D7X/+KF+Zs8LokEQmCwl388vMhO9u/UlR29lELeScnJjA2txvnndaQ37y2mMlzFfAi0S6ocDezAWa2wsxWm9l9FbS50syWmtkSM3shtGVKWOXnQ16ef31X5/yveXlHBXxKoo9xud04p20mv35tMVMLNnpYsIhUptJZIc3MB6wEzgc2AfOAoc65pWXatAGmAuc553abWUPn3Lbjva9mhYwi2dnlL9ydlQXr1h2162BRCbc8V8CM1Tt46PLOXH5Gs4iUKCJ+oZwVsjuw2jm3xjl3GJgMDD6mzS3AWOfcboDKgl2izIYKulnK2V8jyceT1+XQu3UG9778Ja8t2BTm4kSkKoIJ96ZA2c/gmwL7ymoLtDWzmWY228wGlPdGZpZnZgVmVrB9u1YCihotWpzQ/iMB36tVA+6Z+iVvLNwcxuLiwHHuZ4iES6huqCYCbYBzgaHAk2ZW99hGzrnxzrkc51xOZmZmiE4tJ230aEhNPXpfaqp/fwVqJvuYcH0O3VvW57+nLOTNL7eEucgYFcT9DJFwCCbcNwPNy2w3C+wraxMwzTlX5Jxbi7+Pvk1oSpSwy82F668Hn8+/7fP5tytZEjA1OZGJN5xJTlZ97pqykLcXfROBYmPMqFFw4JiFUA4c8O8XCaNgwn0e0MbMWppZMjAEmHZMm9fxX7VjZhn4u2nWhLBOCaf8fHj2WSgp8W+XlPi3g7i6TE1O5Okbz6Rr87rcOXkB732lgD/KCdzPEAmlSsPdOVcMjADeB5YBU51zS8zsATMbFGj2PrDTzJYCHwP3Oud2hqtoCbGTvLpMS0nkmZu607lZHUa8sIB/LdkahiJj1AnezxAJFS2QLf4bfeX9f2AGpaVBv83eg0Vc+9RclmzZw2O5Z9C/Q6MQFhmjjvS5l/3jmZoK48dX2u0lUh4tkC3BC9HVZa0aSTw3rDsdmtTmtvwv+Hi5RsSSm+sP8qws/x/LrCwFu0SEwl2qNFqmIrVrJPHcsB60a1yL4ZPm88kKBTy5uf6HwUpL/a8KdokAhbtUebRMRerUTGLSsO60aZRO3qT5fLpSzzSIRJrCXU5qtExF6qYm8/ywHrTOTPdPV7BqR4iKFZFgKNwlbGOx66Ulk39zD1pmpHHzc/P4fLUCXiRSFO4S1rHY9QMBn1U/jZuencfsNRohK9VYBKeiULhL2MdiN0hPIf+WHjSvl8qNT89j7tpdIXlfkZgS4akoFO4S0tEyFclIT+GFW3pySt0a3PD0XArWKeClmonwVBQKd4nYWOzMWim8eEtPGteuwfUT5zJ//e6Qvn/U0qyQAhGfikJPqErEFX53kCHjZ7N97yEmDetO1xb1vC4pfPSEqhxxAoviHI+eUJWo1ah2DV64pQcN0pO57qm5fLnxW69LCh/NCilHRKD7syyFu/hFuOugSZ2avHhLT+qmJXHtU3NYvGlPWM/nGc0KKUdEeCoKhbt4tqDEKXX9AV+7ZhLXPDWHrzbHYcDH4qyQukcQPhGcikLhLp52HTSrl8qLt/QkPSWRa56aw9It34X9nBEV4Y/iJ00rR8UNhbt43nXQvL4/4FOTfOROmM3yrXEU8LE2K+SoUZQe+J4VGVkUWyAedI8gJmm0jITsLv7JWr9zP1c9MZuiklJezOtJ20a1Inbu6u7A4WJmrNrBh/f+mY9ancmO9Hpc9eX7PPjeP/0NTnBufwkfjZaR4EVJ10FWgzRezOtJos+4+snZrCrcG9HzVzdb9xwkf856bnpmHl0f+IC8SfN5t11vem1YxOAlnzCl83/xTrve/sb163tbrJywRK8LkChwpItg1Ch/V0yLFv5g96DroGVGGi/c0pMh42cz9Mk5TM7ryakN0yNeRzxyzrFky3d8uKyQ6cu2sThwA7tF/VSu7tGC89s34sxePyFpxzaKEnysq9eE+wbcQZctKzgFbz7hS9WpW0ai0upt+xgyfjYJBpPzetIqUwFfFQeLSpi1ZicfLi3ko+Xb+GbPQcygW4t69GvfkPPbN+LUhumYmf8byiy5uK5uEy68YQw/KfyaF6f8Fl9JsYf/JXJEsN0yCneJWqsK9zL0ydn4EozJeb1omZH2n4P5+VHxSSMa7dh3iI+Wb2P6skI+W7WDA4dLSE32cXabTPq1b8hPT2tIRnpK+d98zP2Xlzuex/+78G7uXTSN2995IjL/AXJcCneJCyu2+gM+2ZfAlOE9yWqQpkf6j+GcY9W2fXywtJDpywpZsPFbnIMmdWrQr31D+rdvRM9WDaiR5Kv8zY752Trgzkt+zTttz+Ll23rH91QRMULhLnFj+dbvGDp+NjWTfEzO60WLMzpExegeLxWVlDJ37S5/oC8vZOOu7wE4vWkd+rdvRL/2DfnJKbX/091yIo75VLTngf/lgi2N8SUY74zsS3qKbtV5SeEucWXplu+4esJs0pITmfyXq2m+p/DHjeJ8uN63Bw7zyYrtfLiskH+v2M7eQ8WkJCbQ+9SMHwK9Ue0aYTl3wbpdXPnELC7u2pSHr+wSlnNIcIINd/0JlpjQ4ZTaPD+sB7kT5jD0mgeZ8ty9NN17zMLb0fxIfxWt3bGfD5cW8uGyQgrW76ak1JGRnswFpzehX/uG9GmTQWpy+H+Nc7Lrc8d5bRgzfRXntM1kcJemYT+nnBxduUtMWbxpD7njPqXu7m1Mzr+PU/YG1mWNkz734pJSvtjwLdOXFfLBskLWbN8PwGmNa/3Qf965WV0SEqrQ3RKC2q4aP5uVW/fyzsi+NK+fWvk3ScipW0bi1pcbv+Wax2ZQf88Obv8sn1q100i/ZihpF/yMWimJpNdIJC0lkbTkRHwehOCJ2nuwiE9X7mD6skI+WrGNbw8UkeQzerZqQL/TGtKvfaOoCdKNuw5wwZjPaNe4FpPzepLo03OQkaZwl7i2YMNubnh6Hnu+Lzpuu7RkH2mBwK+V4g/99GO3yzmWfszXNZN8Vbs5CeUO29w48BKmLytk+vJtzF6zk6ISR93UJH7azn91fnbbDGrVSKra+U5WJcNM31i4mZGTF3JX/zbc1b+tNzVWYwp3iXsHi0rYvvcQ+w8Xs+9gMXsP+V/3Hypm36Fi9pb9+lDg64NljgW+r7i08t+BBONHgZ+Wkkitsl+X+dSQHjiW9unHpI9+gFp7drEztQ7TT+3Oh217sTwjC4BWmWn0b9+I/u0b0a1FXe+vhIMcZnr3lIW8vnAzU4f3IidbUxNEksJdJAjOOQ4Vl7Lv2OA/3h+FCo7tP1xMZb9OCaUl5Gxayvk7V9Hv+THR9+RtkJPI7T1YxIWPzKCk1PHuXX2p7dWnjGpI4S4SYaWljgNFJYGwL2LfoRL2nf1T9iXXZF9yTVKKi+izbiH1Du6N3mGbZaYfOEo59X6xYTdXPD6LC09vwpghXarebSUnJKRDIc1sADAG8AETnHN/qaDdZcDLwJnOOSW3VCsJCfZD1w0Expu73bBy4Y8bR+uwzRYtyr9yL6febi3qcVe/Nvz9g5Wc2y6TS7s1i0CBEqxKO/jMzAeMBQYCHYChZtahnHa1gJHAnFAXKRKzomQ65aCdYL23/fRUumfX5/7Xv2L9zv0RKFCCFczdm+7AaufcGufcYWAyMLicdn8EHgQOhrA+kdgWaysxnWC9vgTjH0O64EswRk5eSFFJFHY1VVPBhHtTYGOZ7U2BfT8ws25Ac+fc2yGsTSQ+RHBR5JA4wXqb1q3Jny/txMKN3zLmw1URKVEqd9LjrswsAXgYuCeItnlmVmBmBdu3b6+suYjEiAs7NeHKnGaM/WQ1s9fs9LocIbhw3ww0L7PdLLDviFpAR+ATM1sH9ASmmdmP7uY658Y753KcczmZmZlVr1pEos7vfv4Tshuk8d9TFrLnwPEfLpPwCybc5wFtzKylmSUDQ4BpRw465/Y45zKcc9nOuWxgNjBIo2VEqpe0lETGDOnC9r2H+PVri/BqmLX4VRruzrliYATwPrAMmOqcW2JmD5jZoHAXKCKxo1Ozutzzs3a8s3grLxVs8rqcai2oce7OuXeAd47Z9z8VtD335MsSkVg1/OxWfLZqO7+btoSc7HrR9xRuNaEp3UQkpBISjIev7EJKUgIjJy/kcLGGR3pB4S4iIde4Tg3+cmknFm/ew98/WOF1OdWSwl1EwmJAx8Zc3aMF4z9dw8zVO7wup9pRuItI2Nx/YQdaZaRx99SF7N5/2OtyqhWFu8Sm/Hz/9LQJCf7X/HyvK5Jy1Ez2MWZIV3bvL+JXr2h4ZCQp3CX2HFlQYv16//S069f7txXwUalj0zr8ckA7/rW0kBfmbvC6nGpD4S6xZ9Soo1cKAv/2qFHe1COVuql3S/q2yeCPby1l9ba9XpdTLSjcJfZsqODqr6L94rmEBOPvV3QmNTmRO15cyKHiEq9LinsKd4k9FS10Ea0LYAgADWvX4KHLO7Hsm+946D0Njww3hbvEntGjIemYNTuTkqJ3AYxYE8ab1f3aN+K6XllMmLGWf6/UzLDhpHCX2HTsep1avzM0InCz+jcXtKdto3TumfolO/YdCtn7ytEU7hJ7Ro2Cw8eMmT58OHpvqMbSsM0I3KyukeTjkaFd+e5gEb98WcMjw0XhLrEnlm6oxtqwzQj9bE9rXJvfDDyNj5Zv47lZ5SzILSdN4S6xJ5ZuqMbasM0I/myvPyubn7bLZPQ7y1ixVcMjQ03hLrFn9GhITT16X2pqdN5QjaVPGRDRn62Z8dAVnaldI4k7X1zAwSINjwwlhbvEntxcGD8esrL8N1Kzsvzb0bjwdCx9yoCI/2wz0lP42xWdWFG4l7+8uzws56iuzKubGTk5Oa6gQCvxSZw70udetmsmNTV6/xh55IE3lzJx5lom3pDDeac18rqcqGZm851zP1qj+li6chcJp1j6lOGhXw1sR/smtbn3pUVs23vQ63LigsJdJNxyc2HdOigt9b8q2H8kJdHHI0O6sO9QMfdM/ZLSUg2PPFkKdxGJCm0a1eL+izrw2aodTJy51utyYp7CXUSiRm6PFpzfoRF/fW8FS7bs8bqcmKZwF5GoYWY8eFkn6qb6h0d+f1jDI6tK4S4iUaV+WjL/uKoLa3bs509vL/W6nJilcBeRqNP71Azy+rYif84G3l+y1etyYpLCXUSi0j0/a0fHprX51SuL2LpHwyNPlMJdRKJScmICY4Z05VBRKfe8tFDDI0+Qwl1EolbrzHR+P6gDM1fv5MnP1nhdTkxRuItIVLsypzkDOzbmofdXsGjTt0cfjKW58iNM4S4iUc3M+POlp5NZK4WRkxey/1Cx/0CszZUfYQp3EYl6dVP9wyPX7dzPA28GhkfG2lz5EaZwF5GY0LNVA247tzVTCjbyzuJvYm+u/AgLKtzNbICZrTCz1WZ2XznH7zazpWa2yMymm1lW6EsVkerurv5t6dy8Lve9sogtzU8tv1H9+pEtKkpVGu5m5gPGAgOBDsBQM+twTLMFQI5zrhPwMvDXUBcqIpLkS2DMVV0oKXXcde5wSkydDxUJ5ifTHVjtnFvjnDsMTAYGl23gnPvYOXek82s20Cy0ZYqI+GVnpPHA4I7MbXIaj/e47McNdu2KfFFRKJhwbwpsLLO9KbCvIsOAd8s7YGZ5ZlZgZgXbt28PvkoRkTIu7daUn68v4OG+17CgSdujD0brEoYRFtLPNGZ2DZADPFTecefceOdcjnMuJzMzM5SnFpFqxMz408A2NN63k5E/v5d9yTX9B6J1oXQPBBPum4HmZbabBfYdxcz6A6OAQc65Q6EpT0SkfHWuu5oxXWqyqU4jfj1gBCXZ2VrCsIzEINrMA9qYWUv8oT4EuLpsAzPrCjwBDHDObQt5lSIi5cjJG8I9H6/moYQEDlxyBWMu70q610VFiUqv3J1zxcAI4H1gGTDVObfEzB4ws0GBZg8B6cBLZrbQzKaFrWIRkTJu/+mp/HHwT/hk5XYuf+xzNn/7vdclRQVzzpuZ1nJyclxBQYEn5xaR+PPvldsZkf8FKUk+JlyfQ5fmdb0uKSzMbL5zLqeydhokKiJx4Zy2mbx621nUTE7gqidm8faib7wuyVMKdxGJG20a1eL123pzetM63P7CFzz60Sq86p3wmsJdROJKg/QUnr+5B5d0bcrf/rWSu6d+yaHi6rfQdjCjZUREYkqNJB8PX9mZVhlp/P2DlWzcdYAnrj2DBukpXpcWMbpyF5G4ZGbc0a8Nj17dlcWb93DxuJmsKtzrdVkRo3AXkbh2UadTmDK8F98fLuXScZ/z2arqMfWJwl1E4l6X5nV5Y0RvmtaryQ1Pz2PS7PVelxR2CncRqRaa1q3Jy7eexTltM7n/9a/4w5tLKCmN35E0CncRiV0nuEB2ekoiT16Xw7A+LXl65jpufnYeew8WRaTUSFO4S2zSqvdSxQWyfQnG/Rd14E8Xd+TTVTu4/LFZbNp94LjfE4s0/YDEniO/1GUXR05N1YyA1U12tj/Qj5WVBevWBfUWM1bt4Nb8+aQkJjD+uhy6tagX0hLDQdMPSPzSqvcCIVkgu0+bDF677SxSkxMZMn42b365JUTFeU/hLrFHq94LVLzi0gmuxHRqw1q8fntvOjerwx0vLmDMh/ExZYHCXWJPiH6pJcaNHu3vjiuriisx1U9L5vmbe3Bpt6b848OV3DVlIQeLYnvKAoW7xJ4Q/lJLDMvN9d9nycoCM//rSdx3SUn08fcrOnPvf7XjjYVbuPrJ2ezYF7uLyumGqsSm/Hx/H/uGDf4r9tGjdTNygJMEAAAHf0lEQVRVQuadxd9w99SFZKSnMPGGM2nbqJbXJf1AN1QlvuXm+kdElJb6XxXsEkIXnN6EKXm9OFRcymXjPuffK0M0ZUEEh/Aq3EVEytG5eV3euL03zeqncuPTc3lu1rqTe8MqjsuvKoW7iEgFTqlbk5d/0YvzTmvI/7yxhN+98RXFJaVVe7MID+FVuIuIHEdaSiJPXJvDLX1b8uys9Qx7tqBqUxZEeAivwl1EpBK+BGPUhR3486WnM3P1Di577HM27jrBKQsiPIRX4S4iEqSh3Vvw7E3d2brnIBePncn89buD/+YID+FVuIuInIDep2bw6m29Sa+RyNAnZ/PGws3BfWOIx+VXRuPcRUSqYPf+wwx/fj5z1+5iZL823NW/DWYW9vNqnLuISBjVS0vm+WE9uPyMZoyZvoo7J0fXlAWJXhcgIhKrkhMTeOjyTrTOTOfB95azafcBxl+bQ2atFK9L05W7iMjJMDNuPbc1j1/TjWXffMfFY2eyYuter8tSuIuIhMKAjk14afhZFJWUctljn/Px8m2e1qNwFxEJkdOb1eGNEb3JapDKsGfn8fTMtZ7NDa9wFxEJoSZ1ajJ1eC/6tW/EH95cyv+8saTqUxachKDC3cwGmNkKM1ttZveVczzFzKYEjs8xs+xQFyoiEivSUhJ54pozGH5OKybNXs+Nz8zju6pMWXASKg13M/MBY4GBQAdgqJl1OKbZMGC3c+5U4B/Ag6EuFNCK9yJytCjOhIQE49cD2/PgZacz6+udXDbuczbcfg8kJvofYkpMhNtuC9/5g2jTHVjtnFvjnDsMTAYGH9NmMPBs4OuXgX4W6tH8EZ4uU0SiXIxkwlVntuC5Yd3ZVriLi31nUNC4rf9ASQk89ljYAj6YcG8KbCyzvSmwr9w2zrliYA/QIBQF/kAr3otIWTGUCWe1zuC1iSOpc3AfVw/5X95p1/s/B8ePD8s5I3pD1czyzKzAzAq2bz/BlU204r2IlBVjmdBqx0Zem3QPfdctoMXub/5zoCQ8T7UGE+6bgeZltpsF9pXbxswSgTrAzmPfyDk33jmX45zLyczMPLFKteK9iJQVa5ng81H34D6eeuUBOm5bc9T+cAgm3OcBbcyspZklA0OAace0mQZcH/j6cuAjF+rBnVrxXkTKirVMyMs7sf0nqdJwD/ShjwDeB5YBU51zS8zsATMbFGj2FNDAzFYDdwM/Gi550iI8XaaIRLlYy4Rx4+DWW/9zpe7z+bfHjQvL6TTlr4hIDNGUvyIi1ZjCXUQkDincRUTikMJdRCQOKdxFROKQZ6NlzGw7sL6K354B7AhhOeEWS/XGUq0QW/XGUq0QW/XGUq1wcvVmOecqfQrUs3A/GWZWEMxQoGgRS/XGUq0QW/XGUq0QW/XGUq0QmXrVLSMiEocU7iIicShWwz08c2SGTyzVG0u1QmzVG0u1QmzVG0u1QgTqjck+dxEROb5YvXIXEZHjiLlwr2yx7mhiZhPNbJuZfeV1LZUxs+Zm9rGZLTWzJWY20uuaKmJmNcxsrpl9Gaj1D17XFAwz85nZAjN7y+tajsfM1pnZYjNbaGZRP7ufmdU1s5fNbLmZLTOzXl7XVB4zaxf4mR75952Z3RW288VSt0xgse6VwPn4l/ubBwx1zi31tLAKmNnZwD7gOedcR6/rOR4zawI0cc59YWa1gPnAxdH4sw2sz5vmnNtnZknADGCkc262x6Udl5ndDeQAtZ1zF3ldT0XMbB2Q45yLiXHjZvYs8JlzbkJgzYlU59y3Xtd1PIEs2wz0cM5V9Xmf44q1K/dgFuuOGs65T4FdXtcRDOfcN865LwJf78U/d/+xa+VGBee3L7CZFPgX1VcpZtYMuBCY4HUt8cTM6gBn419TAufc4WgP9oB+wNfhCnaIvXAPZrFuOUlmlg10BeZ4W0nFAl0cC4FtwAfOuaitNeD/gF8CpV4XEgQH/MvM5ptZeJYJCp2WwHbg6UCX1wQzS/O6qCAMAV4M5wliLdwlzMwsHXgFuMs5953X9VTEOVfinOuCf03f7mYWtd1eZnYRsM05N9/rWoLUxznXDRgI3B7oXoxWiUA34DHnXFdgP+FYCS6EAl1Hg4CXwnmeWAv3YBbrlioK9F+/AuQ75171up5gBD6CfwwM8LqW4+gNDAr0ZU8GzjOz570tqWLOuc2B123Aa/i7Q6PVJmBTmU9uL+MP+2g2EPjCOVcYzpPEWrgHs1i3VEHgJuVTwDLn3MNe13M8ZpZpZnUDX9fEf4N9ubdVVcw592vnXDPnXDb+/2c/cs5d43FZ5TKztMANdQLdGz8Dona0l3NuK7DRzNoFdvUDom4QwDGGEuYuGfB/pIkZzrliMzuyWLcPmOicW+JxWRUysxeBc4EMM9sE/M4595S3VVWoN3AtsDjQlw3wG+fcOx7WVJEmwLOBEQcJ+Bdtj+rhhTGkEfCa/289icALzrn3vC2pUncA+YELvjXAjR7XU6HAH8zzgeFhP1csDYUUEZHgxFq3jIiIBEHhLiIShxTuIiJxSOEuIhKHFO4iInFI4S4iEocU7iIicUjhLiISh/4/vY+XN+8g3XoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "xy = normalization(xy)\n",
    "print(xy)\n",
    "x_train = xy[:, 0:-1]\n",
    "y_train = xy[:, [-1]]\n",
    "\n",
    "plt.plot(x_train, 'ro')\n",
    "plt.plot(y_train)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Tensorflow Session\n",
    "### 위 Data를 기준으로 Linear Regression 모델을 만들도록 하겠습니다\n",
    "* Tensorflow data API를 통해 학습시킬 값들을 담는다 (Batch Size는 한번에 학습시킬 Size로 정한다)\n",
    "* X(features),Y(labels)는 실재 학습에 쓰일 Data (연산을 위해 Type를 맞춰준다)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(len(x_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* W와 b은 학습을 통해 생성되는 모델에 쓰이는 Wegith와 Bias (초기값을 variable : 0이나 Random값으로 가능 tf.random_normal / tf.zeros)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "W = tf.Variable(tf.random_normal([4, 1]), dtype=tf.float32)\n",
    "b = tf.Variable(tf.random_normal([1]), dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Linear Regression의 Hyphthesis를 정의한다 (y=Wx+b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def linearReg_fn(features):\n",
    "    hypothesis = tf.matmul(features, W) + b\n",
    "    return hypothesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### L2 loss를 적용할 함수를 정의한다\n",
    "* Weight의 수가 많아지면 수만큼 더한다 tf.nn.l2_loss(W2) + tf.nn.l2_loss(W3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def l2_loss(loss, beta = 0.01):\n",
    "    W_reg = tf.nn.l2_loss(W) # output = sum(t ** 2) / 2\n",
    "    loss = tf.reduce_mean(loss + W_reg * beta)\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* 가설을 검증할 Cost 함수를 정의합니다(Mean Square Error를 사용)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loss_fn(hypothesis, features, labels, flag = False):\n",
    "    cost = tf.reduce_mean(tf.square(hypothesis - labels))\n",
    "    if(flag):\n",
    "        cost = l2_loss(cost)\n",
    "    return cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Rate 값을 조정하기 위한 Learning Decay 설정 \n",
    "#### 5개 파라미터 설정\n",
    "* starter_learning_rate : 최초 학습시 사용될 learning rate (0.1로 설정하여 0.96씩 감소하는지 확인)\n",
    "* global_step : 현재 학습 횟수\n",
    "* 1000 : 곱할 횟수 정의 (1000번에 마다 적용)\n",
    "* 0.96 : 기존 learning에 곱할 값\n",
    "* 적용유무 decayed_learning_rate = learning_rate * decay_rate ^ (global_step / decay_steps)\n",
    "* decayed_learning_rate = learning_rate * decay_rate ^ (global_step / decay_steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "is_decay = True\n",
    "starter_learning_rate = 0.1\n",
    "\n",
    "if(is_decay):\n",
    "    global_step = tf.Variable(0, trainable=False)\n",
    "    learning_rate = tf.train.exponential_decay(starter_learning_rate, global_step, 50, 0.96, staircase=True)\n",
    "    optimizer = tf.train.GradientDescentOptimizer(learning_rate)\n",
    "else:\n",
    "    optimizer = tf.train.GradientDescentOptimizer(learning_rate=starter_learning_rate)\n",
    "\n",
    "def grad(hypothesis, features, labels, l2_flag):\n",
    "    with tf.GradientTape() as tape:\n",
    "        loss_value = loss_fn(linearReg_fn(features),features,labels, l2_flag)\n",
    "    return tape.gradient(loss_value, [W,b]), loss_value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensorflow를 통한 실행을 위해 Session을 실행합니다. (L2 loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter: 0, Loss: 1.7346, Learning Rate: 0.10000000\n",
      "Iter: 10, Loss: 0.0745, Learning Rate: 0.10000000\n",
      "Iter: 20, Loss: 0.0438, Learning Rate: 0.10000000\n",
      "Iter: 30, Loss: 0.0273, Learning Rate: 0.10000000\n",
      "Iter: 40, Loss: 0.0181, Learning Rate: 0.10000000\n",
      "Iter: 50, Loss: 0.0128, Learning Rate: 0.09600000\n",
      "Iter: 60, Loss: 0.0099, Learning Rate: 0.09600000\n",
      "Iter: 70, Loss: 0.0080, Learning Rate: 0.09600000\n",
      "Iter: 80, Loss: 0.0068, Learning Rate: 0.09600000\n",
      "Iter: 90, Loss: 0.0060, Learning Rate: 0.09600000\n",
      "Iter: 100, Loss: 0.0054, Learning Rate: 0.09216000\n"
     ]
    }
   ],
   "source": [
    "EPOCHS = 101\n",
    "\n",
    "for step in range(EPOCHS):\n",
    "    for features, labels  in tfe.Iterator(dataset):\n",
    "        features = tf.cast(features, tf.float32)\n",
    "        labels = tf.cast(labels, tf.float32)\n",
    "        grads, loss_value = grad(linearReg_fn(features), features, labels, False)\n",
    "        optimizer.apply_gradients(grads_and_vars=zip(grads,[W,b]), global_step=global_step)\n",
    "        if step % 10 == 0:\n",
    "            print(\"Iter: {}, Loss: {:.4f}, Learning Rate: {:.8f}\".format(step, loss_value, optimizer._learning_rate()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}