deep-learning-with-python-notebooks/6.3-advanced-usage-of-recurrent-neural-networks.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'2.0.8'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced usage of recurrent neural networks\n",
    "# 循环神经网络的高级用法\n",
    "\n",
    "This notebook contains the code samples found in Chapter 6, Section 3 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff). Note that the original text features far more content, in particular further explanations and figures: in this notebook, you will only find source code and related comments.\n",
    "\n",
    "---\n",
    "\n",
    "In this section, we will review three advanced techniques for improving the performance and generalization power of recurrent neural \n",
    "networks. By the end of the section, you will know most of what there is to know about using recurrent networks with Keras. We will \n",
    "demonstrate all three concepts on a weather forecasting problem, where we have access to a timeseries of data points coming from sensors \n",
    "installed on the roof of a building, such as temperature, air pressure, and humidity, which we use to predict what the temperature will be \n",
    "24 hours after the last data point collected. This is a fairly challenging problem that exemplifies many common difficulties encountered \n",
    "when working with timeseries.\n",
    "\n",
    "We will cover the following techniques:\n",
    "\n",
    "* *Recurrent dropout*, a specific, built-in way to use dropout to fight overfitting in recurrent layers.\n",
    "* *Stacking recurrent layers*, to increase the representational power of the network (at the cost of higher computational loads).\n",
    "* *Bidirectional recurrent layers*, which presents the same information to a recurrent network in different ways, increasing accuracy and mitigating forgetting issues.\n",
    "\n",
    "本节将介绍提高循环神经网络的性能和泛化能力的三种高级技巧。学完本节，你将会掌握 用 Keras 实现循环网络的大部分内容。我们将在温度预测问题中介绍这三个概念。在这个问题中， 数据点时间序列来自建筑物屋顶安装的传感器，包括温度、气压、湿度等，你将要利用这些数 据来预测最后一个数据点24小时之后的温度。这是一个相当有挑战性的问题，其中包含许多处 理时间序列时经常遇到的困难。\n",
    "\n",
    "我们将会介绍以下三种技巧。\n",
    "* 循环 dropout（recurrent dropout）。这是一种特殊的内置方法，在循环层中使用 dropout来降低过拟合。\n",
    "* 堆叠循环层（stacking recurrent layers）。这会提高网络的表示能力（代价是更高的计算负荷）。\n",
    "* 双向循环层（bidirectional recurrent layer）。将相同的信息以不同的方式呈现给循环网络，可以提高精度并缓解遗忘问题。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A temperature forecasting problem\n",
    "\n",
    "Until now, the only sequence data we have covered has been text data, for instance the IMDB dataset and the Reuters dataset. But sequence \n",
    "data is found in many more problems than just language processing. In all of our examples in this section, we will be playing with a weather \n",
    "timeseries dataset recorded at the Weather Station at the Max-Planck-Institute for Biogeochemistry in Jena, Germany: http://www.bgc-jena.mpg.de/wetter/.\n",
    "\n",
    "In this dataset, fourteen different quantities (such air temperature, atmospheric pressure, humidity, wind direction, etc.) are recorded \n",
    "every ten minutes, over several years. The original data goes back to 2003, but we limit ourselves to data from 2009-2016. This dataset is \n",
    "perfect for learning to work with numerical timeseries. We will use it to build a model that takes as input some data from the recent past (a \n",
    "few days worth of data points) and predicts the air temperature 24 hours in the future.\n",
    "\n",
    "##  温度预测问题\n",
    "\n",
    "到目前为止，我们遇到的唯一一种序列数据就是文本数据，比如 IMDB 数据集和路透社 数据集。但除了语言处理，其他许多问题中也都用到了序列数据。在本节的所有例子中，我 们将使用一个天气时间序列数据集，它由德国耶拿的马克思 • 普朗克生物地球化学研究所的气象站记录。\n",
    "\n",
    "在这个数据集中，每 10 分钟记录 14 个不同的量（比如气温、气压、湿度、风向等），其中包含多年的记录。原始数据可追溯到 2003 年，但本例仅使用 2009—2016 年的数据。这个数据集非常适合用来学习处理数值型时间序列。我们将会用这个数据集来构建模型，输入最近的一 些数据（几天的数据点），可以预测 24 小时之后的气温。\n",
    "\n",
    "备注：我们下载好了，文件路径： 'data/jena_climate_2009_2016.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the data:\n",
    "\n",
    "来观察一下数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\"Date Time\"', '\"p (mbar)\"', '\"T (degC)\"', '\"Tpot (K)\"', '\"Tdew (degC)\"', '\"rh (%)\"', '\"VPmax (mbar)\"', '\"VPact (mbar)\"', '\"VPdef (mbar)\"', '\"sh (g/kg)\"', '\"H2OC (mmol/mol)\"', '\"rho (g/m**3)\"', '\"wv (m/s)\"', '\"max. wv (m/s)\"', '\"wd (deg)\"']\n",
      "420551\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "\n",
    "data_dir = 'data/'\n",
    "fname = os.path.join(data_dir, 'jena_climate_2009_2016.csv')\n",
    "\n",
    "f = open(fname)\n",
    "data = f.read()\n",
    "f.close()\n",
    "\n",
    "lines = data.split('\\n')\n",
    "header = lines[0].split(',')\n",
    "lines = lines[1:]\n",
    "\n",
    "print(header)\n",
    "print(len(lines))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's convert all of these 420,551 lines of data into a Numpy array:\n",
    "\n",
    "接下来，将 420 551 行数据转换成一个 Numpy 数组。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "float_data = np.zeros((len(lines), len(header) - 1))\n",
    "for i, line in enumerate(lines):\n",
    "    values = [float(x) for x in line.split(',')[1:]]\n",
    "    float_data[i, :] = values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, here is the plot of temperature (in degrees Celsius) over time:\n",
    "\n",
    "比如，温度随时间的变化如下图所示（单位：摄氏度）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fname.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD8CAYAAAB0IB+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXd4FVX6x79vGjUhlFADBhAIRWqkiaIogoJl/eEuuuu6irru2nbVVSzr2kW36NrW1cW2igtWkCZdRZHeSyD0QCChBQIkIeT8/rhzk8nN3DZzZubcue/nefLk3nNn5rz33Jl3zrznLSSEAMMwDON9EtwWgGEYhnEGVvgMwzBxAit8hmGYOIEVPsMwTJzACp9hGCZOYIXPMAwTJ7DCZxiGiROkKXwiSiSi1UQ0XXvfnoiWEtE2IppMRCmy+mIYhmGiR+YM/z4Am3XvXwTwshCiE4CjAMZJ7IthGIaJEpIRaUtEmQA+APAcgPsBXAWgCEBLIUQFEQ0C8KQQYkSo4zRr1kxkZWVZlodhGCaeWLly5SEhREa47ZIk9fcKgIcApGrvmwI4JoSo0N7nA2hjtCMR3QHgDgBo164dVqxYIUkkhmGY+ICIdkeynWWTDhGNBlAohFipbzbY1PBRQgjxthAiRwiRk5ER9gbFMAzDmETGDP8CAFcT0ZUA6gJIg2/Gn05ESdosPxPAfgl9MQzDMCaxPMMXQjwihMgUQmQBGAtggRDilwAWAhijbXYzgKlW+2IYhmHMY6cf/sMA7ieiPPhs+hNt7IthGIYJg6xFWwCAEGIRgEXa6x0A+ss8PsMwDGMejrRlGIaJE1jhMwzDxAms8BmGUZo9h0/h+21FbovhCaTa8BmGYWRz0V8XAgB2TRjlsiSxD8/wGYaJCUrPnHVbhJiHFT7DMDHFu4t3Yt+x026LEZOwwmcYJmY4eLwUT0/fhFveW+a2KDEJK3wmrvj9xyvx4ZJdbovBmORspS8l14nSijBbMkawwmfiipnrD+CJqRvdFgMAcKL0DN5clIfKSuspyuMFHilrsMJnGJd4fuYWvDQ7F3M2HXBbFCZOYIUfQ5SUVeDuSatwuKTMbVEYCZwq95klSs9UuiwJEy+wwo8hJi/fi+nrCvDagjy3RWEkkEC+shGVEqrOxRtGBTeY8LDCjyH8tt6kBD7dY5HNBcfR8dGZVS6Fmr7H/VPWuihV7JD959lYtvOw22IAAL5eux/D/r7IcP3l9g9XYNLSPS5IFR5W+DFEhXZy7Tx0EjJqETPOMmnpHpytFJi/+SAAgHieGjVzNx10WwQAwAOfrsWOopMoP1vbHDd300E8+uV6F6QKDyv8GKJCO7nmbynEO9/vcFkaxiqkgL7fffgk+jw9B3uPnKpqyz96CsWnz7goVXD085yPftqNJ6ZucEWO8orYXHdhhR9DzFhfUPV6bX6xi5IwMlDBMjd5+V4cPXUGU9fsq2ob8uJCjHzlOxelCo7/JnnkVDke/2oDPlwSUe1uRoMVfoR8u7WoKujDLbYcOOFq/9GwYV8xxvzrR85/EoIEBab4/jOaAmQpKC51XpgIOHrS9+TBnk3mYIUfAYtyC3Hzu8vw5kJ1vGPcVxWheWLqBqzYfRQb97v/JDJx8U5c9dpit8Wowm+WCFSybrBhn+/32VF00mVJImPJDjUWbf3E2lIaK/wIKDzh83vfrbNzus3pcrVnzqv2HAMAJWzBz0zfhPX7ilFSplY4vgL6Ht1apQEAOmQ0cFkSxglY4cco87cUui1CRKhkhlq5+6jbIgCoVvQK6HslnjIY52CFHwUqPb5d1au12yJEhFuuh0IIbC8qCZBFDfznkQo2fAVEMORQSRm2HlRnshAMEZDd54yBm6ZKsMJnPMlnK/Nx6d+/rVEaz+37daByVclEqBqX/G0RLn9ZTU8hPYGTQFb4HsLN2ZDqJ1Iw3Boz/2Lk9sKSMFu6x3dbq29GbuVHOqStT/204zBmri/A3ZNWuSJHIF5If5w1fgYWKmZ6tazwiaguES0jorVEtJGIntLa2xPRUiLaRkSTiSjFurjxy9FT5W6LYAo39P2p8gqsUMRer6fwuE+5btp/vNZn8za7E0G6ao9vnL7fdgi//3gVpq8rCLMHoyfcU+Mt7y93RI5IkTHDLwMwTAjRC0BvACOJaCCAFwG8LIToBOAogHES+nIVN234lbE5wbd9hv/Jsj34ZFnNvCUPTFmLjZpS1f9kbqejWJjrm+1NXrG31mduLZ4eKonNiYQbPDltI2ZvqJnK+vmZmzHmXz9WpctQHcsKX/jwPzcna38CwDAAn2ntHwC41mpfbvHTdvd9f+dvqXlCKbrW5jiPfLEej3xRM2/JRoMZNKC2r3miSwpfBbfZWOH9H3fhzo9W1mibtHQPVuw+inEfrHBJquiQYsMnokQiWgOgEMBcANsBHBNC+A1x+QDayOjLDb5Y7Qs7/3xVPl6avcUVGSrO1pydur0AGSlueOnolZi+d/06yOGSMhQUO1sIuyJEpHYCr6Z5glOKx8dIOc2EEGeFEL0BZALoD6Cr0WZG+xLRHUS0gohWFBUVGW2iFG8u2u5ofyt2HcHsDQW1TCNuLgZVnK2MOPDLqYmrPpmVXuHrTzr9xdjv2XkY9MICJ0SrIlRqDhVcNAO5VTH7cyywdu8xt0UIidR5hRDiGIBFAAYCSCeiJO2jTAD7g+zzthAiRwiRk5GRIVMcTzDmrSW486PanhP+qNHp6/Zj2c4jjsp016RV6PrEbEf7DEfnx2eh6ERtT5eDx2OjOliilknt1fnbsHK3s79nMBYo5mESC6h449Yjw0sng4jStdf1AFwGYDOAhQDGaJvdDGCq1b7imWCFt++etBo///cSR2X5ZmPoBao8nSukk4uR/sIiet761tknMrP4bfj/mLsV//cvZ39PlYm55Htq63skhd8kLK0AfEBEifDdQKYIIaYT0SYA/yOiZwGsBjBRQl9MDPCibp1DpfP/n/O3uS1CUDjFgTH9npnrtgjRofjimgwvnXVCiD5CiJ5CiB5CiKe19h1CiP5CiHOFENcLIWLj2ZqJisnL9yBr/IyqgtxAzapET0/fhPUO5e5/ZvomCCHQrGEdR/qTyXH2ljHkpOKLoIEEplpQDfYNYEwzefmeqkXs+ZuD23uvet2Z1MQrdx9F4Yky1EuJvdO6bkqi2yIYsrnA2MVVJVTKHKt62crYuzIcZGFuIbLGz3BbDGV5+PNq//d7PlkNAFiwxd0AlJdm5yqV5C5S7v1kNZYqlusdAK745/eO9nf0ZHnUAXInStV4OsorLFHLhmkAK/wA9h07jazxM7B6z1Hc8p7abmm5CqQe1nslCCFw6/vuBqB8virf1f4j5biBkjKKwLWLA8Wlyk1mCopPo88zc6N2fVbl/n7bB8vx4Y+73BYjJHGv8CsrBV5fsK3qAlysZVcMDNd3g3AznTFv/eiQJMHZeag6etVf9MRtYmH9c+mO2q6Xc8J4P8nk73NyHesrUvxlFfVrQJGgyhPdrsOnsDBX7ViiuFf487cU4m9ztuKpaZtqtKvgvx2uhG5g9K37qCaPuhgVMHeyIlevtumO9WU3lapo/Bgg7hW+P0Lz9BnfxeY/d77d6v6dOtyJfFo5H+UYmForgtsBOql1ZXhkqwGr+8iJe4UfyH9/2u22CFWECsVXkVgwpajCCZfr65ZVxGj6VQMqY+w6cZO4V/j7jtWsOqRSDdZYe1I9HCLVbuHxUhzQbLR2o56pqzb3al5NbjHx+52u9m+Ef76g+q/XTbG0ItEQ1wq/8EQpnp9ZM/tlJLPqdfnOLE7Gmm3y9g+De+j0f34+Br4w3xE5CkzcWKat3Y/nZ26WLstl//gW73y3Q/pxrZKrYL3YqmjjKM97py8T1TNihiKuFf6xU+b8d6c45D53NsYUfixz7yer8bYkxXyopKyqZGFeYQmes+FG4jTj3l+OiYvVeyoA1I9uVYm4Vvh6orlrFxxzxjQhvGNmjStynp2Hfs/Oc1sMqczfUohnpm8Kv6EL8LwocuJa4evXGFfsirwG6nyH0saqZtK5f8oaDHjeW4qMiY7ZGwrwp0/X2nJsfz6maM96ta4StYlrha8iWeNn4MlpvlTIqin8L1btsxyfsHqPesXFmci586NV+HSlPdHMN76zFED0M3bVrhOVYYWvoZJL4fuKh2dbYeVutRT+D3mHpB7v4HFnzH1e5mSULqus7yOHFb7GidIKfBdFsFW5h/yYnUQ1/+9f/mep1OPpzwsZxVeKTpThpolLcfRkcJdXr7HjUHTF5m+auFTpYuz9n5vnaBR1KOJa4QfO6n/97rKI9/39x7XLDrrJoRL5qSAmLd2DXk/NiTp7IeNjwqxql1+zQXT/WbwD3287hE+Wu5/byS46Pz4LT31tXNEtEgqKS/HfJbukyRPI+vxiS4VYCk+UYcM+Z2pChCPuFP6xU+Uoq/B75Ji348zb7G4a4EA+X5kvvZzfo1+uV3rmpCLB7o1mffz9KRhUu+fKTINdXlGJ937YZekYf5uzVY4wBlz1+mIctviEpYpFIO4Ufu+n5+LW99VOewz40udGcxK/MGtLjRmlqqj4tLCjqCT8RhESzCd8xroCU8dLqIpFsjZuR0zkmQ+F22mwY406SWqoWjWkcJgf8tQrNBFIzyfnmErRbEeOcwV1tFRkFoEPNlYHTC7m+isoWUkXs6OoBH2fmYsPTDoD/Ch5YZtxj7hS+Mt2Vucg19dgNUvW+BlYsat2XnOvcaKsAit3y/meKhbrPlkmL1Re9r0xwVy2gRrsOuxbBDWbAfbGIAvb329zP6NsrKDKnCmuFL5+JvfQZ+ukrJx/ssxamoXyikrlXfl6PTUH//cvObNgK2aFk2UVePizddLXFWSG5gf7fsmJJm905J/hq6Iyqrlp4jKUKpeimwlFXCl8PdPXFeD+KWssH2fjfmur7w99thYDnp+PsoqzWOhQBG+s8uGS3Zi8Yi/+FWUJvHCUnrF/Qe2MyQyeMjJI+tOG2OEaqOKNiAlO3Cp8ANhRFJ2/rxFdW6VZ2n+OVs6t4qzA6wvzLMsDoCpxl4pYMenEgnKRb9KxbtPxe8As33UUX6/dj+UeNUN2f2K2EoWLVCauFb4MZFYualQvWcpxzKQHdgoZniLVbrXysVpM49AJuTdbvw3filj6GIB7PlmN69+St0h99es/YO1eNWoZnyw/i3/MleOeeeZspdTCKqoUabGs8ImoLREtJKLNRLSRiO7T2psQ0Vwi2qb9b2xdXCYSVJ4IW5HNH7xi1Wc7FFYL4MiuUuafT2y1kL9+0/7jkqSpTV5hCZ6doWYWTSt0emwWxn0gz337jFcUPoAKAA8IIboCGAjgLiLqBmA8gPlCiE4A5mvvmQD89lWB+KgIa+WB6Hip+kFgsi9rvwnMb/qLlsnL96D8rBpBP0ZIr4ImcbazMFeeeUiVa9uywhdCFAghVmmvTwDYDKANgGsAfKBt9gGAa632xcQ+Kj99ANY9dmR/P6sWw4c/Xy9HEJuQHgSpoNsv4FG3TCLKAtAHwFIALYQQBYDvpgCgucy+mOCoXAHIimRO3Cys9iF77GWuEdkFWZi/Hj1lT1I4IYQy6QwAdSLMpSl8ImoI4HMAfxBCRGw0JKI7iGgFEa0oKorfFXaZJ4Qi55Z0vPq9QqG+ulcL/wLyS9/kovPjs5SJE5gwawuG/X2R22LIUfhElAyfsv9YCPGF1nyQiFppn7cCYOhkLoR4WwiRI4TIycjIkCGOo8ia0Z2WeGJe88YPOKOw3dYsKj+5+JF9U7Iyw//MpkIlsYA/LclpEwXH7ZiNbzlwQoobuFVkeOkQgIkANgsh/qH7aBqAm7XXNwOYarUvFVmfLyft6QNT5JaNU2VmIxP9dfjnrza4J0gIZKsKKy62D9pUilAmdj/BmPk9ihSOY7GKjBn+BQBuAjCMiNZof1cCmABgOBFtAzBce+85thXKybT4/bZD2C/RY0HVubCVzJT67/Tfn3ajoPi0dYEkI3t2+O4PO6UezxYsaG2Z57yes1pks6nfQ9WLRwJJVg8ghFiM4D/5pVaPH09sLpDnL62qvfurNfvxytg+pvbVJ78DgEEvLJAhUg2sL9rK4WylQGJCjFjwFTzXTlhII9GwrmW1qCwcaetR9AFA5RWVyngJqI7ldQJJw9zx0ZmYtd5cDn2nWaZYqgb9uZ5rMZDOzqhuN2CF71H6aiXZjpwsR+fHZ+Gd73e4LFE1v3lvGZ6cZr6kXbwwc8MBt0WwjYqzldLXrfzoa+L+bU6upWNNWuqt0pKs8C2i+lO3P5Lxi1X7XJakmkW5RXjfZDEO1ZHpSfT12v3SjqUa6/cV4/NV9ngR6Z9uV+2xlufHa95urPAtonpgzJerfRcVW3ScodJb+iEu8fK1wgrfIior/JW7j+Cd731eHrkWkm/FE16+2JnoeX6m+nWio4EVvkWs6Pv9x+x1K9SX7mvfrIGtfZlhYa56BV+s6nu+X4Tnx7xDVWUX7SAW6ia4hXf9jxzCygx/vs0VrvSi1UlS797+xoI8XNLFWymWZKdH9iLBauTKYsY6a95Ne46ckiSJeqinBWIM/aLt9HX78ez0yHODq1IUwS06tWjotgi1MOO+WqQrevLnqfZFAOc8O8+2Y9vBotxCrMt3vjiK1SjzXYfse/r4y9QNptI9yIIVvkUSdBr/7kmr8Z/FkUdGyq7NGshNE5eZ2m9hbmHVYi8TmsXbDuH85+ZhzkafC2WR5IpXeg7FWMj/b95bjqtf/8FtMaK+ids5DftgyW58uGSXjT2EhhW+Raws2R447lwpwiMnI09De8t7y/HHyfbnYfGCqXWtNoNdrUiZP6Y2S7Yfjmp7u8/Lsy6e+KzwLaJyNSE9hTbOPM0ydY16fuZmL0Uv3Ly8gt8zzU+016iXF31Z4Vuk9ExsKHwVkZkSWhZG1/r324qCRlz6F8ZjIXVzvOIvEymEiMi+b7fCd/N+wgrfJaIxsTDuctPEZXj0S+NSgVaqPdmFEMLx1BV5herGefiX2SYv34vsP8/GbhtdQlWHFb4LbNhXXJXrhomMJ2z0fjFizd5j2LAviloHCk3w520udDx1xWX/+A75R9V0Z/Tb8Md/4btp7wjjheNhiw4rfKu0alQ36n22ctRr1Hy4ZLdDPfmu9mvf+AGjX1scdutqk446/G+ZOwm/hry40JV+w/FmgDdcYpjYGbvNc2VnzrpWy4EVvkUOHC9FZaWo4fr1fiwUrWBM8/zMzZimSGKz46VncNekVTimFQM/UXoG+UfVKwzjNv+Yu7Xqdbg6A3bnQ3p1QR4GvbAAJ0rP2NuRAazwLdK6UT10eHQmnpm+uapt8gr2Yfcyb3+3A/d+stptMXCguBTPTd+MGesK8PLcrag4W4nznpzjet6kEgvFR+zi1fnbql6HU/hO2finuKAnWOFbZJ+WD0dfii7cMp6XbYTRcqq8AndPWoVCB2MSQhHtb+P/rZ0uMCOEwMAX5mPyir0AfAE9KrgIb9xfjB5/+UaZJyAj/Ar/wyW7kP3nWSgpq8A3WuDc+vxivLogzxE53ChKxArfBhROoFmD/cdOu77QNm3NfkxfV2C5UIUsor0E3cqO8fDn62q1qZCpY+N+X5nO77YWuSxJcPz5r56YuhGlZyrx0Gdr8dv/rkReYQmuej38uo0s9h07jazxM7A+PwrnAIt4XuEXnz7j+J00VhT+4AkLai207Tns7A3A/8uo6N4YCa8v8JkKVu4+amr//90x0NR+RuYAlRK3qfwUG2jS8SdLO1XurClqgZY8cZKDi+yeVvj7j51Gr6fmOF7eLxaV1+7DJ3HweKnjmQL9isF/k3xl3tbo3CFtkidSTmqJsA4eNxfJPLBDU1P7GSFzYpNqspB3LJz5R0+W14gbcOt6deOm6GmF7/dWmLvpoKP9xsoMX8/Qvy7CgOfnm77QzeKPtq1W+NtquUP+mHfIUZnMoEKReJkT/L+O6SnvYIrx9PRNKCiuXjM66fDM3o8baZg9rfDdYp2DNjmz6BXUU1+7V1D8map00sHvkg8Z2KvtYvG2IlvT49qJTJPOsOwWlvZXOdXEzkMna4zVjiLrv/cdF3Uwve8ny/ag2xOzLcsQCazw45T3fthl+NrNp5Ngs2QnJ8+vLsjDxX9b5FyHURLqSULmU0ZKUgLaNakf9X6HSnzxAN/mqrtoSwRUnK09VlaGLyXRmio95VCOfCkKn4jeJaJCItqga2tCRHOJaJv2v7GMvsxw5GR5Vb5yFVBh7vP5KrViBYjUXugDQqfZdUr0LQeC+9jLXrOtn5IY9T5+vXdY4VxRSQmECoPoqvmbnTX9uoGsGf77AEYGtI0HMF8I0QnAfO29K2wvOok7/rvSre5r8cZCZ/x89YzoXvMR3e8+F0huCIViJwkUXGmqkq72hnd+CvqZUyKeCeFrL2uc/jSiCwDgPzfnRL1v89TQqUbcrPbkh4jw7dba60LFp52PfHUaKSt0QojviCgroPkaABdrrz8AsAjAwzL6c4rUOkk4YUPU4E6FbcQ/7TjiSr8ECmqSCKXkVMEpm/Xx08HPR1kK/65LzgUAZDaO3qQTzCRYUlaBce8vx9Kd7pxfegjA6j213WjJgj1T5TULPXba8FsIIQoAQPsfM9Wqr+vTBrsmjEI9E4+0sY6MEzcjtU7U+5SUVQTt2W8XVhmnZvg/7QhhVlJY58zbdFAJZe+nvEL9SYQduL5oS0R3ENEKIlpRVKTGQs/Pz28LoDoizwx2enp0zGgQ9T4R+xpLUBpf/G5w1PtsLypRWmGFw4zoP44fFvU+r4cwB9odeJXdMjXsNnbW9NXTt1266X3LKirRzGBSkpwYg/7UUWKnwj9IRK0AQPtfaLSREOJtIUSOECInIyNDSscVZytRUlZh2uMkWVt5Oi+zkWkZ7PT08D9y24EMs4CZcSciWx6L7720k/RjyqJ1er0a75++prul47lZK9XPszM2h99IAh+OG4CrerU2vX+jesm12gJLI3oROxX+NAA3a69vBjDVxr5q8MCna9HjL9+Y3r9Fmu/uf1FnOTcg2QzLjt46FqkSjnaSOLpnK4O+otf4a/cew5sLt4ffMEruH95Z+jFlMLhj7Qjb3m3T8duh5v25K22e4fc9x7yj3QHJyfEa1klC+2bRP+n6seLC+ouctqb3dRtZbpmfAFgCoAsR5RPROAATAAwnom0AhmvvHcFfHNvsb9oyzedpMPb8tvjjZZ2x+s/DZYlmmd5t05FePwW7JozC13cPkX78o6eis5en1vXNlB4a2aWqzeyD8T91KWxjjWjPtWBPUpH4vgczm/xq4tLohIiSJvVTTO23ZPthvPOdvPQmfTRzjhUDTL0U8/4qD1+RjXn3D4U+JU9jk2PjNLK8dG4I8tGlMo7vFsmJCbjvMnVMAs1T6+Cruy6oel8nOfL7daQFw7/fZi6NgX69w8raR6xSFmVBdr29/bw2jbBeyx90Y/92eOzL0OUc3w1SYMdsPp9IMWsyWhJioTladk0YVfW6V1vzJtevLaRvbtIgBU0aVCv4CdedhzH9Mh0zZ1nB9UVbOzly0twFYMU9S8+GfcVSfXsDxercIhX3RWijXpRbhLIKO3ygayuBFml1cMsFWZaO+qmW5z1WiNZ9t2kDY0+mSM69fy2Sb/qKBLPrOwTrvgC/v7gj3vl1zbiAtibcRq0yoH2Tqtf+32pMv0wkWYy0dYrYkNIkZivKGFXEGdm9ZdTHGf3aYvzyP8GDdaIlvV7tx8abBp0T8f52uKJVZbvUtRERHh6Zbem4f/rMlz/H7epJh0pqThqspi+4sFMzAEDThrV/S7fXXJ//2XnY+cKVQT/PaBi9u60sHhqZjeHdagYPZlmw4Zslq6nzfcrE0wq/pDR6ZWG0eg8AjRsYt4djwz7jiFYzGE3+olESdphaLtEWkHu3Tcd/x/XH36/vJfX4x6JcU/CTY7DA+NoNffDqDX3w0bgBER9nxMvf1Xhv1fVxZA/fxEF/GFUsYGP6ZYZ8wri2TxsHpalm1n0XGrYnuzCrfvTKrrXajM6I3wzOsl0WMzibC9dhjpsoEhzcF9f9q9LoySMaV0bZisVvT1335OVIq1vzhiirL6MkV5Hw33EDcOy072bx1V0XIK1uEjpkNAQAFJ+K/LwIzAmjT6trBv9NV/+k8MoveuOtb7eje+s0S8e2wsIHL0ZKkj0K1Oq54PaTj56GuvThob7Wk1d3x/s/7rJdnmjx1Ay/pKwC7y6uXtAKlWgqWoLN/J3Abwa4/UIDlz2XZ/gAail7AEiU1FeFyRl1vZREtGrk83Pv3Ta9StkD1qKJ1+Yfq3ptpjxk5xa+4KW+7aqfQDpkNMRLY3q5age24uIYjoPHS1ERA+kxIkF/Vl/a1fd0G+y6qhuFU4VTeGqG//TXGyVUgjf+8do28SmP6/q2wRer9lnsIzrqJPlSPBhlL0xz8UYUChnKa/G2Q2iWqpa7m96DZtvBkqj373dOY/wwfhhaNwqdZEw1Xv6FeVPd7sOncNyEedWP/9pTjX+O7YNDJWW1nryn/HYQAKBnZjqWKZROAvDYDP9YFI/qwQg2MfWnJqhj02NvaILPSOsmR57vRxVbcaRsOXAcJywoCjvQe11F67Xid6ltk15PmieY3fh12dW92pg2rVhxFtg1YVRVrIdq1E1ONEww11/z5Bmo8+iJhILi01LkCoWnFL4MrutrvDDlvz6t2hOj9fKYfs8QXd1Xa0oi1mrtHjlZjkW5hhk5lCDac6F3W/P5X9zirV/1Q/+sJlr6anMnv50pHxY+eDEWPnixbccPJMFgHU0W09cW2HZsP54y6ciYNA3tZJxOwe8RkGQxwdLExcZBM8Ho0aZR1WUWW+raOm8u2o5re5vPlxIMv4nMKgqtJVoiVKqOy7u3xOUmXJL1mF14jwQ71x6c5qDk9BNGeGqGb3aRVp+iYPC5zQy3uaZ3a/x2aAc8ZMG/fMa6AkvReFZvaDFiRajBGRvyw9RLScTFXaznSQo06RyRVOVp7ROXSzlOpJzXxnzEaiSYXXiPeaK84AJjPuzAUwp/92FzVeD9PvZt0oMvDiUnJuCRK7oaeqSE41S5zw5916RVpuTrp/mUt2mOuXYqAAAZYklEQVSszuLVoA61k3/ZgV3eHS3CVGaKhEBLRd9n5lo+JgA0qu+czXrGvUMizyhqUm+fNSgn6EW++cNFeEaX8bRzi4Yhtq7NV2vMp3uIFM8o/NIoc5n4eePGvpIlqY3VxeTfDe2IhQ9ejOyW7vlpm+H5n52HT+8cZOkY32xUuc5o7M9cu7duZBjfYYT+26bWjdwabDY6tVur2Drfu7RMxU2Dsqrej+7ZOmjQmFt4RuGbfWxs16R+1Itvfx3T01Rf0eIPxElIIGm2yk1BatnawY0D2uH8rCboquCFe//l1tMmx7qlwmytgOapdaIyba7YXbucYCSc2zy6GbKKqHbue0bhm8WM58H1OW3x+KjaIdaqM23Nflz56veYvcG6N8DPgngzGaHi0kGLNOsmHbNPlapgpVbAVQZ1EIJhdm0jxu+nNWhuouynHcS9wgd8xRQAYGAUdunbjKJegyBgTjmYWWR99toeQT/bcsA3u99eZK384q4Jo/DzKIpAGBX78AL3T1kb8bbLHovpTOEAgHRtbeFPI7og3ab87+c0rfZrl1WUXRY39DdX+CT32ZFY/HD05SztwFNumWYQAmjcIAXzHxiKTJsWRSfM2mIq/3akfvObnh6BlbuP4kLNpfTxr4zzqftNEN9sPBC1LFYYf0U2fj0oCxf9daGj/drBpdnNMX9LdLEBv8hpi+YSFondpk5SYo189HagT8nRuXn4GrpO8sJ15ky5fjfg5ETCGRtdVCMh7hW+n44Z9tkLrRRbiIT6KUlVyj4UuZrb6rr8YlvlCSQpMQHtmtqXu7xNej3sO+aLUnzsyq4otzFvi5kMjXbU6vUsujnO3cPsq93sBtkt06oK3bhF3Jt04ulS3HnImilHVfSLe7df1CHiIu+LH74Ec/94UVR9pUfoMvn13UPwq4Htojo2U03OOY0j9h6KFVSIg/GMwldgLKVjpYSbEZGWOYw1zN60MxvXR6cW0ZkNIrUrn5fZCLde0B6ALw8NExnDu/qKnIyOYlHYCTpI8JJTIX+SZ0w6J8vVSrJllfuHd8adQztKPaasSFDVcHIiGI0rZoeMhrbbvL3G+Cuy8cfhnV1KUhic+nWsp+PYXOCcS3Qw1BpVC6yPwi694akR6JChdg6O7q3TbCtI4TVuGugr82h3igAA+Gyl1fTbTCiICHWTE5WYDeuR4TA06jz3n1riUqM0rJOEVM0V02qNUrtQVCwlqaeliPa71zKMbGQUD7rjoshdue3CMwqfFSTDxC9/vMx65HQourRUy0XULJ5R+F4jy0NpX+0mWwtfv3VIe5clYdwiOcleE9CV51lLEQ3EiZcOEY0kolwiyiOi8Xb3FzEqjH4QVj5+mSfyiDhFkwYp2DVhFIZ3a2Fq/5n3Xoihna2nS7aLVjFWDtEN7H7Cj7XiQcGwVeETUSKANwBcAaAbgBuIqJsdfUUbhp2ksI9vY5vC1mXQwKCubqzTrXUaPri1v9tiGHJN79YoKLa/MIYZVFozqYyBTHYqFGuxe4bfH0CeEGKHEKIcwP8AXGNznxHxz7G9MW5Ie/TKlFd2rqPinj9myXvuCrdFiFtSEhOUzUWk0pTJdhOohC8rq9KaFexW+G0A7NW9z9faqiCiO4hoBRGtKCoqMt3RoZLofMwzG9fHn0d3s1SjMjAh1vR75OS+Vs3alJSYUJXtr7sDro9MdW3lrq3SpJ8Pdw7tiE9uHyjteC//ope0Y5lldM9WuEZyOcxG9ZJxkWbqU+ySNI3dCt9onGo8ewkh3hZC5AghcjIyzNtR3/0hulqxMghMiFVPkrlDNR9kABikzTLNZgwEgPOzGssSx/PcM6wT2qTXsyXidPwV2VW/pxX8p+mwbHNrJ4E8eZV5ay8RVVWGk4XZNSGVsVvh5wPQa4hMALZkEssrLLHjsDFJn3bVZqrslqnKBJlFklK3aQN11y+cpH2zBvhh/DA0T6ur7ILhlDsH4c6hHZEWRfWrUPzmAjleVjf0l5fDyB+no+IkzAx2K/zlADoRUXsiSgEwFsA0m/uMe778/QVVr1+47jykmMjwGIiM0z0S6xlHF9dGVV2T3TIN46/IVk4ZygqmpCCvYxlbry4hRAWAuwF8A2AzgClCiI129snUJNVE0fVQWLmWIolWtDuAxiyxVl81kOREr6is4Pi/YbAbUJv06OpdtGtiX0pvt7B9OiWEmCmE6CyE6CiEeM7u/phAhJSwcBmzuEjkGNChieV+7ODz3w12re/+WWqOiZ7bFAp6k+Fx/d5vzsfvLpabvFAF+PlZEr8ZnOW2CEFJkPAr+5W1Fb0fyb5JEsxPdiBrQd4Md11yLt6/5XxT+75+Yx8A9gcOPT7aWniNallFL8lujqTEBLw0piduGniOsq6x0aLm1RUlq/ccdVsEPHl1d7dFqMGrN/gu9Nbp9aTY8B8b1RU3DTwHo84z7/rWWZd73mhxtlPzhlE/dscDCQmEi7s0N7XvZV2952niJK0a1cMz1/ZQdiISLZ74FgeiiERs28RehXJ9v0zT+3aSmE7h6l6tsWvCKNRPScJrN/Y1fRy/7bdJgxQ8c20PS4uq/jTGAFA3ufaM2cws6sHL1bT528HGp0ZEtN1r2s0eqPlU9e5vcvDWr8yfC3bx0bgBcg4kafH4PZNPU7GAOrHRFgj3O/9qYDt89NMeAMDs+6IraRctL/5fTyzOOxRVOPz9wzujYZ0kjO3fFifL5FelsjJrlll8u7FuVt+rbaOqOrR+IplFdW+dhtdu6INhf/8WAHD3sE7S5FOdBgapDJY9ein6Pz+/RttVvVqjQ0YDlFVUIlmz590z7Fxp/vIy+eDW/hjSqZmUY/kXWf1Pkk0bpOCwiaI/l5h8moqEhnWSUFLmXrEmjyj80Bp/cMdmVQrf6KKRSUIC4c6hHfGXaZE7I917abXSqp+i1k+y4MGhUo+XkpSA8opKwwXcDC2aNxR92zVGBxsLzscazdOMb8jdW1dHRKtmHweAxATC2UohtR7F0M4Z+OquC9ArsxEGdWyKpg1S0O/ZeVWfv/yL3vj5v5dI688MbvtKecKkE877Q4aXSjR8s/GAo/3Ziez8H/5fwqhA9UWdwkdaX97d/lnqqzf0iQnb97JHLw2/kaJccK5vVi875VnvtukgInRukYqmDasnEP8c2xv927vv7TS6l7tVrzyh8EOp85ZpdQ2Vi538etA54TeKcxIDbsK7JoxCt9ahfd13TRiFCyO4KVjl6l6tMbKH9fzndhNsdu8Wj13ZNeJtu7TwPaXZHVl9+4U+d9HBHSMzG3125yA7xcEz1/Sw9fjhUMt+YJKKysqgnwmIiEwFZvn4tgFoEXDhJcnwg1QAfYoGWXz+u8H4eu3+qO2Ys/9QMzGdf93DLuIhUEk2kRb6blgnCQ+NzMalXVugp8RstUY8NqobHhsVuctojs0xD257+3hC4X+8dE/Qz85WAr0yG+GWC7Jwy2D5wSH+R1M9qZJyi7jJRZ0zqny4ZdKjTSP0aNMIczcdDPm7BZLdsubsX7/uoRot0uybYMQidZMTUHqmelI2674LkZyYgIEdnPdtb1QvGcWnzzjeryp4Yipaeia4Z4sQAkSEv1zVHe2aOhMqPcCFE1k2dw7tgDTJaRn0DO/WAn+xkB1RZRY+eLHtfXTXmb+GGEw63MAouOvj2wZg3v3VC/8NUhLR1sWUBWcDCqX8rE+bIFt6E08o/FBRhOrXwVEUBwbuliDZEVW4CK2kknDC02r6PUOqXn90myQ/dosYDdkF5zZDZmN1ctIErq9dn2M+biYWiX3bA0L74Udb+jDeyG6Zii0HTtRqd2rUAh/3AWMPHqdpk67WgmggwW5IvdraaxMPxdW9WuPb3CLMDuGl9uCILg5KVJvRPVvjzUXbq94P7tgMuyaMQnlFJUQcTA+9McMPoR8yG3Oofii6tEw1bHfqPjnz3gvx1zE9a7S5r+6Bfue478JnBjfHrkGdJLx1U7+q90bXntvmp26t07DzhStrtackJbheglBWXYFQeELhh/Kz79oyttPauoVTs50OGQ1xfU7NKlpjLKSnsELnFhzQJYNHrsgG4AuECkRm5LZZVMvfv+Lxyxzry5MKf9Z91S58iv22MYOblrABHZri67uHhN9QMlPvGoKVUV58b+tmtE4QzlVWhfO9vpZZ1EiWRvXtcwSIVZIS/Jlo7f/xPKHwA8epq65Yharl4VQhmGJ325rphuKql5JYIzozEroGFEaRmQDPiI9vG4DFD19iax92kWpzWpNY4d8OTxL0eOIXcDp1gpcIpthl5jjxMoEuhrILaQdSPyVJuXxLkfDJ7QOR1Uwdbx03GdHdvSju2DtzDGB9Lx+39b1KN/FJt0fu9uhGMJEedUatJoM8UkDEDhrVS8ZtQ9rjur72r115w6QT6jNVrwBFCDaTd9tFTQXXTD/RBKCda7NJJxyqLUgyxnzzh+o07USEx0d3C5tLSgaeUPihZoN8/ocmuEnHUTFqoeLvVt/FMoexiNvnkMoEc4e2G08o/PlbCt0WQXnGnt/WsD2YXnX7YlVJ3wcWamHCoOLdmgHgEYXPhCfYNXhem0a12n6R0xZDu9ifhjgUKumMI1rVJIVECooKMqbX85nAovV4YuzHE4u27ZrUx54jp4J8qsIloALG41DfwFXuxYDIVzdQwRZN5HvSCYwODX2+uUs7FxOT+RndsxXKKipxdS/zBe8Ze/DEDN8od/ktF2QBUGumaIRT0Z3BxkHV4VFBrhZaVGhSwPk1497gQWFumcJuG+JLROeWbVgPEWFMv0xLBe/jgbVPXI61T1zuaJ+WfhEiup6INhJRJRHlBHz2CBHlEVEuEY2wJmZojDw6Oipe99QfsHND/3aO9BfM6UXVG2JaPXUiMt1ez4gGldxZmdA0qp/seOSx1VvwBgDXAfhO30hE3QCMBdAdwEgAbxKRbS4OsXSSn59lb2BOMIJFHKsaidxMQfuvCmamYPjTvCssIqMAlhS+EGKzECLX4KNrAPxPCFEmhNgJIA9Afyt9hcLo0dE/KVPt/HdLwX69br9heystDfDNXIc3YkJN+O0spxkKt+MmmNjALiNbGwB7de/ztbZaENEdRLSCiFYUFRWZ6uyNG/vWbtSew1Wd8TiduuDYqdpl3a7vl4lLujTHpNsG4C9XdXdUnlgg8Nzxm8WEAP45tjf+FJDb/dLs5mjZyJ1skMO7tQDAEa1MaMJ66RDRPABGyR8eE0JMDbabQZuhhhNCvA3gbQDIyckxpQXdLJkWNQrdgPz1eAcrUiJPVfwn5ZQ7B+Gr1fuRVjcJ1/SuPX9xs/aCv5AHw4QirMIXQphJ1pwPQB/pkwnA2KbAuAZXA4uO7JZpGH9F7fD31DpJOFFWgdsu7OCCVAwTOXaZdKYBGEtEdYioPYBOAJbZ1FcN2qRrsyzteVy1BV2VpGF9H5qC4lIAQOmZsyG3q5Psu4zqJnPqBUZtLAVeEdHPALwGIAPADCJaI4QYIYTYSERTAGwCUAHgLiFE6KtGEv4w+Ov7ZSL3wHE8MNzdGprhcFPpsr6PjMMl5ejobuAxw0jBksIXQnwJ4Msgnz0H4Dkrx7dC3eREPHvteW51HxNwzvvIaNYwxW0RGEYKnguFUyG0PBJ+rtVx9XtXuAHr+9D4F2GTEyO7TNg1klEdT+TS0aN6Clv/kkJ2q1TXvSp40TY02S1TkX/0dJWNPjh+f03bRWIYS3hO4ascDekmbdLr1Urzy/opNK+M7YP1+cVonuqObz3DyMZzJp2YUfcOa1uj++BgDtIJScM6SRzIxHgKzyl81XErtYKReypnM5QDP1QysQJf8Q6z+/BJAEBZRaWj/Rply6xkm44U/LEfKtXhZRgjPGjDd1uC0OzXgnmW7TqCS7KbO9av0Qyf3TLlMPHmHCzZcZgrPDHK47kZvupVdjpmNADgvP3cKAqU9b0cmjasg9E91T7vmJo8c20P11KVu4nnFH7rdPcSWEVCizSfx4fTKR9uv8hXEal9swZVbazwmXjlpoHn4NM7B7sthuN4TuGrbtIZp5Wiy3a4FF3j+r5o0czG9dC2ie+mGBgo1KQBR5QyjJfxng1fMcfMr+8egqteX1z1/tKuLVwNuCKiqjEKnOHPu38ojp4qd0EqhmGcgGf4NtOjTRoeGN7ZbTGqEEJUF/II+KxJgxTlawEzDGMe7yl8twUIgIhwz6Wd3BajRgRyal1f4WTVxophGHvxnkmHtZghLdJ8LoM9MxvhVwPPwbQ1+3FO09hINMcwjBw8p/D7ndPEbRGUJLtlGmbcOwTZLdOQmED47dCObovEMIzDeE7hh89sGL90b93IbREYhnER1o4MwzBxgucUPpvwGYZhjPGcwvd7oDAMwzA18ZzCV4UGilfeYhgm/vDcoq0qrHtyBMp1KZDf+lVfLMotclEihmHiHVb4NpGYQKinm+WP7NEKI3u0clEihmHiHTbpMAzDxAmWFD4R/ZWIthDROiL6kojSdZ89QkR5RJRLRCOsi8owDMNYweoMfy6AHkKIngC2AngEAIioG4CxALoDGAngTSLiVUyGYRgXsaTwhRBzhBAV2tufAGRqr68B8D8hRJkQYieAPAD9rfTFMAzDWEOmDf9WALO0120A7NV9lq+1MUzUNONasQwjhbBeOkQ0D0BLg48eE0JM1bZ5DEAFgI/9uxlsb1hQj4juAHAHALRr1y4CkZl44/qczPAbMQwTlrAKXwhxWajPiehmAKMBXCpEVQ2lfABtdZtlAtgf5PhvA3gbAHJycrjKKlMLDmJjGDlY9dIZCeBhAFcLIU7pPpoGYCwR1SGi9gA6AVhmpS8mfqmo5HkAw8jAqg3/dQCpAOYS0RoiegsAhBAbAUwBsAnAbAB3CSHOWuyLiTNeGtMTANArMz3MlgzDRIKlSFshxLkhPnsOwHNWjs/ENz/PaYtLujRHRiov2jKMDDjSllEaVvYMIw9W+AzDMHECK3yGYZg4gRU+wzBMnMAKn2EYJk7wTD78D2/tj+OlZ9wWg2EYRlk8o/Av6pzhtggMwzBKwyYdhmGYOIEVPsMwTJzACp9hGCZOYIXPMAwTJ7DCZxiGiRNY4TMMw8QJrPAZhmHiBFb4DMMwcQJVVyV0HyIqArDb5O7NABySKI4X4TEKDY9PeHiMQuPW+JwjhAgbfaqUwrcCEa0QQuS4LYfK8BiFhscnPDxGoVF9fNikwzAMEyewwmcYhokTvKTw33ZbgBiAxyg0PD7h4TEKjdLj4xkbPsMwDBMaL83wGYZhmBB4QuET0UgiyiWiPCIa77Y8siGid4mokIg26NqaENFcItqm/W+stRMRvaqNxToi6qvb52Zt+21EdLOuvR8Rrdf2eZWIKFQfqkFEbYloIRFtJqKNRHSf1s5jpEFEdYloGRGt1cboKa29PREt1eSfTEQpWnsd7X2e9nmW7liPaO25RDRC1254HQbrQ0WIKJGIVhPRdO29t8ZHCBHTfwASAWwH0AFACoC1ALq5LZfk73gRgL4ANujaXgIwXns9HsCL2usrAcwCQAAGAliqtTcBsEP731h73Vj7bBmAQdo+swBcEaoP1f4AtALQV3udCmArgG48RjXGiAA01F4nA1iqffcpAMZq7W8B+J32+vcA3tJejwUwWXvdTbvG6gBor117iaGuw2B9qPgH4H4AkwBMDyV7rI6P6wMs4QcaBOAb3ftHADzitlw2fM8s1FT4uQBaaa9bAcjVXv8bwA2B2wG4AcC/de3/1tpaAdiia6/aLlgfqv8BmApgOI9R0PGpD2AVgAHwBQklae1V1xKAbwAM0l4nadtR4PXl3y7YdajtY9iHan8AMgHMBzAMwPRQssfq+HjBpNMGwF7d+3ytzeu0EEIUAID2v7nWHmw8QrXnG7SH6kNZtEfrPvDNYHmMdGjmijUACgHMhW/GeUwIUaFtov9eVWOhfV4MoCmiH7umIfpQjVcAPASgUnsfSvaYHB8vKHwyaItn16Ng4xFte8xBRA0BfA7gD0KI46E2NWjz/BgJIc4KIXrDN5PtD6Cr0Wbaf1ljFBNjR0SjARQKIVbqmw02jenx8YLCzwfQVvc+E8B+l2RxkoNE1AoAtP+FWnuw8QjVnmnQHqoP5SCiZPiU/cdCiC+0Zh4jA4QQxwAsgs+Gn05ESdpH+u9VNRba540AHEH0Y3coRB8qcQGAq4loF4D/wWfWeQUeGx8vKPzlADppK90p8C2gTHNZJieYBsDvRXIzfHZrf/uvNU+UgQCKNVPDNwAuJ6LGmifJ5fDZCgsAnCCigZrnya8DjmXUh1Jock8EsFkI8Q/dRzxGGkSUQUTp2ut6AC4DsBnAQgBjtM0Cx8j/vcYAWCB8RuZpAMZqXirtAXSCb0Hb8DrU9gnWhzIIIR4RQmQKIbLgk32BEOKX8Nr4uL1QImmx5Ur4PDO2A3jMbXls+H6fACgAcAa+mcI4+Gx/8wFs0/430bYlAG9oY7EeQI7uOLcCyNP+btG15wDYoO3zOqoD8gz7UO0PwBD4HoPXAVij/V3JY1RjjHoCWK2N0QYAT2jtHeBTSHkAPgVQR2uvq73P0z7voDvWY9o45ELzVtLaDa/DYH2o+gfgYlR76XhqfDjSlmEYJk7wgkmHYRiGiQBW+AzDMHECK3yGYZg4gRU+wzBMnMAKn2EYJk5ghc8wDBMnsMJnGIaJE1jhMwzDxAn/D0MA15+1ZpeKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "temp = float_data[:, 1]  # temperature (in degrees Celsius)（温度（单位：摄氏度））\n",
    "plt.plot(range(len(temp)), temp)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "On this plot, you can clearly see the yearly periodicity of temperature.\n",
    "\n",
    "Here is a more narrow plot of the first ten days of temperature data (since the data is recorded every ten minutes, we get 144 data points \n",
    "per day):\n",
    "\n",
    "在这张图中，你可以清楚地看 到温度每年的周期性变化。\n",
    "\n",
    "上图给出了前 10 天温度数据的图像。因为每 10 分钟记录一个数据，所以每天有 144 个 数据点。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD8CAYAAAB0IB+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXecG+W193+P+korbe/e9a7ttY17xxhMNb0TSCC5BAJ5CaQn75sE4pt7Q0u4gRRCCpBAyKUmgZBCiQE7YKqNjcG9t117vb1JWvXn/WNmtCOtumZUz/fz8cfSzEhzPJZ+OnOeUxjnHARBEETho8m2AQRBEERmIMEnCIIoEkjwCYIgigQSfIIgiCKBBJ8gCKJIIMEnCIIoEkjwCYIgigQSfIIgiCKBBJ8gCKJI0GXbADnV1dW8tbU122YQBEHkFZs3b+7jnNfEO051wWeMXQDgQQBaAL/nnN8X7djW1lZs2rRJbZMIgiAKCsbYkUSOUzWkwxjTAvg1gAsBzAJwHWNslprnJAiCICKjdgx/GYD9nPODnHMPgOcAXK7yOQmCIIgIqC34TQA6ZM87xW0EQRBEhlFb8FmEbSH9mBljtzDGNjHGNvX29qpsDkEQRPGituB3AmiWPZ8E4Lj8AM75o5zzJZzzJTU1cReZCYIgiBRRW/A/BNDOGGtjjBkAXAvgHyqfkyAIgoiAqmmZnHMfY+yrANZASMt8nHO+Q81zEgRBEJFRPQ+fc/4KgFfUPg+Rn3xwsB8Wgw5zJ5Vl2xSCKHiotQKhGgd77Wi9/WVc++j7Effv6x7FtY9+gEt/9U6GLSOI4oQEn8B7B/rw1Wc+QiCg3EB7p8eHzzz6AQDgg4MD4Hz8vdfu6sbv3z6Ic3++PrjN4wsodm6CICKTU710iOzw2d9tAADce+VclJXoFXnP772wDb2j7uDz4TEvys0GDDo8uPmPE9tn7OwawYLmckXOTRBEZMjDJ4I4PT5F3sft82Ptrm40lJlw1+WzAQAH+xz486YO3P/anpBjf/bp+dAwYN2ubkXOTRBEdMjDJ4I43MoI/sZDA3B6/HjouoXQawWf4vt/3YbdJ0aDx1wyrwG3njEVc5rK8OKWY3jx42P41rnTwVikWj2CIJSAPHwiiMPtV+R91u3ugVGnwYqp1agvMwEAdp8YRZ3NCAC4YkEjfvXZRZjTJGTmXLmwCR0DY9h0ZFCR8xMEERny8IucDw8PBB8n4+G/va8XJr0WS1srg9s45/jHJ8fxv+8fwent1SgxaNFeWwq9lsHr57jvqnlYMa0KOk2on3HurDoAwDUPv4+nv3gyTp1WDUDI4qm1mfCVpz+C1aTD8ilV+PSSZpQYtACA7ceGMaXGArOBPsYEkQj0TSlyrnl4PGXSnqDgu7x+XP/YRgDA4fsuDm5/ccsxfPvPnwAAPn9KKwCAMYbHbliKnV0jOH16DbSaiSEbq0mPhS3l2HJ0CJ/7/QbcesZUWE063L8mNN7/6vYT+O9/7MCTNy/DoNOLrz+7BWfOqMETX1iW1L+ZIIoVEvwixuUNDeE4PYmFdN7c0zNhmz/Acec/dwIAHr1+Mc6aWRvcd/r0Gpw+PXafpGf/z3Ks292Du1/aiYffOhDcvnhyBdprS+H1c9hKdPjDu4eDPzaCLb346OggFrVUgHOOAEfEHxWCIEjwi5ohpzfkeaIe/vsH+oOPXV4/dBqGR9YfxPCYF6tOqsN5s+uTtsWk1+KiuQ04e2YtLnnoHezvseOpm0/Gae3VIcfdesZUvLqtC2t2dOP7F52Em/74IW54bCP+z+lT8KcPO+APcPzja6ei1mpK2gaCKHRI8IuYoTFPyPNEYvicc/xrx4ng81GXD//acSIYfvnVZxemZZNJr8Ub3z4j6v46mwk3ntqGG09tAwD833On4/a/bsPPXt8bPObfu3vwmaUtadlBEIUICX4RE+7hOxII6byxqwfdI260VplxuN8Ju9uHHceGAQDXL58Mk16riq3RuHZZC6bWlqLSYkBblQXz73oNWzuH8ZmlGTWDIPICSsvMEXz+ADYfGcSgwxP/YIWYIPgJePibxdTJ75w/EwBw1gNv4rkPO3ByWyXuvmKO8kYmwNLWSkytKYVGwzBvUhm2iT9ABEGEQoKfA2w/Noxpq1/Fp377Hr767EcZO++WjtC89z2ywqhoHB1wYEq1BdWlhpDtp02rjvKKzDK3qRy7ukbg9ilTU0AQqbK3O/73KdOQ4OcAlzw03i1yzwl7xs77yFsHQ56/s78P/jgN1I4OONFcaYbVFNpz55L5jYrblwrzJ5XB6+fBOxGCyAYvbunEeT9fj39HyGjLJiT4OYY/EAjpLJkpWirNAICeUVfUYwIBjkO9DrRVW9BYHpoF01plVtW+RDlzRi3KzXq8sPlYtk0hipiNhwSHo2PAmWVLQiHBzzLhnuig05sx79SkH//vv375ZACx2ysc7nfA4fFjVoMN5WYDXvraacF9udIDp8SgxawGGw72Ze5OiSDCkVKcSzKcxBAPEvws889Pxme611iN0DBg/b6+jJxb/mGsE3veRFu49Qc4vvP8VgAI9sCZ01SG/750Fm5c0aquoUnSUmnOOc+KKC6k71GixYyZgtIys8wnnUOY31yO65Y2Y2lbJb7x3BZ8cKAffBVX3WuWfxilRdhIgu/1B3DTEx9i85FBtFSacVKDNbjvC2I+fC7RXGlGn92DUZd3wloDQWSCgBiW3X1iJMuWhEIefhbpGXFhy9EhzG2yCfnkNaU4e0YtNh4ewMW/fAdev3pToAIBDrcvgIvnNuCeK+ag1Cj89kfKxX//QD/e3teHJZMr8MJtK3ImfBON2Y02AMBD6/Zn2RKiWJEmuO3tzq3QIgl+Fnlqw1EAwMVzxzNcvnp2O246tQ07u0Zw78u7VDv3mNhHZ+6kMvzH8skw6oTwTqRRg1K65sPXL0aN1aiaTUohddt8dP1BSs8ksoJb/B7lWmiRBD9L+AMcv1y7DwAwp8kW3G7QafD9i2ZixdQqPPHeYby3X514viT4ZrHVsEEnfBQ8/okCufnIIJorS1BdmvtiDwB6rQa//uwiAMD/vncky9YQxYjUmLBn1J1wj6pMQIKfJXYcH68GDY8z67Qa/O7zS2AxaPHYO4fAOYdP4fDOmBi6kVohGEXBd3tDz8M5x6Yjg1g6uRL5xKnTqgAA9/1rd8hsXYLIBPJOtI/Iur9mGxL8LPGO6Llv+s9VEfdbjDp8Y1U71u7uwcK7X8e01a/irb29ip0/uocfKviPv3sYfXY3Frbk14DxcrMBL355BfwBjqX3voGu4bFsm0QUEW5fABfNFbrGRgqTZgsS/Cyxu2sUTeWxwyQ3nzYFy6dUBnvefP+v2xCIUwmbKJKHXxLm4Yd/OD86KtQEnH1SnSLnzSQLWypw52XCEPWnPqDQDpE5XN4AykoMKCvRT5g7kU1I8LPErq6RkPTGSGg1DM98cTk2rj4H91wxB8eGxvCbN5XJPJE8fEnwJQ/fHSb4NpMe1aVGNJWXKHLeTHPDilYsba3AK9tO5NQXj8gMIy6vqtlu0XD7/DDqNDDpNXB5ycMvalxePw72OTCz3hb3WI2GodZqwudObsH0ulK8uUeZsE7Qw5dCOlophh8qiiNjXpSV5He5xg0rWnGoz4FLH3oHoy5v/BcQBcO8H76G//j9hoyf1+0NwKTXwqTXwpVDmWIk+Flgf48d/gDHSQ3xBV+CMYZTplRhV9eIImGdoIcvCj5jDDaTDsNjgiB2DDjRevvLeHlbV94XL10yrxE/vHQW9vXYsX5vZqqYidxhw6GBjJ5vxOWFxx+ASa+BSacNOle5AAl+hnn8nUP4f38RBn3PjBPSCWd2YxkcHj8O9zvStiM8hg8A1VYj+uweuH1+XP/YuFck/QjkM589eTJ0GhaSHUUUNkqtdyXLqp++BQAw6rRCSIcWbYuTruEx3PXSTuw+MYq5TWVoq7Ik9fpZYgXp9uPpl2s7wzx8AKguNaLP7sa/tp/A4f7xgpF7szTYREkMOg3aqi3Y15NblY+FjNcfQL89eymx4etRmaJHTAPWaxkMOg08FNLJf+xuX0IfZofbF+ynseGgcGt5wymT8diNS6DRJNeiYHqdFeVmPV6TzZRNFVckD7/UgD67G2t39aDKYsC/vrkSO+48HytyZLhJukytKcXrO7tp8TZDrH5xGxbf80bW0hKz8f8snycR4BxGnZbSMguBT/3mPSy+543g8yP9Dvzgb9snjCi87emPcMEv3sagw4NnNhxFS6UZ/33pbNRaTeFvGReDToOT2yoVGeHnjCD4NaVGHOh14NXtXThzRi1m1ttgMeb3gq2cxZMrAAC/+Tf12MkEL24RZhIc7ndg0+GBuMN1lCYbi6XyqlqDViN4+FnIEooGCX6K7BHHl1344NsYcnpwxv1v4skPjuDMB97EDY9vxKDDg0CAY71YLLXw7tex8fAAPndyS9KevZzqUiOO9Duxdld3WvaPef0waDXQacc/As3iEBSvn+OyBbkxwUpJbjqtDaVGHV7bmd61IxJD6s/0pw87cPXD7+MP7x7K6Pm3dWZ+veZdWSsUi1EHg1ZDHn6+s/nI+Kr/rq4RLLjr9eDz4TEv3trbi79uOYYPDvZPeO2Np7amde7rlrUAAO5fsyet93F5/SEDUABgsmxNQZqAVUhoNQzfOnc6dp8YxaG+9Be+idiUlQjZXY+9Iwj9kf7MNRLjnOPvH4/PmjjQm5m1my8/LcykNuk1uGReoxjDJ8HPa9btnjinstSowyf/dR6qLEJf+f099uDkqvduPxt3XDgTO+48P+j1pMqcpjLcdfls7D4xmlaPDqfHF7JgC4SOKay0GMJfUhBcNLceGga8sLkz26YUPLaS0HTeUlPmwoPrdvfg5W1dwefniJkzmeL3n1+KEoO2eASfMfZDxtgxxtjH4p+L1DpXptl8ZBDzJpVhzTdPD2575PrFKDPr8eZ3zsRJDTY8u/Eofvr6XgBAvc2EL50xVbF4+JULmwAAf9rUkfJ7jHkDMBtC7WmWefW2DH45M0lDWQlObqvKueHShciYJ7RLpF6bOf9yIGwtLdMYxbvnYovh/5xzvkD884rK51IVl9ePQIDjxS2d+ODgAOY2lWFGvRUv3LYCXzpjCk5uE7pJWk16fEE28q+t2pJWzD4SVpMeXz5zKo72O1MuGx/z+IOdMiVMei2uXjwJD1wzP+eHnKTDsrZK7OwaQc9I9IHtRPqEj/ezuzLXJtgYYZZsJrJ2pGLKRS1CgoBBq8laemgkCtONUxi3z4+zHngTXcPjAiHlxC+eXBHM/pBYNasOeEF4/MA181SxaVptKXwBjiP9TkyrLU369WNeX7BTppwHrpmvhHk5zfmz6/Hg2n14c08vPr20OdvmFCxOjz8kpNE94oLXH8iIpy81A5QTyclRmkqLHksmV0ArOnnGYgnpiHyVMbaVMfY4Y6wi/uG5ybf/9EmI2APAZfOjZ7FUWgzBZmPtdclV0yZKo/j+3Sl6qWMef0hKZjExo96KEr0Wu8VJXoTycM7h9PhQLVsLenlbF259cnNGzh/p/jQToRWXNxAM5wDjIR3Os1P1G05aHj5j7A0A9RF2rQbwWwB3A+Di3z8FcFOE97gFwC0A0NLSko45qtA56Awu/ly1sAktVWbcdFpb3P4yz92yHDotg02lPjTS4nB/irFKp8ePSkt+TLBSGq2GYUa9FZvF1s+E8rh9AQQ4UFVqxHGZs7Q2QsKDGnj9EwU2fLiPGrh9/mB2EiCEdDgHfAEOvTb7YdK0BJ9zHnl6RxiMsd8BeCnKezwK4FEAWLJkSW78DEJIt/zThx144r3DAIBL5zfi/mvmB2/V4tGsclqjlEXzzIYjuHhuQ8J2Sbi8/oghnWLhrBm1+PkbezHi8qr2o1zMSL2aqkqzk+0VaW0rE/ON3d5ASDjJIJszkclF62iomaXTIHt6JYDtap1LaR556wAufPDtoNhfvXgSHrpuYdKiqiblZuGL9MHBgaCdyTDmLd6QDgAsECd4bVegapmYiNSrqaEsO3MU5HFz6XubicVTty8Qsk5giDJYKFuo+ZPzE8bYNsbYVgBnAfiWiudSlB+/ujvkeS4uZMp/fA6nUETk9Pgn5OEXE3ObygAAW7NQjVkMOMUWA8unVOI/Lz4Jf7hxKQBgQXNmRmXK4/WPi+fOhIfv8voje/g5kpqpWpYO5/x6td5bTQ6KFXlVFgP++bXTMOjMbj5vLK5d2oznPuxAIIUFIZe3uAW/0mLAzHor/rblGL50+pSCTkPNBlJKpsWgw+UrhbqRk9sqkamYrTykY4wyzU0N3L6wkI62eDz8vMIntnL98yahAvN7F85EY3kJZjeWZdmy6Nz3qXmYUWfF/iRb/nr9AXj9vKhDOgDwqUWTsPvEaMoL30R0JME3G8c/Yya9dsJENbXInuCHpn5KxZbhNQnZgvLwRW7/6zY8L5bbT6oowRULmrJsUWKcMrUKz2w8ikCAJ1zgFT7PtliZN0n4MV+3q4fy8RVmzCuEdOTV3CZ95oqQJI+63mYKtjNRO0uHcy6kZco8/FJR8HNltCZ5+CLPy3qr/PGmZcHYW67TXlcKjy+AE0nk4w/YBY+2okD75STKsrZKWI06bD02lG1TCo6ghy8LGxp12oz1qPeIaZnvfO+s4HdZ7Ri+1KCtxjbe+txqkgQ/c1XGscgPVVOZ8KKIqTXJV65mi1axw+WK+9bhobX7EnpNrzi4pcZanHn4EowJ+fh7qABLcSLNWzDpNXBlIBceAHpHXdBqGHRaTcZCOieGhe/VDFmxpVSvM+omwc8ZRsRf36+f045DP86vHm+TZR0uf/r6Xvz63/vj3j72iSPYakqLW/ABYHFrBT46OoShHF6cz0fGonj4mciUsbt9eHZjR3DgilT5qrbgO8RmcRbZusW4h08hnZxBmlLVWmXOu2yN8Dzn+9fswR/ePRzzNZKHX20t7pAOAJwxvQb+AMcnlJ6pCGt3deOmJz4MTn6Sd4jNlIcfPnVOiuGrnSnjlARftm5BIZ0cZMCZvzHtSMVgkQavyOkbdUPDgKoiba0g56R6oQneXgrrKMLNf9yEdbt7cGxoDIyFNjEz6QUPX+2+MsNjod60MUMxfId74l1NiV4LrYZltFNoLEjwMe4RVJrzT/AjEW+yUK/djUqLIacqh7NFhcWAGqsxOLKSUIYj/Q6Y9dqQO2ajToMAj9znRklGogm+yncXkodvlt3VMMZQXqLPmdRfEnwAB3uFStWG8uQHi+cCK9urAQC3nD4Fnzu5BceHx2JmQ/SOelBN8fsgM+qs2EuCryjv7u+f0LZbyk9Xe7i4W8zBf/DaBQAE0TXo1E8JlTz88HTnpooSdA5mbrxjLEjwIYRAplRbUGvNT8F/5PrFePu7Z+H7F52EhS0V4BwT2jnLGXR6UFEgdzNKMKepDFs7h/FJB6VnpkMgEOq5y2ckA5nztL2isMt/cIxajeohHafHFwzhyGmuNKNzcEzVcydK0Qs+5xxbOoYmDDHJJ8wGXbA7Z7nYmjX8tlbOkNODCgt1iJT41CKhyO6hdfuzbEl+E+65h89FlqZQqZ2LL4WMDLLulMYMFH05PP6QDB2J5gozOgedwayhbFK0gi8tHHUOjmHA4cH8DDV1UhtpcPRIjDSwIac32G2TEIbULG2tQM8ojTxMh7Gw9gGnTasOeS6FdNQWXqmtgrwdsVGnVT9Lx+2bMCcaAJorS+D185SHFSlJUQo+5xzn/nw9TvnxWmwRb+PnTyoMwZeGLww5Iws+5xxDY15UmMnDl9NaZcmJL2Q+Mxbmua+aVRfyXArpqO3hS50p9Tq54Kvv4Q86I3+vmiuEu++OgezH8YtS8LtH3NjfY0fXsAt/2dQBg06DGfXqjCLMNNLC87GhyDHDEZcP/gBHeQl5+HJqrEb02T05M4ouH4kn5FK6otqNxMY9/PFYukGnUb1xW7/DjaoIyRBSuDUXEgOKUvCPyn5p397Xh+VTqvKmd048bCY9Ki0GHOmP3CN/WPT8y8nDD6GsRA9/gMORI10N85ExjyC0Z82owft3nD1hv9TKo1es9FYLadE2NIavVd3D77d7gqNH5UyuNKPOZsQHhwZUPX8iFIbKJcnxMO93TqMtS5aoQ0ulOWouvtTfn7J0QilLYLG7EOGcY8+JUUUWFKWQzs2nTYk46apOzIJTO3QmLdqGxvDVzdLhnAuCH8HD12gYWqss6MmBkGFRCv7HYtz+i6e1ARAEspBorUpA8ClLJwRpEbvfnhsFMpniu89vxfm/WI+Xth5P+72CbbejDNYpN+th0GlUF3xPxEVbjaqLtqNuHzz+AKqjzPCtLzMl1dFWLQpK8O/85w489s6hmMdsPzYcnAH73Qtm4r6r5uJTiydlwLrM0VxpxvHhsYiDnIeCIR3y8OVMqxVyxvf3Zj/OmilcXj/+IrYFP9Cb/JjMcMbEStNocxYYY6izGdUXfN/EGL7ai7ZSQ8JoQ9vrbSZ0j7izvkZUUIL/h3cP4+6XdsY8Rj4dyqDT4NplLTkxTV5JGspKwHnkWKnUFVLK1ycEJldZYNBqsLuIeur8/eNjir6f1CBMahgWiTqr+p6u1x+AXsvC2jqoG8OXWidE609VZzPB4wtgMEr2XKYoGKVLND4nVaA+dfPJapqTVRrKhFjps+IkLDnSB66MBD8EvVaDOU02bDiY/YW1TNEzMu4Q/HLtPjjS7NkudcgsNcYQ/DJTyHnVQBD8UGlTO4YfLxmiXvxOnohRAZ8JCkLw7W4f/r27N/g81m1Tx6AT5WY9TmuvjnpMviN9uB5atx+v7TwRsm/Q6YHNpIOuwO5qlGBlew0+6RyKWbRWSDz3YUfI842H0/uxkzz80jgefiYWbcMF32LUYWRMvY6VUpVxtHBWnS0zC9bxKIhv/f4eO259anPwebSe26MuL57ZcBRt1ZaI+wsFycMHgM1HBkP29dndqC7ySVfRWDRZ6EO0vUh640u1Gk9/Ubjb9aYZ8rC7fTDpNTFDpPVlRjg8flUHgngiePiN5SUYHvOmfRcTDanK2BRF8CUnjARfAWxhHsWoO/KHSVrQXdZaqbpN2UQerjkclq3TZ6dOmdGY1yQMNd/ZNZJlSzLHxXMbgllqQ2mmpI66fCg1xg4Vjnu66oV1vL4ADNrQBmZNFUKaaLSCxHSRis6iCX6t1YhSow6fdGa3QV9hCH5YPDradBlpwfabq6arblM2YYzh+uWTAUzMK++zu2m0YRQqLAaUGnU509lQTQbERcbZTbbx/ktpCr7d7Yu5YAuMC/7RgfSzgqLh9QdC2ioAwCRR8OPNikgVKaoQLSVVr9VgSo0lZhfbTFAQgh/+IYs2XWZftx3nzKyN+p9SSNx9xRysOqkuOK8XAPrtbhzsdURNHSOEcFjXcOEL/rWPvg9AGMdnNerAGPDEe4fT+rePurwxF2wBYEFzOUx6Ddbv7Uv5PPGIFMOfWW+FVsOwfm9vlFelh1SDYIpRsW8x6OB0Z7eSuyAEX5pZKWGPEKfz+QM42GdHe11h9MxJhHKzHgMO4dbZ4wtg8T1vABBuL4nINJaX4PhQ9gtk1CQQ4NjbLdztnj+7HhoNA+dC59jbX9iW8vvaXfE9fJNei2VtVXhtx4mYx6VDpBi+2aCDP8Dx5AdHcLhP+bsLp8cPg04TMxnCYtQGB51ni4IQfAB4/MYl+K9LZgEAfviPHRM60x0ZcMLr52gPm8JTyEyqKEH3iBtPfXAEQ2PjFaT/IYZ7iInU23KjIlJNpFqD//nU3OBiooQvkPrCrd3ti+vhA8Cy1gocH3ap1jXT658YwwfG17bUiOM7Pb6QWbaRMBt0qjeOi0fBCP7ZM+twrtiOdV+PHV955qOQ/dIYw6lFJPifWdoMANhxfCQYn/3ldQupyjYG1VYDBhyeCfULhYTUtXHx5PHkhRduWwEgeoZbIoy6fDFTMiUay4V4enhPK6WIlIcPjGcjDagwX9bh9sMSoRe+HItRGzH6kEkKRvCB8W58ALC1cxgPvrEv+HyNeAsZqZtdodJQVoKTGmzoHXVheIwKrhKhutQIf4AHew4VIn12Icwn/74snlyBC+fUp7VwO+rywpqAh99Urm7GjNc3MYYPjIcy081GikTCHj4JvnKY9Fp8ZklzcNLOz9/YC0Bolva82DPEksAHspCotRrRPeKGXVwsKo0wgo0YR0pZ7SvgJmq9o24YtJoJ6cxlJfqgY5AsLq8fIy5fQim/wRRJlbKhPBGydADAahKcHTVqABweP8xxtMVi1MHpFWoQUr3O6VJQgg8A/3P1PDx58zIsEWfUdgw4Q/6DI82cLGRqrUb0jLpkja2K6wcvWSTB6rerW/6fTXrtbtRYjSG9ZgAhvTnVKmOpoCh8TSASdTYTNExFDz9KDN+k10CnYVHTttPB6fbBEsfDtxi04Bw464E3Mf/O1xS3IREKTvABIQ999cUnAQB2dY2EpGmGZ/QUOnU2E3pHxz38eLedxU6NVQj59Raw4AvFdxNDm2Uleri8gZR6zhzoFbJ+ptTEr2LXazUoNxtUC5t5fJFj+IwxWE069Tz8ODF86Q5Aunv0RehmqzYFKfjA+OLsI+sPqhKzyxdqbUYE+Pg8TRL82BRLSCdS6EUK8Vz04NtJv6dUUNRUnthsCUF41YlnR1u0BYTmgU99cFTxJmZOjy9u9KAyLFkiG2GdghV8mxiv23xkEAd77XGOLlxqxSlDD64VFrCLoegsHcpKMjOkI1v02d3Y1TUSMuZTokT0UA/0OpLu2y6Jt60ksZChmoIfzcOXs3Z3t2Ln45zD7oq/aBte8JiNVskFK/hyfve20EPn55+Zn2VLMk+tLdSTi3fbWewwxtBcUTKhjqNQkNqLrGyvmbDPKFvoTLZ3/KjLC62GRe0WGU5ZiT44m0FJhse8ODHiiltNrmTWbdewC/0OD9prYxd1hofR1Pj3xyMtwWeMXcMY28EYCzDGloTtu4Mxtp8xtocxdn56ZqbGW985M+T5lQsLa7JVIki9SyS0momLWUQoLZXmiB5wIdAjDsW5dlnzhH0GmeAn21XS7hKKrsIXgqPRVF6iSs+iPrsbAQ7MjjKn+jvnzwCQ/L/ypCPLAAAgAElEQVQvFtIUucby2AvWlWHDUYby0MPfDuAqAOvlGxljswBcC2A2gAsA/IYxlvFYwuQqSzDXONoHoNChRmnJ01JpxtF+Z9bH0amBNEhbGiguRy74iVaE+gMcIy4vRhNoqyCnoawEPaPuiGM408EeZ+rWbWdMBQBFq3yldgnxUr7Dp8xlo9YjLcHnnO/inO+JsOtyAM9xzt2c80MA9gNYls65UmWu2PL2K2dNy8bps44hRjMnIjLNlWaMun1Z8cDUpmfUDaNOEzHWLg/pJNrz5cn3D2PeD1/DtmPDwTz3RJBmMgwqXPUqVbJGq3rVaBgMWk1aFcXRzhkvXKrRMPz1yyuw/jtnASisRdsmAPJxOp3itoxz6xlT0VJpLloPHwBe+9bp2TYhr5D6wxdiWOeZDUfBGCKGXkIEP8Gujmt39wAQ2pnokggXVosV70pnQwXHLMa42zDpNYp6+FIHzERqfBa1VAQLz174SNmZwokQ9x6MMfYGgPoIu1Zzzv8e7WURtkW8P2aM3QLgFgBoaWmJZ07SLGurxPrvnqX4++YT0+us+O4FMzClwCd9KUVLlSD4h/sdmN9cnmVrlMPu9sXs5aLVyEM6iXn48swUTRKCXyUVuDmUrXeQQjqxmriZ9FpF59tKrZHNCRY1Sutou7pGcKjPkdEJfHEt5JyvSuF9OwHIV4UmATge5f0fBfAoACxZsqTwgqY5wpfPLM6QVipMrSmFQafBts5hXL4gKzemqiC1BV4ZZZ5ze20pGstMOD7sStjDH3R6sbClHItaKnDlwsSvlZRF06+Whx9D8EsMWkW7Vkp3C0Z98gGTvd2jGRV8tUI6/wBwLWPMyBhrA9AOYKNK5yIIRdFrNZjTaMv6ODqlkVJNb79wZsT9FqMOz96yHEDiHn6f3Y2GMhN+cMkszBHXyxKh2iIVuCns4bvjL6CWl+gVzYEPjjdMoYo/0+m/6aZlXskY6wRwCoCXGWNrAIBzvgPAnwHsBPAvAF/hnGe3ETRBJMHClgp80jkczFsvBKQ0yEkV0athpYVHR4IecF+Uqt142EqEKVvPbDgKv4JJ8Xa3D3otC1mPCKfSYggOBlICqWYhGQ9fWlfrHBzDC5s7sfxHazPSkjvdLJ0XOeeTOOdGznkd5/x82b57OedTOeczOOevpm8qQWSOm09rg1Gnwa/W7Yt/cJ7QOeiEzaSL2SJbWnj86Mggbn7iw5hN5Ny+xDtkhsOYMGXrYJ8Dr27vSvr10XC449cDVJgNimZgubx+MIaYPzLhTK+zYma9FR0DTvzfv3yCEyMu2DMwDYty9ggiAo3lJThlShW2dg5n2xTF6LN7QnrgR0IKS7y45RjW7u7BOT97K+qx0iCRVAQfACaLi+NvKzjf1u7yxc2Ht4iD6pUahOLy+mHUaRIuOpNoLC8JGWqu9HpGJEjwCSIKc5rKcLDPoWhVZjYZdfvi5sqHZ9rE8oT7RiXBT22o0MtfX4mT2yrx/sH+lF4fidEExixKPwin/HitIud0eQMwJdhSQk6dzRisfAaAm574UBF7YkGCTxBRkFr9KpGP//ePj2Hx3a+rNsc1Eewub1LVsAAwqyF6/Yq04Fod564hGqVGHeY3l+PEiEuxqmZHAoIvDQFKtl9QNFxef0oLtlaTHnb3+A/qjLrYvXiUgASfIKIwuVIQ/CP96Qv+N577GP0OD/aIA8SzQaJDxu+7am7wsSaGQkgzA6otqbfvqCk1wuMLKBZTt7vjz9VNNvQSD5cvAFMKKZlmgzak4veuy2craVZESPAJIgpSAdbRAYdi73nLk5sUe69kkRqcxeOqRZPwk0/Nw4Vz6qPmq3PO8d3ntwIQBr+nilTYtuHQQMrvISeRf2OFWdm51i6vP6WQjtzO65dPRq0t/rSwdCHBJ4golJXoUWrU4fiQcr3xu0eyN0lr1B1/QRMQ+i99emkzrCZdsG1AOIf6xn8E02m5PW9SGRiDYnc+idzFXLNE6Jpbr5DAurx+GFMQfPl1SybDJx1I8AkiBg1lJnQNqzN7NZNwzmF3J9fR0mzQRS3AWr+3FwBwxYLGtOwy6bUoL9Gj167Mj2oigq/XanDdsmb4FVo3cHsDMKUg2FJPHQBQOMoUFRJ8gohBQ1jqXL7i9PjBeeyWA+GUGLTBPjESXn8AHl8AXSMu6LUMP/v0grRtqyo1onc0/Tsff4DD6fEndBdj0mvhUqi9gsuXWkhnwaTxPk0lGRpMRIJPEDFosJkUEXx5/5pMVFSGk0gXyXAsBi28fo5+uzuYkXP2T9/E3B+uQfewC7VWU1IN06LhdPuwZkc3frf+YFrvI7V0TuQuxmzQYtTtU6RFsdub2qJtmVmPD1evwpdOn4IvrmxL245EIMEniBg0lJvQZ3fDk2YKX0AWPnAp2KkxUUYT6CIZjuR1Lr7nDSy55w0AQMfAGNy+ALpH3KgvUyYGfuflcwAAv3hjb1rpmVKnzEQ8fGkU4/w7X0t7fnGqHj4A1FiNuOOik4IzuNWGBJ8gYtBQZgLnSFsUfP5xIVOyU2OiSB5+MjH8cPGVP+8ecSm26HnurDrcceFMODz+tK6NI4FOmRJygV4n9vRPlVTz8LMBCT5BxKChTFhYSzesI28Q9rctxzI+PvFvW4RhG6XGxD3J08LaKI+4xhdwD/Y5JsxLTgcp1BSrX388RpMIW5XI+vi/uacnrf8PV4ohnWyQH1YSRJaQBlOnm6njkwn+PS/vwt8+zty0o02HB/DEe4cBCOX8iTKzPrTKdiQs3l1fpty8ZMkrH3WlLviJDD+RaCofz5BZs6Mba3acSOmcHl8AIy5vyiGdTEOCTxAxqBc9/HRz8X2BAOTrm0pU7ybKYdm5pDuWRLHIPOERV6jgK+nhSzHsUVfqi6hD4g9SIoJ/6rRqfGvV9ODz3SnWAew5MQrOgfYMtEVQAhJ8gohBqVGHKosBh/rS64vv83PYZG2J9drMffXkvd+THWr/0tdX4urFQqHSoCNUjCdVJPfjEQslQjofiE3YKi3xK2n1Wg2+sao9+PxEiiE7yd5GhRaw1YYEnyDiMKvRhh3HR9J6D3+AhyyYjmVw4XZAFOobV7Qm/dq2akvwdZ2Dwp2ClOEyoz56Y7Vkka5NOiEdjy8Am0mX0p3HsaHUQnZjXsFe+ZpALkOCTxBxmNVow75ue1qpmb4AR42sb7wjA8MuJFxeP8pK9PjhZak155IGpkhhj3uvnINtPzwvqRTPeEjvZU9D8F1ef9K9+T97cgsA4O19ffhXCoNYpKyidNpLZBISfIKIw5zGMnj8Aew+kbqX7/b6MaWmFD+5eh6AzHr4bp8/rV4tUihKWvgtN+vj9tVPlnKxodmAM/UhIKk0MfvRlXPx/YuEGb+3PvVR0uccF3zy8AmiIFjaWglAGPuXKm5fAEadBp9e0oy2akvCM2OVwO0LJB27lxM+ErGsRNluk4Dg4ZsN2pRj6UDq6ZGfP6U1+PhwX3KdUaUfbgrpEESBUGczwmbSYV8aA80FwRdEwWzQwpnBKVrSj006fGZJc/BxrJm46TC70YaXtnbB608tdDY8llp6pEmvxZM3LwMAfPvPHyf1WvLwCaLAYIxhep0V+7rTEXw/jKL3aTHoMlpt6/aO/9ikSqMsb73crI7gX7VoEvrs7pS8/C1HB7Ht2DA2p3gXNq9JaGT20dGhpF43Jq7FUKUtQRQQ7XVW7O0ZTaki0x/g8Pp50Ms2G7VR2w6rgdvnTyukAwh96yXU8vClYqgTKbSxeHufMAg91bGFZbIfsWT+j50eP0r0WkWayGUCEnyCSICFzeUYcnpTSs+UsnskL9tm0qe1OJnK+dMN6Zw5oyb4WK0agoYyqao5ecF3K9CQ7geXzAIQe3B7OE6vP2/COQAJPkEkxKpZddBqGP61PfkSfMmbLxFDOpOrzOgYGEs5/JAsbl8gpYlMchhj2Lj6HLxw2ykKWTURqfvm8RRy4qWQivyHKVmkH5zjSbTRGPP482bBFiDBJ4iEqLQYcHJbJV5NIVd7vFOlEDa4ZrGwALpud7dyBsbA7QvAoIBXXms1YfHkSgUsiozVpEedzYi9KbQ5kFoV/eZzi9I4v5BLn8z6itPjgyVPcvABEnyCSJiL5jbgQK8DO44PJ/U6qXpUEpSWKjPqbSb0ZGi+rUe2YJzrzGqwYWdXCmEzvx8all4BlFRBnJzgk4dPEAXJebPrAACPv3M4qYW94PARWWuFSosBA47MxPHdvgCMGezdkw6zGm3Y32NPOibvSbPWABjPpf/gYH/C/79jHorhE0RBUms14ZbTp+CFjzqxpSPx9L1gSEfWi77UpEurUVgyCDH8/Piqz2oogy/Ak06B9fp52mErycP/7ZsH8PePjyf0GicJPkEULl89exoMOg0ee/tQwl6g1PJX3jzNasyc4Ht86efhZ4pZjUJDtm3HkgubCdXE6f0b5eGgb/7p44T+f8e8/owNIFcCEnyCSAKbSY8vnNqKl7d1JZxlE2mAeCY9fJc3/Tz8TDG50oxysx6vJpkNpUTqaVVpaMuIDYcG4r7G6fHBnCfDTwASfIJImltPnwpgvP96PMIXbQGhd0w6nSETxecPwO0L5E3YQaNhOGdmHXYluXDr8Qeg16ZX/BReX3CwN35fHVq0JYgCp8JiwMx6K9aL1Z3xGHX5YNBqQsIqmfLwpSZtSrYyVpvpdaXoHXVjeCzxAiiPAtXEwPggE8YSq/ilRVuCKAIaykzYeGgAb+3tjXus3e2dMFjbatTB7Quk1WM/ERzuxOe85grTaksBAPuTaFbn9XNFBP/1b5+BLT84FzaTHr9cuw93/XNn1GM9vgB8AU6CTxCFztfOEcbjvbI1fiHWW3t7gxkgEsGBHyp7+ZLgW/JI8GfUC/NhX9mWeJGbR6HiMotRhwqLIXh38fi7h3DWA2/CF6GD53hr5Py5tiT4BJECi1oqsLK9GlvjZJN0DDjRMTA2YYSeNPBjUOWeOvY89PAnVZhx3bJm/OHdQ+hKsM2BEnn40TjU50Dn4EQ7nOJ4Q/LwCaIIWNhcjj0nRoJedCQ+FvP1v7Vqesh2aRRf36i61bb2PPTwAeCW06ciwIE1CWbruP0BRZu6PXjtgpDnuyO0e3C486sXPpCm4DPGrmGM7WCMBRhjS2TbWxljY4yxj8U/D6dvKkHkFgtbKhDgsXPGv/bsFgDAF1e2hWyvtgoefp9dXQ9/PKSTP6IEAK1VQnrm3gTj+F4F0jLlXL6gCRtXn4P/vPgkAMCtT21GIBCalx8M6RRRWuZ2AFcBWB9h3wHO+QLxz61pnocgco6ZDUKsOdokLJdXEISm8pIJHrbk4d/3r10qWgjY3fmXpQMI3Tmn1ZQmvHDr8Ssf0qm1mvDFlVOCz/+9pwe/f/sgAGDI6cGlv3oHQP4MMAfSFHzO+S7O+R6ljCGIfKLeZoLZoMXB3sii1CuGa247c+qEfRViDL9jIPlWwMkw4BBsqLAoP4dWbVqqzNh4aABbjsYvcHN5/apNnfrxVXMBADf/cRPueXkXnB4fjg44g/spD1+gjTG2hTH2FmNsZbSDGGO3MMY2McY29fbGT3EjiFyBMYa2asuEiluX14/pq1/Fyp/8G0DkkYBaDcPK9moAmBAqUJKeETdK9FpY88zDB8avyxee+DDusUNOb8jUKiWZP6k85Pn+HntIz/58iuHH/RQwxt4AUB9h12rO+d+jvKwLQAvnvJ8xthjA3xhjsznnE8rnOOePAngUAJYsWaLeJ58gVGDI6cWxoTEc7nOgtdoCQPDsPbI0vnDBkFjZXo239/VhzOtXbVG1e9SNWpsRjOXHCD45Wo3gj/rj/CB6fAHY3T5UmtW5iwlvuXDZr94NeZ5Pgh/Xw+ecr+Kcz4nwJ5rYg3Pu5pz3i483AzgAYHq04wkiX/nuBTMAAJ9/fGNw25h3vLXv+u+cheZKc8TXSvnbDhXn2/aMuFBnNan2/mry5bOEUNhssaFaNEbE5nQ2lWbtVsT5ISn6kA5jrIYxphUfTwHQDuCgGuciiGxy+YImGHUaHB1wBqtmR2U9clqqIos9gGDTrbEkBm4kS8+oGzU2o2rvryZTa0px3qw6DDpit1iQrp9ad0nxFoPzaUE83bTMKxljnQBOAfAyY2yNuOt0AFsZY58AeB7ArZzz+K3nCCIPeeCa+QCAvd1CrraU+37p/MaYr5NCAclMWEqWfPbwAaDWZkSvPXatgnSHpGZo5bMnt6DWGvmHM5+ydNKylHP+IoAXI2x/AcAL6bw3QeQLUsime8SFOU1lONwndFn87vkzYr6uJCj46oR0XF4/HB7/hBh0PmEz6THg8KBz0IlJFZHvlpzBFgfqCf6PrpyLaxZPwpW/eS+47blblqOpvES1c6oBVdoSRJpInl+PmIb55p4eVJj1ccVACkGo5eHnY+O0cKT12nteil6v4BRrDdQeJi735F/62mlYPqUq6vpMrkKCTxBpIi0WSpOtDvc7sWJqNTSa2JkxUoVmsuP8EkX6IcmnLJJwvnb2NACx/w3ODIR0gNCK2jlNZaqeSy1I8AkiTaTFV7vbj44BJw71OYKj+mIhDUS566XoLXjTwanyYmYmsBh1aKu24K9bjuG1HSfQb3fj0fUHQmoXpKwo1QU/j384JUjwCSJNNBoGs0ELp9uHv398DIwBVyxsivu6Flk4INH5uMmQicXMTNAgDiX5/TuHcMuTm/GjV3bjgFjd/OzGo/jGcx8DUH/xVPqB/uaqdlXPoyYk+AShABajDg6PD7u6RtFaZUloMU9eDOVWYRBKMLadxx4+ADx03UIAQHttKToHhZYGXcMuuLx+/OfftgePU9sDN+m12H/vhfjGOST4BFHUlBp1sLv96BoeQ2N54mmQd18xBwAwksQ4v0QpFA+/qtSIJZMr8HHHEHRi9e3nH9+Izz++MaQoKxP/Tp1Wk5dVyxIk+AShABajENIZ8waSCi2UiQu+UrWokkiLmWpnr2SCRZMrJnQl3XhoAHW28R9XJfvhFyp0hQhCAcwGYSi5y+uHKYn+6FJ8emfXxAEb6RIc0JFnvfAjMa22FB5fYMLksDiJUEQYJPgEoQClRh26R1xim97Ev1aLWypQZzPitR2JTXZKhkLy8NvFwebhuLzC2se7t5+dSXPyFhJ8glCA6XVWHO53omvYldTioUbDMLuxLOFBH8kgefj5NJEpGu111ojb39rbi/ba0ryreM0WJPgEoQCnTqsKPk4mpAMI4YqDfY64bYCTxenxwWzQxi0AywdKjTpsXH1OxH3RJo4REyHBJwgFWNZWmfJrp9UI8ekO2RQlJXB4/HmfoSOn1mrCoR9fNGH7tCjhHmIiJPgEoQBGnRaf/Nd5uGx+I25c0ZrUa6eKgqV0WMfh9hVEdaicSCmRL355RRYsyU9I8AlCIcrMevzyuoVoTDKeLHmoB6LMxk2VY4NjaLAVXmz77Jm1AIAbV7Tiz186BVaTOoNPCpH8X74niDynrESPcrMeHYPKhnQ6B8dwmjg3t5D4zecWYcDhSfqHlSAPnyByguYKMzoGxuIfmAQjLi/KVRr7l01Mei2JfYqQ4BNEDjCpoiTYJ0YJvP4AnB6/anNeifyEBJ8gcgBB8McU65opzdWVOjwSBECCTxA5waQKM9y+AH70yq6QXu+pUgjTrgjlIcEniByguVKISf/u7UN4fVd32u8ndcokwSfkkOATRA4gH9A9psCM2/HGaST4xDgk+ASRA8h7wSjRYkEK6VgKrPCKSA8SfILIAeRTqZQQ/GCnTPLwCRkk+ASRYxwZcKT1eo8vgONDLgCF0RqZUA4SfILIEV647RQAwPZjI2m9z21PbcZdL+0EUBjDTwjlIMEniBxh8eRKXL14EnYcH04rH3/t7p7gY8rSIeSQ4BNEDtFeW4o+uwd2cdE1XYxJTN8iCh/6NBBEDlFrMwIAukdcKb+HQRR5q1EXsZ0wUbyQ4BNEDtFYJqRnrpOFZZJlZr0wDnBUobsEonAgwSeIHEKanDXo9Kb8HrVWEwCgACYbEgpDgk8QOQRjDDVWIwbsnpTfIyAu+L789ZVKmUUUCCT4BJFjVFkM6HekLvhefwCLWspxUoNNQauIQoAEnyByjH09dryxqzvloeZuXwB6LX21iYnQp4IgcgyptcK2Y8Mpvd7rDwQzdQhCDn0qCCLH+POXhIrbVHPxvf4ADOThExFI61PBGLufMbabMbaVMfYiY6xctu8Oxth+xtgextj56ZtKEMVBe20pAMDuSlHwfZxCOkRE0v1UvA5gDud8HoC9AO4AAMbYLADXApgN4AIAv2GMUVMPgkiAUnEsYToevp5COkQE0vpUcM5f45xLn8oPAEwSH18O4DnOuZtzfgjAfgDL0jkXQRQLeq0GJr0Go67UcvE9/gD0WkrCJyaipBtwE4BXxcdNADpk+zrFbRNgjN3CGNvEGNvU29uroDkEkb9YTXqK4ROKE/dTwRh7gzG2PcKfy2XHrAbgA/C0tCnCW0Vs/8c5f5RzvoRzvqSmpiaVfwNBFBxWow6jYgzf5w8kNdjcQ2mZRBTi9k7lnK+KtZ8xdgOASwCcw8d7unYCaJYdNgnA8VSNJIhio9Q0LvjTVr+KBc3l+NtXTk3otV4/LdoSkUk3S+cCAN8DcBnnXF4l8g8A1zLGjIyxNgDtADamcy6CKCasJh3sbh8GxIrbjzuGEn6th/LwiSikOx3hVwCMAF4X27B+wDm/lXO+gzH2ZwA7IYR6vsI596d5LoIoGkqNOry7vxuL7n49qddxzsUYPi3aEhNJS/A559Ni7LsXwL3pvD9BFCtWkz6l1/kDHJyDQjpEROhTQRA5SIU5NcH3+oVlNMrDJyJBnwqCyEEmV1kmbNt+bDhmLN/rD2DlT9YBIA+fiAx9KggiB7l0XuOEbZc89A6u+PW7UV/TZ3ejT+yjTzF8IhIk+ASRg5SlENIZkPXQJw+fiAR9KggiR/njTcl1I/nVuv3Bx5SWSUSCPhUEkaOcMb0GzZUlCR/fbycPn4gNfSoIIod58qaTJ3jrXn8g5PnqF7fh0fUH0Gt3B7eR4BORSLfwiiAIFWmttuDb507Hfa/uDm4b8/qDgu71B/D0hqMAhGItCSOFdIgI0KeCIHKcEn3oKAmXZ7xo3SHrqCnvrmnU01ebmAh9Kggix/nM0mZ86Ywp+MElswAATo8fz208CpfXH7WFcviPBEEAFNIhiJzHpNfijgtPwivbugAAL2/rwv1r9uCTziEsnlwZ9TUEEQ4JPkHkCZLX3jU8BgB4dmMHnt3YEfFYEnwiEhTSIYg8QRJxhzt+41kTxfCJCJCHTxB5QolBEPwXtxyLeszK9mp0DbtQZzVlyiwijyDBJ4g8IZGF2NvOnIoVU6szYA2Rj9B9H0HkCYkIPuXfE7GgTwdB5AkmQ+Sva1v1eCtlo44Wa4noUEiHIPKE8hJD8PFvP7cIF85tAOfCwJO2O14BAMxqsGXFNiI/IMEniDxB3lOn1iYsyoqzpLHx++cgwAGNhvrgE9EhwSeIPKS+LDQLR/oBIIhYUAyfIPKQOqsx2yYQeQh5+ASRRzx188k4PjQGHbU/JlKABJ8g8ojT2inHnkgdchMIgiCKBBJ8giCIIoEEnyAIokggwScIgigSSPAJgiCKBBJ8giCIIoEEnyAIokggwScIgigSmNRtLxdgjPUCOJLGW1QD6FPIHDUhO5UlX+wE8sdWslN51LR1Mue8Jt5BOSX46cIY28Q5X5JtO+JBdipLvtgJ5I+tZKfy5IKtFNIhCIIoEkjwCYIgioRCE/xHs21AgpCdypIvdgL5YyvZqTxZt7WgYvgEQRBEdArNwycIgiCiUBCCzxi7gDG2hzG2nzF2e5ZtaWaM/ZsxtosxtoMx9g1xeyVj7HXG2D7x7wpxO2OM/VK0fStjbFGG7dUyxrYwxl4Sn7cxxjaIdv6JMWYQtxvF5/vF/a0ZtrOcMfY8Y2y3eG1PycVryhj7lvj/vp0x9ixjzJQr15Qx9jhjrIcxtl22LelryBi7QTx+H2PshgzZeb/4f7+VMfYiY6xctu8O0c49jLHzZdtV1YVIdsr2/T/GGGeMVYvPs3Y9Q+Cc5/UfAFoABwBMAWAA8AmAWVm0pwHAIvGxFcBeALMA/ATA7eL22wH8j/j4IgCvAmAAlgPYkGF7vw3gGQAvic//DOBa8fHDAG4TH38ZwMPi42sB/CnDdv4RwBfFxwYA5bl2TQE0ATgEoER2LW/MlWsK4HQAiwBsl21L6hoCqARwUPy7QnxckQE7zwOgEx//j8zOWeJ33gigTdQCbSZ0IZKd4vZmAGsg1BRVZ/t6htiWiS+Cyh/iUwCskT2/A8Ad2bZLZs/fAZwLYA+ABnFbA4A94uNHAFwnOz54XAZsmwRgLYCzAbwkfhj7ZF+s4LUVP8CniI914nEsQ3baRCFlYdtz6ppCEPwO8curE6/p+bl0TQG0hglpUtcQwHUAHpFtDzlOLTvD9l0J4Gnxccj3XbqmmdKFSHYCeB7AfACHMS74Wb2e0p9CCOlIXzKJTnFb1hFv0RcC2ACgjnPeBQDi37XiYdm0/xcAvgsgID6vAjDEOfdFsCVop7h/WDw+E0wB0AvgD2L46feMMQty7Jpyzo8BeADAUQBdEK7RZuTmNZVI9hrmwvftJgjeMmLYkxU7GWOXATjGOf8kbFdO2FkIgs8ibMt66hFjrBTACwC+yTkfiXVohG2q288YuwRAD+d8c4K2ZPM66yDcOv+Wc74QgANC+CEa2bqmFQAuhxBaaARgAXBhDFty8rMrEs22rNrMGFsNwAfgaWlTFHsybidjzAxgNYD/irQ7ij0ZtbMQBL8TQsxMYhKA41myBQDAGNNDEPunOed/FTd3M8YaxP0NAHrE7dmy/1QAlzHGDgN4DkJY5xcAyhlj0nB7uS1BO8X9ZQAGMmCndO5OzvkG8fnzEH4Acu2argJwiHPeyzn3AvgrgBXIzd1jTDYAAAHDSURBVGsqkew1zNr3TVzQvATA57gY/8gxO6dC+LH/RPxeTQLwEWOsPlfsLATB/xBAu5gJYYCw+PWPbBnDGGMAHgOwi3P+M9mufwCQVuBvgBDbl7Z/XlzFXw5gWLrFVhPO+R2c80mc81YI12wd5/xzAP4N4Ooodkr2Xy0enxHPjnN+AkAHY2yGuOkcADuRY9cUQihnOWPMLH4OJDtz7prKSPYargFwHmOsQryjOU/cpiqMsQsAfA/AZZxzZ5j914oZT20A2gFsRBZ0gXO+jXNeyzlvFb9XnRASOE4gV66nWosDmfwDYQV8L4RV+dVZtuU0CLdkWwF8LP65CEJsdi2AfeLfleLxDMCvRdu3AViSBZvPxHiWzhQIX5j9AP4CwChuN4nP94v7p2TYxgUANonX9W8QMhpy7poCuBPAbgDbATwJIXskJ64pgGchrC14IYjRzalcQwgx9P3iny9kyM79EGLd0nfqYdnxq0U79wC4ULZdVV2IZGfY/sMYX7TN2vWU/6FKW4IgiCKhEEI6BEEQRAKQ4BMEQRQJJPgEQRBFAgk+QRBEkUCCTxAEUSSQ4BMEQRQJJPgEQRBFAgk+QRBEkfD/ATyGJJoJE7qrAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(1440), temp[:1440])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "On this plot, you can see daily periodicity, especially evident for the last 4 days. We can also note that this ten-days period must be \n",
    "coming from a fairly cold winter month.\n",
    "\n",
    "If we were trying to predict average temperature for the next month given a few month of past data, the problem would be easy, due to the \n",
    "reliable year-scale periodicity of the data. But looking at the data over a scale of days, the temperature looks a lot more chaotic. So is \n",
    "this timeseries predictable at a daily scale? Let's find out.\n",
    "\n",
    "在这张图中，你可以看到每天的周期性变化，尤其是最后 4 天特别明显。另外请注意，这10天一定是来自于很冷的冬季月份。 \n",
    "\n",
    "如果你想根据过去几个月的数据来预测下个月的平均温度，那么问题很简单，因为数据具有可靠的年度周期性。但从几天的数据来看，温度看起来更混乱一些。以天作为观察尺度，这个时间序列是可以预测的吗？我们来寻找这个问题的答案。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the data\n",
    "\n",
    "\n",
    "The exact formulation of our problem will be the following: given data going as far back as `lookback` timesteps (a timestep is 10 minutes) \n",
    "and sampled every `steps` timesteps, can we predict the temperature in `delay` timesteps?\n",
    "\n",
    "We will use the following parameter values:\n",
    "\n",
    "* `lookback = 720`, i.e. our observations will go back 5 days.\n",
    "* `steps = 6`, i.e. our observations will be sampled at one data point per hour.\n",
    "* `delay = 144`, i.e. our targets will be 24 hours in the future.\n",
    "\n",
    "To get started, we need to do two things:\n",
    "\n",
    "* Preprocess the data to a format a neural network can ingest. This is easy: the data is already numerical, so we don't need to do any \n",
    "vectorization. However each timeseries in the data is on a different scale (e.g. temperature is typically between -20 and +30, but \n",
    "pressure, measured in mbar, is around 1000). So we will normalize each timeseries independently so that they all take small values on a \n",
    "similar scale.\n",
    "* Write a Python generator that takes our current array of float data and yields batches of data from the recent past, alongside with a \n",
    "target temperature in the future. Since the samples in our dataset are highly redundant (e.g. sample `N` and sample `N + 1` will have most \n",
    "of their timesteps in common), it would be very wasteful to explicitly allocate every sample. Instead, we will generate the samples on the \n",
    "fly using the original data.\n",
    "\n",
    "We preprocess the data by subtracting the mean of each timeseries and dividing by the standard deviation. We plan on using the first \n",
    "200,000 timesteps as training data, so we compute the mean and standard deviation only on this fraction of the data:\n",
    "## 准备数据\n",
    "这个问题的确切表述如下：一个时间步是 10 分钟，每 steps 个时间步采样一次数据，给定过去 lookback 个时间步之内的数据，能否预测 delay 个时间步之后的温度？用到的参数值 如下。\n",
    "\n",
    "* lookback = 720：给定过去 5 天内的观测数据。\n",
    "* steps = 6：观测数据的采样频率是每小时一个数据点。\n",
    "* delay = 144：目标是未来 24 小时之后的数据。 开始之前，你需要完成以下两件事。\n",
    "* 将数据预处理为神经网络可以处理的格式。这很简单。数据已经是数值型的，所以不需要做向量化。但数据中的每个时间序列位于不同的范围（比如温度通道位于 -20 到 +30 之间，但气压大约在 1000 毫巴上下）。你需要对每个时间序列分别做标准化，让它们在相似的范围内都取较小的值。\n",
    "* 编写一个 Python 生成器，以当前的浮点数数组作为输入，并从最近的数据中生成数据批量，同时生成未来的目标温度。因为数据集中的样本是高度冗余的（对于第 N 个样本和第 N+1 个样本，大部分时间步都是相同的），所以显式地保存每个样本是一种浪费。相反，我们将使用原始数据即时生成样本。\n",
    " \n",
    "预处理数据的方法是，将每个时间序列减去其平均值，然后除以其标准差。我们将使用前200 000 个时间步作为训练数据，所以只对这部分数据计算平均值和标准差。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean = float_data[:200000].mean(axis=0)\n",
    "float_data -= mean\n",
    "std = float_data[:200000].std(axis=0)\n",
    "float_data /= std"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Now here is the data generator that we will use. It yields a tuple `(samples, targets)` where `samples` is one batch of input data and \n",
    "`targets` is the corresponding array of target temperatures. It takes the following arguments:\n",
    "\n",
    "* `data`: The original array of floating point data, which we just normalized in the code snippet above.\n",
    "* `lookback`: How many timesteps back should our input data go.\n",
    "* `delay`: How many timesteps in the future should our target be.\n",
    "* `min_index` and `max_index`: Indices in the `data` array that delimit which timesteps to draw from. This is useful for keeping a segment \n",
    "of the data for validation and another one for testing.\n",
    "* `shuffle`: Whether to shuffle our samples or draw them in chronological order.\n",
    "* `batch_size`: The number of samples per batch.\n",
    "* `step`: The period, in timesteps, at which we sample data. We will set it 6 in order to draw one data point every hour.\n",
    "\n",
    "代码清单给出了将要用到的生成器。它生成了一个元组 (samples, targets)，其 中 samples 是输入数据的一个批量，targets 是对应的目标温度数组。生成器的参数如下。\n",
    "* data：浮点数数据组成的原始数组，在代码清单 6-32 中将其标准化。\n",
    "* lookback：输入数据应该包括过去多少个时间步。\n",
    "* delay：目标应该在未来多少个时间步之后。\n",
    "* min_index 和 max_index：data 数组中的索引，用于界定需要抽取哪些时间步。这有助于保存一部分数据用于验证、另一部分用于测试。\n",
    "* shuffle：是打乱样本，还是按顺序抽取样本。\n",
    "* batch_size：每个批量的样本数。\n",
    "* step：数据采样的周期（单位：时间步）。我们将其设为 6，为的是每小时抽取一个数据点。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generator(data, lookback, delay, min_index, max_index,\n",
    "              shuffle=False, batch_size=128, step=6):\n",
    "    if max_index is None:\n",
    "        max_index = len(data) - delay - 1\n",
    "    i = min_index + lookback\n",
    "    while 1:\n",
    "        if shuffle:\n",
    "            rows = np.random.randint(\n",
    "                min_index + lookback, max_index, size=batch_size)\n",
    "        else:\n",
    "            if i + batch_size >= max_index:\n",
    "                i = min_index + lookback\n",
    "            rows = np.arange(i, min(i + batch_size, max_index))\n",
    "            i += len(rows)\n",
    "\n",
    "        samples = np.zeros((len(rows),\n",
    "                           lookback // step,\n",
    "                           data.shape[-1]))\n",
    "        targets = np.zeros((len(rows),))\n",
    "        for j, row in enumerate(rows):\n",
    "            indices = range(rows[j] - lookback, rows[j], step)\n",
    "            samples[j] = data[indices]\n",
    "            targets[j] = data[rows[j] + delay][1]\n",
    "        yield samples, targets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Now let's use our abstract generator function to instantiate three generators, one for training, one for validation and one for testing. \n",
    "Each will look at different temporal segments of the original data: the training generator looks at the first 200,000 timesteps, the \n",
    "validation generator looks at the following 100,000, and the test generator looks at the remainder.\n",
    "\n",
    "下面，我们使用这个抽象的 generator 函数来实例化三个生成器：一个用于训练，一个用于验证，还有一个用于测试。每个生成器分别读取原始数据的不同时间段：训练生成器读取前200 000 个时间步，验证生成器读取随后的 100 000 个时间步，测试生成器读取剩下的时间步。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "lookback = 1440\n",
    "step = 6\n",
    "delay = 144\n",
    "batch_size = 128\n",
    "\n",
    "train_gen = generator(float_data,\n",
    "                      lookback=lookback,\n",
    "                      delay=delay,\n",
    "                      min_index=0,\n",
    "                      max_index=200000,\n",
    "                      shuffle=True,\n",
    "                      step=step, \n",
    "                      batch_size=batch_size)\n",
    "val_gen = generator(float_data,\n",
    "                    lookback=lookback,\n",
    "                    delay=delay,\n",
    "                    min_index=200001,\n",
    "                    max_index=300000,\n",
    "                    step=step,\n",
    "                    batch_size=batch_size)\n",
    "test_gen = generator(float_data,\n",
    "                     lookback=lookback,\n",
    "                     delay=delay,\n",
    "                     min_index=300001,\n",
    "                     max_index=None,\n",
    "                     step=step,\n",
    "                     batch_size=batch_size)\n",
    "\n",
    "# This is how many steps to draw from `val_gen`\n",
    "# in order to see the whole validation set:\n",
    "# 为了查看整个验证集，需要 从 val_gen 中抽取多少次\n",
    "val_steps = (300000 - 200001 - lookback) // batch_size\n",
    "\n",
    "# This is how many steps to draw from `test_gen`\n",
    "# in order to see the whole test set:\n",
    "# 为了查看整个测试集，需要从 test_gen 中抽取多少次\n",
    "test_steps = (len(float_data) - 300001 - lookback) // batch_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A common sense, non-machine learning baseline\n",
    "\n",
    "\n",
    "Before we start leveraging black-box deep learning models to solve our temperature prediction problem, let's try out a simple common-sense \n",
    "approach. It will serve as a sanity check, and it will establish a baseline that we will have to beat in order to demonstrate the \n",
    "usefulness of more advanced machine learning models. Such common-sense baselines can be very useful when approaching a new problem for \n",
    "which there is no known solution (yet). A classic example is that of unbalanced classification tasks, where some classes can be much more \n",
    "common than others. If your dataset contains 90% of instances of class A and 10% of instances of class B, then a common sense approach to \n",
    "the classification task would be to always predict \"A\" when presented with a new sample. Such a classifier would be 90% accurate overall, \n",
    "and any learning-based approach should therefore beat this 90% score in order to demonstrate usefulness. Sometimes such elementary \n",
    "baseline can prove surprisingly hard to beat.\n",
    "\n",
    "In our case, the temperature timeseries can safely be assumed to be continuous (the temperatures tomorrow are likely to be close to the \n",
    "temperatures today) as well as periodical with a daily period. Thus a common sense approach would be to always predict that the temperature \n",
    "24 hours from now will be equal to the temperature right now. Let's evaluate this approach, using the Mean Absolute Error metric (MAE). \n",
    "Mean Absolute Error is simply equal to:\n",
    "\n",
    "## 一种基于常识的、非机器学习的基准方法\n",
    "\n",
    "开始使用黑盒深度学习模型解决温度预测问题之前，我们先尝试一种基于常识的简单方法。 它可以作为合理性检查，还可以建立一个基准，更高级的机器学习模型需要打败这个基准才能 表现出其有效性。面对一个尚没有已知解决方案的新问题时，这种基于常识的基准方法很有用。 一个经典的例子就是不平衡的分类任务，其中某些类别比其他类别更常见。如果数据集中包含90% 的类别 A 实例和 10% 的类别 B 实例，那么分类任务的一种基于常识的方法就是对新样本 始终预测类别“A”。这种分类器的总体精度为 90%，因此任何基于学习的方法在精度高于 90% 时才能证明其有效性。有时候，这样基本的基准方法可能很难打败。\n",
    "\n",
    "本例中，我们可以放心地假设，温度时间序列是连续的（明天的温度很可能接近今天的温度），并且具有每天的周期性变化。因此，一种基于常识的方法就是始终预测 24 小时后的温度 等于现在的温度。我们使用平均绝对误差（MAE）指标来评估这种方法。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "np.mean(np.abs(preds - targets))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's our evaluation loop:\n",
    "\n",
    "下面是评估的循环代码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.289735972991\n"
     ]
    }
   ],
   "source": [
    "def evaluate_naive_method():\n",
    "    batch_maes = []\n",
    "    for step in range(val_steps):\n",
    "        samples, targets = next(val_gen)\n",
    "        preds = samples[:, -1, 1]\n",
    "        mae = np.mean(np.abs(preds - targets))\n",
    "        batch_maes.append(mae)\n",
    "    print(np.mean(batch_maes))\n",
    "    \n",
    "evaluate_naive_method()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It yields a MAE of 0.29. Since our temperature data has been normalized to be centered on 0 and have a standard deviation of one, this \n",
    "number is not immediately interpretable. It translates to an average absolute error of `0.29 * temperature_std` degrees Celsius, i.e. \n",
    "2.57˚C. That's a fairly large average absolute error -- now the game is to leverage our knowledge of deep learning to do better. \n",
    "\n",
    "得到的 MAE 为 0.29。因为温度数据被标准化成均值为 0、标准差为 1，所以无法直接对这个值进行解释。它转化成温度的平均绝对误差为0.29×temperature_std 摄氏度，即 2.57℃。绝对误差还是相当大的。接下来的任务是利用深度学习知识来改进结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A basic machine learning approach\n",
    "\n",
    "In the same way that it is useful to establish a common sense baseline before trying machine learning approaches, it is useful to try \n",
    "simple and cheap machine learning models (such as small densely-connected networks) before looking into complicated and computationally \n",
    "expensive models such as RNNs. This is the best way to make sure that any further complexity we throw at the problem later on is legitimate \n",
    "and delivers real benefits.\n",
    "\n",
    "Here is a simply fully-connected model in which we start by flattening the data, then run it through two `Dense` layers. Note the lack of \n",
    "activation function on the last `Dense` layer, which is typical for a regression problem. We use MAE as the loss. Since we are evaluating \n",
    "on the exact same data and with the exact same metric as with our common sense approach, the results will be directly comparable.\n",
    "\n",
    "## 一种基本的机器学习方法\n",
    "\n",
    "在尝试机器学习方法之前，建立一个基于常识的基准方法是很有用的；同样，在开始研究 复杂且计算代价很高的模型（比如 RNN）之前，尝试使用简单且计算代价低的机器学习模型也是很有用的，比如小型的密集连接网络。这可以保证进一步增加问题的复杂度是合理的，并且会带来真正的好处。\n",
    "\n",
    "下面的代码清单给出了一个密集连接模型，首先将数据展平，然后通过两个Dense 层并运行。 注意，最后一个 Dense 层没有使用激活函数，这对于回归问题是很常见的。我们使用 MAE 作 为损失。评估数据和评估指标都与常识方法完全相同，所以可以直接比较两种方法的结果。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "500/500 [==============================] - 15s 30ms/step - loss: 0.9271 - val_loss: 0.4560\n",
      "Epoch 2/20\n",
      "500/500 [==============================] - 13s 25ms/step - loss: 0.3413 - val_loss: 0.3067\n",
      "Epoch 3/20\n",
      "500/500 [==============================] - 13s 25ms/step - loss: 0.2850 - val_loss: 0.3491\n",
      "Epoch 4/20\n",
      "500/500 [==============================] - 13s 25ms/step - loss: 0.2651 - val_loss: 0.3085\n",
      "Epoch 5/20\n",
      "500/500 [==============================] - 13s 25ms/step - loss: 0.2547 - val_loss: 0.3041\n",
      "Epoch 6/20\n",
      "500/500 [==============================] - 13s 25ms/step - loss: 0.2488 - val_loss: 0.3594\n",
      "Epoch 7/20\n",
      "500/500 [==============================] - 13s 25ms/step - loss: 0.2419 - val_loss: 0.3169\n",
      "Epoch 8/20\n",
      "500/500 [==============================] - 13s 26ms/step - loss: 0.2360 - val_loss: 0.3223\n",
      "Epoch 9/20\n",
      "500/500 [==============================] - 13s 25ms/step - loss: 0.2286 - val_loss: 0.3235\n",
      "Epoch 10/20\n",
      "500/500 [==============================] - 13s 25ms/step - loss: 0.2257 - val_loss: 0.3144\n",
      "Epoch 11/20\n",
      "500/500 [==============================] - 13s 25ms/step - loss: 0.2210 - val_loss: 0.3662\n",
      "Epoch 12/20\n",
      "500/500 [==============================] - 13s 25ms/step - loss: 0.2185 - val_loss: 0.3404\n",
      "Epoch 13/20\n",
      "500/500 [==============================] - 13s 26ms/step - loss: 0.2151 - val_loss: 0.3560\n",
      "Epoch 14/20\n",
      "500/500 [==============================] - 13s 26ms/step - loss: 0.2134 - val_loss: 0.3339\n",
      "Epoch 15/20\n",
      "500/500 [==============================] - 13s 25ms/step - loss: 0.2093 - val_loss: 0.3196\n",
      "Epoch 16/20\n",
      "500/500 [==============================] - 13s 25ms/step - loss: 0.2077 - val_loss: 0.3328\n",
      "Epoch 17/20\n",
      "500/500 [==============================] - 13s 26ms/step - loss: 0.2065 - val_loss: 0.3285\n",
      "Epoch 18/20\n",
      "500/500 [==============================] - 13s 26ms/step - loss: 0.2048 - val_loss: 0.3366\n",
      "Epoch 19/20\n",
      "500/500 [==============================] - 13s 25ms/step - loss: 0.2032 - val_loss: 0.3241\n",
      "Epoch 20/20\n",
      "500/500 [==============================] - 13s 26ms/step - loss: 0.2004 - val_loss: 0.3510\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras import layers\n",
    "from keras.optimizers import RMSprop\n",
    "\n",
    "model = Sequential()\n",
    "model.add(layers.Flatten(input_shape=(lookback // step, float_data.shape[-1])))\n",
    "model.add(layers.Dense(32, activation='relu'))\n",
    "model.add(layers.Dense(1))\n",
    "\n",
    "model.compile(optimizer=RMSprop(), loss='mae')\n",
    "history = model.fit_generator(train_gen,\n",
    "                              steps_per_epoch=500,\n",
    "                              epochs=20,\n",
    "                              validation_data=val_gen,\n",
    "                              validation_steps=val_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's display the loss curves for validation and training:\n",
    "\n",
    "我们来显示验证和训练的损失曲线"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xt8FNX9//HXh7vcEVARJIBSFZBLTBGvgFqLN6jWC4hVvEBRqVZrK1VrlWq1atWi1J/Uam1BkNqvFi2VasVirQJBAQVEIqJEFAMKinhL8vn9cTZhCZtkkmx2k837+XjsY3dmzsx8dnb3M2fPmYu5OyIiklkapTsAERFJPiV3EZEMpOQuIpKBlNxFRDKQkruISAZSchcRyUBK7pKQmTU2s+1m1j2ZZdPJzA4ws6Qf+2tmx5vZ+rjhNWZ2dJSy1VjXg2Z2bXXnr2C5N5vZn5K9XEmfJukOQJLDzLbHDbYEvgKKYsM/dPeZVVmeuxcBrZNdtiFw9wOTsRwzuxg4192HxS374mQsWzKfknuGcPfS5BqrGV7s7s+VV97Mmrh7YSpiE5HUU7NMAxH72/2Ymc0ys8+Ac83scDN7xcy2mtkHZjbVzJrGyjcxMzezHrHhGbHp/zSzz8zsZTPrWdWyseknmtlbZrbNzO41s5fMbFw5cUeJ8Ydmlmdmn5jZ1Lh5G5vZ3Wa2xczeBkZUsH2uN7PZZcZNM7O7Yq8vNrPVsffzdqxWXd6y8s1sWOx1SzP7Syy2lcChCda7LrbclWY2Mjb+EOA+4OhYk9fmuG17Y9z8E2PvfYuZPWlmXaJsm8qY2fdi8Ww1s+fN7MC4adea2UYz+9TM3ox7r0PM7NXY+E1mdkfU9UktcHc9MuwBrAeOLzPuZuBr4FTCTn0P4NvAYYR/cL2At4BJsfJNAAd6xIZnAJuBHKAp8Bgwoxpl9wI+A0bFpl0FfAOMK+e9RInx70A7oAfwccl7ByYBK4FuQEdgYfjKJ1xPL2A70Cpu2R8BObHhU2NlDDgW+ALoH5t2PLA+bln5wLDY6zuBF4AOQBawqkzZs4Ausc/knFgMe8emXQy8UCbOGcCNsdcnxGIcCLQAfg88H2XbJHj/NwN/ir0+OBbHsbHP6NrYdm8K9AXeBfaJle0J9Iq9XgKMib1uAxyW7t9CQ36o5t6w/Nfdn3L3Ynf/wt2XuPsidy9093XAdGBoBfM/7u657v4NMJOQVKpa9hRgmbv/PTbtbsKOIKGIMd7q7tvcfT0hkZas6yzgbnfPd/ctwG0VrGcd8AZhpwPwHWCru+fGpj/l7us8eB74N5Cw07SMs4Cb3f0Td3+XUBuPX+8cd/8g9pk8Stgx50RYLsBY4EF3X+buXwKTgaFm1i2uTHnbpiKjgbnu/nzsM7oNaEvYyRYSdiR9Y01778S2HYSddG8z6+jun7n7oojvQ2qBknvDsiF+wMwOMrN/mNmHZvYpMAXoVMH8H8a93kHFnajlld03Pg53d0JNN6GIMUZaF6HGWZFHgTGx1+cQdkolcZxiZovM7GMz20qoNVe0rUp0qSgGMxtnZstjzR9bgYMiLhfC+ytdnrt/CnwCdI0rU5XPrLzlFhM+o67uvgb4CeFz+CjWzLdPrOgFQB9gjZktNrOTIr4PqQVK7g1L2cMAHyDUVg9w97bADYRmh9r0AaGZBAAzM3ZNRmXVJMYPgP3ihis7VPMx4PhYzXcUIdljZnsAjwO3EppM2gP/ihjHh+XFYGa9gPuBS4COseW+Gbfcyg7b3Eho6ilZXhtC88/7EeKqynIbET6z9wHcfYa7H0lokmlM2C64+xp3H01oevst8Dcza1HDWKSalNwbtjbANuBzMzsY+GEK1vk0kG1mp5pZE+AKoHMtxTgH+LGZdTWzjsA1FRV2903Af4GHgTXuvjY2qTnQDCgAiszsFOC4KsRwrZm1t3AewKS4aa0JCbyAsJ+7mFBzL7EJ6FbSgZzALOAiM+tvZs0JSfZFdy/3n1AVYh5pZsNi6/4poZ9kkZkdbGbDY+v7IvYoIryBH5hZp1hNf1vsvRXXMBapJiX3hu0nwPmEH+4DhJprrYol0LOBu4AtwP7Aa4Tj8pMd4/2EtvHXCZ19j0eY51FCB+mjcTFvBa4EniB0Sp5B2ElF8UvCP4j1wD+BP8ctdwUwFVgcK3MQEN9O/SywFthkZvHNKyXzP0NoHnkiNn93Qjt8jbj7SsI2v5+w4xkBjIy1vzcHbif0k3xI+KdwfWzWk4DVFo7GuhM4292/rmk8Uj0WmjxF0sPMGhOaAc5w9xfTHY9IplDNXVLOzEaYWbvYX/tfEI7AWJzmsEQyipK7pMNRwDrCX/sRwPfcvbxmGRGpBjXLiIhkINXcRUQyUNouHNapUyfv0aNHulYvIlIvLV26dLO7V3T4MJDG5N6jRw9yc3PTtXoRkXrJzCo70xpQs4yISEZSchcRyUBK7iIiGUh3YhJpIL755hvy8/P58ssv0x2KRNCiRQu6detG06blXVqoYkruIg1Efn4+bdq0oUePHoSLcUpd5e5s2bKF/Px8evbsWfkMCdSrZpmZM6FHD2jUKDzPrNItn0Uati+//JKOHTsqsdcDZkbHjh1r9C+r3tTcZ86ECRNgx44w/O67YRhgbI2vgyfSMCix1x81/azqTc39uut2JvYSO3aE8SIisqt6k9zfe69q40WkbtmyZQsDBw5k4MCB7LPPPnTt2rV0+Ouvo132/YILLmDNmjUVlpk2bRozk9Rme9RRR7Fs2bKkLCvV6k2zTPfuoSkm0XgRSb6ZM8M/4/feC7+zW26pWRNox44dSxPljTfeSOvWrbn66qt3KePuuDuNGiWudz788MOVrueyyy6rfpAZpN7U3G+5BVq23HVcy5ZhvIgkV0kf17vvgvvOPq7aOIghLy+Pfv36MXHiRLKzs/nggw+YMGECOTk59O3blylTppSWLalJFxYW0r59eyZPnsyAAQM4/PDD+eijjwC4/vrrueeee0rLT548mcGDB3PggQfyv//9D4DPP/+c73//+wwYMIAxY8aQk5NTaQ19xowZHHLIIfTr149rr70WgMLCQn7wgx+Ujp86dSoAd999N3369GHAgAGce+65Sd9mUdSb5D52LEyfDllZYBaep09XZ6pIbUh1H9eqVau46KKLeO211+jatSu33XYbubm5LF++nGeffZZVq1btNs+2bdsYOnQoy5cv5/DDD+ehhx5KuGx3Z/Hixdxxxx2lO4p7772XffbZh+XLlzN58mRee+21CuPLz8/n+uuvZ8GCBbz22mu89NJLPP300yxdupTNmzfz+uuv88Ybb3DeeecBcPvtt7Ns2TKWL1/OfffdV8OtUz2RknvszjlrzCzPzCYnmJ5lZv82sxVm9kLs7vFJN3YsrF8PxcXhWYldpHakuo9r//3359vf/nbp8KxZs8jOziY7O5vVq1cnTO577LEHJ554IgCHHnoo69evT7js008/fbcy//3vfxk9ejQAAwYMoG/fvhXGt2jRIo499lg6depE06ZNOeecc1i4cCEHHHAAa9as4YorrmD+/Pm0a9cOgL59+3Luuecyc+bMap+EVFOVJvfYPS6nAScCfYAxZtanTLE7gT+7e3/CDXtvTXagIpI65fVl1VYfV6tWrUpfr127lt/97nc8//zzrFixghEjRiQ83rtZs2alrxs3bkxhYWHCZTdv3ny3MlW9SVF55Tt27MiKFSs46qijmDp1Kj/84Q8BmD9/PhMnTmTx4sXk5ORQVFRUpfUlQ5Sa+2Agz93Xxe5kPhsYVaZMH8Jd5gEWJJguIvVIOvu4Pv30U9q0aUPbtm354IMPmD9/ftLXcdRRRzFnzhwAXn/99YT/DOINGTKEBQsWsGXLFgoLC5k9ezZDhw6loKAAd+fMM8/kpptu4tVXX6WoqIj8/HyOPfZY7rjjDgoKCthRto0rBaIcLdMV2BA3nA8cVqbMcuD7wO+A04A2ZtbR3bckJUoRSamSJs9kHi0TVXZ2Nn369KFfv3706tWLI488Munr+NGPfsR5551H//79yc7Opl+/fqVNKol069aNKVOmMGzYMNydU089lZNPPplXX32Viy66CHfHzPjNb35DYWEh55xzDp999hnFxcVcc801tGnTJunvoTKV3kPVzM4EvuvuF8eGfwAMdvcfxZXZF7gP6AksJCT6vu6+rcyyJgATALp3737ou4mObRSRWrF69WoOPvjgdIdRJxQWFlJYWEiLFi1Yu3YtJ5xwAmvXrqVJk7p1dHiiz8zMlrp7TmXzRnkn+cB+ccPdgI3xBdx9I3B6bMWtge+XTeyxctOB6QA5OTm6M7eIpMX27ds57rjjKCwsxN154IEH6lxir6ko72YJ0NvMegLvA6OBc+ILmFkn4GN3LwZ+DiQ+JklEpA5o3749S5cuTXcYtarSDlV3LwQmAfOB1cAcd19pZlPMbGSs2DBgjZm9BewN6NQiEZE0ivQ/xN3nAfPKjLsh7vXjwOPJDU1ERKqr3pyhKiIi0Sm5i4hkICV3EUmJYcOG7XZC0j333MOll15a4XytW7cGYOPGjZxxxhnlLjs3N7fC5dxzzz27nEx00kknsXXr1iihV+jGG2/kzjvvrPFykk3JXURSYsyYMcyePXuXcbNnz2bMmDGR5t933315/PHqd+2VTe7z5s2jffv21V5eXafkLiIpccYZZ/D000/z1VdfAbB+/Xo2btzIUUcdVXrceXZ2Nocccgh///vfd5t//fr19OvXD4AvvviC0aNH079/f84++2y++OKL0nKXXHJJ6eWCf/nLXwIwdepUNm7cyPDhwxk+fDgAPXr0YPPmzQDcdddd9OvXj379+pVeLnj9+vUcfPDBjB8/nr59+3LCCSfssp5Eli1bxpAhQ+jfvz+nnXYan3zySen6+/TpQ//+/UsvWPaf//yn9GYlgwYN4rPPPqv2tk0ks47aF5FIfvxjSPYNhgYOhFheTKhjx44MHjyYZ555hlGjRjF79mzOPvtszIwWLVrwxBNP0LZtWzZv3syQIUMYOXJkufcRvf/++2nZsiUrVqxgxYoVZGdnl0675ZZb2HPPPSkqKuK4445jxYoVXH755dx1110sWLCATp067bKspUuX8vDDD7No0SLcncMOO4yhQ4fSoUMH1q5dy6xZs/jDH/7AWWedxd/+9rcKr89+3nnnce+99zJ06FBuuOEGbrrpJu655x5uu+023nnnHZo3b17aFHTnnXcybdo0jjzySLZv306LFi2qsLUrp5q7iKRMfNNMfJOMu3PttdfSv39/jj/+eN5//302bdpU7nIWLlxYmmT79+9P//79S6fNmTOH7OxsBg0axMqVKyu9KNh///tfTjvtNFq1akXr1q05/fTTefHFFwHo2bMnAwcOBCq+rDCE68tv3bqVoUOHAnD++eezcOHC0hjHjh3LjBkzSs+EPfLII7nqqquYOnUqW7duTfoZsqq5izRAFdWwa9P3vvc9rrrqKl599VW++OKL0hr3zJkzKSgoYOnSpTRt2pQePXokvMxvvES1+nfeeYc777yTJUuW0KFDB8aNG1fpciq6vlbJ5YIhXDK4smaZ8vzjH/9g4cKFzJ07l1/96lesXLmSyZMnc/LJJzNv3jyGDBnCc889x0EHHVSt5SeimruIpEzr1q0ZNmwYF1544S4dqdu2bWOvvfaiadOmLFiwgMouKnjMMceU3gT7jTfeYMWKFUC4XHCrVq1o164dmzZt4p///GfpPG3atEnYrn3MMcfw5JNPsmPHDj7//HOeeOIJjj766Cq/t3bt2tGhQ4fSWv9f/vIXhg4dSnFxMRs2bGD48OHcfvvtbN26le3bt/P2229zyCGHcM0115CTk8Obb75Z5XVWRDV3EUmpMWPGcPrpp+9y5MzYsWM59dRTycnJYeDAgZXWYC+55BIuuOAC+vfvz8CBAxk8eDAQ7qo0aNAg+vbtu9vlgidMmMCJJ55Ily5dWLBgQen47Oxsxo0bV7qMiy++mEGDBlXYBFOeRx55hIkTJ7Jjxw569erFww8/TFFREeeeey7btm3D3bnyyitp3749v/jFL1iwYAGNGzemT58+pXeVSpZKL/lbW3Jycryy41JFJHl0yd/6pyaX/FWzjIhIBlJyFxHJQEruIg1Iupphpepq+lkpuYs0EC1atGDLli1K8PWAu7Nly5Yandiko2VEGohu3bqRn59PQUFBukORCFq0aEG3bt2qPb+Su0gD0bRpU3r27JnuMCRF1CwjIpKBlNxFRDJQpORuZiPMbI2Z5ZnZ5ATTu5vZAjN7zcxWmNlJyQ9VRESiqjS5m1ljYBpwItAHGGNmfcoUux6Y4+6DgNHA75MdqIiIRBel5j4YyHP3de7+NTAbGFWmjANtY6/bARuTF6KIiFRVlOTeFdgQN5wfGxfvRuBcM8sH5gE/SrQgM5tgZrlmlqvDsUREak+U5J7oVihlz4IYA/zJ3bsBJwF/MbPdlu3u0909x91zOnfuXPVoRUQkkijJPR/YL264G7s3u1wEzAFw95eBFkAnREQkLaIk9yVAbzPraWbNCB2mc8uUeQ84DsDMDiYkd7W7iIikSaXJ3d0LgUnAfGA14aiYlWY2xcxGxor9BBhvZsuBWcA41wUsRETSJtLlB9x9HqGjNH7cDXGvVwFHlp1PRETSQ2eoiohkICV3EZEMpOQuIpKBlNxFRDKQkruISAZSchcRyUBK7iIiGUjJXUQkAym5i4hkICV3EZEMpOQuIpKBlNxFRDKQkruISAZSchcRyUBK7iIiGUjJXUQkAym5i4hkICV3EZEMFCm5m9kIM1tjZnlmNjnB9LvNbFns8ZaZbU1+qCIiElWl91A1s8bANOA7QD6wxMzmxu6bCoC7XxlX/kfAoFqIVUREIopScx8M5Ln7Onf/GpgNjKqg/BhgVjKCExGR6omS3LsCG+KG82PjdmNmWUBP4Plypk8ws1wzyy0oKKhqrCIiElGU5G4Jxnk5ZUcDj7t7UaKJ7j7d3XPcPadz585RYxQRkSqKktzzgf3ihrsBG8spOxo1yYiIpF2U5L4E6G1mPc2sGSGBzy1byMwOBDoALyc3RBERqapKk7u7FwKTgPnAamCOu680sylmNjKu6BhgtruX12QjIiIpUumhkADuPg+YV2bcDWWGb0xeWCIiUhM6Q1VEJAMpuYuIZCAldxGRDKTkLiKSgZTcRUQykJK7iEgGUnIXEclASu4iIhlIyV1EJAMpuYuIZCAldxGRDKTkLiKSgZTcRUQykJK7iEgGUnIXEclASu4iIhlIyV1EJAMpuYuIZKBIyd3MRpjZGjPLM7PJ5ZQ5y8xWmdlKM3s0uWGKiEhVVHoPVTNrDEwDvgPkA0vMbK67r4or0xv4OXCku39iZnvVVsAiIlK5KDX3wUCeu69z96+B2cCoMmXGA9Pc/RMAd/8ouWGKiEhVREnuXYENccP5sXHxvgV8y8xeMrNXzGxEogWZ2QQzyzWz3IKCgupFLCIilYqS3C3BOC8z3AToDQwDxgAPmln73WZyn+7uOe6e07lz56rGKiIiEUVJ7vnAfnHD3YCNCcr83d2/cfd3gDWEZC8iImkQJbkvAXqbWU8zawaMBuaWKfMkMBzAzDoRmmnWJTNQERGJrtLk7u6FwCRgPrAamOPuK81sipmNjBWbD2wxs1XAAuCn7r6ltoIWEZGKmXvZ5vPUyMnJ8dzc3LSsW0SkvjKzpe6eU1k5naEqIpKBlNxFRDKQkruISAZSchcRyUBK7iIiGUjJXUQkAym5i4hkICV3EZEMpOQuIpKBlNxFRDKQkruISAZSchcRyUBK7iIiGUjJXUQkAym5i4hkICV3EZEMpOQuIpKBlNxFRDJQpORuZiPMbI2Z5ZnZ5ATTx5lZgZktiz0uTn6oIiISVZPKCphZY2Aa8B0gH1hiZnPdfVWZoo+5+6RaiFFERKooSs19MJDn7uvc/WtgNjCqdsMSEZGaiJLcuwIb4obzY+PK+r6ZrTCzx81sv0QLMrMJZpZrZrkFBQXVCFdERKKIktwtwTgvM/wU0MPd+wPPAY8kWpC7T3f3HHfP6dy5c9UiFRGRyKIk93wgvibeDdgYX8Ddt7j7V7HBPwCHJic8ERGpjijJfQnQ28x6mlkzYDQwN76AmXWJGxwJrE5eiCIiUlWVHi3j7oVmNgmYDzQGHnL3lWY2Bch197nA5WY2EigEPgbG1WLMIiJSCXMv23yeGjk5OZ6bm5uWdYuI1FdmttTdcyorpzNURUQykJK7iEgGUnIXEclASu4iIhlIyV1EJAPVu+S+YQM8/HC6oxARqdvqXXL/y1/gwgth7dp0RyIiUnfVu+R+wQXQuDE8+GC6IxERqbvqXXLv0gVOOQX+9Cf4+ut0RyMiUjfVu+QOMH48fPQRPPVUuiMREamb6mVyHzECunWDP/wh3ZGIiNRN9TK5N24cOlX/9S9Yvz7d0YiI1D31MrlDSO4ADz2U3jhEROqiepvcs7Lgu98Nyb2wMN3RiIjULfU2uUPoWH3/fXjmmXRHIiJSt9Tr5H7qqbD33upYFREpq14n96ZNYdw4+Mc/YOPGSouLiDQY9Tq5A1x8MRQV6XozIiLxIiV3MxthZmvMLM/MJldQ7gwzczOr9BZQyXLAATB8OPzxj1BcnKq1iojUbZUmdzNrDEwDTgT6AGPMrE+Ccm2Ay4FFyQ6yMuPHwzvvwL//neo1i4jUTVFq7oOBPHdf5+5fA7OBUQnK/Qq4HfgyifFFctppsOee6lgVESkRJbl3BTbEDefHxpUys0HAfu7+dBJji6xFCzjvPHjySSgoSEcEIiJ1S5TkbgnGeelEs0bA3cBPKl2Q2QQzyzWz3IIkZ+Hx4+Gbb+CRR5K6WBGReilKcs8H9osb7gbEH3jYBugHvGBm64EhwNxEnaruPt3dc9w9p3PnztWPOoE+feCII8J13t0rLy8iksmiJPclQG8z62lmzYDRwNySie6+zd07uXsPd+8BvAKMdPfcWom4AuPHw5o18OKLqV6ziEjdUmlyd/dCYBIwH1gNzHH3lWY2xcxG1naAVXHmmdC2rTpWRUTM09SGkZOT47m5ya/cX3ppOKFp40bo0CHpixcRSSszW+rulZ5LVO/PUC1r/Hj48kuYMSPdkYiIpE/GJfdBg+DQQ0PTjDpWRaShyrjkDqH2/vrrsHhxuiMREUmPjEzuY8ZAy5bqWJXkmzMHevaEK66AvLx0RyNSvoxM7m3bwujRMHs2fPZZuqOpe75M+QUi6j93+O1v4eyzwz18778fvvUtGDUKFixQE6DUPRmZ3CE0zXz+Ocyale5I6g53+OlPw87v178Ol0qWyhUVhZr61VfDGWfAG2/Au+/CddfB//4Hxx4b+nr+9Kfa33EWF8OGDdqZSATunpbHoYce6rWpuNi9Xz/3nJxaXU29UVjofvHF7hC2C7gPH+6en5/uyOq2HTvcTzstbK8rr3QvKtp9+oMP7tyme+3l/stfun/4YfJiePtt9wcecD/rLPeOHcN6evd2v+UW9/feS956JDWKi8OjuoBcj5BjMza5u7v/7nfhHb72WvKWuWOH+9KlyVteKnz1lfvZZ4dtcd114Yv10EPurVq577mn+5NPpjvCuqmgwP3ww93N3O+5p+KyxcXuzz3nfsopYTs3a+Z+/vnV++4VFLg/9pj7+PHuPXuG5YH7vvuGZd55p/vQoWGcmft3vuP+6KPhuyl119dfu8+c6T5okPszz1R/OUru7r5li3vz5u6XXhqGZ8xwz8oKP4isrDBcFcuXu/fpE7bamDFh+XXd55+7n3RSiPn223edtmaNe3Z2mHbJJUoO8fLyQu24RQv3xx+v2rxr1rhfdpl7y5Zh2w4bFnaghYWJy3/+efix//Sn4YdfkszbtnUfNcr93nvdV6/evbb39tvhX0JW1s7yEya4v/xyzWqGklyffeZ+993u3buHz+mgg9znzav+8pTcY8aOdW/Xzv2Pf9z5Yyt5tGwZLcEXF4d/Ac2bu3fp4n755e5NmoSa1Pz5tf8eqmvbNvdjjgk7s+nTE5f56iv3n/wkbI8+fdxXrEhtjHXRokXunTuHfzUvvVT95Xz8sfsdd+z8UffqFf4BfPyx+yuvuN98c0j8zZqF6U2bhuGbbw4J+ptvoq2nqMj9+efdzztv53f8wAPdb71VzW7ptHGj+89/7t6+ffhMjj7afe7c3Zv2qkrJPeaFF8K7LGmrLPvIyqp4/k2bdtZ8Tz3V/aOPwvjcXPeDDw7jL73Uffv2Wn8rVVJQ4H7ooWEnNGtW5eXnz3ffe++wA7v33oZb85s7132PPUJzyJtvJmeZ33zjPmeO+xFH7P79GzjQ/eqrQ809Gd+hTz8NFZmjjw7Lb9TIfcQI99mz3b/4oubLl8qtWuV+4YVhp92okfsZZ4SdebIouccUF7t/61uJE3tJm2V54hPeffftnvB27AidbCUdXC+/XLvvJar8/FALb9HC/emno89XdkdWUFB7MdZFv/99+DHm5CS3QzTeokWh32P27J0Vhdqydq379de777df+Ezbtw/Nb6+8UvPaY32wbZv7q6+GHeutt7pfdFH4Z7T//u4nnBC2zVNPhe99TRUXu//nPzv7XPbYI1T68vJqvuyyoib3jLtwWCJ33AE/+1niaVlZsH79ruO++ioc5vbb30LfvuFwykMOKX/5CxbAuHGQnw/XXgu/+AU0a5as6Ktm3To4/njYvBmeegqGDq3a/O4wdWrYXh07wp//HJaXyYqLw+d9221wyinh/IhWrdIdVfIUF4fv6MMPw//9H3zxRbio3tFHwzHHhMegQdCkSe2sv7AwnDH+v//Byy/DsmXQvHn4fkV5tG0LluCWQe7hzmt5efD22zufS15v3rxr+b32ggMOgK5d4a23QkzFxWFaVhYcdhgMHhyes7PDiZCVKSqCJ54IOWbxYujUCSZNgssuC69rQ9QLh2V8zd097JkbNw5NFJW1ub/55s5Orcsui97JuHWr+7hxYb7sbPc33kj++6jM66+HPoE993RfsqRmy1q2LHT8mLn/7GehbT4Tffml+znnhM9t4sTo7dz11dat7n/+c6jF9u6987fQunWozd58s/vChTVrwtm8OdSqjQj0AAAOE0lEQVSIr7021JRbtdq5ni5dwr/Ck05yP+ww9wMOcO/QIXzPyvt33aRJOMT04IPdjzoqzDtgQIi57L/wrCz3Y48NRxr95jehM3zZstBcVdb27eG93nmn+5ln7uyYhpAvBgwIHdQPPhh+W/Ed4p9/Hv7N9+oVyh9wgPv994fxtQ3V3Hd15pnwzDPhRtobNkD37nDLLTB2bJjuDg89BJdfDnvsEV6PrMbV6p94AiZMCGfG/vrX8OMfQ6MUnCq2ZAmMGBFqRM8+G/5x1NSOHXDllTB9OuTkhH8wBxxQ8+W6h9rcN9/sfHz9dfRhM+jVK8TSvHn149i6Ndxc/YUX4NZb4ZprEtcQM9mHH4ab2yxcGB6vvx4+n2bNQg22pGZ/+OHQps3u8xcVwapVoUZeUjN/660wrXHj8I/g8MPD44gjwu8u0TYuKoJPPoEtW3Z/fPzxrsNbt0KXLuHz33//nc89etTs+wCwaVOogS9eDIsWhedt28K0Vq3C76B37/A737IFhgwJJwaOGhXebypErbk3mOT+r3/Bd78bEtTo0btO++STkJAffxyOOy40Rey7b/XXtWlTOEO2pFnkkUfC377a8sILcOqp0LkzPPdcSHzJ9Le/7bxH7X33hZuRl/2BFhaGv8ibNoXHhx/ufF12ePPm5Jxh2ahRuM7LQQfBgQfu+ty5c8WJ+r334KSTQiJ66CE499yax5MJPv4YXnppZ7JfujQk3saNQ1PF0UeHq66+9VZI5osWwaefhnk7d941kefkRGvaqMuKi0MTT0miX7w4nKF8/PEhqR95ZOorBEruZRQXh717r17w73/vHP/ii6H2/sEHoSZ/9dXJqWm7h9PRr7giDE+dCuefn/wvwtNPh1Pi998/1NhrslOqyIYNIQEuXBiSYocO0RL2HnvAPvvA3nuHxz77hLbI5s1D7bBp052PssPljSsqCu2qb74Zbqv45psh2cSf+t++feKkv//+oaZ58smwfXuogR17bO1ss0ywfTu88srOZP/KK6FPqlGj0A91xBE7E/r++ze8fz7poOSewM03h87OvLxQk54yJST0Xr3g0Ufh299O/jrXrw9JfeHC8Ndt+vTQsZMMs2aFWvTAgaHJqWPH5Cy3PEVFoalp6tTwFz0+YSd6vffe0Lp1an7wxcWhNl6S7OOfN8bdzr1x4/DYay+YN6/ijnLZ3Vdfhe3aq1fiZhqpfUruCbz/fmjz+8EPQk3v5ZfDUS4lyaq2FBfD3XeHI2natQuvBw7ceTRA06ZVX+b06TBxYmgPnTs3HFEgiX36afi8S5L91q2hfb1bt3RHJlJ1SU3uZjYC+B3QGHjQ3W8rM30icBlQBGwHJrj7qoqWmY7kDqGT9KmnQjJ84IHd299r0xtvhB3LsmW7jm/TJjRVVHZIWEmZxx4Lyenkk+Gvfw1NHyLSMERN7pUe2WpmjYFpwHeAfGCJmc0tk7wfdff/Fys/ErgLGFGtyGvZTTeFI2ZuvDH0rqdSv36hY+all+Cjj3Y/KmDz5vC8dm14LumlT2T06NDxW51av4hkviinLQwG8tx9HYCZzQZGAaXJ3d0/jSvfCqizV5suue52ujRrBsOHRytbWLj7YWBbtoTOyNGjU3folYjUP1GSe1dgQ9xwPnBY2UJmdhlwFdAMSHj8gZlNACYAdO/evaqxNjhNmoSOv2R1wIpIwxHloL9ExzrsVjN392nuvj9wDXB9ogW5+3R3z3H3nM6dO1ct0jpg5szQlNOoUXieOTPdEYmIJBal5p4P7Bc33A3YWE5ZgNnA/TUJqi6aOTOc6LRjRxh+990wDDvPchURqSui1NyXAL3NrKeZNQNGA3PjC5hZ77jBk4G1yQuxbrjuup2JvcSOHWG8iEhdU2nN3d0LzWwSMJ9wKORD7r7SzKYQLmAzF5hkZscD3wCfAOfXZtDp8N57VRsvIpJOkS7y6e7zgHllxt0Q9/qKJMdV53TvHppiEo0XEalrUnC9wsxwyy27XwSpZcswPip1yIpIqii5RzR2bDjlPysrXCslKysMR+1MLemQfffdcIGtkg5ZJXgRqQ0N6toy6dSjR+JmnUR3ghIRKU/Uyw+o5p4iyeiQVbOOiESl5J4i5XW8Ru2QVbOOiFSFknuK1LRDNhnH2avmL9JwKLmnSE07ZGvarKOav0jDouSeQmPHhs7T4uLwXJXLFtS0WUc1f5GGRcm9nqhps05dqPlr5yCSOkru9URNm3XSXfNXs5BIaim51yM1adZJd81fzUIiqaXk3kCku+avZiGR1FJyb0DSWfPPhGYh7RykPlFyl0hqWvOv781Cyeoz0A5CUsbd0/I49NBDXRqWGTPcs7LczcLzjBnR583Kcg9pdddHVla0+c0Sz2+WmvW7h/fbsuWu87dsWbXtUJNtmIz5Jf0I99GoNMcquUu9UNPEmO6dQzJiqOk2SMbORdIvanJXs4zUC+luFqppnwGkv2mpLhyxlO75G5Qoe4DaeKjmLqlWkyaJZNR60/3voabzp/ufg5q1ApLZLAOMANYAecDkBNOvAlYBK4B/A1mVLVPJXeqbZCSGdDYtNfT5M2Hn4p7E5E64KfbbQC+gGbAc6FOmzHCgZez1JcBjlS1XyV0aonT+e6jp/On+55DuTvF071xKJDO5Hw7Mjxv+OfDzCsoPAl6qbLlK7iJVl85mhXQnx/rerJWMI67coyf3KB2qXYENccP5sXHluQj4Z6IJZjbBzHLNLLegoCDCqkUkXk1ORKvp/DXtlE73/DXtFE/3WdpVFSW5W4JxnrCg2blADnBHounuPt3dc9w9p3PnztGjFJG0q+kRS+mev77vXKqssqo9EZtlgOOB1cBeUf4yqFlGRFItnc1aqW5zt1C2fGbWBHgLOA54H1gCnOPuK+PKDAIeB0a4+9ooO5WcnBzPzc2t0o5IRKQ+mzkznFfw3nuhxn7LLVVvWjOzpe6eU1m5JpUVcPdCM5sEzCccOfOQu680symEPchcQjNMa+CvZgbwnruPrFrIIiKZbezYqifz6qo0uQO4+zxgXplxN8S9Pj7JcYmISA3o8gMiIhlIyV1EJAMpuYuIZCAldxGRDFTpoZC1tmKzAuDdas7eCdicxHCSTfHVjOKruboeo+Krvix3r/Qs0LQl95ows9wox3mmi+KrGcVXc3U9RsVX+9QsIyKSgZTcRUQyUH1N7tPTHUAlFF/NKL6aq+sxKr5aVi/b3EVEpGL1teYuIiIVUHIXEclAdTq5m9kIM1tjZnlmNjnB9OZm9lhs+iIz65HC2PYzswVmttrMVprZFQnKDDOzbWa2LPa4IdGyajHG9Wb2emzdu11f2YKpse23wsyyUxjbgXHbZZmZfWpmPy5TJuXbz8weMrOPzOyNuHF7mtmzZrY29tyhnHnPj5VZa2bnpyi2O8zszdjn94SZtS9n3gq/C7Uc441m9n7c53hSOfNW+Huvxfgei4ttvZktK2felGzDpIly0fd0PIh2Y+5Lgf8Xez2aCDfmTmJ8XYDs2Os2hGvel41vGPB0GrfheqBTBdNPItwS0YAhwKI0ftYfEk7OSOv2A44BsoE34sbdDkyOvZ4M/CbBfHsC62LPHWKvO6QgthOAJrHXv0kUW5TvQi3HeCNwdYTvQIW/99qKr8z03wI3pHMbJutRl2vug4E8d1/n7l8Ds4FRZcqMAh6JvX4cOM5iF5Svbe7+gbu/Gnv9GeEuVBXdW7YuGgX82YNXgPZm1iUNcRwHvO3u1T1jOWncfSHwcZnR8d+zR4DvJZj1u8Cz7v6xu38CPAuMqO3Y3P1f7l4YG3wF6JbMdVZVOdsviii/9xqrKL5Y7jgLmJXs9aZDXU7uUW7MXVom9gXfBnRMSXRxYs1Bg4BFCSYfbmbLzeyfZtY3pYGFe93+y8yWmtmEBNOrevPz2jKa8n9Q6dx+JfZ29w8g7NSBvRKUqQvb8kLKuTk9lX8XatukWNPRQ+U0a9WF7Xc0sMnLv5tcurdhldTl5B7lxtyRb95dW8ysNfA34Mfu/mmZya8SmhoGAPcCT6YyNuBId88GTgQuM7NjykyvC9uvGTAS+GuCyeneflWR1m1pZtcBhcDMcopU9l2oTfcD+wMDgQ8ITR9lpf27CIyh4lp7OrdhldXl5J4P7Bc33A3YWF4ZC/d6bUf1/hJWi5k1JST2me7+f2Wnu/un7r499noe0NTMOqUqPnffGHv+CHiC8Nc3XpRtXNtOBF51901lJ6R7+8XZVNJcFXv+KEGZtG3LWOftKcBYjzUOlxXhu1Br3H2Tuxe5ezHwh3LWndbvYix/nA48Vl6ZdG7D6qjLyX0J0NvMesZqd6OBuWXKzAVKjko4A3i+vC93ssXa5/4IrHb3u8ops09JH4CZDSZs7y0piq+VmbUpeU3oeHujTLG5wHmxo2aGANtKmh9SqNzaUjq3Xxnx37Pzgb8nKDMfOMHMOsSaHU6IjatVZjYCuAYY6e47yikT5btQmzHG9+OcVs66o/zea9PxwJvunp9oYrq3YbWku0e3ogfhaI63CL3o18XGTSF8kQFaEP7O5wGLgV4pjO0owt/GFcCy2OMkYCIwMVZmErCS0PP/CnBECuPrFVvv8lgMJdsvPj4DpsW27+tAToo/35aEZN0ublxatx9hR/MB8A2hNnkRoR/n38Da2POesbI5wINx814Y+y7mARekKLY8Qlt1yXew5OixfYF5FX0XUrj9/hL7fq0gJOwuZWOMDe/2e09FfLHxfyr53sWVTcs2TNZDlx8QEclAdblZRkREqknJXUQkAym5i4hkICV3EZEMpOQuIpKBlNxFRDKQkruISAb6/3p+cZNGFh4zAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(len(loss))\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Some of our validation losses get close to the no-learning baseline, but not very reliably. This goes to show the merit of having had this baseline in the first place: it turns out not to be so easy to outperform. Our \n",
    "common sense contains already a lot of valuable information that a machine learning model does not have access to.\n",
    "\n",
    "You may ask, if there exists a simple, well-performing model to go from the data to the targets (our common sense baseline), why doesn't \n",
    "the model we are training find it and improve on it? Simply put: because this simple solution is not what our training setup is looking \n",
    "for. The space of models in which we are searching for a solution, i.e. our hypothesis space, is the space of all possible 2-layer networks \n",
    "with the configuration that we defined. These networks are already fairly complicated. When looking for a solution with a space of \n",
    "complicated models, the simple well-performing baseline might be unlearnable, even if it's technically part of the hypothesis space. That \n",
    "is a pretty significant limitation of machine learning in general: unless the learning algorithm is hard-coded to look for a specific kind \n",
    "of simple model, parameter learning can sometimes fail to find a simple solution to a simple problem.\n",
    "\n",
    "部分验证损失接近不包含学习的基准方法，但这个结果并不可靠。这也展示了首先建立这个基准方法的优点，事实证明，超越这个基准并不容易。我们的常识中包含了大量有价值的信息，而机器学习模型并不知道这些信息。\n",
    "\n",
    "你可能会问，如果从数据到目标之间存在一个简单且表现良好的模型（即基于常识的基准 方法），那为什么我们训练的模型没有找到这个模型并进一步改进呢？原因在于，这个简单的解决方案并不是训练过程所要寻找的目标。我们在模型空间（即假设空间）中搜索解决方案，这个模型空间是具有我们所定义的架构的所有两层网络组成的空间。这些网络已经相当复杂了。 \n",
    "\n",
    "如果你在一个复杂模型的空间中寻找解决方案，那么可能无法学到简单且性能良好的基准方法， 虽然技术上来说它属于假设空间的一部分。通常来说，这对机器学习是一个非常重要的限制： 如果学习算法没有被硬编码要求去寻找特定类型的简单模型，那么有时候参数学习是无法找到 简单问题的简单解决方案的。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A first recurrent baseline\n",
    "\n",
    "\n",
    "Our first fully-connected approach didn't do so well, but that doesn't mean machine learning is not applicable to our problem. The approach \n",
    "above consisted in first flattening the timeseries, which removed the notion of time from the input data. Let us instead look at our data \n",
    "as what it is: a sequence, where causality and order matter. We will try a recurrent sequence processing model -- it should be the perfect \n",
    "fit for such sequence data, precisely because it does exploit the temporal ordering of data points, unlike our first approach.\n",
    "\n",
    "Instead of the `LSTM` layer introduced in the previous section, we will use the `GRU` layer, developed by Cho et al. in 2014. `GRU` layers \n",
    "(which stands for \"gated recurrent unit\") work by leveraging the same principle as LSTM, but they are somewhat streamlined and thus cheaper \n",
    "to run, albeit they may not have quite as much representational power as LSTM. This trade-off between computational expensiveness and \n",
    "representational power is seen everywhere in machine learning.\n",
    "\n",
    "## 第一个循环网络基准\n",
    "\n",
    "第一个全连接方法的效果并不好，但这并不意味着机器学习不适用于这个问题。前一个方法首先将时间序列展平，这从输入数据中删除了时间的概念。我们来看一下数据本来的样子： 它是一个序列，其中因果关系和顺序都很重要。我们将尝试一种循环序列处理模型，它应该特别适合这种序列数据，因为它利用了数据点的时间顺序，这与第一个方法不同。\n",
    "\n",
    "我们将使用 Chung 等人在2014 年开发的 GRU 层，而不是上一节介绍的 LSTM 层。门控循环单元（GRU，gated recurrent unit）层的工作原理与 LSTM 相同。但它做了一些简化，因此运行的计算代价更低（虽然表示能力可能不如 LSTM）。机器学习中到处可以见到这种计算代价与表示能力之间的折中。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "500/500 [==============================] - 229s 458ms/step - loss: 0.3056 - val_loss: 0.2703\n",
      "Epoch 2/20\n",
      "500/500 [==============================] - 228s 457ms/step - loss: 0.2851 - val_loss: 0.2690\n",
      "Epoch 3/20\n",
      "500/500 [==============================] - 230s 460ms/step - loss: 0.2774 - val_loss: 0.2658\n",
      "Epoch 4/20\n",
      "500/500 [==============================] - 228s 456ms/step - loss: 0.2743 - val_loss: 0.2642\n",
      "Epoch 5/20\n",
      "500/500 [==============================] - 230s 459ms/step - loss: 0.2690 - val_loss: 0.2678\n",
      "Epoch 6/20\n",
      "500/500 [==============================] - 229s 458ms/step - loss: 0.2668 - val_loss: 0.2622\n",
      "Epoch 7/20\n",
      "500/500 [==============================] - 227s 455ms/step - loss: 0.2613 - val_loss: 0.2647\n",
      "Epoch 8/20\n",
      "500/500 [==============================] - 229s 458ms/step - loss: 0.2583 - val_loss: 0.2673\n",
      "Epoch 9/20\n",
      "500/500 [==============================] - 230s 459ms/step - loss: 0.2524 - val_loss: 0.2685\n",
      "Epoch 10/20\n",
      "500/500 [==============================] - 233s 466ms/step - loss: 0.2495 - val_loss: 0.2697\n",
      "Epoch 11/20\n",
      "500/500 [==============================] - 231s 462ms/step - loss: 0.2447 - val_loss: 0.2739\n",
      "Epoch 12/20\n",
      "500/500 [==============================] - 232s 464ms/step - loss: 0.2400 - val_loss: 0.2754\n",
      "Epoch 13/20\n",
      "500/500 [==============================] - 233s 466ms/step - loss: 0.2376 - val_loss: 0.2836\n",
      "Epoch 14/20\n",
      "500/500 [==============================] - 233s 465ms/step - loss: 0.2323 - val_loss: 0.2851\n",
      "Epoch 15/20\n",
      "500/500 [==============================] - 232s 464ms/step - loss: 0.2294 - val_loss: 0.2792\n",
      "Epoch 16/20\n",
      "500/500 [==============================] - 232s 464ms/step - loss: 0.2268 - val_loss: 0.2859\n",
      "Epoch 17/20\n",
      "500/500 [==============================] - 232s 465ms/step - loss: 0.2233 - val_loss: 0.2892\n",
      "Epoch 18/20\n",
      "500/500 [==============================] - 231s 463ms/step - loss: 0.2211 - val_loss: 0.2941\n",
      "Epoch 19/20\n",
      "500/500 [==============================] - 231s 463ms/step - loss: 0.2181 - val_loss: 0.2911\n",
      "Epoch 20/20\n",
      "500/500 [==============================] - 233s 465ms/step - loss: 0.2147 - val_loss: 0.2956\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras import layers\n",
    "from keras.optimizers import RMSprop\n",
    "\n",
    "model = Sequential()\n",
    "model.add(layers.GRU(32, input_shape=(None, float_data.shape[-1])))\n",
    "model.add(layers.Dense(1))\n",
    "\n",
    "model.compile(optimizer=RMSprop(), loss='mae')\n",
    "history = model.fit_generator(train_gen,\n",
    "                              steps_per_epoch=500,\n",
    "                              epochs=20,\n",
    "                              validation_data=val_gen,\n",
    "                              validation_steps=val_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let look at our results:\n",
    "\n",
    "看下结果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmYFNXZ9/HvzS6CYAARQRaNUQERcSQaURCNggsYHlQQoqLEYHCPiTyCvkrkiRFjDIaoxOAGioohEjfcUDQGZVgcNhFUVAQBieyIDNzvH6eGaYaZoWemp3u6+/e5rr6mu+pU1d3VPXedPnXqlLk7IiKSHaqlOgAREUkeJX0RkSyipC8ikkWU9EVEsoiSvohIFlHSFxHJIkr6UiZmVt3MNptZy0SWTSUz+6GZJbzvspmdYWbLY14vMbNT4ilbjm09bGa3lHf5UtZ7p5k9muj1SurUSHUAUrnMbHPMy7rAdmBn9PqX7j6xLOtz951AvUSXzQbufmQi1mNmg4GB7t4tZt2DE7FuyXxK+hnO3Xcn3agmOdjdXy+pvJnVcPf8ZMQmIsmn5p0sF/18f9rMnjKzTcBAMzvJzGaa2XozW2VmY8ysZlS+hpm5mbWOXk+I5r9sZpvM7D9m1qasZaP5Pc3sYzPbYGb3m9m/zeyyEuKOJ8ZfmtkyM/vWzMbELFvdzP5kZuvM7BOgRyn7Z4SZTSoybayZ3Rs9H2xmi6P380lUCy9pXSvMrFv0vK6ZPRHFthA4vpjtfhqtd6GZ9YqmHwP8BTglajr7Jmbf3h6z/JDova8zs3+aWbN49s2+mNn5UTzrzexNMzsyZt4tZrbSzDaa2Ucx7/VEM5sTTV9tZqPj3Z5UAnfXI0sewHLgjCLT7gS+B84jVAL2A04Afkz4JXgY8DFwdVS+BuBA6+j1BOAbIAeoCTwNTChH2YOATUDvaN6NwA7gshLeSzwxPg80AFoD/y1478DVwEKgBdAImBH+FYrdzmHAZmD/mHWvAXKi1+dFZQzoDmwDOkTzzgCWx6xrBdAten4P8BZwINAKWFSk7IVAs+gzuTiKoWk0bzDwVpE4JwC3R8/PjGLsCNQB/gq8Gc++Keb93wk8Gj0/Ooqje/QZ3RLt95pAO+Bz4OCobBvgsOj5LKB/9Lw+8ONU/y9k80M1fQF4193/5e673H2bu89y9/fdPd/dPwXGAV1LWX6yu+e6+w5gIiHZlLXsucA8d38+mvcnwgGiWHHG+Ht33+DuywkJtmBbFwJ/cvcV7r4OuKuU7XwKLCAcjAB+Cqx399xo/r/c/VMP3gTeAIo9WVvEhcCd7v6tu39OqL3HbvcZd18VfSZPEg7YOXGsF2AA8LC7z3P374BhQFczaxFTpqR9U5p+wFR3fzP6jO4CDiAcfPMJB5h2URPhZ9G+g3DwPsLMGrn7Jnd/P873IZVASV8Avox9YWZHmdmLZva1mW0ERgKNS1n+65jnWyn95G1JZQ+JjcPdnVAzLlacMca1LUINtTRPAv2j5xcTDlYFcZxrZu+b2X/NbD2hll3avirQrLQYzOwyM/swakZZDxwV53ohvL/d63P3jcC3QPOYMmX5zEpa7y7CZ9Tc3ZcAvyZ8Dmui5sKDo6KDgLbAEjP7wMzOjvN9SCVQ0hcIP/djPUSo3f7Q3Q8AbiM0X1SmVYTmFgDMzNgzSRVVkRhXAYfGvN5Xl9KngTOimnJvwkEAM9sPmAz8ntD00hB4Nc44vi4pBjM7DHgAuApoFK33o5j17qt76UpCk1HB+uoTmpG+iiOusqy3GuEz+wrA3Se4+8mEpp3qhP2Cuy9x936EJrw/As+ZWZ0KxiLlpKQvxakPbAC2mNnRwC+TsM0XgE5mdp6Z1QCuA5pUUozPANebWXMzawTcXFphd18NvAs8Aixx96XRrNpALWAtsNPMzgVOL0MMt5hZQwvXMVwdM68eIbGvJRz/BhNq+gVWAy0KTlwX4yngCjPrYGa1Ccn3HXcv8ZdTGWLuZWbdom3/hnAe5n0zO9rMTou2ty167CS8gZ+bWePol8GG6L3tqmAsUk5K+lKcXwOXEv6hHyLUdCtVlFgvAu4F1gGHA3MJ1xUkOsYHCG3v8wknGSfHscyThBOzT8bEvB64AZhCOBnal3Dwisf/I/ziWA68DDwes948YAzwQVTmKCC2Hfw1YCmw2sxim2kKln+F0MwyJVq+JaGdv0LcfSFhnz9AOCD1AHpF7fu1gbsJ52G+JvyyGBEtejaw2ELvsHuAi9z9+4rGI+VjoelUpGoxs+qE5oS+7v5OquMRyRSq6UuVYWY9zKxB1ERwK6FHyAcpDkskoyjpS1XSBfiU0ETQAzjf3Utq3hGRclDzjohIFlFNX0Qki1S5AdcaN27srVu3TnUYIiJpZfbs2d+4e2ndnIEqmPRbt25Nbm5uqsMQEUkrZravK8sBNe+IiGQVJX0RkSyipC8ikkWU9EVEsoiSvohIFsmYpD9xIrRuDdWqhb8Ty3S7bxGR7FDlumyWx8SJcOWVsHVreP355+E1wIAKjy0oIpI5MqKmP3x4YcIvsHVrmC4iIoUyIul/8UXZpouIZKuMSPotS7jZXUnTRUSyVUYk/VGjoG7dPafVrRumi4hIoYxI+gMGwLhx0KoVmIW/48bpJK6ISFEZ0XsHQoJXkhcRKV1G1PRFRCQ+SvoiIllESV9EJIso6YuIZJG4kr6Z9TCzJWa2zMyGFTN/iJnNN7N5ZvaumbWNmfe/0XJLzOysRAYvIiJls8+kb2bVgbFAT6At0D82qUeedPdj3L0jcDdwb7RsW6Af0A7oAfw1Wp+IiKRAPDX9zsAyd//U3b8HJgG9Ywu4+8aYl/sDHj3vDUxy9+3u/hmwLFqfiIikQDz99JsDX8a8XgH8uGghMxsK3AjUArrHLDuzyLLNyxWpiIhUWDw1fStmmu81wX2sux8O3AyMKMuyZnalmeWaWe7atWvjCElERMojnqS/Ajg05nULYGUp5ScB55dlWXcf5+457p7TpEmTOEISEZHyiCfpzwKOMLM2ZlaLcGJ2amwBMzsi5uU5wNLo+VSgn5nVNrM2wBHABxUPW0REymOfbfrunm9mVwPTgOrAeHdfaGYjgVx3nwpcbWZnADuAb4FLo2UXmtkzwCIgHxjq7jsr6b2IiMg+mPteTewplZOT47m5uakOQ0QkrZjZbHfP2Vc5XZErIpJFlPRFRLKIkr6ISBZR0hcRSbFdu8Ld/kaPrvxtKemLiKTQ7Nlw0knwy1/C66+HA0BlUtIXEUmB9evh6qvhhBPg889hwgR45RWoVslZWUlfRASYOROmTYOdlXwlkTs8/jgceSQ88EBI/B99FO7xbcUNXJNgSvoiktU2bgxNKyedBD16wOGHw//9H6xenfhtLVgAXbvCpZfCYYdBbi6MGQMNGyZ+WyVR0heRrPX663DMMfDww/Cb38Czz8IPfwjDh8Ohh0K/fvD226F2XhGbNsFNN0HHjrBoUdjev/8Nxx2XmPdRFkr6IpJ1Nm2Cq66Cn/4U9tsvJOC774a+fcOB4KOPQrPLq69Ct27Qvj3cf39ohy8Ld3jmGTjqKPjjH+Hyy2HJErjiispvuy+Jkr6IZJU33wy1+4cegl//GubOhRNP3LPMkUfCvffCV1/BI49AvXpw7bXQvDkMHhx63OzLxx/DWWfBRRdB06bwn/+EbpmNGlXO+4qXkr6IZIXNm2HoUDj9dKhVC955B+65J9T0S7LffnDZZfD++6H9/eKL4cknIScHOncOB4StW/dcZutWGDEiHFg++CD8Qpg1a+8DS6oo6YtIxnvrLejQIfSWueEGmDcPTj65bOs4/nj4299g5Ur485/DQeTyy0Pt/4YbQrPNv/4F7drBqFGhhr9kSWgmql6F7gyupC8iGWvLFrjmGjjttJB4Z8wIzTZ165Z/nQ0bhqaehQvDweSss2Ds2NBu36tXWPdbb4VumU2bJuqdJE4898gVEUk7M2bAoEHw2Wdw3XWhG2ZFkn1RZqH7ZdeuoXvnY4+Ftv9f/AJq1kzcdhJNSV9EMsqWLXDLLaEtvU2bUOs+9dTK3WbTpvDb31buNhJFzTsikjHefTf0hR8zJrSl5+VVfsJPN0r6IpL2li4NV9WeemoYRmH69JD4998/1ZFVPUr6IpKW3EO7/fnnh371jz5aWLvv1i3V0VVdatMXkbSyY0e4yvXee2HOnHCx04gR8KtfwcEHpzq6qk9JX0TSwrffhita778/XCl75JHhqtqBAxPbKyfTKemLSJW2bFm4GGr8+HC16+mnh+Tfo0fqxq9JZ0r6IlLluIdhEv70J3j+eahRIwyBcMMNcOyxqY4uvSnpi0iVsWNHGN743nvDoGY/+EHocz90KDRrluroMoN+HEUmToTWrcPPxdatw2sRSZ6HHw43FhkwIAx9/MAD8OWXcOedSviJpJo+IcFfeWXhaHmffx5eQ/gCikjlcYdhw8J49iefHJL92Wervb6ymFf0ljAJlpOT47m5uUndZuvWIdEX1aoVLF+e1FBEkmb79tCnfdasMATwnDnw4x+Hk6bJ6g2zY0cYn/7xx8NNTe6/v2qNSJlOzGy2u+fsq5xq+sAXX5Rtuki62bkz3A3qgw9Ckp81Cz78MCRdgCZNwpDAf/97GDd+ypRQGapMW7bAhRfCSy/ByJGhr30ybgye7ZT0gZYti6/pt2yZ/FhEKso9jCxZkNxnzQonRbdsCfPr1w83AbnhBjjhhPBo2TIk3BdfDL1kcnLg6adD98jKsG4dnHNOiO2hhwqbU6XyKekTbngQ26YP4eftqFGpi0mkrB59NCTqWbNCUgWoXTsMQDZoUGGCP/LIktvLCxLxz34GZ54Jo0eHg0Mia+BffBHGoP/sM5g8OWxLkkdJn8KTtcOHhy9ky5Yh4eskrqSLBx8MbeJHHgm9e4db+Z1wQrihd61aZVvXj34EM2fCpZeGe8jOnh3uGJWIdv4FC0LC37Il3HRcI2Amn07kiqS5KVOgb9/Q42XKlHAhUyLs2gW//z3cemu4IKqi7fzvvBPuLLXffjBtWriHrCROvCdy1SlKJI3NmAH9+4ea/dNPJy7hQ2gCGj483Pf1s89CO/8bb5RvXc8/H5qLDjoI3ntPCT+VlPRF0tT8+aHm3KYNvPBC5XWzLGjnb9o0JO577w0ni+P18MPQp0+4Mfm//135vYKkdEr6Imnoiy+gZ89wk5Bp08LwwpXpiCNCO//554d2/oED9+z4UBz3cDXtL34RDhZvvgmNG1dunLJvcSV9M+thZkvMbJmZDStm/o1mtsjM8szsDTNrFTPvbjNbaGaLzWyMmXriilTEunXhZOjmzfDKK8nrWly/fuhtM2oUPPVUuHq2pIsXd+6Ea64J5wMGDoSpU3UXq6pin0nfzKoDY4GeQFugv5m1LVJsLpDj7h2AycDd0bI/AU4GOgDtgROArgmLXiTLbN0K554b2tinTk1+27hZGADthRdKbuffvh369YOxY+Gmm+Cxx6BmzeTGKSWLp6bfGVjm7p+6+/fAJKB3bAF3n+7uBT/2ZgItCmYBdYBaQG2gJrA6EYGLZJv8fLjoonBV7ZNPpra749lnF9/Ov3FjaHaaPBnuuSf089cYOlVLPOf6mwNfxrxeAfy4lPJXAC8DuPt/zGw6sAow4C/uvrjoAmZ2JXAlQEtdBiuyF/dw4+8XXggDkvXpk+qICtv5L7sstPPPmhWGeliwAJ54IjTrSNUTzzG4uDb4Ys/dm9lAIAcYHb3+IXA0oebfHOhuZnvVT9x9nLvnuHtOkyZN4o1dJGvcemu4c9Stt8KQIamOplBsO//TT8PSpaGLpxJ+1RVPTX8FcGjM6xbAyqKFzOwMYDjQ1d23R5N/Bsx0981RmZeBE4EZFQlaJJv85S8hqf7iF3DHHamOZm8F7fynnQYNGkDbomf8pEqJp6Y/CzjCzNqYWS2gHzA1toCZHQc8BPRy9zUxs74AuppZDTOrSTiJu1fzTibQTVikMkyeDNdeG/rj//WvVXsUypNOUsJPB/us6bt7vpldDUwDqgPj3X2hmY0Ect19KqE5px7wbNQj8wt370XoydMdmE9oEnrF3f9VOW8ldXQTFqkMb70Vvj8nnRS6SCbyalvJXhp7JwF0ExZJtA8/DL1zmjeHd98N94oVKY3G3kki3YRFEmn5cujRI5wknTZNCV8SS0k/AUrqZarep1JW33wTrrb97ruQ8A89dN/LiJSFkn4CjBq192BXugmLlNWWLWFwsy++CN0e27VLdUSSiZT0E2DAABg3LrThm4W/48aV7SSuev9kn4LbGj77LNx8M/zkJ+H+tE89BV26pDo6yVTqD5AgAwaUv6eOev9kPndYsSIk9djHf/8b5tesGW5UMmFCGMlSpLKo904VoN4/mWfVqr0T/JroCpbq1cNAaTk5hY/27cP9bEXKK97eO6rpVwHq/ZP+tm8P95F97bWQ4FdG16xXqxYuWDr77MIE36FDuGWgSCoo6VcBLVsWX9NX75+qzz2cdL3xRvjkk3BT8e7dCxN8x44aR16qFiX9KmDUqD3b9EG9f9LBwoVwww2hdn/UUfDyy6F/vUhVpt47VUAiev9I8vz3v+GuUMceG4YT/vOfIS9PCV/Sg2r6VURFev9IcuTnw4MPwm23wYYNYXz7kSN131dJL0r6InF47bXQlLNwYWizv+++5N+qUCQR1LwjUoply6B373BLwG3bYMoUeP11JXxJX0r6IsXYuBF++9vQ3fLNN+Guu2DRonDhVFUe015kXzKmeSc/H664IvyTtmsXLnZp2VI3ZZay2bkTHn003AlqzRoYNCj0omrWLNWRiSRGxiT91ath+nR4/PHCafXqhYNA+/bhUXAwaNYs82prEyfC8OHhgq6WLUOi0onh+LiHi6nmzIHbbw9/f/ITePHF0NdeJJNkTNJv3jwkvA0bwsm2hQthwYLwePHFcFPpAgceWHgAiD0YpGsvDI3dE59du8J3ZNGi8Fi8uPD5xo2hTIsW8OST0K9f5lUMRCCLxt5Zu3bvg8GCBbB+fWGZpk2hb1+47jo44oiEh1BpNHbPnvLz4dNP907uH3205wVwTZuGX4Jt28LRR4fHSSdpiARJT/GOvZM1Sb847mFgrAULwsHg/ffhH/8ISePcc8Ol9V27Vv0aX7Vq4b0UZRZqt5nq++9h6dLC5F7w+PjjMK/AoYfumdwL/uqOVJJJNOBaHMzgkEPC48wzw7Svv4a//hUeeABOOy2MnXLDDeHnfq1aqY23JJk+ds/27SGRFyT1hQvD36VLwwEawmfZpk1oqjv77MIkf9RR4baDIhJkdU2/NNu2hbHN77svJJhmzWDo0HAVZlVr+7/33tC9cOfOwml165ZtKIeqcCL4u+9gyZLCpF7wWLas8L1VqwaHHx6Se0FiL0juapaRbKbmnQRxh1dfDYn11VehTh249FK4/vqQaFJpyZKQnJ98MiTDmjVDm3W1auEg8Pvfx7eeoieCoewHjfLavj3cC/app2Dq1MIYqlcP51UKknpBkv/Rj8JnICJ7ijfp4+5V6nH88cd7VbVggfvgwe61a7uDe8+e7q++6r5rV3LjmD/fvV8/dzP3/fZzv/FG95UrC+cddVSYd/vt7vn5+15fq1bh/RR9tGpVOfHn57u/9pr75Ze7N2wYttWokfuQIe5PPx3ew/btlbNtkUwF5HocOTblSb7ooyon/QKrV7vfcYf7QQeFPXjMMe5//7v7tm2Vu925c9379AnbrFfP/eabQyxFbdrk/vOfh3Ldu7uvWlX6es2KT/pmiYt91y73995zv+Ya96ZNC9/Dz3/u/tJL7t9/n7htiWQjJf0k+O4790cece/QIezJgw5yv/RS9wceCAl6x47EbOeDD9zPOy9s44AD3EeMcP/mm9KX2bXLffz48Evg4IPd33yz5LKVVdPftcv9ww/dhw1zb906rLN27XDgevZZ961bK7Z+ESmkpJ9Eu3a5v/56SGaNGxcmzbp13U891f23v3X/xz/cv/qqbOv997/de/QI6zrwQPeRI92//bZs6yho7qlWLfw6Ka65Z8KEEGtswq9bN0wvj6VL3X/3O/e2bcO6qlcP7+Oxx9zXry/fOkWkdEr6KbJrl/snn7hPnBiaMk44wb1mzcJkeuih7hdc4P7HP7q/+27xtd233grNMhAOIr//vfuGDeWPadMm94EDw/rOOMP966/3LjNhQqjZm4W/ZU34X3wR3tMJJxS+11NOcf/rX93XrCl/7CISn3iTvnrvJMF338HcuTBzZrgAbObMwn71NWqEawF+/OPQQ+Wpp+Cdd8LVor/5DQwZkph7rLqHoSiuvhoaNgzb6datYutcuRKefRaeeQbeey9M69QJ+veHiy4KF0WJSHKoy2YV9/XXhQeAmTPDbfe2bAljCN18MwweXDn9zufPhwsuCBc23XEH/O//hu6RZYl78uSQ6N99NxxMOnQISf6CC9Jr+AqRTKKkn2Z27oRPPgnj5dSuXbnb2rw5/IKYOBF++tNwEdpBB5Vcfs0aeO65kOjffjsk+vbt4cILQ6JP9fUKIqJhGNJO9erhwqNkqFcPnngiNO9cc01oXnrqqTDOUIFvvgnjED3zTBiyeteukNxvuy0k+nbtkhOriCSWkn6WMgtNSJ07hyTevXsYS/6QQ0Kif+ON8OvjiCPCDUUuvDDU7qv64HMiUjrdVyrLdegAubmhTf6228KBYNmyMIzD3LlhqIff/S7cE7a0hD9xYhjiuVq18HfixGS9AxEpC9X0hfr1Q5IePBgaNAg9cMpSo9dNXETSR1w1fTPrYWZLzGyZmQ0rZv6NZrbIzPLM7A0zaxUzr6WZvWpmi6MyrRMXviSKWWjiOf74sjfhDB++52BtEF4PH564+EQkMfaZ9M2sOjAW6Am0BfqbWdsixeYCOe7eAZgM3B0z73FgtLsfDXQG1iQicKk6vviibNNFJHXiqel3Bpa5+6fu/j0wCegdW8Ddp7t7QV1vJtACIDo41HD316Jym2PKSYYo6WYtmXITF5FMEk/Sbw58GfN6RTStJFcAL0fPfwSsN7N/mNlcMxsd/XKQDDJqVBh/P1bdumG6iFQt8ST94lp4i72iy8wGAjnA6GhSDeAU4CbgBOAw4LJilrvSzHLNLHft2rVxhCRVyYAB4YYrrVqF8wGtWpX9Bizq/SOSHPH03lkBxI6i0gJYWbSQmZ0BDAe6uvv2mGXnuvunUZl/AicCf49d1t3HAeMgXJFbxvcgVcCAAeXvqaPePyLJE09NfxZwhJm1MbNaQD9gamwBMzsOeAjo5e5riix7oJk1iV53BxZVPGzJJOr9I5I8+0z67p4PXA1MAxYDz7j7QjMbaWa9omKjgXrAs2Y2z8ymRsvuJDTtvGFm8wlNRX+rhPchaUy9f0SSJ66Ls9z9JeClItNui3l+RinLvgZ0KG+AkvlatiwcarrodBFJLA3DICmn3j8iyaOkLymXiN4/IhIfjb0jVUJFev+ISPxU0xcRySJK+iIiWURJX0Qkiyjpi4hkESV9EZEsoqQvGUEDtonER102Je1pwDaR+KmmL2lPA7aJxE9JX9KeBmwTiZ+SvqQ93a5RJH5K+pL2NGCbSPyU9CXt6XaNIvFT7x3JCLpdo0h8VNOXrKfeP5JNlPQl66n3j2QTJX3Jeur9I9lESV+ynnr/SDZR0pesp9s1SjZR7x0RdLtGyR6q6YuIZBElfRGRLKKkLyKSRZT0RRJAwzhIutCJXJEK0jAOkk5U0xepIA3jIOlESV+kgjSMg6QTJX2RCtIwDpJOlPRFKkjDOEg6UdIXqSAN4yDpRL13RBJAwzhIulBNX0Qkiyjpi4hkkbiSvpn1MLMlZrbMzIYVM/9GM1tkZnlm9oaZtSoy/wAz+8rM/pKowEUyia7olWTZZ9I3s+rAWKAn0Bbob2ZtixSbC+S4ewdgMnB3kfm/A96ueLgimafgit7PPwf3wit6lfilMsRT0+8MLHP3T939e2AS0Du2gLtPd/eCaxJnAi0K5pnZ8UBT4NXEhCySWXRFryRTPEm/OfBlzOsV0bSSXAG8DGBm1YA/Ar8pbQNmdqWZ5ZpZ7tq1a+MISSRz6IpeSaZ4kr4VM82LLWg2EMgBRkeTfgW85O5fFld+98rcx7l7jrvnNGnSJI6QRDKHruiVZIon6a8ADo153QJYWbSQmZ0BDAd6ufv2aPJJwNVmthy4B7jEzO6qUMQiGSYRV/TqRLDEK56Ls2YBR5hZG+AroB9wcWwBMzsOeAjo4e5rCqa7+4CYMpcRTvbu1ftHJJsVXNQ1fHho0mnZMiT8eC/20tDOUhbmXmxLzZ6FzM4G7gOqA+PdfZSZjQRy3X2qmb0OHAOsihb5wt17FVnHZYSkf3Vp28rJyfHc3NyyvxORLNW6dUj0RbVqBcuXJzsaSRUzm+3uOfssF0/STyYlfZGyqVYtdPUsygx27Up+PJIa8SZ9XZErkuZ0IljKQklfJM1paGcpCyV9kTSnoZ2lLDS0skgG0NDOEi/V9EVEsoiSvohIFlHSFxHJIkr6IiJZRElfRDR2TxZR7x2RLKexe7KLavoiWU43cckuSvoiWU43cckuSvoiWU5j92QXJX2RLKexe7KLkr5IlkvE2D3q/ZM+1HtHRCo0do96/6QX1fRFpELU+ye9KOmLSIWo9096UdIXkQpR75/0oqQvIhWi3j/pRUlfRCpEvX/SS1r03tmxYwcrVqzgu+++S3UoEoc6derQokULatasmepQJEnU+yd9mLunOoY95OTkeG5u7h7TPvvsM+rXr0+jRo0wsxRFJvFwd9atW8emTZto06ZNqsORNNC6dUj0RbVqBcuXJzua9GVms909Z1/l0qJ557vvvlPCTxNmRqNGjfSrTOKm3j/JlRZJH1DCTyP6rKQs1PsnudIm6YtIZlLvn+TKyKSf6J4A69ato2PHjnTs2JGDDz6Y5s2b7379/fffx7WOQYMGsWTJklLLjB07lokJ6rbQpUsX5s2bl5B1iVSmRPT+kfilRe+dsqiMngCNGjXanUBvv/126tWrx0033bRHGXfH3alWrfjj6COPPLLP7QwdOrR8AYqkuYr0/oHwfz98eDgP0LJl+JWgg0bxMq6mn8z/ucyAAAAP1klEQVRxQJYtW0b79u0ZMmQInTp1YtWqVVx55ZXk5OTQrl07Ro4cubtsQc07Pz+fhg0bMmzYMI499lhOOukk1qxZA8CIESO47777dpcfNmwYnTt35sgjj+S9994DYMuWLfzP//wPxx57LP379ycnJ2efNfoJEyZwzDHH0L59e2655RYA8vPz+fnPf757+pgxYwD405/+RNu2bTn22GMZOHBgwveZSKIVVPQ+/xzcCyt66utfvIyr6Se7J8CiRYt45JFHePDBBwG46667+MEPfkB+fj6nnXYaffv2pW3btnsss2HDBrp27cpdd93FjTfeyPjx4xk2bNhe63Z3PvjgA6ZOncrIkSN55ZVXuP/++zn44IN57rnn+PDDD+nUqVOp8a1YsYIRI0aQm5tLgwYNOOOMM3jhhRdo0qQJ33zzDfPnzwdg/fr1ANx99918/vnn1KpVa/c0kaqstIqeavt7y7iafrJ7Ahx++OGccMIJu18/9dRTdOrUiU6dOrF48WIWLVq01zL77bcfPXv2BOD4449neQmdkfv06bNXmXfffZd+/foBcOyxx9KuXbtS43v//ffp3r07jRs3pmbNmlx88cXMmDGDH/7whyxZsoTrrruOadOm0aBBAwDatWvHwIEDmThxoi6ukrSgLp9lk3FJP9k9Afbff//dz5cuXcqf//xn3nzzTfLy8ujRo0ex/dVr1aq1+3n16tXJz88vdt21a9feq0xZL6YrqXyjRo3Iy8ujS5cujBkzhl/+8pcATJs2jSFDhvDBBx+Qk5PDzp07y7Q9kWRTl8+yybikn8qeABs3bqR+/foccMABrFq1imnTpiV8G126dOGZZ54BYP78+cX+koh14oknMn36dNatW0d+fj6TJk2ia9eurF27Fnfnggsu4I477mDOnDns3LmTFStW0L17d0aPHs3atWvZWvR3s0gVk4iKXjaN/ZNxbfpQ8Z4A5dWpUyfatm1L+/btOeywwzj55JMTvo1rrrmGSy65hA4dOtCpUyfat2+/u2mmOC1atGDkyJF069YNd+e8887jnHPOYc6cOVxxxRW4O2bGH/7wB/Lz87n44ovZtGkTu3bt4uabb6Z+/foJfw8iiVTwv17e3jvZNvZPXGPvmFkP4M9AdeBhd7+ryPwbgcFAPrAWuNzdPzezjsADwAHATmCUuz9d2raKG3tn8eLFHH300XG/qUyWn59Pfn4+derUYenSpZx55pksXbqUGjWq1vFbn5mki0wZ+yfesXf2mSnMrDowFvgpsAKYZWZT3T22XWEukOPuW83sKuBu4CJgK3CJuy81s0OA2WY2zd3VLaScNm/ezOmnn05+fj7uzkMPPVTlEr5IOsm2E8HxZIvOwDJ3/xTAzCYBvYHdSd/dp8eUnwkMjKZ/HFNmpZmtAZoASvrl1LBhQ2bPnp3qMEQyRsuWxdf0M/VEcDwncpsDX8a8XhFNK8kVwMtFJ5pZZ6AW8Ekx8640s1wzy127dm0cIYmIJEa2jf0TT9IvbsjEYk8EmNlAIAcYXWR6M+AJYJC779prZe7j3D3H3XOaNGkSR0giIomRbWP/xJP0VwCHxrxuAawsWsjMzgCGA73cfXvM9AOAF4ER7j6zYuGKiCTegAHhpO2uXeFvWRN+OnX5jKdNfxZwhJm1Ab4C+gEXxxYws+OAh4Ae7r4mZnotYArwuLs/m7CoRUSqiHTr8rnPmr675wNXA9OAxcAz7r7QzEaaWa+o2GigHvCsmc0zs6nR9AuBU4HLounzom6caaVbt257XWh133338atf/arU5erVqwfAypUr6du3b4nrLtpFtaj77rtvj4ukzj777ISMi3P77bdzzz33VHg9ItksmYM8JkJcff3c/SXgpSLTbot5fkYJy00AJlQkwKqgf//+TJo0ibPOOmv3tEmTJjF69OhSlip0yCGHMHny5HJv/7777mPgwIHUjc42vfTSS/tYQkSSJd26fKZdB+/rr4dE3xukY0eIRjQuVt++fRkxYgTbt2+ndu3aLF++nJUrV9KlSxc2b95M7969+fbbb9mxYwd33nknvXv33mP55cuXc+6557JgwQK2bdvGoEGDWLRoEUcffTTbtm3bXe6qq65i1qxZbNu2jb59+3LHHXcwZswYVq5cyWmnnUbjxo2ZPn06rVu3Jjc3l8aNG3Pvvfcyfvx4AAYPHsz111/P8uXL6dmzJ126dOG9996jefPmPP/88+y3334lvsd58+YxZMgQtm7dyuGHH8748eM58MADGTNmDA8++CA1atSgbdu2TJo0ibfffpvrrrsOCLdGnDFjhq7clayVbl0+M27sncrQqFEjOnfuzCuvvAKEWv5FF12EmVGnTh2mTJnCnDlzmD59Or/+9a9LHRTtgQceoG7duuTl5TF8+PA9+tyPGjWK3Nxc8vLyePvtt8nLy+Paa6/lkEMOYfr06UyfPn2Pdc2ePZtHHnmE999/n5kzZ/K3v/2NuXPnAmHwt6FDh7Jw4UIaNmzIc889V+p7vOSSS/jDH/5AXl4exxxzDHfccQcQhoqeO3cueXl5u4ePvueeexg7dizz5s3jnXfeKfVgIpLp0q3LZ9rV9EurkVemgiae3r17M2nSpN21a3fnlltuYcaMGVSrVo2vvvqK1atXc/DBBxe7nhkzZnDttdcC0KFDBzp06LB73jPPPMO4cePIz89n1apVLFq0aI/5Rb377rv87Gc/2z3SZ58+fXjnnXfo1asXbdq0oWPHcPqktOGbIYzvv379erp27QrApZdeygUXXLA7xgEDBnD++edz/vnnA3DyySdz4403MmDAAPr06UOLFi3i2YUiGamiY/8km2r6cTr//PN54403mDNnDtu2bdt985KJEyeydu1aZs+ezbx582jatGmxwynHMtv70ofPPvuMe+65hzfeeIO8vDzOOeecfa6ntF8UBcMyQ+nDN+/Liy++yNChQ5k9ezbHH388+fn5DBs2jIcffpht27Zx4okn8tFHH5Vr3SKZIp26fCrpx6levXp069aNyy+/nP79+++evmHDBg466CBq1qzJ9OnT+by4xr0Yp5566u6bny9YsIC8vDwgDMu8//7706BBA1avXs3LLxde1Fy/fn02bdpU7Lr++c9/snXrVrZs2cKUKVM45ZRTyvzeGjRowIEHHsg777wDwBNPPEHXrl3ZtWsXX375Jaeddhp3330369evZ/PmzXzyySccc8wx3HzzzeTk5Cjpi1RAsm/3mHbNO6nUv39/+vTpw6RJk3ZPGzBgAOeddx45OTl07NiRo446qtR1XHXVVQwaNIgOHTrQsWNHOnfuDIS7YB133HG0a9dur2GZr7zySnr27EmzZs32aNfv1KkTl1122e51DB48mOOOO67UppySPPbYY7tP5B522GE88sgj7Ny5k4EDB7JhwwbcnRtuuIGGDRty6623Mn36dKpXr07btm133wVMRMou2bd7jGto5WTS0MqZQZ+ZSHyqVQs1/KLMQnNRvOIdWlnNOyIiKZTs2z0q6YuIpFCyu3ymTdKvas1QUjJ9ViLxS/Yon2lxIrdOnTqsW7eORo0aFdvdUaoOd2fdunXUqVMn1aGIpI1k3tc7LZJ+ixYtWLFiBbrBSnqoU6eOLtgSqaLSIunXrFmTNm3apDoMEZG0lzZt+iIiUnFK+iIiWURJX0Qki1S5K3LNbC1Q+gA2pWsMfJOgcCqD4qsYxVcxiq9iqnJ8rdy9yb4KVbmkX1FmlhvPpcipovgqRvFVjOKrmKoeXzzUvCMikkWU9EVEskgmJv1xqQ5gHxRfxSi+ilF8FVPV49unjGvTFxGRkmViTV9EREqgpC8ikkXSMumbWQ8zW2Jmy8xsWDHza5vZ09H8982sdRJjO9TMppvZYjNbaGbXFVOmm5ltMLN50eO2ZMUXE8NyM5sfbT+3mPlmZmOifZhnZp2SGNuRMftmnpltNLPri5RJ6j40s/FmtsbMFsRM+4GZvWZmS6O/B5aw7KVRmaVmdmkS4xttZh9Fn98UM2tYwrKlfhcqMb7bzeyrmM/w7BKWLfX/vRLjezomtuVmNq+EZSt9/yWUu6fVA6gOfAIcBtQCPgTaFinzK+DB6Hk/4OkkxtcM6BQ9rw98XEx83YAXUrwflwONS5l/NvAyYMCJwPsp/Ly/Jlx4krJ9CJwKdAIWxEy7GxgWPR8G/KGY5X4AfBr9PTB6fmCS4jsTqBE9/0Nx8cXzXajE+G4Hborj8y/1/72y4isy/4/Abanaf4l8pGNNvzOwzN0/dffvgUlA7yJlegOPRc8nA6dbkgbid/dV7j4ner4JWAw0T8a2E6w38LgHM4GGZtYsBXGcDnzi7hW5SrvC3H0G8N8ik2O/Z48B5xez6FnAa+7+X3f/FngN6JGM+Nz9VXfPj17OBFI23nUJ+y8e8fy/V1hp8UW540LgqURvNxXSMek3B76Meb2CvZPq7jLRl34D0Cgp0cWImpWOA94vZvZJZvahmb1sZu2SGljgwKtmNtvMrixmfjz7ORn6UfI/W6r3YVN3XwXhYA8cVEyZqrIfLyf8civOvr4LlenqqPlpfAnNY1Vh/50CrHb3pSXMT+X+K7N0TPrF1diL9juNp0ylMrN6wHPA9e6+scjsOYTmimOB+4F/JjO2yMnu3gnoCQw1s1OLzK8K+7AW0At4tpjZVWEfxqMq7MfhQD4wsYQi+/ouVJYHgMOBjsAqQhNKUSnff0B/Sq/lp2r/lUs6Jv0VwKExr1sAK0sqY2Y1gAaU76dluZhZTULCn+ju/yg63903uvvm6PlLQE0za5ys+KLtroz+rgGmEH5Gx4pnP1e2nsAcd19ddEZV2IfA6oImr+jvmmLKpHQ/RieOzwUGeNQAXVQc34VK4e6r3X2nu+8C/lbCdlO9/2oAfYCnSyqTqv1XXumY9GcBR5hZm6gm2A+YWqTMVKCgl0Rf4M2SvvCJFrX//R1Y7O73llDm4IJzDGbWmfA5rEtGfNE29zez+gXPCSf8FhQpNhW4JOrFcyKwoaApI4lKrGGleh9GYr9nlwLPF1NmGnCmmR0YNV+cGU2rdGbWA7gZ6OXuW0soE893obLiiz1H9LMSthvP/3tlOgP4yN1XFDczlfuv3FJ9Jrk8D0LPko8JZ/WHR9NGEr7cAHUITQLLgA+Aw5IYWxfCz888YF70OBsYAgyJylwNLCT0RJgJ/CTJ+++waNsfRnEU7MPYGA0YG+3j+UBOkmOsS0jiDWKmpWwfEg4+q4AdhNrnFYTzRG8AS6O/P4jK5gAPxyx7efRdXAYMSmJ8ywjt4QXfw4IebYcAL5X2XUhSfE9E3608QiJvVjS+6PVe/+/JiC+a/mjBdy6mbNL3XyIfGoZBRCSLpGPzjoiIlJOSvohIFlHSFxHJIkr6IiJZRElfRCSLKOmLiGQRJX0RkSzy/wFrS7YrDq3dnQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(len(loss))\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Much better! We are able to significantly beat the common sense baseline, such demonstrating the value of machine learning here, as well as \n",
    "the superiority of recurrent networks compared to sequence-flattening dense networks on this type of task.\n",
    "\n",
    "Our new validation MAE of ~0.265 (before we start significantly overfitting) translates to a mean absolute error of 2.35˚C after \n",
    "de-normalization. That's a solid gain on our initial error of 2.57˚C, but we probably still have a bit of margin for improvement.\n",
    "\n",
    "效果好多了！远优于基于常识的基准方法。这证明了机器学习的价值，也证明了循环网络与序列展平的密集网络相比在这种任务上的优势。\n",
    "\n",
    "新的验证 MAE 约为 0.265（在开始显著过拟合之前），反标准化转换成温度的平均绝对误差为 2.35℃。与最初的误差 2.57℃相比，这个结果确实有所提高，但可能仍有改进的空间。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using recurrent dropout to fight overfitting\n",
    "\n",
    "\n",
    "It is evident from our training and validation curves that our model is overfitting: the training and validation losses start diverging \n",
    "considerably after a few epochs. You are already familiar with a classic technique for fighting this phenomenon: dropout, consisting in \n",
    "randomly zeroing-out input units of a layer in order to break happenstance correlations in the training data that the layer is exposed to. \n",
    "How to correctly apply dropout in recurrent networks, however, is not a trivial question. It has long been known that applying dropout \n",
    "before a recurrent layer hinders learning rather than helping with regularization. In 2015, Yarin Gal, as part of his Ph.D. thesis on \n",
    "Bayesian deep learning, determined the proper way to use dropout with a recurrent network: the same dropout mask (the same pattern of \n",
    "dropped units) should be applied at every timestep, instead of a dropout mask that would vary randomly from timestep to timestep. What's \n",
    "more: in order to regularize the representations formed by the recurrent gates of layers such as GRU and LSTM, a temporally constant \n",
    "dropout mask should be applied to the inner recurrent activations of the layer (a \"recurrent\" dropout mask). Using the same dropout mask at \n",
    "every timestep allows the network to properly propagate its learning error through time; a temporally random dropout mask would instead \n",
    "disrupt this error signal and be harmful to the learning process.\n",
    "\n",
    "Yarin Gal did his research using Keras and helped build this mechanism directly into Keras recurrent layers. Every recurrent layer in Keras \n",
    "has two dropout-related arguments: `dropout`, a float specifying the dropout rate for input units of the layer, and `recurrent_dropout`, \n",
    "specifying the dropout rate of the recurrent units. Let's add dropout and recurrent dropout to our GRU layer and see how it impacts \n",
    "overfitting. Because networks being regularized with dropout always take longer to fully converge, we train our network for twice as many \n",
    "epochs.\n",
    "\n",
    "## 使用循环 dropout 来降低过拟合\n",
    "\n",
    "从训练和验证曲线中可以明显看出，模型出现过拟合：几轮过后，训练损失和验证损失就开始显著偏离。我们已经学过降低过拟合的一种经典技术——dropout，即将某一层的输入单元随机设为 0，其目的是打破该层训练数据中的偶然相关性。但在循环网络中如何正确地使用 dropout，这并不是一个简单的问题。人们早就知道，在循环层前面应用 dropout，这种正则化会妨碍学习过程，而不是有所帮助。2015 年，在关于贝叶斯深度学习的博士论文中，Yarin Gal 确定了在循环网络中使用 dropout 的正确方法：对每个时间步应该使用相同的 dropout 掩码（dropout mask，相同模式的舍弃单元），而不是让 dropout 掩码随着时间步的增加而随机变化。此外，为了对 GRU、LSTM 等循环层得到的表示做正则化，应该将不随时间变化的 dropout 掩码应用于层的内部循环激活（叫作循环 dropout 掩码）。对每个时间步使用相同的 dropout 掩码，可以让网络沿着时间正确地传播其学习误差，而随时间随机变化的 dropout 掩码则会破坏这个误差信号，并且不利于学习过程。\n",
    "\n",
    "Yarin Gal 使用 Keras 开展这项研究，并帮助将这种机制直接内置到 Keras 循环层中。Keras 的每个循环层都有两个与 dropout 相关的参数：一个是dropout，它是一个浮点数，指定该层输入单元的 dropout 比率；另一个是 recurrent_dropout，指定循环单元的 dropout 比率。我们向 GRU 层中添加 dropout和循环 dropout，看一下这么做对过拟合的影响。因为使用 dropout 正则化的网络总是需要更长的时间才能完全收敛，所以网络训练轮次增加为原来的 2 倍。\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/40\n",
      "500/500 [==============================] - 259s 519ms/step - loss: 0.3349 - val_loss: 0.2725\n",
      "Epoch 2/40\n",
      "500/500 [==============================] - 259s 517ms/step - loss: 0.3136 - val_loss: 0.2728\n",
      "Epoch 3/40\n",
      "500/500 [==============================] - 258s 515ms/step - loss: 0.3066 - val_loss: 0.2724\n",
      "Epoch 4/40\n",
      "500/500 [==============================] - 257s 515ms/step - loss: 0.3028 - val_loss: 0.2685\n",
      "Epoch 5/40\n",
      "500/500 [==============================] - 258s 515ms/step - loss: 0.2978 - val_loss: 0.2663\n",
      "Epoch 6/40\n",
      "500/500 [==============================] - 259s 518ms/step - loss: 0.2970 - val_loss: 0.2685\n",
      "Epoch 7/40\n",
      "500/500 [==============================] - 258s 517ms/step - loss: 0.2948 - val_loss: 0.2654\n",
      "Epoch 8/40\n",
      "500/500 [==============================] - 257s 514ms/step - loss: 0.2915 - val_loss: 0.2660\n",
      "Epoch 9/40\n",
      "500/500 [==============================] - 257s 514ms/step - loss: 0.2928 - val_loss: 0.2671\n",
      "Epoch 10/40\n",
      "500/500 [==============================] - 257s 515ms/step - loss: 0.2909 - val_loss: 0.2671\n",
      "Epoch 11/40\n",
      "500/500 [==============================] - 258s 517ms/step - loss: 0.2910 - val_loss: 0.2643\n",
      "Epoch 12/40\n",
      "500/500 [==============================] - 257s 515ms/step - loss: 0.2866 - val_loss: 0.2636\n",
      "Epoch 13/40\n",
      "500/500 [==============================] - 257s 514ms/step - loss: 0.2871 - val_loss: 0.2657\n",
      "Epoch 14/40\n",
      "500/500 [==============================] - 257s 513ms/step - loss: 0.2842 - val_loss: 0.2670\n",
      "Epoch 15/40\n",
      "500/500 [==============================] - 255s 511ms/step - loss: 0.2854 - val_loss: 0.2639\n",
      "Epoch 16/40\n",
      "500/500 [==============================] - 261s 521ms/step - loss: 0.2845 - val_loss: 0.2637\n",
      "Epoch 17/40\n",
      "500/500 [==============================] - 262s 523ms/step - loss: 0.2830 - val_loss: 0.2634\n",
      "Epoch 18/40\n",
      "500/500 [==============================] - 261s 521ms/step - loss: 0.2843 - val_loss: 0.2654\n",
      "Epoch 19/40\n",
      "500/500 [==============================] - 263s 525ms/step - loss: 0.2820 - val_loss: 0.2636\n",
      "Epoch 20/40\n",
      "500/500 [==============================] - 260s 520ms/step - loss: 0.2806 - val_loss: 0.2659\n",
      "Epoch 21/40\n",
      "500/500 [==============================] - 262s 524ms/step - loss: 0.2810 - val_loss: 0.2656\n",
      "Epoch 22/40\n",
      "500/500 [==============================] - 262s 523ms/step - loss: 0.2791 - val_loss: 0.2642\n",
      "Epoch 23/40\n",
      "500/500 [==============================] - 262s 524ms/step - loss: 0.2782 - val_loss: 0.2743\n",
      "Epoch 24/40\n",
      "500/500 [==============================] - 261s 521ms/step - loss: 0.2781 - val_loss: 0.2706\n",
      "Epoch 25/40\n",
      "500/500 [==============================] - 261s 522ms/step - loss: 0.2784 - val_loss: 0.2681\n",
      "Epoch 26/40\n",
      "500/500 [==============================] - 261s 522ms/step - loss: 0.2778 - val_loss: 0.2678\n",
      "Epoch 27/40\n",
      "500/500 [==============================] - 261s 522ms/step - loss: 0.2758 - val_loss: 0.2663\n",
      "Epoch 28/40\n",
      "500/500 [==============================] - 263s 525ms/step - loss: 0.2749 - val_loss: 0.2661\n",
      "Epoch 29/40\n",
      "500/500 [==============================] - 261s 523ms/step - loss: 0.2752 - val_loss: 0.2626\n",
      "Epoch 30/40\n",
      "500/500 [==============================] - 262s 524ms/step - loss: 0.2752 - val_loss: 0.2684\n",
      "Epoch 31/40\n",
      "500/500 [==============================] - 260s 520ms/step - loss: 0.2752 - val_loss: 0.2643\n",
      "Epoch 32/40\n",
      "500/500 [==============================] - 260s 520ms/step - loss: 0.2730 - val_loss: 0.2669\n",
      "Epoch 33/40\n",
      "500/500 [==============================] - 261s 523ms/step - loss: 0.2724 - val_loss: 0.2637\n",
      "Epoch 34/40\n",
      "500/500 [==============================] - 261s 523ms/step - loss: 0.2715 - val_loss: 0.2676\n",
      "Epoch 35/40\n",
      "500/500 [==============================] - 260s 521ms/step - loss: 0.2730 - val_loss: 0.2701\n",
      "Epoch 36/40\n",
      "500/500 [==============================] - 261s 521ms/step - loss: 0.2696 - val_loss: 0.2743\n",
      "Epoch 37/40\n",
      "500/500 [==============================] - 260s 519ms/step - loss: 0.2707 - val_loss: 0.2643\n",
      "Epoch 38/40\n",
      "500/500 [==============================] - 258s 516ms/step - loss: 0.2707 - val_loss: 0.2668\n",
      "Epoch 39/40\n",
      "500/500 [==============================] - 260s 519ms/step - loss: 0.2694 - val_loss: 0.2671\n",
      "Epoch 40/40\n",
      "500/500 [==============================] - 260s 520ms/step - loss: 0.2701 - val_loss: 0.2702\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras import layers\n",
    "from keras.optimizers import RMSprop\n",
    "\n",
    "model = Sequential()\n",
    "model.add(layers.GRU(32,\n",
    "                     dropout=0.2,\n",
    "                     recurrent_dropout=0.2,\n",
    "                     input_shape=(None, float_data.shape[-1])))\n",
    "model.add(layers.Dense(1))\n",
    "\n",
    "model.compile(optimizer=RMSprop(), loss='mae')\n",
    "history = model.fit_generator(train_gen,\n",
    "                              steps_per_epoch=500,\n",
    "                              epochs=40,\n",
    "                              validation_data=val_gen,\n",
    "                              validation_steps=val_steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VNX5+PHPQ2SRRVBAQRCCS5UAEeIIqCiLli+4gFsVBFcsotJaab8Vd6XlV4tWKYp+S63USpRaWytVFLe44AIksoMURJAIagDZN0Oe3x/nTjIJs9xkJplJ5nm/XvPK3P2Zm+SZc88951xRVYwxxqSHeskOwBhjTM2xpG+MMWnEkr4xxqQRS/rGGJNGLOkbY0wasaRvjDFpxJK+qRQRyRCRXSLSIZHrJpOInCgiCW+7LCLnici6kOlVInK2n3WrcKynReSuqm4fZb+/FZG/Jnq/JnkOS3YApnqJyK6QycbAfuCgN32TquZWZn+qehBomuh104GqnpyI/YjIjcBIVe0Xsu8bE7FvU/dZ0q/jVLU06XolyRtV9e1I64vIYapaXBOxGWNqnlXvpDnv8v3vIvKCiOwERorIGSLyqYhsE5FNIjJFROp76x8mIioimd70DG/56yKyU0Q+EZFOlV3XWz5YRP4rIttF5HER+UhErosQt58YbxKRNSLyvYhMCdk2Q0QeE5EtIvIFMCjK+blHRGZWmDdVRB713t8oIiu9z/OFVwqPtK9CEennvW8sIs95sS0HTgtz3LXefpeLyBBvfjfgCeBsr+psc8i5fSBk+zHeZ98iIv8WkbZ+zk0sInKxF882EXlXRE4OWXaXiGwUkR0i8nnIZ+0tIp95878VkYf9Hs9UA1W1V5q8gHXAeRXm/RY4AFyEKwQcDpwO9MJdCR4P/BcY661/GKBApjc9A9gMBID6wN+BGVVY92hgJzDUWzYO+AG4LsJn8RPjK0BzIBPYGvzswFhgOdAeaAl84P4Vwh7neGAX0CRk398BAW/6Im8dAQYAe4Fsb9l5wLqQfRUC/bz3jwDvAUcCHYEVFda9Amjr/U6u8mI4xlt2I/BehThnAA947wd6MXYHGgFPAu/6OTdhPv9vgb967zt7cQzwfkd3eee9PtAFWA+08dbtBBzvvV8ADPfeNwN6Jft/IZ1fVtI3AHNV9T+qWqKqe1V1garOU9ViVV0LTAP6Rtn+JVXNV9UfgFxcsqnsuhcCi1T1FW/ZY7gviLB8xvg7Vd2uqutwCTZ4rCuAx1S1UFW3AA9FOc5aYBnuywjgx8A2Vc33lv9HVdeq8y7wDhD2Zm0FVwC/VdXvVXU9rvQeetwXVXWT9zt5HveFHfCxX4ARwNOqukhV9wHjgb4i0j5knUjnJpphwCxVfdf7HT0EHIH78i3GfcF08aoIv/TOHbgv75NEpKWq7lTVeT4/h6kGlvQNwIbQCRE5RUReE5FvRGQHMAFoFWX7b0Le7yH6zdtI6x4bGoeqKq5kHJbPGH0dC1dCjeZ5YLj3/ircl1UwjgtFZJ6IbBWRbbhSdrRzFdQ2Wgwicp2ILPaqUbYBp/jcL7jPV7o/Vd0BfA+0C1mnMr+zSPstwf2O2qnqKuCXuN/Dd151YRtv1euBLGCViMwXkfN9fg5TDSzpG3CX+6H+hCvdnqiqRwD34aovqtMmXHULACIilE9SFcUT4ybguJDpWE1K/w6c55WUh+K+BBCRw4GXgN/hql5aAG/6jOObSDGIyPHAU8DNQEtvv5+H7DdW89KNuCqj4P6a4aqRvvYRV2X2Ww/3O/saQFVnqOpZuKqdDNx5QVVXqeowXBXeH4B/ikijOGMxVWRJ34TTDNgO7BaRzsBNNXDMV4EcEblIRA4DbgNaV1OMLwK/EJF2ItISuCPayqr6LTAXmA6sUtXV3qKGQAOgCDgoIhcC51YihrtEpIW4fgxjQ5Y1xSX2Itz33424kn7Qt0D74I3rMF4ARolItog0xCXfD1U14pVTJWIeIiL9vGP/L+4+zDwR6Swi/b3j7fVeB3Ef4GoRaeVdGWz3PltJnLGYKrKkb8L5JXAt7h/6T7iSbrXyEuuVwKPAFuAEYCGuX0GiY3wKV/e+FHeT8SUf2zyPuzH7fEjM24DbgZdxN0Mvx315+XE/7opjHfA68LeQ/S4BpgDzvXVOAULrwd8CVgPfikhoNU1w+zdw1Swve9t3wNXzx0VVl+PO+VO4L6RBwBCvfr8hMAl3H+Yb3JXFPd6m5wMrxbUOewS4UlUPxBuPqRpxVafGpBYRycBVJ1yuqh8mOx5j6gor6ZuUISKDRKS5V0VwL65FyPwkh2VMnWJJ36SSPsBaXBXBIOBiVY1UvWOMqQKr3jHGmDRiJX1jjEkjKTfgWqtWrTQzMzPZYRhjTK1SUFCwWVWjNXMGUjDpZ2Zmkp+fn+wwjDGmVhGRWD3LAaveMcaYtGJJ3xhj0oglfWOMSSMpV6dvjKlZP/zwA4WFhezbty/ZoRgfGjVqRPv27alfP9LQS9FZ0jcmzRUWFtKsWTMyMzNxg5uaVKWqbNmyhcLCQjp16hR7gzDqTPVObi5kZkK9eu5nbqUe921M+tq3bx8tW7a0hF8LiAgtW7aM66qsTpT0c3Nh9GjYs8dNr1/vpgFGxD22oDF1nyX82iPe31WdKOnffXdZwg/as8fNN8YYU6ZOJP2vvqrcfGNM6tiyZQvdu3ene/futGnThnbt2pVOHzjgb9j966+/nlWrVkVdZ+rUqeQmqN63T58+LFq0KCH7qml1onqnQwdXpRNuvjEmsXJz3VX0V1+5/7GJE+OrRm3ZsmVpAn3ggQdo2rQpv/rVr8qto6qoKvXqhS+nTp8+PeZxbr311qoHWYfUiZL+xInQuHH5eY0bu/nGmMQJ3j9bvx5Uy+6fVUfDiTVr1tC1a1fGjBlDTk4OmzZtYvTo0QQCAbp06cKECRNK1w2WvIuLi2nRogXjx4/n1FNP5YwzzuC7774D4J577mHy5Mml648fP56ePXty8skn8/HHHwOwe/duLrvsMk499VSGDx9OIBCIWaKfMWMG3bp1o2vXrtx1110AFBcXc/XVV5fOnzJlCgCPPfYYWVlZnHrqqYwcOTLh58yPOpH0R4yAadOgY0cQcT+nTbObuMYkWk3fP1uxYgWjRo1i4cKFtGvXjoceeoj8/HwWL17MW2+9xYoVKw7ZZvv27fTt25fFixdzxhln8Mwzz4Tdt6oyf/58Hn744dIvkMcff5w2bdqwePFixo8fz8KFC6PGV1hYyD333ENeXh4LFy7ko48+4tVXX6WgoIDNmzezdOlSli1bxjXXXAPApEmTWLRoEYsXL+aJJ56I8+xUTZ1I+uAS/Lp1UFLiflrCNybxavr+2QknnMDpp59eOv3CCy+Qk5NDTk4OK1euDJv0Dz/8cAYPHgzAaaedxrp168Lu+9JLLz1knblz5zJs2DAATj31VLp06RI1vnnz5jFgwABatWpF/fr1ueqqq/jggw848cQTWbVqFbfddhtz5syhefPmAHTp0oWRI0eSm5tb5c5V8aozSd8YU/0i3SerrvtnTZo0KX2/evVq/vjHP/Luu++yZMkSBg0aFLa9eoMGDUrfZ2RkUFxcHHbfDRs2PGSdyj5UKtL6LVu2ZMmSJfTp04cpU6Zw0003ATBnzhzGjBnD/PnzCQQCHDx4sFLHSwRL+sYY35J5/2zHjh00a9aMI444gk2bNjFnzpyEH6NPnz68+OKLACxdujTslUSo3r17k5eXx5YtWyguLmbmzJn07duXoqIiVJWf/OQnPPjgg3z22WccPHiQwsJCBgwYwMMPP0xRURF7KtaV1YA60XrHGFMzgtWmiWy941dOTg5ZWVl07dqV448/nrPOOivhx/jZz37GNddcQ3Z2Njk5OXTt2rW0aiac9u3bM2HCBPr164eqctFFF3HBBRfw2WefMWrUKFQVEeH3v/89xcXFXHXVVezcuZOSkhLuuOMOmjVrlvDPEEvKPSM3EAioPUTFmJqzcuVKOnfunOwwUkJxcTHFxcU0atSI1atXM3DgQFavXs1hh6VW+Tjc70xEClQ1EGvb1PokxhiTRLt27eLcc8+luLgYVeVPf/pTyiX8ePn6NCIyCPgjkAE8raoPVVg+BrgVOAjsAkar6goR6QlMC64GPKCqLycqeGOMSaQWLVpQUFCQ7DCqVcwbuSKSAUwFBgNZwHARyaqw2vOq2k1VuwOTgEe9+cuAgDd/EPAnEalbX5vGGFOL+Gm90xNYo6prVfUAMBMYGrqCqu4ImWwCqDd/j6oG20s1Cs43xhiTHH5K3e2ADSHThUCviiuJyK3AOKABMCBkfi/gGaAjcHXIl4Axxpga5qekH27w5kNK7Ko6VVVPAO4A7gmZP09VuwCnA3eKSKNDDiAyWkTyRSS/qKjIf/TGGGMqxU/SLwSOC5luD2yMsv5M4OKKM1V1JbAb6Bpm2TRVDahqoHXr1j5CMsbUFf369Tuko9XkyZO55ZZbom7XtGlTADZu3Mjll18ecd+xmoBPnjy5XCep888/n23btvkJPaoHHniARx55JO79JJqfpL8AOElEOolIA2AYMCt0BRE5KWTyAmC1N79T8MatiHQETgbWJSBuY0wdMXz4cGbOnFlu3syZMxk+fLiv7Y899lheeumlKh+/YtKfPXs2LVq0qPL+Ul3MpO/VwY8F5gArgRdVdbmITBCRId5qY0VkuYgswtXrX+vN7wMs9ua/DNyiqpsT/imMMbXW5Zdfzquvvsr+/fsBWLduHRs3bqRPnz6l7eZzcnLo1q0br7zyyiHbr1u3jq5dXQXC3r17GTZsGNnZ2Vx55ZXs3bu3dL2bb765dFjm+++/H4ApU6awceNG+vfvT//+/QHIzMxk82aXph599FG6du1K165dS4dlXrduHZ07d+anP/0pXbp0YeDAgeWOE86iRYvo3bs32dnZXHLJJXz//felx8/KyiI7O7t0oLf333+/9CEyPXr0YOfOnVU+t+H4aj6pqrOB2RXm3Rfy/rYI2z0HPBdPgMaYmvOLX0CiHwjVvTt4+TKsli1b0rNnT9544w2GDh3KzJkzufLKKxERGjVqxMsvv8wRRxzB5s2b6d27N0OGDIn4nNinnnqKxo0bs2TJEpYsWUJOTk7psokTJ3LUUUdx8OBBzj33XJYsWcLPf/5zHn30UfLy8mjVqlW5fRUUFDB9+nTmzZuHqtKrVy/69u3LkUceyerVq3nhhRf485//zBVXXME///nPqOPjX3PNNTz++OP07duX++67jwcffJDJkyfz0EMP8eWXX9KwYcPSKqVHHnmEqVOnctZZZ7Fr1y4aNTrkNmhcbMA1Y0zShVbxhFbtqCp33XUX2dnZnHfeeXz99dd8++23EffzwQcflCbf7OxssrOzS5e9+OKL5OTk0KNHD5YvXx5zMLW5c+dyySWX0KRJE5o2bcqll17Khx9+CECnTp3o3r07EH34ZnDj+2/bto2+ffsCcO211/LBBx+UxjhixAhmzJhR2vP3rLPOYty4cUyZMoVt27YlvEewdZQyxpSKViKvThdffDHjxo3js88+Y+/evaUl9NzcXIqKiigoKKB+/fpkZmaGHU45VLirgC+//JJHHnmEBQsWcOSRR3LdddfF3E+0ccmCwzKDG5o5VvVOJK+99hoffPABs2bN4je/+Q3Lly9n/PjxXHDBBcyePZvevXvz9ttvc8opp1Rp/+FYSd8Yk3RNmzalX79+3HDDDeVu4G7fvp2jjz6a+vXrk5eXx/pwD8MOcc4555Q+/HzZsmUsWbIEcMMyN2nShObNm/Ptt9/y+uuvl27TrFmzsPXm55xzDv/+97/Zs2cPu3fv5uWXX+bss8+u9Gdr3rw5Rx55ZOlVwnPPPUffvn0pKSlhw4YN9O/fn0mTJrFt2zZ27drFF198Qbdu3bjjjjsIBAJ8/vnnlT5mNFbSN8akhOHDh3PppZeWa8kzYsQILrroIgKBAN27d49Z4r355pu5/vrryc7Opnv37vTs2RNwT8Hq0aMHXbp0OWRY5tGjRzN48GDatm1LXl5e6fycnByuu+660n3ceOON9OjRI2pVTiTPPvssY8aMYc+ePRx//PFMnz6dgwcPMnLkSLZv346qcvvtt9OiRQvuvfde8vLyyMjIICsrq/QpYIliQysbk+ZsaOXaJ56hla16xxhj0oglfWOMSSOW9I0xlX4guEmeeH9XlvSNSXONGjViy5YtlvhrAVVly5YtcXXYstY7xqS59u3bU1hYiI1wWzs0atSI9u3bV3l7S/rGpLn69evTqVOnZIdhaohV7xhjTBqxpG+MMWnEkr4xxqQRS/rGGJNGLOkbY0wasaRvjDFpxJK+McakEUv6xhiTRnwlfREZJCKrRGSNiIwPs3yMiCwVkUUiMldEsrz5PxaRAm9ZgYgMSPQHMMYY41/MpC8iGcBUYDCQBQwPJvUQz6tqN1XtDkwCHvXmbwYuUtVuwLXYQ9KNMSap/JT0ewJrVHWtqh4AZgJDQ1dQ1R0hk00A9eYvVNWN3vzlQCMRaYgxxpik8DP2TjtgQ8h0IdCr4koiciswDmgAhKvGuQxYqKr7w2w7GhgN0KFDBx8hGWOMqQo/Jf1DHy3vleTLzVCdqqonAHcA95TbgUgX4PfATeEOoKrTVDWgqoHWrVv7CMkYY0xV+En6hcBxIdPtgY0R1gVX/XNxcEJE2gMvA9eo6hdVCdIYY0xi+En6C4CTRKSTiDQAhgGzQlcQkZNCJi8AVnvzWwCvAXeq6keJCdkYY0xVxUz6qloMjAXmACuBF1V1uYhMEJEh3mpjRWS5iCzC1etfG5wPnAjc6zXnXCQiRyf+YxhjjPFDUu0RaYFAQPPz85MdhjHG1CoiUqCqgVjrWY9cY4xJI5b0jTEmjVjSN8aYNGJJ3xhj0oglfWOMSSOW9I0xJo1Y0jfGmDRiSd8YY9JIWiT93FzIzIR69dzP3NxkR2SMMcnhZ2jlWi03F0aPhj173PT69W4aYMSI5MVljDHJUOdL+nffXZbwg/bscfONMSbd1Pmk/9VXlZtvjDF1WZ1P+pEexGUP6DLGpKM6n/QnToTGjcvPa9zYzTfGmHRT55P+iBEwbRp07Agi7ue0aXYT1xiTnup86x1wCd6SvDHGpEFJ3xhjTBlL+sYYk0Z8JX0RGSQiq0RkjYiMD7N8jIgs9Z6BO1dEsrz5LUUkT0R2icgTiQ7eGGNM5cRM+iKSAUwFBgNZwPBgUg/xvKp2U9XuwCTgUW/+PuBe4FeJC9kYY0xV+Snp9wTWqOpaVT0AzASGhq6gqjtCJpsA6s3frapzccnfGGNMkvlpvdMO2BAyXQj0qriSiNwKjAMaAAMqE4SIjAZGA3SwXlPGGFNt/JT0Jcw8PWSG6lRVPQG4A7inMkGo6jRVDahqoHXr1pXZ1BhjTCX4SfqFwHEh0+2BjVHWnwlcHE9QxhhjqoefpL8AOElEOolIA2AYMCt0BRE5KWTyAmB14kI0xhiTKDHr9FW1WETGAnOADOAZVV0uIhOAfFWdBYwVkfOAH4DvgWuD24vIOuAIoIGIXAwMVNUVif8oxhhjYhHVQ6rnkyoQCGh+fn6ywzDGmFpFRApUNRBrPeuRa4wxacSSPvYMXWNM+kiLUTajsWfoGmPSSdqX9O0ZusaYdJL2Sd+eoWuMSSdpn/TtGbrGmHSS9knfnqFrjEknaZ/07Rm6xph0kvatd8CeoWuMSR9pX9I3xph0YknfGGPSiCV9H6zHrjGmrrA6/Risx64xpi6xkn4M1mPXGFOXWNKPwXrsGmPqEkv6MViPXWNMXWJJPwbrsWuMqUss6cdgPXaNMXWJr6QvIoNEZJWIrBGR8WGWjxGRpSKySETmikhWyLI7ve1Wicj/JDL4mjJiBKxbByUl7qclfGNMbRUz6YtIBjAVGAxkAcNDk7rneVXtpqrdgUnAo962WcAwoAswCHjS258xxpgk8FPS7wmsUdW1qnoAmAkMDV1BVXeETDYBgk9bHwrMVNX9qvolsMbbnzHGmCTw0zmrHbAhZLoQ6FVxJRG5FRgHNAAGhGz7aYVt21UpUmOMMXHzU9KXMPP0kBmqU1X1BOAO4J7KbCsio0UkX0Tyi4qKfIRkjDGmKvwk/ULguJDp9sDGKOvPBC6uzLaqOk1VA6oaaN26tY+QjDHGVIWfpL8AOElEOolIA9yN2VmhK4jISSGTFwCrvfezgGEi0lBEOgEnAfPjD9sYY0xVxEz6qloMjAXmACuBF1V1uYhMEJEh3mpjRWS5iCzC1etf6227HHgRWAG8Adyqqger4XMklY3CaYypLUT1kCr2pAoEApqfn5/sMHyrOAonuB671oHLGFOTRKRAVQOx1rMeuXHyMwqnXQkYY1KFjacfp1ijcNp4/MaYVGIl/TjFGoXTxuM3xqQSS/pxijUKp43Hb4xJJZb04xRrFE4bj98Yk0os6SdAtFE4bTx+Y0wqsaRfzWJdCVjLHmNMTbLWOzVgxIjwLXWsZY8xpqZZST+JrGWPMaamWdJPImvZY4ypaZb0k8ha9hhjapol/SRKRMseuxFsjKkMS/pJFKtlTyzBG8Hr14Nq2Y1gS/zGmEhslM1aLDPTJfqKOnZ0/QWMMenDRtlMA3Yj2BhTWZb0U1y0Onu7EWyMqSxL+iksVp29nxvBsW702o1gY9KMqqbU67TTTlPjdOyo6tJ9+VfHjmXrzJjhpkXczxkzyi9r3Lj8to0bl60Ta7kxpvYA8tVHjrUbuSmsXj2XiisScYO7xRLrRq/dCDam7kjojVwRGSQiq0RkjYiMD7N8nIisEJElIvKOiHQMWfZ7EVnmva6s3MdIb/HW2ce60Ws3go1JPzGTvohkAFOBwUAWMFxEsiqsthAIqGo28BIwydv2AiAH6A70Av5XRI5IXPh1W7ydt2J9adiNYGPSj5+Sfk9gjaquVdUDwExgaOgKqpqnqsGhwz4F2nvvs4D3VbVYVXcDi4FBiQm97ou381asLw0b69+Y9OMn6bcDNoRMF3rzIhkFvO69XwwMFpHGItIK6A8cV3EDERktIvkikl9UVOQv8jQR7QEtfraN9qUR75eKMab28TOevoSZF/bur4iMBAJAXwBVfVNETgc+BoqAT4DiQ3amOg2YBu5Grq/IjS+RxvL3u9wYU7f4KekXUr503h7YWHElETkPuBsYoqr7g/NVdaKqdlfVH+O+QFbHF7Ixxpiq8pP0FwAniUgnEWkADANmha4gIj2AP+ES/nch8zNEpKX3PhvIBt5MVPDGGGMqJ2b1jqoWi8hYYA6QATyjqstFZAKuM8As4GGgKfAPEQH4SlWHAPWBD715O4CRqnpI9Y4xxpia4esZuao6G5hdYd59Ie/Pi7DdPlwLHlMH5ea6Rzt+9ZVr5jlxot0fMCbV2dg7JqpIY/PYWP7G1E6+SvomPQUTe/Dh7cHEDtEf6m6lfWNSl5X0TUTRErsN4WBM7WRJ30QULbEnYggHG9bZmJpnSd9EFC2xxzuEg90TMCY5LOmbiKIl9niHcIhWdWSMqT6W9E1Efsbuqeq4QH7uCcRT/WNVR8aEZ0nfRBVPYo/n+b7xVP9Y1ZExkVnSN9Ui3uf7xlP9Y1VHxkRmSd9Ui1iJN1bVUTxNQq05qTGRWdI31cJP4o1WdeSnSWik6iN7IpgxkVnSN9Ui3sQbq/onWvWRPRHMmMgs6ZtqEW/ijVX9E2sYCHsimDHhiWpqPagqEAhofn5+ssMwCVCdo3DWq+dK+BWJuOqiZMZmTDKISIGqBmKtZwOumWpTnY9i7NDBVemEmx9LtIHkLPGbus6qd0ytFE/1kTXpNOnMkr6pleKpt09Ek07r8WtqK0v6ptaqam/heJqDBpdZj19TW/lK+iIySERWicgaERkfZvk4EVkhIktE5B0R6RiybJKILBeRlSIyRbwH5hqTLPE0BwWrHjK1W8ykLyIZwFRgMO55t8NFpOJzbxcCAVXNBl4CJnnbngmcBWQDXYHTgb4Ji96YKoinOShYj19Tu/kp6fcE1qjqWlU9AMwEhoauoKp5qhr8N/kUaB9cBDQCGgANgfrAt4kI3Jh4RKsaipXUrcevqc38JP12wIaQ6UJvXiSjgNcBVPUTIA/Y5L3mqOrKihuIyGgRyReR/KKiIr+xG1MtYiV1Py2H7EavSVV+kn64OviwPbpEZCQQAB72pk8EOuNK/u2AASJyziE7U52mqgFVDbRu3dpv7MZUi1hJPVb1kN3oNanMT9IvBI4LmW4PbKy4koicB9wNDFHV/d7sS4BPVXWXqu7CXQH0ji9kY6qXn+ag0aqHYt0TiHUVEO9yY6JS1agvXK/dtUAnXN38YqBLhXV6AF8AJ1WYfyXwtreP+sA7wEXRjnfaaaepMbWZiKor45d/iajOmKHauHH5+Y0bu/mq8S8PrtOxoztex47ll5m6C8jXGPlcVf2NvSMi5wOTgQzgGVWdKCITvIPMEpG3gW64enuAr1R1iNfy50ngHFyV0BuqOi7asWzsHVPbZWaGHyKio9eQOdKydeuib+tnecUhJsBVTdmAc3Wf37F3bMA1YxIsWuK9+uroA8XFGkgu1vJYXwqm7vKb9K1HrjEJFu2eQKyWQfEutz4EJhZL+sZUg0g3emO1DIp3ufUhMLFY0jemBsVqGRTvcntqmInF6vSNqWNiPSDGHiBTN1mdvjFpKlofgng7jlkfgzrAT7vOmnxZO31jqk/HjuH7EHTs6JZHa+NfE30MTNWRyHb6Ncmqd4ypPtGafD73XPQ2/vH2IbDmpNXL2ukbYw4RT8exePsQxPswexOd1ekbYw4RrXVPvENKx7vc1AxL+sYkwezZcOyxsGJFzR43no5j1d3HINl27YKePeGJJ5IdSTXzU/Ffky+7kWvSwciR7kZmVpbqrl3JjsZJxGBu1b083s8Xbd+33OI+c21NQfi8kZv0JF/xZUnf1HUHD6q2bq3atatLQNdem+yIyiRzhM7qbN0Ta99vv+3mtW3rfn7zTfzHrGmW9I1JUfn57j/vuedU77vPvZ8+PdlRJV88zUnj2ff27aodOqj+6Eeqc+e6+c8+m+hPV/38Jv3Dkly7lHD790Nhobv5VPFVVORaL3Tu7F6nnOJeTZu0selfAAAUZklEQVQmO2qTTt54w/0cOBCGD4cPP4RbboFAALp2TW5syRTtRnLFkUuDncrAX2/iaPv+5S9dzpg7F3r1gmOOgddfh2uuqfxnqBX8fDPU5KuqJf2vv1Zt0yb8t3mbNqo9e6oOHqx6yimqhx1Wfvlxx6kOHKg6caK79DamOvXpU77eeNMm1WOOcX+bO3cmL65ki1Yaj3UVoBr9SiDS9kcf7X7++tdl6157repRR6kWF1f/Z04k0q16Z/9+1RtuUH3wQXep/M47qqtXq+7bF37dFStU//Uvl+hHjlTt3t2djYcfrtLhjfHl++9VMzJU7767/Py333bJ6uqrVUtKkhNbskWrd4/2NLJY20Zafvjhqkce6W6m791bFsfMmW75xx/X/DmIR9ol/XiVlKhedpm7Cpg3LykhmDTwj3+4/7oPPzx02QMPuGV/+UvNx5UqIpXWY5X0q3Il0KeP+wJesKB8DFu2qNarp3rvvdX7WRPNkn4VbN3qbugcf7zqtm1JC8PUYaNGqTZvrvrDD4cuKy5WPfdc1UaNVJcsqfnYUlmsknysK4GKXnnFLb/nnvDLzzxTNRA4NIZo4xIl+7nECU36wCBgFbAGGB9m+ThgBbAE9/Dzjt78/sCikNc+4OJox0p2652PPnLf/sOGpe9ltqkeJSWq7dq5K8pIvvnG3YM6+eT0rt8Ppyp19qEl/aDNm909lOxsV9Ubzm9+47b/9tuyY0f60klEU9NEfGkkLOnjHob+BXA80ABYDGRVWKc/0Nh7fzPw9zD7OQrYGlwv0ivZSV/V1fOn+2W2SbylS93f1Z//HH29d9911QtXXBH+isAcqjKJd/hwV427cGHk/QWb1f7tb2463pvMiYo9mkQm/TOAOSHTdwJ3Rlm/B/BRmPmjgdxYx0uFpB+8zD78cHfD15hEePhh9x+3YUPsdSdNcusOHGhVjX75KS0H76lMmBB9+w4dVI84wn1BqEavPvJTtZSoq5RoEpn0LweeDpm+GngiyvpPAPeEmf8ucGGs46VC0ldV3bjR9ZrMzlbdsyfZ0Zi64NxzXS9cv55+2pVIs7JUv/yy2sJKG//4h2rDhq6u/sCB8svClbYzMlSbNnWFwHhK+om+HxFJIpP+T8Ik/ccjrDsS+BRoWGF+W6AIqB9hu9FAPpDfoUOHyn3SajR7tjtDt9yS7EhMbbdzp2qDBqq/+lXltnvnHdUWLVx78k8+qZ7Y0sFjj7kkeuaZrk6/okiJG9x5j6dOP9aXQrCvQCqV9H1V7wDnASuBo8Msuw2Y5iegVCnpB/3yl+4s/etfyY7E1Gb/+Y/7O3r77cpvu3Kl6gknuFLqzJmJj60uO3hQ9fbb3bm/9NLIV+2RStvghspQrXrrnWgl+Y8/doWBevUif2n4lcikfxiwFugUciO3S4V1eng3e0+KsI9Pgf5+Akq1pL9/v7scbNFCdf36qu/n+++tNVA6u/VW948crrOgH0VFrl05uJYl9rcU29697mY4qP7859F72EYqjTdsqHr66fHFEWnfbdu6zmEnnaQ6dWoKtd5x++J84L9eYr/bmzcBGOK9fxv4NqRp5qyQbTOBr4F6fo6VaklfVXXNGtVmzVyv3com/pIS1SeecHWzN9xg/6zp6oQTVC+8ML597NvneuyC+1nVL5Dq9uc/x26hVN22bFE9+2x3rv7wh9j/d5GqaC67zCXi776reizh9t2okUv4bdqorl1b9X2HSmjSr8lXKiZ9VVe/36yZasuWqnPm+Ntm717V6693Z/lHP3I///d/qzdOk3pWr3a/+yeeiH9fJSVlbcibNlXt0kX1/PNVx4xR/X//TzU31/X2/eqr5BQwvvpKtX59lyjz8mr++Kqq69apdu7sqk0qUx0WropmwQJ3rp97Lr6YQvfdvr3qsce6TnqLF8e331CW9KvBqlVlY6A/+GD0wdk2bHCXheC6cxcXlz2kYdKkmovZJN/jj7vf+5o1idvn7NmuyuLii1VzclxhpGL1QSCg+sYbNZv8x451V7WZmW4gw++/r7ljq7q+EG3buurY996Lf38HD7obrVddFf++VN0Dc3r1ctVG77+fmH0GWdKvJrt2lV1iDxoUviXA+++7P5SmTVVffrlsfnGx6pVXum0T2fGrqEj1gw9cCW/uXNer+JNPVD/91I0jtGBB8pqdlpTYyKUXXODqbavbrl2uX8kbb7jWKsG65LPPdn8f1W3jRpfMRo1SnT/fNXlMVLL0Y9s2N4TKsceqLluWuP1efbX7Uo131M1du1zOqFdP9d//TkxsoSzpV6OSEtX/+z93+dihg/sDD85//HFX0vnRj8J37Nq/33W4qVev/BdCVWJ47z03XET9+oeW8iq+srLcEL415cAB15sxK8slvJo8dirZu9d18vvZz2r+2Pv2uSql4JDj//M/rqdpdbn9dpfog1c0wWqo3Fz/+9izx8UcHP7Ar5ISV6DKyHCFnkR6/nn3OWINxLhrl+tbcf/9rrrt0ktVzzpL9cQTXdVw8H+xuu53WNKvAfPnu6TfoIFL9tdd587ohRdG70W5c2fZJV5l6z23bHGluFNOccdq0UL1tttc6e7NN939hjfecJf/r72m+uqr7qqiSRM3nsvXX8f1kWPau1f1ySfd5T246rDGjV0VxI4d1XvsVPTmm+48vPZa8mLYvdtVKR51lJY2XUxkSVjVJenDDy//6McffnDt4ps3d/XssRQVqZ5xhouxRw/3RCu/pk1z2/3ud5UO3VdcIm4U1Eg2bXLPSAgm9pYtXYGnf39XMPv5z93wLm+8kfj4gizp15DNm90lW/CXfe+9/qozNm92N5uaNVP97LPo6x486Eov11zj7vqDau/eqn/9q/uH9uPDD11104knupttibZjh0ssxxxTFt9//uNKYK+95kpgAwdGHuCqrho3zn25p8LDz7dtc6XQYKmzWTPXquiMM1SHDlX96U/dqJNTprjCQ2XuBfz61+7qddWq8vO/+ML93fXtG7165Isv3NVxw4aq48e7q+X+/cuPcx/J0qXu/2LgwOqrSuzVy73CWbHCVaU1buyqbZI1XpIl/Rp08KC7JJ09u3LbbdjgrhRat1b973/dvB07XIJ/6il3iXjmmWX/pE2bqt58s+qiRVWL85NP3HginTolrlv/9u2u80qLFi7GH//YXb1UTBh/+YuWNjWsDc1Wt251465066Y6YICrOhg71t3Af/JJ16X//fdj3yvJynLnJJUUFak+8oi7Qhw+3A0P0a2b+8IO7SQ0ZYr//TVpErn+fvp0t7/f/z788gUL3D2wo45y96RUXWsZcE0mo31Z7NrlznGbNtX7MPMHHnCl/aKi8vPfe8/97R9zzKHj8tc0S/q1xKpVLukfc4y7CRVaD9+8ubsJd+utrlSfiOqR+fPdH2mHDq50FY+PPy6rxrnkkrJ7G5FMmODWHT8+vuP6tWGDSzh+SouhFi1yv4v69V1zyDPOcPclgl9soa82bVQnTw6f/Nev19J24rVFcbFLbEOHuquzt96Kvc1dd7mEuHx5+OUlJaqXX+7OZ8Wr2tdecyXkzEzX8zjUH/7gzt+YMZELCjfc4I5dlZ7OlTFvnh5yfyI3132mzp1TY2wkS/q1SEGBu5S94grV3/5WddYslzCqq0RcUOBKVe3bl11hVEZxsUvgGRnun9XvjbOSEtWbbnJ/dY8/XvnjVsbzz5cl6RNOUH39dX/bPfecq5s+9tjwj8s7cMC1Ulm82D2Io3//yMk/WM8cKRmmsh073P2YI4+M/jeydau7Ev3JT6Lvb/Nmd047dy47R9Omub+hnJzIN/rvuMOdw/vvP3RZbq5bFulBKIl08KBqq1bu0aolJa5PBLhqq61bq//4fljSN1EtWuT+iNu2PbSEFc369WU9HYcPr/ywv8XFrhQp4qpIEi1YLRO8r5CbW9Yx7rLLIt/POHDAtbAB1XPOqVxVQV6ear9+Wtq1/o9/dFcXl17q2qrXhuqscNaudTckO3eO/Hu+/373uf10Mgre1L71VnfvK9jsOdrDYkpKyjo4Tp1aNv+//3XVnWefXXN16CNGuKvy0aNdPFddlVq9oi3pm5iWLXPVSscc45qRff559AT1j3+40nPTpqrPPlv1ZLZ7t6sySXQHlbffdlcvhx3mmgsGk8G+fe4KqlEjV/c8aVL5oXU3biwb1+b22w8ddtevvDz3hQGuVNukiUsQtVlenjufF1xwaN36tm2uCvKSS/zv7xe/0NKqsVGj/J3rH35QvegiV1D4+9/d77NHD3e16ufZBIkyY0ZZ7HfdlXr9TyzpG19Wrix/L6FVK9fL85FHXOeu/fvdzbJRo9zy0093wwrEa/Nm1+y0eXPVF190VUQrVrjL/MrWwe/dWzaS4sknR76htnatSx7BfgvvveduHLZp4+qVX3gh/s+lWj75V/bmfip68kn3We64o/z8YDv8WK3PQu3d6670fve7yhUadu92bd7r11cdPNgdd9Ys/9snwtatrmHFtGk1e1y/LOkb30pKXCn/6addX4MTTyz7Ejj8cHclIKJ6551VLwWHs26dKxGH60zWqJGrKunc2dWbjxjhxi2aPNl9Scyd65L4ggWu7jlYbeCnCesrr5T1Vs3IcJ936dLEfS5Vd043bkzsPpNpzBh3voKjP+7Y4Ura8Q4iVxlbt5b9rm+/veaOW1v4Tfri1k0dgUBA8/Pzkx1G2vvmG5g7171Wr4Zf/hIGDEj8cXbsgJUr4fvv3WvbtvLvt251sWzc6F4HDhy6jzZtYPp0GDTI/3H37IGHHoLCQnj0UWjRInGfqS46cAAGDoRPP4UPP4R334Xx42HePOjZs+bi2LQJXnoJbroJGjSouePWBiJSoKqBmOtZ0je1hSps2VL2BbBxo/vSGDkSWrVKdnR1X1GRS/D798MPP0AgAK+/nuyoTJDfpH9YTQRjTCKIuOTeqhVkZyc7mvTTujW88gqceSbs3g333ZfsiExVWNI3xviWnQ2vvgqffQZnnJHsaExVWNI3xlRKv37uZWqneskOwBhjTM2xpG+MMWnEV9IXkUEiskpE1ojI+DDLx4nIChFZIiLviEjHkGUdRORNEVnprZOZuPCNMcZURsykLyIZwFRgMJAFDBeRrAqrLQQCqpoNvARMCln2N+BhVe0M9AS+S0TgxhhjKs9PSb8nsEZV16rqAWAmMDR0BVXNU9U93uSnQHsA78vhMFV9y1tvV8h6xhhjapifpN8O2BAyXejNi2QUEOyy8SNgm4j8S0QWisjD3pVDOSIyWkTyRSS/qKjIb+zGGGMqyU/SlzDzwnbjFZGRQAB42Jt1GHA28CvgdOB44LpDdqY6TVUDqhpo3bq1j5CMMcZUhZ+kXwgcFzLdHthYcSUROQ+4GxiiqvtDtl3oVQ0VA/8GcuIL2RhjTFX56Zy1ADhJRDoBXwPDgKtCVxCRHsCfgEGq+l2FbY8UkdaqWgQMAKIOrFNQULBZRNZX4jNU1ArYHMf21cliqxqLrWostqqprbF1jDC/HF8DronI+cBkIAN4RlUnisgE3FCes0TkbaAbsMnb5CtVHeJt+2PgD7hqogJgtHdDuFqISL6fQYeSwWKrGoutaiy2qqnrsfkahkFVZwOzK8y7L+T9eVG2fQuw4bGMMSYFWI9cY4xJI3Ux6U9LdgBRWGxVY7FVjcVWNXU6tpR7iIoxxpjqUxdL+sYYYyKwpG+MMWmkziT9WCOBJpOIrBORpSKySESS/gBgEXlGRL4TkWUh844SkbdEZLX388gUiesBEfnaO3eLvObDNU5EjhORPG+02OUicps3PxXOW6TYkn7uRKSRiMwXkcVebA968zuJyDzvvP1dRGr8MedRYvuriHwZct6613RsITFmeEPYvOpNx3/eVLXWv3D9B77ADfPQAFgMZCU7rpD41gGtkh1HSDzn4HpGLwuZNwkY770fD/w+ReJ6APhVCpyztkCO974Z8F/cqLOpcN4ixZb0c4frn9PUe18fmAf0Bl4Ehnnz/w+4OYVi+ytwebL/5ry4xgHPA69603Gft7pS0o85Eqgpo6ofAFsrzB4KPOu9fxa4uEaDImJcKUFVN6nqZ977ncBK3MCDqXDeIsWWdOrs8ibrey/F9c5/yZufrPMWKbaUICLtgQuAp71pIQHnra4k/cqOBFrTFHhTRApEZHSyg4ngGFXdBC6JAEcnOZ5QY70H9DyTjOqTirwHAfXAlQxT6rxViA1S4Nx5VRSLcM/SeAt3Vb5N3XhckMT/14qxqWrwvE30zttjItIwGbHhRkH4NVDiTbckAeetriR93yOBJslZqpqDexDNrSJyTrIDqkWeAk4AuuOG+fhDMoMRkabAP4FfqOqOZMZSUZjYUuLcqepBVe2OG6yxJ9A53Go1G5V30AqxiUhX4E7gFNzIwEcBd9R0XCJyIfCdqhaEzg6zaqXPW11J+r5GAk0WVd3o/fwOeBn3h59qvhWRtgDez5R4wpmqfuv9Y5YAfyaJ505E6uOSaq6q/subnRLnLVxsqXTuvHi2Ae/h6s1biEhwGJik/7+GxDbIqy5TdaMFTyc55+0sYIiIrMNVVw/AlfzjPm91JemXjgTq3c0eBsxKckwAiEgTEWkWfA8MBJZF3yopZgHXeu+vBV5JYiylggnVcwlJOndefepfgJWq+mjIoqSft0ixpcK5E5HWItLCe384cB7unkMecLm3WrLOW7jYPg/5EhdcnXmNnzdVvVNV26tqJi6fvauqI0jEeUv23ekE3uU+H9dq4Qvg7mTHExLX8bjWRIuB5akQG/AC7nL/B9xV0ihcfeE7wGrv51EpEtdzwFJgCS7Btk3SOeuDu5ReAizyXuenyHmLFFvSzx1usMWFXgzLgPu8+ccD84E1wD+AhikU27veeVsGzMBr4ZOsF9CPstY7cZ83G4bBGGPSSF2p3jHGGOODJX1jjEkjlvSNMSaNWNI3xpg0YknfGGPSiCV9Y4xJI5b0jTEmjfx/BSOLdo8qBHUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(len(loss))\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great success; we are no longer overfitting during the first 30 epochs. However, while we have more stable evaluation scores, our best \n",
    "scores are not much lower than they were previously.\n",
    "\n",
    "成功！前 30 个轮次不再过拟合。不过，虽然评估分数更加稳定，但最佳分数并没有比之前低很多。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stacking recurrent layers\n",
    "\n",
    "Since we are no longer overfitting yet we seem to have hit a performance bottleneck, we should start considering increasing the capacity of \n",
    "our network. If you remember our description of the \"universal machine learning workflow\": it is a generally a good idea to increase the \n",
    "capacity of your network until overfitting becomes your primary obstacle (assuming that you are already taking basic steps to mitigate \n",
    "overfitting, such as using dropout). As long as you are not overfitting too badly, then you are likely under-capacity.\n",
    "\n",
    "Increasing network capacity is typically done by increasing the number of units in the layers, or adding more layers. Recurrent layer \n",
    "stacking is a classic way to build more powerful recurrent networks: for instance, what currently powers the Google translate algorithm is \n",
    "a stack of seven large LSTM layers -- that's huge.\n",
    "\n",
    "To stack recurrent layers on top of each other in Keras, all intermediate layers should return their full sequence of outputs (a 3D tensor) \n",
    "rather than their output at the last timestep. This is done by specifying `return_sequences=True`: \n",
    "\n",
    "## 循环层堆叠\n",
    "\n",
    "模型不再过拟合，但似乎遇到了性能瓶颈，所以我们应该考虑增加网络容量。回想一下机 器学习的通用工作流程：增加网络容量通常是一个好主意，直到过拟合变成主要的障碍（假设你已经采取基本步骤来降低过拟合，比如使用 dropout）。只要过拟合不是太严重，那么很可能是容量不足的问题。\n",
    "\n",
    "增加网络容量的通常做法是增加每层单元数或增加层数。循环层堆叠（recurrent layer stacking）是构建更加强大的循环网络的经典方法，例如，目前谷歌翻译算法就是 7 个大型LSTM 层的堆叠——这个架构很大。\n",
    "\n",
    "在 Keras 中逐个堆叠循环层，所有中间层都应该返回完整的输出序列（一个 3D 张量），而不是只返回最后一个时间步的输出。这可以通过指定return_sequences=True 来实现。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/40\n",
      "500/500 [==============================] - 533s 1s/step - loss: 0.3353 - val_loss: 0.2806\n",
      "Epoch 2/40\n",
      "500/500 [==============================] - 531s 1s/step - loss: 0.3133 - val_loss: 0.2737\n",
      "Epoch 3/40\n",
      "500/500 [==============================] - 531s 1s/step - loss: 0.3072 - val_loss: 0.2716\n",
      "Epoch 4/40\n",
      "500/500 [==============================] - 529s 1s/step - loss: 0.3032 - val_loss: 0.2665\n",
      "Epoch 5/40\n",
      "500/500 [==============================] - 534s 1s/step - loss: 0.2978 - val_loss: 0.2668\n",
      "Epoch 6/40\n",
      "500/500 [==============================] - 532s 1s/step - loss: 0.2966 - val_loss: 0.2656\n",
      "Epoch 7/40\n",
      "500/500 [==============================] - 533s 1s/step - loss: 0.2932 - val_loss: 0.2654\n",
      "Epoch 8/40\n",
      "500/500 [==============================] - 533s 1s/step - loss: 0.2921 - val_loss: 0.2707\n",
      "Epoch 9/40\n",
      "500/500 [==============================] - 532s 1s/step - loss: 0.2881 - val_loss: 0.2717\n",
      "Epoch 10/40\n",
      "500/500 [==============================] - 530s 1s/step - loss: 0.2857 - val_loss: 0.2675\n",
      "Epoch 11/40\n",
      "500/500 [==============================] - 531s 1s/step - loss: 0.2822 - val_loss: 0.2677\n",
      "Epoch 12/40\n",
      "500/500 [==============================] - 533s 1s/step - loss: 0.2831 - val_loss: 0.2703\n",
      "Epoch 13/40\n",
      "500/500 [==============================] - 535s 1s/step - loss: 0.2789 - val_loss: 0.2614\n",
      "Epoch 14/40\n",
      "500/500 [==============================] - 534s 1s/step - loss: 0.2778 - val_loss: 0.2669\n",
      "Epoch 15/40\n",
      "500/500 [==============================] - 525s 1s/step - loss: 0.2767 - val_loss: 0.2668\n",
      "Epoch 16/40\n",
      "500/500 [==============================] - 524s 1s/step - loss: 0.2732 - val_loss: 0.2792\n",
      "Epoch 17/40\n",
      "500/500 [==============================] - 528s 1s/step - loss: 0.2728 - val_loss: 0.2648\n",
      "Epoch 18/40\n",
      "500/500 [==============================] - 528s 1s/step - loss: 0.2716 - val_loss: 0.2636\n",
      "Epoch 19/40\n",
      "500/500 [==============================] - 525s 1s/step - loss: 0.2695 - val_loss: 0.2626\n",
      "Epoch 20/40\n",
      "500/500 [==============================] - 524s 1s/step - loss: 0.2695 - val_loss: 0.2647\n",
      "Epoch 21/40\n",
      "500/500 [==============================] - 533s 1s/step - loss: 0.2695 - val_loss: 0.2714\n",
      "Epoch 22/40\n",
      "500/500 [==============================] - 532s 1s/step - loss: 0.2667 - val_loss: 0.2731\n",
      "Epoch 23/40\n",
      "500/500 [==============================] - 532s 1s/step - loss: 0.2655 - val_loss: 0.2696\n",
      "Epoch 24/40\n",
      "500/500 [==============================] - 534s 1s/step - loss: 0.2657 - val_loss: 0.2751\n",
      "Epoch 25/40\n",
      "500/500 [==============================] - 533s 1s/step - loss: 0.2652 - val_loss: 0.2679\n",
      "Epoch 26/40\n",
      "500/500 [==============================] - 532s 1s/step - loss: 0.2650 - val_loss: 0.2675\n",
      "Epoch 27/40\n",
      "500/500 [==============================] - 532s 1s/step - loss: 0.2618 - val_loss: 0.2679\n",
      "Epoch 28/40\n",
      "500/500 [==============================] - 533s 1s/step - loss: 0.2626 - val_loss: 0.2742\n",
      "Epoch 29/40\n",
      "500/500 [==============================] - 532s 1s/step - loss: 0.2614 - val_loss: 0.2740\n",
      "Epoch 30/40\n",
      "500/500 [==============================] - 532s 1s/step - loss: 0.2600 - val_loss: 0.2724\n",
      "Epoch 31/40\n",
      "500/500 [==============================] - 531s 1s/step - loss: 0.2597 - val_loss: 0.2707\n",
      "Epoch 32/40\n",
      "500/500 [==============================] - 532s 1s/step - loss: 0.2596 - val_loss: 0.2695\n",
      "Epoch 33/40\n",
      "500/500 [==============================] - 533s 1s/step - loss: 0.2582 - val_loss: 0.2673\n",
      "Epoch 34/40\n",
      "500/500 [==============================] - 529s 1s/step - loss: 0.2569 - val_loss: 0.2713\n",
      "Epoch 35/40\n",
      "500/500 [==============================] - 532s 1s/step - loss: 0.2582 - val_loss: 0.2811\n",
      "Epoch 36/40\n",
      "500/500 [==============================] - 531s 1s/step - loss: 0.2571 - val_loss: 0.2687\n",
      "Epoch 37/40\n",
      "500/500 [==============================] - 531s 1s/step - loss: 0.2554 - val_loss: 0.2703\n",
      "Epoch 38/40\n",
      "500/500 [==============================] - 531s 1s/step - loss: 0.2577 - val_loss: 0.2706\n",
      "Epoch 39/40\n",
      "500/500 [==============================] - 531s 1s/step - loss: 0.2558 - val_loss: 0.2829\n",
      "Epoch 40/40\n",
      "500/500 [==============================] - 533s 1s/step - loss: 0.2539 - val_loss: 0.2728\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras import layers\n",
    "from keras.optimizers import RMSprop\n",
    "\n",
    "model = Sequential()\n",
    "model.add(layers.GRU(32,\n",
    "                     dropout=0.1,\n",
    "                     recurrent_dropout=0.5,\n",
    "                     return_sequences=True,\n",
    "                     input_shape=(None, float_data.shape[-1])))\n",
    "model.add(layers.GRU(64, activation='relu',\n",
    "                     dropout=0.1, \n",
    "                     recurrent_dropout=0.5))\n",
    "model.add(layers.Dense(1))\n",
    "\n",
    "model.compile(optimizer=RMSprop(), loss='mae')\n",
    "history = model.fit_generator(train_gen,\n",
    "                              steps_per_epoch=500,\n",
    "                              epochs=40,\n",
    "                              validation_data=val_gen,\n",
    "                              validation_steps=val_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at our results:\n",
    "\n",
    "看下结果："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmcFMX5+PHPwy2HoECiglyCBzfrChhQzhgQBSUmyuEVFSX6ixFNRFFjTPxG0SBBSSImGiOriFckRkMMoIiJ3IccIoiLLudCuEFgd5/fH9XDDsvsTM/OuTPP+/Wa10z3VHfX9Ow+U11VXSWqijHGmOxQJdUZMMYYkzwW9I0xJotY0DfGmCxiQd8YY7KIBX1jjMkiFvSNMSaLWNA3URGRqiKyX0SaxTNtKolIaxGJe99lEekvIvlBy2tF5CI/aStwrD+JyP0V3T7Mfn8tIn+J935N6lRLdQZMYonI/qDF2sBhoNhbvlVV86LZn6oWA3XjnTYbqOo58diPiNwMjFTV3kH7vjke+zaZz4J+hlPVY0HXK0nerKr/Li+9iFRT1aJk5M0Yk3xWvZPlvMv3V0XkFRHZB4wUkQtF5BMR2S0iW0RkkohU99JXExEVkRbe8lTv/fdEZJ+I/FdEWkab1nt/oIh8LiJ7RORpEflYRG4oJ99+8niriKwXkV0iMilo26oi8pSI7BSRL4ABYc7PAyIyrcy6ySIywXt9s4is8T7PF14pvLx9FYhIb+91bRF5ycvbKuD8EMfd4O13lYgM9tZ3AJ4BLvKqznYEnduHg7a/zfvsO0XkbyJyup9zE4mIXOHlZ7eIzBaRc4Leu19ENovIXhH5LOizdheRJd76bSLyhN/jmQRQVXtkyQPIB/qXWfdr4AhwOa4QcBJwAdANdyXYCvgcuMNLXw1QoIW3PBXYAeQC1YFXgakVSPstYB8wxHtvDHAUuKGcz+Inj28D9YEWwP8Cnx24A1gFNAUaAnPdv0LI47QC9gN1gva9Hcj1li/30gjQFzgEdPTe6w/kB+2rAOjtvX4S+AA4BWgOrC6T9ofA6d53MtzLw7e9924GPiiTz6nAw97rS7w8dgZqAb8HZvs5NyE+/6+Bv3ivz/Py0df7ju73znt1oB2wETjNS9sSaOW9XggM817XA7ql+n8hmx9W0jcA81T176paoqqHVHWhqs5X1SJV3QBMAXqF2f51VV2kqkeBPFywiTbtZcAyVX3be+8p3A9ESD7z+BtV3aOq+bgAGzjWD4GnVLVAVXcCj4U5zgZgJe7HCOC7wG5VXeS9/3dV3aDObGAWELKxtowfAr9W1V2quhFXeg8+7nRV3eJ9Jy/jfrBzfewXYATwJ1VdpqrfAGOBXiLSNChNeecmnGuAGao62/uOHgNOxv34FuF+YNp5VYRfeucO3I93GxFpqKr7VHW+z89hEsCCvgH4OnhBRM4VkX+IyFYR2Qs8AjQKs/3WoNcHCd94W17aM4LzoaqKKxmH5DOPvo6FK6GG8zIwzHs9HPdjFcjHZSIyX0T+JyK7caXscOcq4PRweRCRG0RkuVeNshs41+d+wX2+Y/tT1b3ALqBJUJpovrPy9luC+46aqOpa4G7c97Ddqy48zUt6I9AWWCsiC0TkUp+fwySABX0D7nI/2LO40m1rVT0ZeAhXfZFIW3DVLQCIiHB8kCorljxuAc4MWo7UpfRVoL9XUh6C+xFARE4CXgd+g6t6aQD8y2c+tpaXBxFpBfwBGA009Pb7WdB+I3Uv3YyrMgrsrx6uGmmTj3xFs98quO9sE4CqTlXVHriqnaq484KqrlXVa3BVeL8F3hCRWjHmxVSQBX0TSj1gD3BARM4Dbk3CMd8BckTkchGpBtwJNE5QHqcDPxWRJiLSELg3XGJV3QbMA14A1qrqOu+tmkANoBAoFpHLgH5R5OF+EWkg7j6GO4Leq4sL7IW437+bcSX9gG1A00DDdQivADeJSEcRqYkLvh+parlXTlHkebCI9PaO/TNcO8x8ETlPRPp4xzvkPYpxH+BaEWnkXRns8T5bSYx5MRVkQd+EcjdwPe4f+llcSTehvMB6NTAB2AmcBSzF3VcQ7zz+AVf3/imukfF1H9u8jGuYfTkoz7uBu4C3cI2hV+F+vPz4Be6KIx94D/hr0H5XAJOABV6ac4HgevD3gXXANhEJrqYJbP9PXDXLW972zXD1/DFR1VW4c/4H3A/SAGCwV79fExiPa4fZiruyeMDb9FJgjbjeYU8CV6vqkVjzYypGXNWpMelFRKriqhOuUtWPUp0fYzKFlfRN2hCRASJS36sieBDXI2RBirNlTEaxoG/SSU9gA66KYABwhaqWV71jjKkAq94xxpgsYiV9Y4zJImk34FqjRo20RYsWqc6GMcZUKosXL96hquG6OQNpGPRbtGjBokWLUp0NY4ypVEQk0p3lgFXvGGNMVrGgb4wxWcSCvjHGZJG0q9M3xiTX0aNHKSgo4Jtvvkl1VowPtWrVomnTplSvXt7QS+FZ0DcmyxUUFFCvXj1atGiBG9zUpCtVZefOnRQUFNCyZcvIG4SQMdU7eXnQogVUqeKe86Ka7tuY7PXNN9/QsGFDC/iVgIjQsGHDmK7KMqKkn5cHo0bBwYNueeNGtwwwIuaxBY3JfBbwK49Yv6uMKOmPG1ca8AMOHnTrjTHGlMqIoP/VV9GtN8akj507d9K5c2c6d+7MaaedRpMmTY4tHznib9j9G2+8kbVr14ZNM3nyZPLiVO/bs2dPli1bFpd9JVtGVO80a+aqdEKtN8bEV16eu4r+6iv3P/boo7FVozZs2PBYAH344YepW7cu99xzz3FpVBVVpUqV0OXUF154IeJxbr/99opnMoNkREn/0Uehdu3j19Wu7dYbY+In0H62cSOolrafJaLjxPr162nfvj233XYbOTk5bNmyhVGjRpGbm0u7du145JFHjqUNlLyLiopo0KABY8eOpVOnTlx44YVs374dgAceeICJEyceSz927Fi6du3KOeecw3/+8x8ADhw4wPe//306derEsGHDyM3NjViinzp1Kh06dKB9+/bcf//9ABQVFXHttdceWz9p0iQAnnrqKdq2bUunTp0YOXJk3M+ZHxkR9EeMgClToHlzEHHPU6ZYI64x8Zbs9rPVq1dz0003sXTpUpo0acJjjz3GokWLWL58Oe+//z6rV68+YZs9e/bQq1cvli9fzoUXXsjzzz8fct+qyoIFC3jiiSeO/YA8/fTTnHbaaSxfvpyxY8eydOnSsPkrKCjggQceYM6cOSxdupSPP/6Yd955h8WLF7Njxw4+/fRTVq5cyXXXXQfA+PHjWbZsGcuXL+eZZ56J8exUTEYEfXABPj8fSkrcswV8Y+Iv2e1nZ511FhdccMGx5VdeeYWcnBxycnJYs2ZNyKB/0kknMXDgQADOP/988vPzQ+576NChJ6SZN28e11xzDQCdOnWiXbt2YfM3f/58+vbtS6NGjahevTrDhw9n7ty5tG7dmrVr13LnnXcyc+ZM6tevD0C7du0YOXIkeXl5Fb65KlYZE/SNMYlXXjtZotrP6tSpc+z1unXr+N3vfsfs2bNZsWIFAwYMCNlfvUaNGsdeV61alaKiopD7rlmz5glpop1Uqrz0DRs2ZMWKFfTs2ZNJkyZx6623AjBz5kxuu+02FixYQG5uLsXFxVEdLx4s6BtjfEtl+9nevXupV68eJ598Mlu2bGHmzJlxP0bPnj2ZPn06AJ9++mnIK4lg3bt3Z86cOezcuZOioiKmTZtGr169KCwsRFX5wQ9+wC9/+UuWLFlCcXExBQUF9O3blyeeeILCwkIOlq0rS4KM6L1jjEmOQLVpPHvv+JWTk0Pbtm1p3749rVq1okePHnE/xv/7f/+P6667jo4dO5KTk0P79u2PVc2E0rRpUx555BF69+6NqnL55ZczaNAglixZwk033YSqIiI8/vjjFBUVMXz4cPbt20dJSQn33nsv9erVi/tniCTt5sjNzc1Vm0TFmORZs2YN5513XqqzkRaKioooKiqiVq1arFu3jksuuYR169ZRrVp6lY9DfWcislhVcyNtm16fxBhjUmj//v3069ePoqIiVJVnn3027QJ+rHx9GhEZAPwOqAr8SVUfK/P+bcDtQDGwHxilqqtFpCswJZAMeFhV34pX5o0xJp4aNGjA4sWLU52NhIrYkCsiVYHJwECgLTBMRNqWSfayqnZQ1c7AeGCCt34lkOutHwA8KyKZ9bNpjDGViJ/eO12B9aq6QVWPANOAIcEJVHVv0GIdQL31B1U10F+qVmC9McaY1PBT6m4CfB20XAB0K5tIRG4HxgA1gL5B67sBzwPNgWuDfgSCtx0FjAJoZgPmGGNMwvgp6YcavPmEEruqTlbVs4B7gQeC1s9X1XbABcB9IlIrxLZTVDVXVXMbN27sP/fGGGOi4ifoFwBnBi03BTaHST8NuKLsSlVdAxwA2keTQWNMZuvdu/cJN1pNnDiRH//4x2G3q1u3LgCbN2/mqquuKnffkbqAT5w48bibpC699FJ2797tJ+thPfzwwzz55JMx7yfe/AT9hUAbEWkpIjWAa4AZwQlEpE3Q4iBgnbe+ZaDhVkSaA+cA+XHItzEmQwwbNoxp06Ydt27atGkMGzbM1/ZnnHEGr7/+eoWPXzbov/vuuzRo0KDC+0t3EYO+Vwd/BzATWANMV9VVIvKIiAz2kt0hIqtEZBmuXv96b31PYLm3/i3gx6q6I+6fwhhTaV111VW88847HD58GID8/Hw2b95Mz549j/Wbz8nJoUOHDrz99tsnbJ+fn0/79q4C4dChQ1xzzTV07NiRq6++mkOHDh1LN3r06GPDMv/iF78AYNKkSWzevJk+ffrQp08fAFq0aMGOHS5MTZgwgfbt29O+fftjwzLn5+dz3nnnccstt9CuXTsuueSS444TyrJly+jevTsdO3bkyiuvZNeuXceO37ZtWzp27HhsoLcPP/zw2CQyXbp0Yd++fRU+t6H46j6pqu8C75ZZ91DQ6zvL2e4l4KVYMmiMSZ6f/hTiPSFU587gxcuQGjZsSNeuXfnnP//JkCFDmDZtGldffTUiQq1atXjrrbc4+eST2bFjB927d2fw4MHlzhP7hz/8gdq1a7NixQpWrFhBTk7OsfceffRRTj31VIqLi+nXrx8rVqzgJz/5CRMmTGDOnDk0atTouH0tXryYF154gfnz56OqdOvWjV69enHKKaewbt06XnnlFZ577jl++MMf8sYbb4QdH/+6667j6aefplevXjz00EP88pe/ZOLEiTz22GN8+eWX1KxZ81iV0pNPPsnkyZPp0aMH+/fvp1atE5pBY2IDrhljUi64iie4akdVuf/+++nYsSP9+/dn06ZNbNu2rdz9zJ0791jw7dixIx07djz23vTp08nJyaFLly6sWrUq4mBq8+bN48orr6ROnTrUrVuXoUOH8tFHHwHQsmVLOnfuDIQfvhnc+P67d++mV69eAFx//fXMnTv3WB5HjBjB1KlTj93526NHD8aMGcOkSZPYvXt33O8IthuljDHHhCuRJ9IVV1zBmDFjWLJkCYcOHTpWQs/Ly6OwsJDFixdTvXp1WrRoEXI45WChrgK+/PJLnnzySRYuXMgpp5zCDTfcEHE/4cYlCwzLDG5o5kjVO+X5xz/+wdy5c5kxYwa/+tWvWLVqFWPHjmXQoEG8++67dO/enX//+9+ce+65Fdp/KFbSN8akXN26denduzc/+tGPjmvA3bNnD9/61reoXr06c+bMYWOoybCDXHzxxccmP1+5ciUrVqwA3LDMderUoX79+mzbto333nvv2Db16tULWW9+8cUX87e//Y2DBw9y4MAB3nrrLS666KKoP1v9+vU55ZRTjl0lvPTSS/Tq1YuSkhK+/vpr+vTpw/jx49m9ezf79+/niy++oEOHDtx7773k5uby2WefRX3McKykb4xJC8OGDWPo0KHH9eQZMWIEl19+Obm5uXTu3DliiXf06NHceOONdOzYkc6dO9O1a1fAzYLVpUsX2rVrd8KwzKNGjWLgwIGcfvrpzJkz59j6nJwcbrjhhmP7uPnmm+nSpUvYqpzyvPjii9x2220cPHiQVq1a8cILL1BcXMzIkSPZs2cPqspdd91FgwYNePDBB5kzZw5Vq1albdu2x2YBixcbWtmYLGdDK1c+sQytbNU7xhiTRSzoG2NMFrGgb4yJekJwkzqxflcW9I3JcrVq1WLnzp0W+CsBVWXnzp0x3bBlvXeMyXJNmzaloKCAwsLCVGfF+FCrVi2aNm1a4e0t6BuT5apXr07Lli1TnQ2TJFa9Y4wxWcSCvjHGZBEL+sYYk0Us6BtjTBbxFfRFZICIrBWR9SIyNsT7t4nIpyKyTETmiUhbb/13RWSx995iEel74t6NMcYkS8SgLyJVgcnAQKAtMCwQ1IO8rKodVLUzMB6Y4K3fAVyuqh1ws2nZhCrGGJNCfkr6XYH1qrpBVY/gJj4fEpxAVfcGLdYB1Fu/VFUDk6ivAmqJSE2MMcakhJ9++k2Ar4OWC4BuZROJyO24+XFrAKGqcb4PLFXVwyG2HQWMAmjWrJmPLBljjKkIPyX9UJNRnnC/tqpOVtWzgHuBB47bgUg74HHg1lAHUNUpqpqrqrmNGzf2kSVjjDEV4SfoFwBnBi03BTaXkxZc9c8VgQURaQq8BVynql9UJJPGGGPiw0/QXwi0EZGWIlIDuAaYEZxARNoELQ4C1nnrGwD/AO5T1Y/jk2VjjDEVFTHoq2oRcAcwE1gDTFfVVSLyiIgM9pLdISKrRGQZrl7/+sB6oDXwoNedc5mIfCv+H8MYY4wfNl2iMcZkAJsu0RhjzAks6BtjTBaxoG+MMVnEgr4xxmQRC/rGGJNFLOgbY0wWsaBvjDFZxIK+McZkkawI+nl50KIFVKninvPyUp0jY4xJDT9DK1dqeXkwahQcPOiWN250ywAjRqQuX8YYkwoZX9IfN6404AccPOjWG2NMtsn4oP/VV9GtN8aYTJbxQb+8ibhsgi5jTDbK+KD/6KNQu/bx62rXduuNMSbbZHzQHzECpkyB5s1BxD1PmWKNuMaY7JTxvXfABXgL8sYY47OkLyIDRGStiKwXkbEh3r9NRD71ZsaaJyJtvfUNRWSOiOwXkWfinXljjDHRiRj0RaQqMBkYCLQFhgWCepCXVbWDqnYGxgMTvPXfAA8C98Qvy8YYYyrKT0m/K7BeVTeo6hFgGjAkOIGq7g1arAOot/6Aqs7DBX9jjDEp5qdOvwnwddByAdCtbCIRuR03KXoNoG80mRCRUcAogGbWl9IYYxLGT0lfQqw7YTZ1VZ2sqmcB9wIPRJMJVZ2iqrmqmtu4ceNoNjXGGBMFP0G/ADgzaLkpsDlM+mnAFbFkyhhjTGL4CfoLgTYi0lJEagDXADOCE4hIm6DFQcC6+GUx8WwUTmNMtohYp6+qRSJyBzATqAo8r6qrROQRYJGqzgDuEJH+wFFgF3B9YHsRyQdOBmqIyBXAJaq6Ov4fpWJsFE5jTDYR1ROq51MqNzdXFy1alLTjtWjhAn1ZzZtDfn7SsmGMMTERkcWqmhspXcYPwxCJjcJpjMkmWR/0bRROY0w2yfqgb6NwGmOySdYHfRuF0xiTTbJilM1IbBROY0y2yPqSvjHGZBML+sYYk0Us6Ptgd+waYzKF1elHYHfsGmMyiZX0Ixg3rjTgBxw86NYbY0xlY0E/Artj1xiTSSzoR2B37BpjMokF/Qjsjl1jTCaxoB+B3bFrjMkk1nvHB7tj1xiTKXyV9EVkgIisFZH1IjI2xPu3icinIrJMROaJSNug9+7ztlsrIt+LZ+aNMcZEJ2LQF5GqwGRgINAWGBYc1D0vq2oHVe0MjAcmeNu2xU2v2A4YAPze258xxpgU8FPS7wqsV9UNqnoEN/H5kOAEqro3aLEOEJiOawgwTVUPq+qXwHpvf8YYY1LAT51+E+DroOUCoFvZRCJyOzAGqAH0Ddr2kzLbNgmx7ShgFEAz6wtpjDEJ46ekLyHWnTCxrqpOVtWzgHuBB6Lcdoqq5qpqbuPGjX1kyRhjTEX4CfoFwJlBy02BzWHSTwOuqOC2lZINyGaMqSz8BP2FQBsRaSkiNXANszOCE4hIm6DFQcA67/UM4BoRqSkiLYE2wILYs50+AgOybdwIqqUDslngN8ako4hBX1WLgDuAmcAaYLqqrhKRR0RksJfsDhFZJSLLcPX613vbrgKmA6uBfwK3q2pxAj5HytiAbMaYykRUT6hiT6nc3FxdtGhRqrPhW5UqroRflgiUlLjXeXnuR+Crr9yYPY8+ajd7GWPiS0QWq2pupHQ2DEOMIg3IZtU/xph0YkE/RpEGZLPqH2NMOrGgH6NIA7LZePzGmHRiA67FQbgB2Zo1c1U6odYbY0yyWUk/wWw8fmNMOrGgn2A2Hr8xJp1Y9U4S2Hj8xph0YSV9Y4zJIhb0U8zG7THGJJNV76RQ4MatQD/+wI1bYNVBxpjEsJJ+Cvm5ccuuBIwx8WQl/RSKdOOWXQkYY+LNSvopFGncHhvCwRgTbxb0UyjSjVs2hIMxJt4s6KdQpBu3Il0JGGNMtHwFfREZICJrRWS9iIwN8f4YEVktIitEZJaINA9673ERWek9ro5n5jPBiBGQn+/G3s/PP76u3oZwMMbEW8SgLyJVgcnAQKAtMExE2pZJthTIVdWOwOvAeG/bQUAO0BnoBvxMRE6OX/Yzmw3hYIyJNz8l/a7AelXdoKpHcBOfDwlOoKpzVDXQ5PgJbgJ0cD8SH6pqkaoeAJYDA+KT9ewQ7krAGGOi5SfoNwG+Dlou8NaV5ybgPe/1cmCgiNQWkUZAH+DMshuIyCgRWSQiiwoLC/3l3BhjTNT89NOXEOtCTqwrIiOBXKAXgKr+S0QuAP4DFAL/BYpO2JnqFGAKuDlyfeXcGGNM1PyU9As4vnTeFNhcNpGI9AfGAYNV9XBgvao+qqqdVfW7uB+QdbFl2QSzO3aNMdHwU9JfCLQRkZbAJuAaYHhwAhHpAjwLDFDV7UHrqwINVHWniHQEOgL/ilfms53dsWuMiVbEkr6qFgF3ADOBNcB0VV0lIo+IyGAv2RNAXeA1EVkmIjO89dWBj0RkNa76ZqS3PxMHdseuMSZaoppeVei5ubm6aNGiVGejUqhSBUJ9fSKut48xJnuIyGJVzY2Uzu7IrcTsjl1jTLQs6FdidseuMSZaFvQrMT937FrvHmNMMBtPv5ILN+m69e4xxpRlJf0MZr17jDFlWdDPYDYevzGmLAv6Gcx69xhjyrKgn8Gsd48xpiwL+hksHuPxW+8fYzKL9d7JcOF690RivX+MyTxW0s9y4Ury1vvHmPgpKYFDh1KdCwv6WS1Qkt+40Y3hEyjJBwK/9f4xJn6eeAJatYJvvkltPizoZ7FIJXnr/WNM/Lz1FmzdCrNmpTYfFvSzWKSSvPX+MSY+9uyBhQvd6zffTG1eLOhnsUgl+Xj0/jHGwIcfujr9Vq3g7behKIWzivgK+iIyQETWish6ERkb4v0xIrJaRFaIyCwRaR703ngRWSUia0RkkoiEmnPXpICfkvyIEZCf7/5g8/Mt4JvK7Y9/hN//PvnHnTULTjoJfv1r2LkTPvoo+XkIiBj0vSkPJwMDgbbAMBFpWybZUiBXVTsCrwPjvW2/A/TATZPYHrgAb9L0eNu1C+6/H9asScTeM5OV5E02OXIE7rsPHn44+ZMMzZoFPXvC4MEu+KeyisdPSb8rsF5VN6jqEWAaMCQ4garOUdVAk+AnuMnTARSoBdQAauKmT9wWj4yXVVwMv/sdPP54Ivaeuawkb7LFnDmwezcUFsKqVck77tat7nj9+kGdOjBggGvUTdXsdn6CfhPg66DlAm9deW4C3gNQ1f8Cc4At3mOmqiakLN6oEdxyi+tuaF0Kk8Pu1jWVyeuvQ61a7nUye9DMnu2e+/Vzz0OHwqZNpQ27yeYn6Ieqgw85sa6IjARycROlIyKtgfNwJf8mQF8RuTjEdqNEZJGILCosLPSb9xOMGeOeJ0yo8C6MT5H6+MfL9u2pbfQymeHoUVe6HjoUWrcuDcTJMGsWNGgAXbq45csug2rVUlfF4yfoFwBnBi03BTaXTSQi/YFxwGBVPeytvhL4RFX3q+p+3BVA97LbquoUVc1V1dzGjRtH+xmOadYMRo6E556DHTsqvBvjQzLu1t271/2DTpwYv32a7PThh64B9aqroG9ft5yMwoSqC/p9+kDVqm5dgwau1P/mm+79ZPMT9BcCbUSkpYjUAK4BZgQnEJEuwLO4gL896K2vgF4iUk1EquMacRPa1Przn7vg8/TTiTyK8XO3bqzVP3Pnwr59qb+ZxVR+r79eWp/et68rUCxenPjjbtjgroL79z9+/dChsH49rFyZ+DyUFTHoq2oRcAcwExewp6vqKhF5REQGe8meAOoCr4nIMhEJ/Ci8DnwBfAosB5ar6t/j/SGCnXceXHGFC/r79yfySNktUh//eFT/BC7BP/kkdY1epvIrLnal6kGDXM+ZPn3c+mRU8QQKLIH6/IAhQ1yPuZRU8ahqWj3OP/98jdUnn6iC6m9/G/OuTDmmTlWtXdud58Cjdm23XlW1efPj3ws8mjf3f4yOHVWrVHHbrV6diE9hssGcOe5v6LXXStd16KDav3/ij/2DH6g2aaJaUnLiexdd5P7G4wVYpD5ibEbekdutm/s1/+1v4fDhyOlN9CL18Y+1+qewEFasgOHD3fJ//5uIT2GyweuvuxL+wIGl6/r2hXnzEjv4WUmJu5ro18/9j5Q1dKj7G1+/PnF5CCUjgz7A2LGweTNMnZrqnGSucH38Y63+mTPHPY8e7Rq+LOibiiguhjfegEsvdXX6Af36uYD/ySeJO/aKFa7xuGzVTsCVV7rnt95KXB5Cydig/93vQk4OjB/vvniTXJGGeIjU+2f2bKhXD7p2he7dLeibivnPf9zNUVdddfz6iy92V5iJrNcvrz4/oHlzOP9896OUTBkb9EVcaf/zz+Fvf0t1brJPrNU/s2dDr16uP/OFF8Lq1W6kQmOiEbgha9Cg49fXrw+5uYntGTZrFpxzDjQJcyvr0KEwfz4UFCQuH2VlbNAHd0LbtIEc3f4AAAAZ+UlEQVTf/CY1/WGzXUWrf77+Gtatc/Wu4IK+qvvnMMavkhJXih4wwF01ltWvHyxY4LoFx9uRI67LcXml/IChQ91zMgumGR30q1Z1/fYXL7a+3ukmXPVP2dvWu3VzVwvZVMXzzTfubmRTcfPnu+EOylbtBPTt627QSsSIlwsWwIEDkYP+uee6bubJ7LqZ0UEf4Npr4YwzXGnfpI9w1T+zZrmxlNq3d2lPPhnatcuuoD9iBHTo4AKHqZjXXoMaNdywB6H06OHeT0S9/qxZ7u+6d+/IaYcOdXcIJ2sUgYwP+jVrujF5Zs92v74mfYSq/lF131Xfvq6hLeDCC7PnJq2ZM13Jb/t2+Otfk398VXfsffsqb7WoqqvPv+QSV38fykknwXe+k7igf/75cOqpkdMOHer+rmfMiJw2HjI+6IPrCtiggQ27XBl8/rm7JO/X7/h+/G++6RpyP/ss1TlMrCNH4Cc/cW1R558PTz2V/B+6F1+Eb3/bXWFVqwannAItW0Lnzq7kOmQI/Oxn7ntKVwsXurahH/wgfLq+fWHZMte1Ml4OHHAFlEhVOwFdurgr3WRV8WRF0K9XD+64w/WHfe65ylt6yQaBUte+fcf34w/8Uz71VGnaTBzaeeJE98P3u9+5wLpuHbzzTvKOf/gwPPQQdOoETzzhJh0ZORIuusg1squ6q7KJE+Gss+DOO2HLluTlz6/XX4fq1eHyy8On69fPfaYPPojfsT/6yI3q6Tfoi7jS/vvvuzGBEs7PbbvJfMRjGIZQdu1S7d3b3Y49cKDqpk0JOYyJ0fe/r9qsmXuEGsahbl2XLtIwEKl25Ijqhx+qHj3qf5tNm9znu/xyt3z0qDsPF1+cmDyG8swz7lz+61/h023YoPqjH6lWrapaq5bqmDGqW7cmJ4+RlJSotmzp/s8jOXJEtU4d1R//OH7Hv/tu1Ro1VA8c8L/NRx+58/7KKxU/Lj6HYUh5kC/7SFTQV1UtLladNEn1pJNUGzRQfeml0GNimNQoLlY99VTVG25QFQkd9MGljcfYPolQVOT+rs46y+Vn+HD3ufwYMUK1Zk3V9etL1/32t24/CxcmJr/BDh5UPf10NyaM3/+LdetUr7/ejZFUu7bqz3+uWliY0GxGtHixO2d//rO/9AMHqp5zTvyO37mzK2BGo6hI9dvfdmP1VJQF/TA+/1z1wgvdp7/yStVt2xJ+SOPDkiXuO3nppfKDOrirtvJ+FETcvqZOdfsQcc+JvgIoLnYDep13nstHp06qo0e717fdFjmIzp3r0j7wwPHr9+xRPflk1WHDEpf3gAkTXB4++CD6bdeuVR050p3vOnVU77/ffU+pMHasarVqqjt3+kv/5JPucxcUxH7swkK3r1/9Kvpt7703tisOC/oRFBWpjh/vLsMaNTp+BD6TGoF/vk2bQlff1Kzpnv/5z/Al/WRW/ZSUqP797650By7ov/Zaael+7Fi3fuzY8vdx9KgbbfHMM0NXCdx9t6tG2bgx/vkP2L9f9VvfUu3XL7b9rF6tevXV7jOfeqrqE0+oHjoUnzz6UVKi2rq16iWX+N8muLARq+nT3b7+85/Y9xUtC/o+rVypev757kxce63/S/FUO3rU1Rnffbfq2WerdusWXR1iOho4UPXcc0uXy5bWp0xxr3/xi/CBPVlVPx9+6M47qLZq5YJGUdHxaUpKSkv8v/lN6P0E6tHLK3hs3OiC/j33xDf/wR57LL7BaskS1e99z+2zaVNX1RJN+0ZFLVvmjjlliv9tgqsVY3Xrrar16iXns5ZlQT8KR464y1Fw/4Dpau9eFxiuvdb9kYJq9equ/lBE9cYbU53DivPboNahQ2kprrwqnEhVP/Hw5ZeuCuHMM12AOXKk/LTFxa5uH1R///vj39u+3bUv9e0bvgrommtcNc+ePXHJ/nH27HF/T34aPqM1e7Zq166lV0FvvpnYdrQf/ci1L2zfHt12gQ4EseatdWvVyy6LbR8VFdegDwwA1gLrgbEh3h8DrAZWALOA5t76PsCyoMc3wBXhjpWKoK/qvuzvftf9Sn/9dUqyUK41a1ypqUYNPXbZfO217gcgEAQeeECjarxKN/Pmufy/8Ub4dLfeqlq/fvgrsmSU9EePdj+4fv9WjhxxvXJEjq9GuOUWV4pfuTL89gsWuM/w1FMVz3N5fvUrt+8FC+K/b1X3v/XGG66xFNzV0RtvqH7zTXyP8+yzbv8//3n0206e7LZdty667UpKVHfscN/fm28m7jvyI25BH6iKm/KwFVADN+1h2zJp+gC1vdejgVdD7OdU4H+BdOU9UhX0VVW/+ML17Bk8OH169RQVqebmqp5yiqvKKa8bYFGRKy3WquUuceNl61Z36b9vX/z2Gcojj7iAGKnx7S9/cX+14YKknzr9SA294d7fvNkF/Lp1o2soPnRItU8fF+Tfftv1yBFRveuuyNuqul41zZvHt+pg1y53pTF4cPz2WZ6jR1X/9CdX3QPub/rWW113xVj/3z76yH0nAwacWMXmx2efuTw9+2z5adauVf3JT1xJPjfXfY7q1U+8mlyzpuKfIxbxDPoXAjODlu8D7guTvgvwcYj1o4C8SMdLZdBXdQ1P4epXk+2Pf3T5efnlyGm3bnVd7lq3Vt29O/ZjL15c+g86cWLs+wunVy/VnJzI6daudfl57rnw6cIF7Ug/CpHeHzToxKsIvw3Fe/eqXnCBa5Q+5xzXTc/vd/XWW+5Y06f7S+/Hgw+6fcazoBDJ0aOq773nuqgGznOLFu5q9bPPot/fxo2uEbpNm4r3GCopUT3jDNcIXda6darXXeeqjWrVco32Awa4NoCxY93/xrRprtfTV19V7PjxEM+gfxXwp6Dla4FnwqR/BnggxPrZwGXlbDMKWAQsatasWaLPTVhHj7rgc9ppqv/7X0qzooWFrjTUu7f/ktDcua4k+f3vx1Z6evVVd9Vz5pnuRyQ3t+L7iuTAAVd15aehsqREtWFDV3dbUZGqf8K9v3Nn+W0GfquPduxQbd/ebfPCC/7zXVTkvotu3eJzJbpjh6vOvOqq2PdVUfv2qf71r66dJjAf8gUXqM6a5W/7Awfc/+vJJ8dewh45UrVx49Kqw7I3oN19d3p3745n0P9BiKD/dDlpRwKfADXLrD8dKASqRzpeqkv6qq6EW7Wqq29NpZtvdo2Fkep7ywpcrVSkbrG4uLR94DvfcVcPgf7bibps/de/3P7fe89f+kGDXKNgRUVq6A33/sMPh34v2obibdtU8/Ki7y0W6Onz8cfRbRfKvfe6PEf795Uomza5m9ECN7bdfrvrSlqekhJ3/4KI6zYbqxdecMf9+9/d/361au6K7M47XZVeukt69Q7QH1gDfCvEe3cCU/xkKB2CvqordULFblSJh//+1x2/It30SkpUhwxxf7TRdMHbt0/1iivccX/0o9KGti1bXCns/vujz4sfgZtp/LYb/PrXLo8VvRKraEn/zDPdlddJJ8VW0o/F/v0uD0OHxrafrVtd1crw4fHJVzwdOOACLbgfgHnzQqd7/HGX5v/+Lz7Hzc8v/S5r1HA/OvG4YStZ4hn0qwEbgJZBDbntyqTp4jX2tilnH58AffxkKF2C/oEDbvyOs89O7s0lqu4yPifH1THu3Vuxfeza5fLftKm/2+I3bHDdIatUcXWUZasPBgxwXdoScR/DBReo9uzpP/2sWdFdGZRV0Tr9YcPc64cfTu24P/fd50q3wcM1ROuuu9x3vXZt/PIVbx984P6GRVzVysGDpe/94x9u/dVXx7fTxY03ujuoE3kjXKLEu8vmpcDnXmAf5617BBjsvf43sC2oa+aMoG1bAJuAKn6OlS5BX7W02qHsrfGJ9vvfu+NOmxbbfhYvdpenl1xyYo+Go0ddSXnjRtV33nH15A0aqM6cGXpfU6e6PH34YWx5KmvXLhd8HnrI/zb79kW/TVnR9t55/nnXzhO4YzXZwzwE27TJ9Rrp3NmViCdNckHws89O7AZZXOwaF//9b/d39dOfuuqxGjXcmDnpbu9e18MH3I178+e7z1m/vvv84ap/sk1cg34yH+kU9FVdq321aqqffpqc4/m9WcevQN/l1q1dqalRo9LhDIIf557rxiQqz/797uapm2+OPU/B/va3iv2YdOrk7qtIlsAPsd8GxkT/KDz9tLsyK3vFIeKqoS6+2A3tULYq6qST3LkbMaJy1FMHzJzprlqrVHE/vo0aueoYU8pv0BeXNn3k5ubqokWLUp2NY3bscHNYtm4N8+a5eXfLOnoUCguhYUM3U1csbrrJzZa0YoU7bqxU4bHH3Hyh9eqVPurWLX1dvz5873uhJ48Odv318PbbsHUr1Krl7/i7d7tZhKpVc1PTVa/ungOPSZNg+nTYtSu6czd6NLz8stuuSoJnhTh6FM4+200s8t//uvHPw8nLc3MBHDxYuq527dLpIONJ1c1y9cUX7rFhg3v+8ks3cdDZZ7sJWc4+2z3OOCPx5ytRdu+Gu+5yfy/vvgu9eqU6R+lFRBaram7EhH5+GZL5SLeSvmpp1cZNN7m7/a67zpUyO3RwXbwCvT0aNHCXoh9/XLFS+scfu/1U5I7CZHj/fZc/v/3Ei4tdV8/yersEHgMGRJ+XF1902ybjCuyvf3XHmjHDX/pY7whOZdVRZZCKcW0qA6ykHz+qbgaef/zDlU5PPx1OO809Aq8bN3alwDffdCW8s86C665zsw61ahX5GEVFcMEF7orhs89cSTzdFBe72ZPOP9/ffJ5//jPcfLMrzffs6aYCDH4cPeqeu3eHpk2jy8u6da7kOmUK3HJLxT6PHyUlboL2atXctHp+SslVqri/mbJE3P7y8mDcOPjqK3c+H3209AogmVcJJrNYST/OAg2fkUrwe/e6oQL69i29ArjoIjco1/z5bqiHPXtO3M/TT0dXik6Ve+5xbRyRegTt2uWugnr0SMyQFiUlrl430YPMvfGG+16imdEolmGf/VwlxDKEhMlcWENu6m3c6PoQBwaaCn7UqOG6ZHbq5HqE1Kun2r9/+oz5U57ly13+I41GeuedLugsWZK4vFx22fFDMcdbSYnrOtu6dXTjucQy7LOfyWFiGULCZC6/Qd+qd5JAFVaudJfzhYWucTjwHHh9+DC88oqrskh3nTrBSSfBJ5+Efn/VKpfmllvgD39IXD7+7/9cNcnOnXDqqfHf/8yZMGAAPPecq6aKRnlVOJGqflq0cJPBl9W8uZuQPNb3Teay6h2TMOPHuxJkqBt7SkrclcsppyR+rtTAcMwtW6qOG+dmbYqXpUvdjXlNm6oePhy//UYq6UcqqccyhITJbPgs6VfSzlsmlYYPdyXTvLwT33vzTddF81e/gkaNEpuPHj3g1Vddl8Tf/AbatnWNzBMmwObNFdvnkSPwi1+4RvU9e+DFF13jfbw8+qhrmA1Wu7ZbD+5qYMoUVzIXcc/BjbjNmoXeb2B9pPeNSXnJvuzDSvqVQ//+borA4DaIAwfcUA0dOiS/W92WLW6AuUAX0SpV3BXH88/7H6NnyRJ3QxO4ERf9TqwdrVgaWq1O35QHa8g1iRToJx88GFZgBMpUDVIX8Nlnbpz4Vq1cfqpXd1MBlvcDcPiwS1+1qrvb8+23k5/naCSy9471/Km8LOibhNq7193Sf+utbjk/3405HmoSilQpKXHdZO+5p7QuvVq10h+AnTtVFy1yVybgbrpLVOm+MrCrhMrNgr5JuOHDXYPtN9+4SVtq107tzEHhlJS4OWB/9jM3S1PgB6BqVdd1Nh7jsVcW5ZXmkzG3sEkcv0HfGnJNhV17rRv75p574I034P774cwzU52r0ERc4+z48W58moULYcwYN5bLypVw2WWpzmFyBO743bjRhfSNG91yXp7rXhpK8Pq8PNcttEoV9xyqMd+kN+unbyqsqMgNn7BtG7RsCatX+x+IzaRGuH78EL6Pvw0Rkd789tO3kr6psGrVYNgw9/qppyzgVwbhSvORupOOG3d8wAe3PG5c/PNpEsdX0BeRASKyVkTWi8jYEO+PEZHVIrJCRGaJSPOg95qJyL9EZI2XpkX8sm9S7cEH4bXXYPDgVOfE+BGuH3+kewSs+idDRKr0B6riZsxqRel0iW3LpOkD1PZejwZeDXrvA+C73uu6gXTlPawh15jEiaWHTqx3EwfS2GBxiUGyJ0YPer8L8LH3ui0wz09GAg8L+sYkVkUDa6wjhNqNZYnlN+hHbMgVkauAAap6s7d8LdBNVe8oJ/0zwFZV/bWIXAHcDBzBTaz+b2CsqhaX2WYUMAqgWbNm528M1ZpkjEm5cHMBJHowORNePBtyQ00OF/KXQkRGArnAE96qasBFwD3ABbgqohtO2JnqFFXNVdXcxo0b+8iSMSYVRoxwAbikxD0H99qJNO5PpDYBP20GJnZ+gn4BENz7uilwwnBWItIfGAcMVtXDQdsuVdUNqloE/A3IiS3Lxph0FKn3T6yDxcXaSGyNzJ5I9T+40voGXPVMoCG3XZk0XXCNvW3KrK/qpW/sLb8A3B7ueFanb0zlFa69IJY6/Vjr+xPdXpAODdDEcxgG4FLgcy+wj/PWPYIr1YOrq98GLPMeM4K2/S6wAvgU+AtQI9yxLOgbk7kq2nsn1iEi4jENZbjPlA4N0H6Dvt2Ra4xJe5EaiSG2RuZY7jZOlwZouyPXGJMx/NT3lzemkJ/tY7nbuLI1QFvQN8akvViHiIi0fSyB289sZenUiGxB3xiT9mIdIiLWaSih/MAd6Qcl0lVI0vmp+E/mwxpyjTHRirWhN9a7hcM1AidrngKsIdcYky3iMexzuIbgWBpr/TRCx4M15Bpjskak6hu/+yjvbuNE1/knkwV9Y0xGCBe0YxVL4I5U5w/Jbei1oG+MMRH4CdzliXQVkuyGXqvTN8YYH8LV+cciXjd3+a3Tt6BvjDEpFK+GXmvINcaYSiDZDb0W9I0xJoViaS+oCAv6xhiTQvHobhqNaonZrTHGGL9GjEhckC/LSvrGGJNFfAV9ERkgImtFZL2IjA3x/hgRWS0iK0Rklog0D3qvWESWeY8Z8cy8McaY6ESs3hGRqsBk3AxYBcBCEZmhqquDki0FclX1oIiMBsYDV3vvHVLVznHOtzHGmArwU9LvCqxXN7n5EWAaMCQ4garOUdXAUEef4CZPN8YYk2b8BP0mwNdBywXeuvLcBLwXtFxLRBaJyCcickWoDURklJdmUWFhoY8sGWOMqQg/vXckxLqQt/GKyEggF+gVtLqZqm4WkVbAbBH5VFW/OG5nqlOAKd4+CkUkxE3JvjUCdsSwfSJZ3irG8lYxlreKqax5a17O+uP4CfoFwJlBy02BzWUTiUh/YBzQS1UPB9ar6mbveYOIfAB0Ab4ou31Q+sZ+Ml4eEVnk51bkVLC8VYzlrWIsbxWT6XnzU72zEGgjIi1FpAZwDXBcLxwR6QI8CwxW1e1B608RkZre60ZADyC4AdgYY0wSRSzpq2qRiNwBzASqAs+r6ioReQQ3PdcM4AmgLvCaiAB8paqDgfOAZ0WkBPcD81iZXj/GGGOSyNcduar6LvBumXUPBb3uX852/wE6xJLBCpiS5ONFw/JWMZa3irG8VUxG5y3thlY2xhiTODYMgzHGZBEL+sYYk0UyJuhHGh8olUQkX0Q+9cYfSvm0YCLyvIhsF5GVQetOFZH3RWSd93xKmuTrYRHZFDR+06XJzpeXjzNFZI6IrBGRVSJyp7c+Hc5beXlL+bkTkVoiskBElnt5+6W3vqWIzPfO26tez8B0ydtfROTLoPOWsmFkRKSqiCwVkXe85djPm6pW+geuV9EXQCugBrAcaJvqfAXlLx9olOp8BOXnYiAHWBm0bjww1ns9Fng8TfL1MHBPGpyz04Ec73U94HOgbZqct/LylvJzh7u5s673ujowH+gOTAeu8db/ERidRnn7C3BVqv/mvHyNAV4G3vGWYz5vmVLSjzg+kCmlqnOB/5VZPQR40Xv9IhByyIxEKidfaUFVt6jqEu/1PmANbjiSdDhv5eUt5dTZ7y1W9x4K9AVe99an6ryVl7e0ICJNgUHAn7xlIQ7nLVOCfrTjAyWbAv8SkcUiMirVmSnHt1V1C7ggAnwrxfkJdoc3bPfzqag+KUtEWuDuLJ9Pmp23MnmDNDh3XhXFMmA78D7uqny3qhZ5SVL2/1o2b6oaOG+PeuftqcANpikwEfg5EJgevSFxOG+ZEvR9jw+UIj1UNQcYCNwuIhenOkOVyB+As4DOwBbgt6nMjIjUBd4Afqqqe1OZl7JC5C0tzp2qFqsbXr0p7qr8vFDJkpsr76Bl8iYi7YH7gHOBC4BTgXuTnS8RuQzYrqqLg1eHSBr1ecuUoO9rfKBU0dLxh7YDb+H+8NPNNhE5HcB73h4hfVKo6jbvH7MEeI4UnjsRqY4Lqnmq+qa3Oi3OW6i8pdO58/KzG/gAV2/eQEQCN4em/P81KG8DvOoyVTeG2Auk5rz1AAaLSD6uurovruQf83nLlKAfcXygVBGROiJSL/AauARYGX6rlJgBXO+9vh54O4V5OSYQUD1XkqJz59Wn/hlYo6oTgt5K+XkrL2/pcO5EpLGINPBenwT0x7U5zAGu8pKl6ryFyttnQT/igqszT/p5U9X7VLWpqrbAxbPZqjqCeJy3VLdOx7GV+1Jcr4UvgHGpzk9QvlrhehMtB1alQ96AV3CX+0dxV0k34eoLZwHrvOdT0yRfLwGfAitwAfb0FJ2znrhL6RXAMu9xaZqct/LylvJzB3TEzay3Ahc8H/LWtwIWAOuB14CaaZS32d55WwlMxevhk6oH0JvS3jsxnzcbhsEYY7JIplTvGGOM8cGCvjHGZBEL+sYYk0Us6BtjTBaxoG+MMVnEgr4xxmQRC/rGGJNF/j+YxHnLFgOkFQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(len(loss))\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the added layers does improve ours results by a bit, albeit not very significantly. We can draw two conclusions:\n",
    "\n",
    "* Since we are still not overfitting too badly, we could safely increase the size of our layers, in quest for a bit of validation loss \n",
    "improvement. This does have a non-negligible computational cost, though. \n",
    "* Since adding a layer did not help us by a significant factor, we may be seeing diminishing returns to increasing network capacity at this \n",
    "point.\n",
    "\n",
    "可以看到，添加一层的确对结果有所改进，但并不显著。我们可以得出两个结论。\n",
    "* 因为过拟合仍然不是很严重，所以可以放心地增大每层的大小，以进一步改进验证损失。 但这么做的计算成本很高。\n",
    "* 添加一层后模型并没有显著改进，所以你可能发现，提高网络能力的回报在逐渐减小。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using bidirectional RNNs\n",
    "\n",
    "\n",
    "The last technique that we will introduce in this section is called \"bidirectional RNNs\". A bidirectional RNN is common RNN variant which \n",
    "can offer higher performance than a regular RNN on certain tasks. It is frequently used in natural language processing -- you could call it \n",
    "the Swiss army knife of deep learning for NLP.\n",
    "\n",
    "RNNs are notably order-dependent, or time-dependent: they process the timesteps of their input sequences in order, and shuffling or \n",
    "reversing the timesteps can completely change the representations that the RNN will extract from the sequence. This is precisely the reason \n",
    "why they perform well on problems where order is meaningful, such as our temperature forecasting problem. A bidirectional RNN exploits \n",
    "the order-sensitivity of RNNs: it simply consists of two regular RNNs, such as the GRU or LSTM layers that you are already familiar with, \n",
    "each processing input sequence in one direction (chronologically and antichronologically), then merging their representations. By \n",
    "processing a sequence both way, a bidirectional RNN is able to catch patterns that may have been overlooked by a one-direction RNN.\n",
    "\n",
    "Remarkably, the fact that the RNN layers in this section have so far processed sequences in chronological order (older timesteps first) may \n",
    "have been an arbitrary decision. At least, it's a decision we made no attempt at questioning so far. Could it be that our RNNs could have \n",
    "performed well enough if it were processing input sequences in antichronological order, for instance (newer timesteps first)? Let's try \n",
    "this in practice and see what we get. All we need to do is write a variant of our data generator, where the input sequences get reverted \n",
    "along the time dimension (replace the last line with `yield samples[:, ::-1, :], targets`). Training the same one-GRU-layer network as we \n",
    "used in the first experiment in this section, we get the following results:\n",
    "\n",
    "## 使用双向 RNN\n",
    "\n",
    "本节介绍的最后一种方法叫作 双向 RNN（bidirectional RNN）。双向RNN 是一种常见的 RNN 变体，它在某些任务上的性能比普通 RNN 更好。它常用于自然语言处理，可谓深度学习对自然语言处理的瑞士军刀。\n",
    "\n",
    "RNN 特别依赖于顺序或时间，RNN 按顺序处理输入序列的时间步，而打乱时间步或反转 时间步会完全改变 RNN 从序列中提取的表示。正是由于这个原因，如果顺序对问题很重要（比如温度预测问题），RNN 的表现会很好。双向 RNN 利用了 RNN 的顺序敏感性：它包含两个普通 RNN，比如你已经学过的 GRU 层和 LSTM 层，每个 RN 分别沿一个方向对输入序列进行处理（时间正序和时间逆序），然后将它们的表示合并在一起。通过沿这两个方向处理序列，双向 RNN 能够捕捉到可能被单向 RNN 忽略的模式。\n",
    "\n",
    "值得注意的是，本节的 RNN 层都是按时间正序处理序列（更早的时间步在前），这可能是一个随意的决定。至少，至今我们还没有尝试质疑这个决定。如果 RNN 按时间逆序处理输入序列（更晚的时间步在前），能否表现得足够好呢？我们在实践中尝试一下这种方法，看一下会发生什么。你只需要编写一个数据生成器的变体，将输入序列沿着时间维度反转（即将最后一行代码替换为 yield samples[:, ::-1, :], targets）。本节第一个实验用到了一个单 GRU 层的网络，我们训练一个与之相同的网络，得到的结果后面代码所示。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reverse_order_generator(data, lookback, delay, min_index, max_index,\n",
    "                            shuffle=False, batch_size=128, step=6):\n",
    "    if max_index is None:\n",
    "        max_index = len(data) - delay - 1\n",
    "    i = min_index + lookback\n",
    "    while 1:\n",
    "        if shuffle:\n",
    "            rows = np.random.randint(\n",
    "                min_index + lookback, max_index, size=batch_size)\n",
    "        else:\n",
    "            if i + batch_size >= max_index:\n",
    "                i = min_index + lookback\n",
    "            rows = np.arange(i, min(i + batch_size, max_index))\n",
    "            i += len(rows)\n",
    "\n",
    "        samples = np.zeros((len(rows),\n",
    "                           lookback // step,\n",
    "                           data.shape[-1]))\n",
    "        targets = np.zeros((len(rows),))\n",
    "        for j, row in enumerate(rows):\n",
    "            indices = range(rows[j] - lookback, rows[j], step)\n",
    "            samples[j] = data[indices]\n",
    "            targets[j] = data[rows[j] + delay][1]\n",
    "        yield samples[:, ::-1, :], targets\n",
    "        \n",
    "train_gen_reverse = reverse_order_generator(\n",
    "    float_data,\n",
    "    lookback=lookback,\n",
    "    delay=delay,\n",
    "    min_index=0,\n",
    "    max_index=200000,\n",
    "    shuffle=True,\n",
    "    step=step, \n",
    "    batch_size=batch_size)\n",
    "val_gen_reverse = reverse_order_generator(\n",
    "    float_data,\n",
    "    lookback=lookback,\n",
    "    delay=delay,\n",
    "    min_index=200001,\n",
    "    max_index=300000,\n",
    "    step=step,\n",
    "    batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "500/500 [==============================] - 241s 483ms/step - loss: 0.4820 - val_loss: 0.4823\n",
      "Epoch 2/20\n",
      "500/500 [==============================] - 240s 480ms/step - loss: 0.4326 - val_loss: 0.4784\n",
      "Epoch 3/20\n",
      "500/500 [==============================] - 239s 477ms/step - loss: 0.3957 - val_loss: 0.4787\n",
      "Epoch 4/20\n",
      "500/500 [==============================] - 240s 480ms/step - loss: 0.3660 - val_loss: 0.4610\n",
      "Epoch 5/20\n",
      "500/500 [==============================] - 239s 479ms/step - loss: 0.3352 - val_loss: 0.4586\n",
      "Epoch 6/20\n",
      "500/500 [==============================] - 240s 481ms/step - loss: 0.3145 - val_loss: 0.4635\n",
      "Epoch 7/20\n",
      "500/500 [==============================] - 240s 480ms/step - loss: 0.3029 - val_loss: 0.4826\n",
      "Epoch 8/20\n",
      "500/500 [==============================] - 240s 480ms/step - loss: 0.2914 - val_loss: 0.4769\n",
      "Epoch 9/20\n",
      "500/500 [==============================] - 238s 477ms/step - loss: 0.2843 - val_loss: 0.4847\n",
      "Epoch 10/20\n",
      "500/500 [==============================] - 238s 476ms/step - loss: 0.2758 - val_loss: 0.4928\n",
      "Epoch 11/20\n",
      "500/500 [==============================] - 240s 480ms/step - loss: 0.2678 - val_loss: 0.4868\n",
      "Epoch 12/20\n",
      "500/500 [==============================] - 238s 475ms/step - loss: 0.2626 - val_loss: 0.5011\n",
      "Epoch 13/20\n",
      "500/500 [==============================] - 239s 477ms/step - loss: 0.2564 - val_loss: 0.4852\n",
      "Epoch 14/20\n",
      "500/500 [==============================] - 239s 477ms/step - loss: 0.2525 - val_loss: 0.5020\n",
      "Epoch 15/20\n",
      "500/500 [==============================] - 240s 480ms/step - loss: 0.2461 - val_loss: 0.4889\n",
      "Epoch 16/20\n",
      "500/500 [==============================] - 240s 480ms/step - loss: 0.2437 - val_loss: 0.4903\n",
      "Epoch 17/20\n",
      "500/500 [==============================] - 241s 482ms/step - loss: 0.2412 - val_loss: 0.4992\n",
      "Epoch 18/20\n",
      "500/500 [==============================] - 240s 481ms/step - loss: 0.2371 - val_loss: 0.4977\n",
      "Epoch 19/20\n",
      "500/500 [==============================] - 240s 480ms/step - loss: 0.2346 - val_loss: 0.5034\n",
      "Epoch 20/20\n",
      "500/500 [==============================] - 242s 483ms/step - loss: 0.2318 - val_loss: 0.5108\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(layers.GRU(32, input_shape=(None, float_data.shape[-1])))\n",
    "model.add(layers.Dense(1))\n",
    "\n",
    "model.compile(optimizer=RMSprop(), loss='mae')\n",
    "history = model.fit_generator(train_gen_reverse,\n",
    "                              steps_per_epoch=500,\n",
    "                              epochs=20,\n",
    "                              validation_data=val_gen_reverse,\n",
    "                              validation_steps=val_steps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xt8VNW5//HPw10EAYF6ASHgpQqIXFLUIwpYL3gDpKhgrLd6EBW12vYnFdqjVE+V+lNLj0elVtpfiUSqtaUXpVZTQS2XIBgFRC6CRhAjyk0QCXl+f6wdGEIuk2Qyk2S+79drXjOz99p7P7Mnefaatdde29wdERFJD41SHYCIiCSPkr6ISBpR0hcRSSNK+iIiaURJX0QkjSjpi4ikESV9qRIza2xmO8ysSyLLppKZHWdmCe+7bGbnmNm6mPcrzezMeMpWY1tPmdnd1V2+gvXeZ2a/TfR6JXWapDoAqV1mtiPmbUtgN7A3en+ju2dXZX3uvhdoleiy6cDdv5mI9ZjZDcBV7j44Zt03JGLd0vAp6Tdw7r4v6UY1yRvc/Z/llTezJu5elIzYRCT51LyT5qKf78+a2Uwz2w5cZWanm9l8M9tiZhvNbKqZNY3KNzEzN7OM6P2MaP6LZrbdzP5tZt2qWjaaf4GZvW9mW83sV2b2hpldW07c8cR4o5mtNrMvzGxqzLKNzewRM9tsZmuAoRXsn0lmllNq2mNm9nD0+gYzWxF9njVRLby8dRWY2eDodUsz+30U2zKgfxnbXRutd5mZDYumnwz8D3Bm1HT2Wcy+vSdm+XHRZ99sZn8ys6Pi2TeVMbMRUTxbzOxVM/tmzLy7zWyDmW0zs/diPutpZvZWNH2Tmf0i3u1JLXB3PdLkAawDzik17T7ga+ASQiXgEOBbwKmEX4LdgfeB8VH5JoADGdH7GcBnQCbQFHgWmFGNst8AtgPDo3l3AnuAa8v5LPHE+GegDZABfF7y2YHxwDKgM9AemBv+FcrcTndgB3BozLo/BTKj95dEZQw4G9gF9I7mnQOsi1lXATA4ev0Q8C+gHdAVWF6q7OXAUdF3cmUUwxHRvBuAf5WKcwZwT/T6vCjGPkAL4H+BV+PZN2V8/vuA30avT4riODv6ju6O9ntToCewHjgyKtsN6B69XgSMiV63Bk5N9f9COj9U0xeA1939L+5e7O673H2Ruy9w9yJ3XwtMAwZVsPxz7p7n7nuAbEKyqWrZi4Gl7v7naN4jhANEmeKM8efuvtXd1xESbMm2LgcecfcCd98MPFDBdtYC7xIORgDnAlvcPS+a/xd3X+vBq8ArQJkna0u5HLjP3b9w9/WE2nvsdme5+8boO3mGcMDOjGO9AFnAU+6+1N2/AiYAg8ysc0yZ8vZNRUYDs9391eg7egA4jHDwLSIcYHpGTYQfRPsOwsH7eDNr7+7b3X1BnJ9DaoGSvgB8FPvGzE40s7+Z2Sdmtg2YDHSoYPlPYl7vpOKTt+WVPTo2Dnd3Qs24THHGGNe2CDXUijwDjIleX0k4WJXEcbGZLTCzz81sC6GWXdG+KnFURTGY2bVm9nbUjLIFODHO9UL4fPvW5+7bgC+ATjFlqvKdlbfeYsJ31MndVwI/IHwPn0bNhUdGRa8DegArzWyhmV0Y5+eQWqCkLxB+7sd6klC7Pc7dDwN+Smi+qE0bCc0tAJiZcWCSKq0mMW4Ejol5X1mX0meBc6Ka8nDCQQAzOwR4Dvg5oemlLfCPOOP4pLwYzKw78DhwE9A+Wu97MeutrHvpBkKTUcn6WhOakT6OI66qrLcR4Tv7GMDdZ7j7GYSmncaE/YK7r3T30YQmvP8LPG9mLWoYi1STkr6UpTWwFfjSzE4CbkzCNv8K9DOzS8ysCXA70LGWYpwFfN/MOplZe+Cuigq7+ybgdWA6sNLdV0WzmgPNgEJgr5ldDHy7CjHcbWZtLVzHMD5mXitCYi8kHP9uINT0S2wCOpecuC7DTOB7ZtbbzJoTku88dy/3l1MVYh5mZoOjbf+IcB5mgZmdZGZDou3tih57CR/gu2bWIfplsDX6bMU1jEWqSUlfyvID4BrCP/SThJpurYoS6xXAw8Bm4FhgCeG6gkTH+Dih7f0dwknG5+JY5hnCidlnYmLeAtwBvEA4GTqKcPCKx38RfnGsA14E/l/MevOBqcDCqMyJQGw7+MvAKmCTmcU205Qs/xKhmeWFaPkuhHb+GnH3ZYR9/jjhgDQUGBa17zcHphDOw3xC+GUxKVr0QmCFhd5hDwFXuPvXNY1HqsdC06lI3WJmjQnNCaPcfV6q4xFpKFTTlzrDzIaaWZuoieAnhB4hC1MclkiDoqQvdclAYC2hiWAoMMLdy2veEZFqUPOOiEgaUU1fRCSN1LkB1zp06OAZGRmpDkNEpF5ZvHjxZ+5eUTdnoA4m/YyMDPLy8lIdhohIvWJmlV1ZDqh5R0QkrSjpi4ikESV9EZE0oqQvIpJGlPRFRNKIkr6ISBpR0hcRSSN1rp++iEi6+fxz+POfYc8eGDu2drelmr6ISAp8+ilMmwbnnQdHHAHXXw+//W3tb1c1fRGRJPn4Y3jhBXjuOZg3D4qL4bjj4Ic/hO98B/r3r/0YlPRFRGrR+vXw/PMh0f/732Fajx4waVJI9CefDFbbd6COoaQvIpJgq1aFRP/881AylFifPvCzn4VEf9JJqYtNSV9EpIaKiyE/H2bPDok+Pz9MHzAAHnwwJPpjj01tjCXiSvpmNhT4JdAYeMrdHyg1/1rgF8DH0aT/cfenonnXsP8Gyfe5++8SELeIJNhHH8Ehh0CHDqmOpO5zhxUr4NVXITcX/vWv0APHDM44Ax55BEaOhC5dUh3pwSpN+tENqh8DzgUKgEVmNtvdl5cq+qy7jy+17OHAfwGZgAOLo2W/SEj0Ig3E9u0wZQp07w7XXpvcNl53+M1v4NZboWlT+OlP4bbboFmz5MVQ17nD6tUhwZc8Nm0K87p2heHDYcgQOOccOOqo1MZamXhq+gOA1e6+FsDMcoDhQOmkX5bzgZfd/fNo2ZcJ9z6dWb1wRRqev/4Vbr451LQh9Nf+9a+hY6W3w6i57dvhxhth5kz49rehRQv40Y9CV8KHH4aLLkruAagm1q4NvWPatQuPww8Pv1yqa/36kNxLavMFBWH6UUeF5D5kCJx9NnTrlpj4kyWepN8J+CjmfQFwahnlvmNmZwHvA3e4+0flLNup9IJmNhYYC9ClLv4eEqkFn3wCt98Os2ZBz57wxhuwYAFMmAC9e8P06TB0aO1tf8kSuPzykCzvuy9st3FjePFFuOMOuOQSOP/80FSRyhOPFdm1C/74x3CQfO21g+c3b77/IFDe4/DD97/+8MP9SX7t2rCODh1Cgi9J8iecUH8OhGWJJ+mX9fFK3039L8BMd99tZuOA3wFnx7ks7j4NmAaQmZmpO7VLg+YOTz8d+mbv3Bl6dPyf/xOaU/7jP0KNOysLLrgAxo8PzT41qbGWtf3//V+4887wayI3F846a//8Cy4INdnHHoN77gldCsePh//6r5AY64K334annoIZM2DLlnCS9L//O/Rz37IFvvgiPD7/fP/rL76ADRtg2bIwfdu2stfdti0MGhQOyEOGhANyo4Z0Gau7V/gATgfmxLz/MfDjCso3BrZGr8cAT8bMexIYU9H2+vfv7yIN1cqV7oMHu4P7WWe5v/de2eV27XK//fZQ7qST3JcsScz2v/jCfeTIsN4LL3QvLKy4/Kefut94o7uZe/v27o8/7l5UlJhYqmrrVvcnn3TPzAzxN2/ufuWV7q++6r53b9XXt2eP+2efua9a5b5wofucOe55ean7fDUF5Hkl+dzDrqs06TcB1gLdgGbA20DPUmWOinl9KTA/en048AHQLnp8ABxe0faU9KUh2r3b/b77QqJq08Z92rT4EtWcOe5HHeXetKn7lCnVS24l5s93z8hwb9LE/aGHqrauJUvCQQrce/d2z82tfhxVUVzs/sYb7tdd596yZdh+r17uv/yl++bNyYmhvkhY0g/r4kJCW/0aYGI0bTIwLHr9c2BZdEDIBU6MWfZ6YHX0uK6ybSnpS7Lt3u0+e7b71Ve733yz+x//GGrEiTJ/fkhU4H7ZZe4bNlRt+cJC90svDcsPGeL+4YdVW37v3pDkmzRx79o1xFMdxcXuf/hDWAe4jxrl/sEH1VtXZQoL3R9+2L1Hj7CtVq3c//M/3RcsCHHIwRKa9JP5UNKXZCgudn/9dfebbgrNFuB++OHuhx4aXjdq5D5ggPvdd4fmg6++qvo2tm1zv/XW0DTSuXM4sNQk3t/8JsTXtq17Tk58yxUWul90UfhMI0e6f/559WMosXOn++TJ7occEn65TJrkvmNHzde7d6/7P/7hfvnl4ZcNuJ9+evjc27fXfP0NXbxJ30LZuiMzM9PzSq5bFkmwFSsgOzs81q0LJ0hHjAgnTs87L5zkXLgQ/vnP8Jg/H/buDeXOPDOc4DznHDjllIpP7pV0wywogFtugfvvh8MOq3n8q1fDVVeFXj7f/S78z/+Uv95582DMGCgsDN0vb745sb1OPvoo9Ph55hno1ClceTpqFGzdGk6mlpxQLXkdz7Svvw69aa6+Gr73PejVK3HxNnRmttjdMystp6QvDd2GDaEfenZ26KbYqBGce25I9CNGQOvW5S+7bRvMnbv/ILBsWZjevn3oZVNyECjpq/3JJ+HCpj/8IfT6eOopOO20xH6ePXvCQeRnPwtXfP7+9zBw4P75xcXw85+Hi6y6dw9dQvv2TWwMsd54I/R0Wby48rJNm4YeQG3b7n+OffTtG76T5s1rL96GSklf0trWraH/dnZ26HftDt/6Vkj0V1wBRx5ZvfVu2BDW989/wssvh/cQkusZZ8Bf/hL6jv/kJ+Eip9q8qvXf/w61/nXr4Mc/Dl0qP/88/AJ4+eVQy3/yyYoPaolSXAzPPgsffHBwIo9N7i1a1O8+7nWZkr6knd27w4VF2dkh+e7eHfpvZ2WFxwknJHZ77rBy5f5fAa+9Bv36weOPJ35b5dm+PdSyp08P2/7443DA+9WvQvOIEmz6SLukn50NEyeGK+q6dAk/f7OyaiFAqVP27g2DXc2cGUY33LIlXHB0xRWhFjxgQHokvueeC7fZO/LI0JyjtvD0E2/SbxBDK2dnw3/+Z/hZDWHMjJL7TCrxJ0dxcTihBzB4MHTuXHvbcg9NGzk5IcFt2gStWoW24DFjQnt906a1t/26aNSoMGRD8+bp99mlahpE0p84cX/CL7FzZ0j8a9aEWk+vXuGnfuPGqYmxIdu1K4wMOWvW/mnHHReS/5Ah4fnoo2u2Dfdw6f3MmaHteP36kOAuvhhGj4YLL4SWLWu2jfquVatURyD1QYNo3mnUKCSFspjtn9eiRRg4quQgUPI45pj0aAKoDZ98EoaVXbQodNk799zQ3JKbG9q4t24N5U44Yf9BYNCg+IefXbky1OhzcuC998JB+7zzQqIfMSIx3SBFGoK0atPPyAg1v9K6doXly0Pf7HffPfBRMkwqhN4NvXqF+1a2bAlFRaGtuLzn8uaZwU03hSaGdJCfH2ramzeHJrYRIw6cv3dvqJ2X3GRi7tz9g1ydeOKBB4Ejjti/3Icf7k/0S5aE/TpoUEj03/mObvIhUpa0SvrZ2aEpZ+fO/dNatgxjgpfXpr9lS+hzHXsgWLEiXBzSuDE0aVL2c0XzPvkkrOPWW+Ghhxr2TSj+9reQhNu0CT1l4ukHXlQES5fuvwnFvHmwY0eY16NH6Gv+7rvw5pth2oABYRuXXx4u/hGR8sWb9FM+7ELpR3WHYZgxI4wJYhaeZ8yo1mpq5Ouv3e+4I1w+fsYZVR9jpT4oLnZ/9NEwTEG/fu4FBdVf1549YRyYBx5wP//8ML5Kr17u99/vvnp14mIWSQdoGIbUyckJfaQPOyxcmRl7tWR9tmdPuNr0iSfg0kvDlaCHHprqqEQE4q/pN6RbA9QZo0eHsVFatw5t1lOnln+iub7YsiXcOu+JJ+Cuu0K/cCV8kfpHSb+W9OoVerRceGG4YvKqq+DLL1MdVfWsXRvu6JSbG+749MADDexOQiJpRP+6tahNG3jhhXD/0Zkz4fTTwyiJ9cnrr8Opp4aT1C+/DNddl+qIRKQmlPRrWaNG4eKxF18M46JkZoZhd+uDGTPCSJLt2oXmqsGDUx2RiNSUkn6SnH8+5OWF0RgvuSQMe7t3b6qjKltxcRgl8rvfDc068+fD8cenOioRSQQl/STq1i2MPX7ttWEs9IsvDkPh1iW7doWLy+67D66/HubMCTe1EJGGQUk/yQ45JJwMffxxeOWV0NyzdGmqowo++SQ04fzhDzBlSrgBSEO+wEwkHTWIAdfqGzMYNw769AmjI55+erjZxdVXJzeOzZtDD6MFC8ItAt94I/TF/+MfDx5SQUQaBiX9FDrttHCLuSuugGuuCcn3Rz8Kg5El+nZxX30VflEsXLg/yZf0JDILwyCMHBkuvurTJ7HbFpG6Q1fk1gFFReGCp4cf3j+tQ4cwHHGnTuU/d+xYdn/54mJYterABL90aajFQzionHrq/kf//hqtUqS+S6sB1xqKBQvCgGMffxzuvbphw/7XmzYdfFVvkyYhgR99dHgccUS4kGrhwnAFLYSrZr/1rTB42amnhufavMGJiKRGWt05q6EoqXmXZc+ekPhjDwSxB4eVK8PwxV26wGWX7V/XSSfpxjEisp+Sfj3RtGmooauWLiI1oS6bIiJpJK6kb2ZDzWylma02swkVlBtlZm5mmdH7DDPbZWZLo8cTiQpcRESqrtLmHTNrDDwGnAsUAIvMbLa7Ly9VrjVwG7Cg1CrWuLs6AYqI1AHx1PQHAKvdfa27fw3kAMPLKPczYArwVQLjExGRBIon6XcCPop5XxBN28fM+gLHuHtZ40d2M7MlZvaamZ1Z1gbMbKyZ5ZlZXmFhYbyxi4hIFcWT9K2Maft6jJtZI+AR4AdllNsIdHH3vsCdwDNmdtBlQO4+zd0z3T2zY8eO8UUuIiJVFk/SLwCOiXnfGdgQ87410Av4l5mtA04DZptZprvvdvfNAO6+GFgDnJCIwEVEpOriSfqLgOPNrJuZNQNGA7NLZrr7Vnfv4O4Z7p4BzAeGuXuemXWMTgRjZt2B44G1Cf8UIiISl0p777h7kZmNB+YAjYGn3X2ZmU0G8tx9dgWLnwVMNrMiYC8wzt3r2AjyIiLpQ2PviIg0APGOvaMrckVE0oiSvohIGlHSFxFJI0r6IiJpREk/kp0NGRnhTlQZGeG9iEhDo/H0CQl+7FjYuTO8X78+vAfIykpdXCIiiaaaPjBx4v6EX2LnzjBdRKQhUdIHPvywatNFROorJX3CfWWrMl1EpL5S0gfuvx9atjxwWsuWYbqISEOipE84WTttGnTtCmbhedo0ncQVkYZHvXciWVlK8iLS8KmmLyKSRpT0RUTSiJK+iEgaUdIXEUkjSvoiImlESV9EJI0o6YuIpBElfRGRNKKkLyKSRpT0RUTSiJK+iEgaUdIXEUkjSvoiImlESV9EJI0o6YuIpJG4kr6ZDTWzlWa22swmVFBulJm5mWXGTPtxtNxKMzs/EUGLiEj1VHoTFTNrDDwGnAsUAIvMbLa7Ly9VrjVwG7AgZloPYDTQEzga+KeZneDuexP3EUREJF7x1PQHAKvdfa27fw3kAMPLKPczYArwVcy04UCOu+929w+A1dH6REQkBeJJ+p2Aj2LeF0TT9jGzvsAx7v7Xqi4bLT/WzPLMLK+wsDCuwOua7GzIyIBGjcJzdnaqIxIROVg8Sd/KmOb7Zpo1Ah4BflDVZfdNcJ/m7pnuntmxY8c4QqpbsrNh7FhYvx7cw/PYsUr8IlL3xJP0C4BjYt53BjbEvG8N9AL+ZWbrgNOA2dHJ3MqWbRAmToSdOw+ctnNnmC4iUpfEk/QXAcebWTcza0Y4MTu7ZKa7b3X3Du6e4e4ZwHxgmLvnReVGm1lzM+sGHA8sTPinSLEPP6zadBGRVKk06bt7ETAemAOsAGa5+zIzm2xmwypZdhkwC1gOvATc0hB77nTpUrXpIiKpYu4HNbGnVGZmpufl5aU6jCopadOPbeJp2RKmTYOsrNTFJSLpw8wWu3tmZeV0RW4CZGWFBN+1K5iFZyV8EamLKr04S+KTlaUkLyJ1n2r6IiJpRElfRCSNKOmLiKQRJX0RkTSipC8ikkaU9EVE0oiSvohIGlHSFxFJI0r6IiJpRElfRCSNKOmLiKQRJX0RkTSipC8ikkaU9OsI3VhdRJJBQyvXAaVvwlJyY3XQcM0ikliq6dcBurG6iCSLkn4doBuri0iyKOnXAbqxuogki5J+HXD//eFG6rFatgzTRUQSSUm/DtCN1UUkWdR7p47QjdVFJBlU0xcRSSNK+iIiaURJX0QkjcSV9M1sqJmtNLPVZjahjPnjzOwdM1tqZq+bWY9oeoaZ7YqmLzWzJxL9AUREJH6Vnsg1s8bAY8C5QAGwyMxmu/vymGLPuPsTUflhwMPA0GjeGnfvk9iwRUSkOuKp6Q8AVrv7Wnf/GsgBhscWcPdtMW8PBTxxIYqISKLEk/Q7AR/FvC+Iph3AzG4xszXAFOC2mFndzGyJmb1mZmeWtQEzG2tmeWaWV1hYWIXwRUSkKuJJ+lbGtINq8u7+mLsfC9wFTIombwS6uHtf4E7gGTM7rIxlp7l7prtnduzYMf7oRUSkSuJJ+gXAMTHvOwMbKiifA4wAcPfd7r45er0YWAOcUL1QRUSkpuJJ+ouA482sm5k1A0YDs2MLmNnxMW8vAlZF0ztGJ4Ixs+7A8cDaRAQuIiJVV2nvHXcvMrPxwBygMfC0uy8zs8lAnrvPBsab2TnAHuAL4Jpo8bOAyWZWBOwFxrn757XxQUREpHLmXrc62mRmZnpeXl6qwxARqVfMbLG7Z1ZWTlfkioikESV9EZE0oqTfQGRnQ0YGNGoUnrOzUx2RiNRFGk+/AcjOhrFj999cff368B40Rr+IHEg1/QZg4sT9Cb/Ezp1huohILCX9BuDDD6s2XUTSl5J+A9ClS9Wmi0j6UtJvAO6/H1q2PHBay5ZhuohILCX9BiArC6ZNg65dwSw8T5umk7gicjD13mkgsrKU5EWkcqrpi4ikESV9EZE0oqQvIpJGlPRFRNKIkr6ISBpR0hcRSSNK+gJolE6RdKF++qJROkXSiGr6olE6RdKIkr5olE6RNKKkLxqlUySNKOmLRukUSSNK+qJROkXSiHrvCKBROkXShWr6IiJpRElfEkIXd4nUD3ElfTMbamYrzWy1mU0oY/44M3vHzJaa2etm1iNm3o+j5Vaa2fmJDF7qhpKLu9avB/f9F3cp8YvUPebuFRcwawy8D5wLFACLgDHuvjymzGHuvi16PQy42d2HRsl/JjAAOBr4J3CCu+8tb3uZmZmel5dXs08lSZWRERJ9aV27wrp1yY5GJD2Z2WJ3z6ysXDw1/QHAandf6+5fAznA8NgCJQk/cihQciQZDuS4+253/wBYHa1PGhBd3CVSf8ST9DsBH8W8L4imHcDMbjGzNcAU4LaqLCv1my7uEqk/4kn6Vsa0g9qE3P0xdz8WuAuYVJVlzWysmeWZWV5hYWEcIUldoou7ROqPeJJ+AXBMzPvOwIYKyucAI6qyrLtPc/dMd8/s2LFjHCFJXaKLu0Tqj3iS/iLgeDPrZmbNgNHA7NgCZnZ8zNuLgFXR69nAaDNrbmbdgOOBhTUPW+qarKxw0ra4ODwr4YvUTZUmfXcvAsYDc4AVwCx3X2Zmk6OeOgDjzWyZmS0F7gSuiZZdBswClgMvAbdU1HNH0pf6+YskR6VdNpNNXTbTT+mbuEA4J6AmIpH4JbLLpkit0k1cRJJHSV9STv38RZJHSV9STv38RZJHSV9SLhH9/HUiWCQ+SvqScjXt568B30Tip947Uu9pwDcR9d6RNKITwSLxU9KXek8ngkXip6Qv9Z4GfBOJn5K+1HuJGPBNvX8kXTRJdQAiiZCVVf0hG0oPA1HS+6dkvSINiWr6kvY0DISkEyV9SXvq/SPpRElf0p56/0g6UdKXtKdhICSd1IsTuXv27KGgoICvvvoq1aFIHFq0aEHnzp1p2rRpqkOJS8nJ2okTQ5NOly4h4Vd1GAidCJb6oF4Mw/DBBx/QunVr2rdvj1lZ91qXusLd2bx5M9u3b6dbt26pDicpNAyE1AUNahiGr776Sgm/njAz2rdvn1a/ynQiWOqTepH0ASX8eiTdvqtEnAjWOQFJlnqT9EXqqpqeCNbQ0JJMDTLpJ7rWtHnzZvr06UOfPn048sgj6dSp0773X3/9dVzruO6661i5cmWFZR577DGyE/SfPnDgQJYuXZqQdUnFajoMhC4Ok2SqF713qqI2elK0b99+XwK95557aNWqFT/84Q8PKOPuuDuNGpV9HJ0+fXql27nllluqF6CkXE2GgdA5AUmmBlfTT2atafXq1fTq1Ytx48bRr18/Nm7cyNixY8nMzKRnz55Mnjx5X9mSmndRURFt27ZlwoQJnHLKKZx++ul8+umnAEyaNIlHH310X/kJEyYwYMAAvvnNb/Lmm28C8OWXX/Kd73yHU045hTFjxpCZmVlpjX7GjBmcfPLJ9OrVi7vvvhuAoqIivvvd7+6bPnXqVAAeeeQRevTowSmnnMJVV12V8H0mB9M5AUmmBlfTT3atafny5UyfPp0nnngCgAceeIDDDz+coqIihgwZwqhRo+jRo8cBy2zdupVBgwbxwAMPcOedd/L0008zYcKEg9bt7ixcuJDZs2czefJkXnrpJX71q19x5JFH8vzzz/P222/Tr1+/CuMrKChg0qRJ5OXl0aZNG8455xz++te/0rFjRz777DPeeecdALZs2QLAlClTWL9+Pc2aNds3TWrX/fcf+OsUqndOQNcJSDwaXE0/2ZfUH3vssXzrW9/a937mzJn069ePfv36sWLFCpYvX37QMocccggXXHABAP3792ddOZ25R44ceVCZ119/ndGjRwNwyimn0LNnzwrjW7BgAWeffTYdOnSgadOmXHnllcydO5fjjjuOlStXcvvttzNnzhzatGkDQM+ePbnqqqvIzs6uNxdX1Xc6JyDJ1OCSfrJvqHEkVZt9AAAOdElEQVTooYfue71q1Sp++ctf8uqrr5Kfn8/QoUPL7K/erFmzfa8bN25MUVFRmetu3rz5QWWqejFdeeXbt29Pfn4+AwcOZOrUqdx4440AzJkzh3HjxrFw4UIyMzPZu3dvlbYn1ZOVFS7kKi4Oz1WpoSfi162ah9JHg0v6ibihRnVt27aN1q1bc9hhh7Fx40bmzJmT8G0MHDiQWbNmAfDOO++U+Usi1mmnnUZubi6bN2+mqKiInJwcBg0aRGFhIe7OZZddxr333stbb73F3r17KSgo4Oyzz+YXv/gFhYWF7CxdhZQ6p6a/btVlNL3E1aZvZkOBXwKNgafc/YFS8+8EbgCKgELgendfH83bC7wTFf3Q3YclKPZy1aQnRU3069ePHj160KtXL7p3784ZZ5yR8G3ceuutXH311fTu3Zt+/frRq1evfU0zZencuTOTJ09m8ODBuDuXXHIJF110EW+99Rbf+973cHfMjAcffJCioiKuvPJKtm/fTnFxMXfddRetW7dO+GeQxKrpOYGKmod0TqABKulqWN6DkOjXAN2BZsDbQI9SZYYALaPXNwHPxszbUdk2Yh/9+/f30pYvX37QtHS1Z88e37Vrl7u7v//++56RkeF79uxJcVQH03eWXDNmuHft6m4WnmfMiH9ZM/dQxz/wYVZb0UptAPI8jhwbT01/ALDa3dcCmFkOMBzY167g7rkx5ecD6utXS3bs2MG3v/1tioqKcHeefPJJmjRpcJ2wpIpq8uu2S5eyB4zT/QQapnja9DsBH8W8L4imled7wIsx71uYWZ6ZzTezEWUtYGZjozJ5hYWFcYSUvtq2bcvixYt5++23yc/P57zzzkt1SFLP6X4C6SWeKmJZo2eV2SXEzK4CMoFBMZO7uPsGM+sOvGpm77j7mgNW5j4NmAZhaOW4IheRhND9BNJLPDX9AuCYmPedgQ2lC5nZOcBEYJi77y6Z7u4boue1wL+AvjWIV0RqQU26jCbiOgH9UkieeJL+IuB4M+tmZs2A0cDs2AJm1hd4kpDwP42Z3s7MmkevOwBnEHMuQETqv5peJ5CILqM6aMSv0qTv7kXAeGAOsAKY5e7LzGyymZV0v/wF0Ar4g5ktNbOSg8JJQJ6ZvQ3kAg+4u5K+SANS0+sEavpLQdcZVE1cF2e5+9/d/QR3P9bd74+m/dTdZ0evz3H3I9y9T/QYFk1/091PdvdTouff1N5HqT2DBw8+6EKrRx99lJtvvrnC5Vq1agXAhg0bGDVqVLnrLn17yNIeffTRAy6SuvDCCxMyLs4999zDQw89VOP1SHqr6Yngmv5S0DAUVdPgrsitDWPGjCEnJ+eAaTk5OYwZMyau5Y8++miee+65am+/dNL/+9//Ttu2bau9PpFEqulV8DX9paBhKKqm3nXw/v73IdH3BunTB6IRjcs0atQoJk2axO7du2nevDnr1q1jw4YNDBw4kB07djB8+HC++OIL9uzZw3333cfw4cMPWH7dunVcfPHFvPvuu+zatYvrrruO5cuXc9JJJ7Fr16595W666SYWLVrErl27GDVqFPfeey9Tp05lw4YNDBkyhA4dOpCbm0tGRgZ5eXl06NCBhx9+mKeffhqAG264ge9///usW7eOCy64gIEDB/Lmm2/SqVMn/vznP3PIIYeU+xmXLl3KuHHj2LlzJ8ceeyxPP/007dq1Y+rUqTzxxBM0adKEHj16kJOTw2uvvcbtt98OhFsjzp07V1fuprmaXCdQ0yuKa3qdQbr1PlJNPw7t27dnwIABvPTSS0Co5V9xxRWYGS1atOCFF17grbfeIjc3lx/84AcVDor2+OOP07JlS/Lz85k4cSKLFy/eN+/+++8nLy+P/Px8XnvtNfLz87nttts4+uijyc3NJTc394B1LV68mOnTp7NgwQLmz5/Pr3/9a5YsWQKEwd9uueUWli1bRtu2bXn++ecr/IxXX301Dz74IPn5+Zx88snce++9QBgqesmSJeTn5+8bPvqhhx7iscceY+nSpcybN6/Cg4lIZWr6S6GmzUvp1vuo3tX0K6qR16aSJp7hw4eTk5Ozr3bt7tx9993MnTuXRo0a8fHHH7Np0yaOPPLIMtczd+5cbrvtNgB69+5N7969982bNWsW06ZNo6ioiI0bN7J8+fID5pf2+uuvc+mll+4b6XPkyJHMmzePYcOG0a1bN/r06QNUPHwzhPH9t2zZwqBB4fKKa665hssuu2xfjFlZWYwYMYIRI8K1dWeccQZ33nknWVlZjBw5ks6dO8ezC0XKVZNfCjW9ziBRvY/qyy8F1fTjNGLECF555RXeeustdu3ate/mJdnZ2RQWFrJ48WKWLl3KEUccUeZwyrHMDr7e7YMPPuChhx7ilVdeIT8/n4suuqjS9VT0i6JkWGaoePjmyvztb3/jlltuYfHixfTv35+ioiImTJjAU089xa5duzjttNN47733qrVukUSpyXUGqe59BMn9paCkH6dWrVoxePBgrr/++gNO4G7dupVvfOMbNG3alNzcXNaX1bgY46yzztp38/N3332X/Px8IAzLfOihh9KmTRs2bdrEiy/uH8midevWbN++vcx1/elPf2Lnzp18+eWXvPDCC5x55plV/mxt2rShXbt2zJs3D4Df//73DBo0iOLiYj766COGDBnClClT2LJlCzt27GDNmjWcfPLJ3HXXXWRmZirpS72W6t5Hye5yWu+ad1JpzJgxjBw58oCePFlZWVxyySVkZmbSp08fTjzxxArXcdNNN3HdddfRu3dv+vTpw4ABA4BwF6y+ffvSs2fPg4ZlHjt2LBdccAFHHXXUAe36/fr149prr923jhtuuIG+fftW2JRTnt/97nf7TuR2796d6dOns3fvXq666iq2bt2Ku3PHHXfQtm1bfvKTn5Cbm0vjxo3p0aPHvruAidRHNW0equmJ5GQPbW0VNRGkQmZmppfut75ixQpOOumkFEUk1aHvTNJF6TZ9CL8U4j0Z3ahRqOGXZhaaq+JlZovdPbPS7cW/ShERKS3V1ylUlZK+iEgN1eREcrLv611vkn5da4aS8um7Eolfsu/rXS9O5LZo0YLNmzfTvn37Mrs7St3h7mzevJkWLVqkOhSReiOZ9/WuF0m/c+fOFBQUoLtq1Q8tWrTQBVsidVS9SPpNmzalW7duqQ5DRKTeqzdt+iIiUnNK+iIiaURJX0QkjdS5K3LNrBCoeACbinUAPktQOLVB8dWM4qsZxVczdTm+ru7esbJCdS7p15SZ5cVzKXKqKL6aUXw1o/hqpq7HFw8174iIpBElfRGRNNIQk/60VAdQCcVXM4qvZhRfzdT1+CrV4Nr0RUSkfA2xpi8iIuVQ0hcRSSP1Mumb2VAzW2lmq81sQhnzm5vZs9H8BWaWkcTYjjGzXDNbYWbLzOz2MsoMNrOtZrY0evw0WfHFxLDOzN6Jtp9Xxnwzs6nRPsw3s35JjO2bMftmqZltM7PvlyqT1H1oZk+b2adm9m7MtMPN7GUzWxU9tytn2WuiMqvM7JokxvcLM3sv+v5eMLO25Sxb4d9CLcZ3j5l9HPMdXljOshX+v9difM/GxLbOzJaWs2yt77+Ecvd69QAaA2uA7kAz4G2gR6kyNwNPRK9HA88mMb6jgH7R69bA+2XENxj4a4r34zqgQwXzLwReBAw4DViQwu/7E8KFJynbh8BZQD/g3ZhpU4AJ0esJwINlLHc4sDZ6bhe9bpek+M4DmkSvHywrvnj+FmoxvnuAH8bx/Vf4/15b8ZWa/3+Bn6Zq/yXyUR9r+gOA1e6+1t2/BnKA4aXKDAd+F71+Dvi2JWkgfnff6O5vRa+3AyuATsnYdoINB/6fB/OBtmZ2VAri+Dawxt1rcpV2jbn7XODzUpNj/85+B4woY9HzgZfd/XN3/wJ4GRiajPjc/R/uXhS9nQ+kbLzrcvZfPOL5f6+xiuKLcsflwMxEbzcV6mPS7wR8FPO+gIOT6r4y0R/9VqB9UqKLETUr9QUWlDH7dDN728xeNLOeSQ0scOAfZrbYzMaWMT+e/ZwMoyn/ny3V+/AId98I4WAPfKOMMnVlP15P+OVWlsr+FmrT+Kj56elymsfqwv47E9jk7qvKmZ/K/Vdl9THpl1VjL93vNJ4ytcrMWgHPA993922lZr9FaK44BfgV8KdkxhY5w937ARcAt5jZWaXm14V92AwYBvyhjNl1YR/Goy7sx4lAEZBdTpHK/hZqy+PAsUAfYCOhCaW0lO8/YAwV1/JTtf+qpT4m/QLgmJj3nYEN5ZUxsyZAG6r307JazKwpIeFnu/sfS893923uviN6/XegqZl1SFZ80XY3RM+fAi8QfkbHimc/17YLgLfcfVPpGXVhHwKbSpq8oudPyyiT0v0YnTi+GMjyqAG6tDj+FmqFu29y973uXgz8upztpnr/NQFGAs+WVyZV+6+66mPSXwQcb2bdoprgaGB2qTKzgZJeEqOAV8v7g0+0qP3vN8AKd3+4nDJHlpxjMLMBhO9hczLii7Z5qJm1LnlNOOH3bqlis4Gro148pwFbS5oykqjcGlaq92Ek9u/sGuDPZZSZA5xnZu2i5ovzomm1zsyGAncBw9x9Zzll4vlbqK34Ys8RXVrOduP5f69N5wDvuXtBWTNTuf+qLdVnkqvzIPQseZ9wVn9iNG0y4Y8boAWhSWA1sBDonsTYBhJ+fuYDS6PHhcA4YFxUZjywjNATYT7wH0nef92jbb8dxVGyD2NjNOCxaB+/A2QmOcaWhCTeJmZayvYh4eCzEdhDqH1+j3Ce6BVgVfR8eFQ2E3gqZtnro7/F1cB1SYxvNaE9vOTvsKRH29HA3yv6W0hSfL+P/rbyCYn8qNLxRe8P+n9PRnzR9N+W/M3FlE36/kvkQ8MwiIikkfrYvCMiItWkpC8ikkaU9EVE0oiSvohIGlHSFxFJI0r6IiJpRElfRCSN/H+/ZetCZCCY5gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(len(loss))\n",
    "\n",
    "plt.figure()\n",
    "\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "So the reversed-order GRU strongly underperforms even the common-sense baseline, indicating that the in our case chronological processing is very \n",
    "important to the success of our approach. This makes perfect sense: the underlying GRU layer will typically be better at remembering the \n",
    "recent past than the distant past, and naturally the more recent weather data points are more predictive than older data points in our \n",
    "problem (that's precisely what makes the common-sense baseline a fairly strong baseline). Thus the chronological version of the layer is \n",
    "bound to outperform the reversed-order version. Importantly, this is generally not true for many other problems, including natural \n",
    "language: intuitively, the importance of a word in understanding a sentence is not usually dependent on its position in the sentence. Let's \n",
    "try the same trick on the LSTM IMDB example from the previous section:\n",
    "\n",
    "逆序 GRU 的效果甚至比基于常识的基准方法还要差很多，这说明在本例中，按时间正序处理对成功解决问题很重要。这非常合理：GRU 层通常更善于记住最近的数据，而不是久远的数据，与更早的数据点相比，更靠后的天气数据点对问题自然具有更高的预测能力（这也是基于常识的基准方法非常强大的原因）。因此，按时间正序的模型必然会优于时间逆序的模型。重要的是，对许多其他问题（包括自然语言）而言，情况并不是这样：直觉上来看，一个单词对理解句子的重要性通常并不取决于它在句子中的位置。我们尝试对上一节 IMDB 示例中的 LSTM 应用相同的技巧。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 167s 8ms/step - loss: 0.4854 - acc: 0.7696 - val_loss: 0.4135 - val_acc: 0.8344\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 166s 8ms/step - loss: 0.3057 - acc: 0.8842 - val_loss: 0.3418 - val_acc: 0.8568\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 166s 8ms/step - loss: 0.2551 - acc: 0.9031 - val_loss: 0.3131 - val_acc: 0.8722\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 166s 8ms/step - loss: 0.2175 - acc: 0.9200 - val_loss: 0.6058 - val_acc: 0.8084\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 166s 8ms/step - loss: 0.1921 - acc: 0.9314 - val_loss: 0.6356 - val_acc: 0.8154\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 166s 8ms/step - loss: 0.1741 - acc: 0.9393 - val_loss: 0.3660 - val_acc: 0.8782\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 166s 8ms/step - loss: 0.1544 - acc: 0.9456 - val_loss: 0.4077 - val_acc: 0.8724\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 166s 8ms/step - loss: 0.1468 - acc: 0.9511 - val_loss: 0.3923 - val_acc: 0.8458\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 166s 8ms/step - loss: 0.1301 - acc: 0.9552 - val_loss: 0.5633 - val_acc: 0.8638\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 167s 8ms/step - loss: 0.1111 - acc: 0.9625 - val_loss: 0.4430 - val_acc: 0.8734\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import imdb\n",
    "from keras.preprocessing import sequence\n",
    "from keras import layers\n",
    "from keras.models import Sequential\n",
    "\n",
    "# Number of words to consider as features（作为特征的单词个数）\n",
    "max_features = 10000\n",
    "# Cut texts after this number of words (among top max_features most common words)\n",
    "# 在这么多单词之后截断文本（这些单词都属于前 max_features 个最常见的单词）\n",
    "maxlen = 500\n",
    "\n",
    "# Load data（加载数据）\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
    "\n",
    "# Reverse sequences（将序列反转）\n",
    "x_train = [x[::-1] for x in x_train]\n",
    "x_test = [x[::-1] for x in x_test]\n",
    "\n",
    "# Pad sequences（填充序列）\n",
    "x_train = sequence.pad_sequences(x_train, maxlen=maxlen)\n",
    "x_test = sequence.pad_sequences(x_test, maxlen=maxlen)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(layers.Embedding(max_features, 128))\n",
    "model.add(layers.LSTM(32))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['acc'])\n",
    "history = model.fit(x_train, y_train,\n",
    "                    epochs=10,\n",
    "                    batch_size=128,\n",
    "                    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "We get near-identical performance as the chronological-order LSTM we tried in the previous section.\n",
    "\n",
    "Thus, remarkably, on such a text dataset, reversed-order processing works just as well as chronological processing, confirming our \n",
    "hypothesis that, albeit word order *does* matter in understanding language, *which* order you use isn't crucial. Importantly, a RNN trained \n",
    "on reversed sequences will learn different representations than one trained on the original sequences, in much the same way that you would \n",
    "have quite different mental models if time flowed backwards in the real world -- if you lived a life where you died on your first day and \n",
    "you were born on your last day. In machine learning, representations that are *different* yet *useful* are always worth exploiting, and the \n",
    "more they differ the better: they offer a new angle from which to look at your data, capturing aspects of the data that were missed by other \n",
    "approaches, and thus they can allow to boost performance on a task. This is the intuition behind \"ensembling\", a concept that we will \n",
    "introduce in the next chapter.\n",
    "\n",
    "A bidirectional RNN exploits this idea to improve upon the performance of chronological-order RNNs: it looks at its inputs sequence both \n",
    "ways, obtaining potentially richer representations and capturing patterns that may have been missed by the chronological-order version alone.\n",
    "\n",
    "模型性能与正序 LSTM 几乎相同。\n",
    "\n",
    "值得注意的是，在这样一个文本数据集上，逆序处理的效果与正序处理一样好，这证实了一个假设：虽然单词顺序对理解语言很重要，但使用哪种顺序并不重要。重要的是，在逆序序列上训练的 RNN 学到的表示不同于在原始序列上学到的表示，正如在现实世界中，如果时间倒流（你的人生是第一天死去、最后一天出生），那么你的心智模型也会完全不同。在机器学习中，如果一种数据表示不同但有用，那么总是值得加以利用，这 种表示与其他表示的差异越大越好，它们提供了查看数据的全新角度，抓住了数据中被其他方法忽略的内容，因此可以提高模型在某个任务上的性能。这是集成（ensembling）方法背后的直觉，我们将在下一章介绍集成的概念。\n",
    "\n",
    "双向 RNN 正是利用这个想法来提高正序 RNN 的性能。它从两个方向查看数据，从而得到更加丰富的表示，并捕捉到仅使用正序 RNN 时可能忽略的一些模式。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![bidirectional rnn](data/bidirectional_rnn.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To instantiate a bidirectional RNN in Keras, one would use the `Bidirectional` layer, which takes as first argument a recurrent layer \n",
    "instance. `Bidirectional` will create a second, separate instance of this recurrent layer, and will use one instance for processing the \n",
    "input sequences in chronological order and the other instance for processing the input sequences in reversed order. Let's try it on the \n",
    "IMDB sentiment analysis task:\n",
    "\n",
    "在 Keras 中将一个双向 RNN 实例化，我们需要使用 Bidirectional 层，它的第一个参数是一个循环层实例。Bidirectional 对这个循环层创建了第二个单独实例，然后使用一个实例按正序处理输入序列，另一个实例按逆序处理输入序列。我们在 IMDB 情感分析任务上来试一下这种方法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import backend as K\n",
    "K.clear_session()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 332s 17ms/step - loss: 0.5594 - acc: 0.7167 - val_loss: 0.4008 - val_acc: 0.8496\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 331s 17ms/step - loss: 0.3295 - acc: 0.8747 - val_loss: 0.3924 - val_acc: 0.8596\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 332s 17ms/step - loss: 0.2672 - acc: 0.9014 - val_loss: 0.4422 - val_acc: 0.8650\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 333s 17ms/step - loss: 0.2308 - acc: 0.9156 - val_loss: 0.3335 - val_acc: 0.8780\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 330s 16ms/step - loss: 0.2089 - acc: 0.9266 - val_loss: 0.3355 - val_acc: 0.8850\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 328s 16ms/step - loss: 0.1839 - acc: 0.9365 - val_loss: 0.4771 - val_acc: 0.7986\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 329s 16ms/step - loss: 0.1711 - acc: 0.9398 - val_loss: 0.3715 - val_acc: 0.8826\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 331s 17ms/step - loss: 0.1572 - acc: 0.9482 - val_loss: 0.5079 - val_acc: 0.7900\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 329s 16ms/step - loss: 0.1469 - acc: 0.9502 - val_loss: 0.3486 - val_acc: 0.8630\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 331s 17ms/step - loss: 0.1367 - acc: 0.9539 - val_loss: 0.3605 - val_acc: 0.8804\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(layers.Embedding(max_features, 32))\n",
    "model.add(layers.Bidirectional(layers.LSTM(32)))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])\n",
    "history = model.fit(x_train, y_train, epochs=10, batch_size=128, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It performs slightly better than the regular LSTM we tried in the previous section, going above 88% validation accuracy. It also seems to \n",
    "overfit faster, which is unsurprising since a bidirectional layer has twice more parameters than a chronological LSTM. With some \n",
    "regularization, the bidirectional approach would likely be a strong performer on this task.\n",
    "\n",
    "Now let's try the same approach on the weather prediction task:\n",
    "\n",
    "这个模型的表现比上一节的普通 LSTM 略好，验证精度超过 88%。这个模型似乎也很快就开始过拟合，这并不令人惊讶，因为双向层的参数个数是正序 LSTM 的 2 倍。添加一些正则化，双向方法在这个任务上可能会有很好的表现。\n",
    "\n",
    "接下来，我们尝试将相同的方法应用于温度预测任务。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/40\n",
      "500/500 [==============================] - 470s 940ms/step - loss: 0.2930 - val_loss: 0.2739\n",
      "Epoch 2/40\n",
      "500/500 [==============================] - 466s 932ms/step - loss: 0.2743 - val_loss: 0.2733\n",
      "Epoch 3/40\n",
      "500/500 [==============================] - 471s 941ms/step - loss: 0.2665 - val_loss: 0.2690\n",
      "Epoch 4/40\n",
      "500/500 [==============================] - 467s 933ms/step - loss: 0.2628 - val_loss: 0.2713\n",
      "Epoch 5/40\n",
      "500/500 [==============================] - 467s 934ms/step - loss: 0.2551 - val_loss: 0.2714\n",
      "Epoch 6/40\n",
      "500/500 [==============================] - 465s 930ms/step - loss: 0.2485 - val_loss: 0.2743\n",
      "Epoch 7/40\n",
      "500/500 [==============================] - 470s 939ms/step - loss: 0.2440 - val_loss: 0.2819\n",
      "Epoch 8/40\n",
      "500/500 [==============================] - 474s 948ms/step - loss: 0.2386 - val_loss: 0.2809\n",
      "Epoch 9/40\n",
      "500/500 [==============================] - 475s 951ms/step - loss: 0.2322 - val_loss: 0.2803\n",
      "Epoch 10/40\n",
      "500/500 [==============================] - 474s 948ms/step - loss: 0.2277 - val_loss: 0.2835\n",
      "Epoch 11/40\n",
      "500/500 [==============================] - 471s 943ms/step - loss: 0.2225 - val_loss: 0.2868\n",
      "Epoch 12/40\n",
      "500/500 [==============================] - 474s 949ms/step - loss: 0.2173 - val_loss: 0.2881\n",
      "Epoch 13/40\n",
      "500/500 [==============================] - 473s 946ms/step - loss: 0.2111 - val_loss: 0.2933\n",
      "Epoch 14/40\n",
      "500/500 [==============================] - 474s 947ms/step - loss: 0.2083 - val_loss: 0.3016\n",
      "Epoch 15/40\n",
      "500/500 [==============================] - 473s 946ms/step - loss: 0.2027 - val_loss: 0.3052\n",
      "Epoch 16/40\n",
      "500/500 [==============================] - 472s 945ms/step - loss: 0.1993 - val_loss: 0.3074\n",
      "Epoch 17/40\n",
      "500/500 [==============================] - 473s 947ms/step - loss: 0.1941 - val_loss: 0.3064\n",
      "Epoch 18/40\n",
      "500/500 [==============================] - 469s 939ms/step - loss: 0.1925 - val_loss: 0.3150\n",
      "Epoch 19/40\n",
      "500/500 [==============================] - 471s 942ms/step - loss: 0.1888 - val_loss: 0.3161\n",
      "Epoch 20/40\n",
      "500/500 [==============================] - 471s 943ms/step - loss: 0.1849 - val_loss: 0.3181\n",
      "Epoch 21/40\n",
      "500/500 [==============================] - 471s 942ms/step - loss: 0.1808 - val_loss: 0.3144\n",
      "Epoch 22/40\n",
      "500/500 [==============================] - 468s 937ms/step - loss: 0.1779 - val_loss: 0.3203\n",
      "Epoch 23/40\n",
      "500/500 [==============================] - 458s 916ms/step - loss: 0.1752 - val_loss: 0.3171\n",
      "Epoch 24/40\n",
      "500/500 [==============================] - 471s 941ms/step - loss: 0.1714 - val_loss: 0.3207\n",
      "Epoch 25/40\n",
      "500/500 [==============================] - 467s 934ms/step - loss: 0.1692 - val_loss: 0.3234\n",
      "Epoch 26/40\n",
      "500/500 [==============================] - 467s 934ms/step - loss: 0.1671 - val_loss: 0.3260\n",
      "Epoch 27/40\n",
      "500/500 [==============================] - 464s 928ms/step - loss: 0.1645 - val_loss: 0.3233\n",
      "Epoch 28/40\n",
      "500/500 [==============================] - 462s 925ms/step - loss: 0.1614 - val_loss: 0.3226\n",
      "Epoch 29/40\n",
      "500/500 [==============================] - 464s 927ms/step - loss: 0.1591 - val_loss: 0.3253\n",
      "Epoch 30/40\n",
      "500/500 [==============================] - 464s 928ms/step - loss: 0.1584 - val_loss: 0.3247\n",
      "Epoch 31/40\n",
      "500/500 [==============================] - 464s 927ms/step - loss: 0.1549 - val_loss: 0.3312\n",
      "Epoch 32/40\n",
      "500/500 [==============================] - 465s 929ms/step - loss: 0.1530 - val_loss: 0.3310\n",
      "Epoch 33/40\n",
      "500/500 [==============================] - 467s 934ms/step - loss: 0.1513 - val_loss: 0.3291\n",
      "Epoch 34/40\n",
      "500/500 [==============================] - 464s 927ms/step - loss: 0.1499 - val_loss: 0.3326\n",
      "Epoch 35/40\n",
      "500/500 [==============================] - 466s 932ms/step - loss: 0.1480 - val_loss: 0.3379\n",
      "Epoch 36/40\n",
      "500/500 [==============================] - 468s 937ms/step - loss: 0.1467 - val_loss: 0.3335\n",
      "Epoch 37/40\n",
      "500/500 [==============================] - 465s 930ms/step - loss: 0.1456 - val_loss: 0.3360\n",
      "Epoch 38/40\n",
      "500/500 [==============================] - 465s 929ms/step - loss: 0.1433 - val_loss: 0.3313\n",
      "Epoch 39/40\n",
      "500/500 [==============================] - 458s 917ms/step - loss: 0.1424 - val_loss: 0.3359\n",
      "Epoch 40/40\n",
      "500/500 [==============================] - 461s 923ms/step - loss: 0.1407 - val_loss: 0.3347\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras import layers\n",
    "from keras.optimizers import RMSprop\n",
    "\n",
    "model = Sequential()\n",
    "model.add(layers.Bidirectional(\n",
    "    layers.GRU(32), input_shape=(None, float_data.shape[-1])))\n",
    "model.add(layers.Dense(1))\n",
    "\n",
    "model.compile(optimizer=RMSprop(), loss='mae')\n",
    "history = model.fit_generator(train_gen,\n",
    "                              steps_per_epoch=500,\n",
    "                              epochs=40,\n",
    "                              validation_data=val_gen,\n",
    "                              validation_steps=val_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "It performs about as well as the regular GRU layer. It's easy to understand why: all of the predictive capacity must be coming from the \n",
    "chronological half of the network, since the anti-chronological half is known to be severely underperforming on this task (again, because \n",
    "the recent past matters much more than the distant past in this case).\n",
    "\n",
    "这个模型的表现与普通 GRU 层差不多一样好。其原因很容易理解：所有的预测能力肯定都来自于正序的那一半网络，因为我们已经知道，逆序的那一半在这个任务上的表现非常糟糕（本例同样是因为，最近的数据比久远的数据更加重要）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Going even further\n",
    "\n",
    "At this stage, there are still many other things you could try in order to improve performance on our weather forecasting problem:\n",
    "\n",
    "* Adjust the number of units in each recurrent layer in the stacked setup. Our current choices are largely arbitrary and thus likely \n",
    "suboptimal.\n",
    "* Adjust the learning rate used by our `RMSprop` optimizer.\n",
    "* Try using `LSTM` layers instead of `GRU` layers.\n",
    "* Try using a bigger densely-connected regressor on top of the recurrent layers, i.e. a bigger `Dense` layer or even a stack of `Dense` \n",
    "layers.\n",
    "* Don't forget to eventually run the best performing models (in terms of validation MAE) on the test set! Least you start developing \n",
    "architectures that are overfitting to the validation set.   \n",
    "\n",
    "As usual: deep learning is more an art than a science, and while we can provide guidelines as to what is likely to work or not work on a \n",
    "given problem, ultimately every problem is unique and you will have to try and evaluate different strategies empirically. There is \n",
    "currently no theory that will tell you in advance precisely what you should do to optimally solve a problem. You must try and iterate.\n",
    "\n",
    "\n",
    "## Wrapping up\n",
    "\n",
    "Here's what you should take away from this section:\n",
    "\n",
    "* As you first learned in Chapter 4, when approaching a new problem, \n",
    "it is good to first establish common sense baselines for your metric of choice. If you don't have a \n",
    "baseline to beat, you can't tell if you are making any real progress.\n",
    "* Try simple models before expensive ones, to justify the additional expense. Sometimes a simple model will turn out to be your best option.\n",
    "* On data where temporal ordering matters, recurrent networks are a great fit and easily outperform models that first flatten the temporal \n",
    "data.\n",
    "* To use dropout with recurrent networks, one should use a time-constant dropout mask and recurrent dropout mask. This is built into Keras \n",
    "recurrent layers, so all you have to do is use the `dropout` and `recurrent_dropout` arguments of recurrent layers.\n",
    "* Stacked RNNs provide more representational power than a single RNN layer. They are also much more expensive, and thus not always worth it. \n",
    "While they offer clear gains on complex problems (e.g. machine translation), they might not always be relevant to smaller, simpler problems.\n",
    "* Bidirectional RNNs, which look at a sequence both ways, are very useful on natural language processing problems. However, they will not \n",
    "be strong performers on sequence data where the recent past is much more informative than the beginning of the sequence.\n",
    "\n",
    "Note there are two important concepts that we will not cover in detail here: recurrent \"attention\", and sequence masking. Both tend to be \n",
    "especially relevant for natural language processing, and are not particularly applicable to our temperature forecasting problem. We will \n",
    "leave them for future study outside of this book.\n",
    "\n",
    "## 更多尝试\n",
    "为了提高温度预测问题的性能，你还可以尝试下面这些方法。\n",
    "* 在堆叠循环层中调节每层的单元个数。当前取值在很大程度上是任意选择的，因此可能不是最优的。\n",
    "* 调节 RMSprop 优化器的学习率。\n",
    "* 尝试使用 LSTM 层代替 GRU 层。\n",
    "* 在循环层上面尝试使用更大的密集连接回归器，即更大的 Dense 层或 Dense 层的堆叠。\n",
    "* 不要忘记最后在测试集上运行性能最佳的模型（即验证 MAE 最小的模型）。否则，你开发的网络架构将会对验证集过拟合。\n",
    "\n",
    "正如前面所说，深度学习是一门艺术而不是科学。我们可以提供指导，对于给定问题哪些方法可能有用、哪些方法可能没用，但归根结底，每个问题都是独一无二的，你必须根据经验对不同的策略进行评估。目前没有任何理论能够提前准确地告诉你，应该怎么做才能最优地解决问题。你必须不断迭代。\n",
    "\n",
    "## 小结\n",
    "下面是你应该从本节中学到的要点。\n",
    "* 我们在第 4 章学过，遇到新问题时，最好首先为你选择的指标建立一个基于常识的基准。 如果没有需要打败的基准，那么就无法分辨是否取得了真正的进步。\n",
    "* 在尝试计算代价较高的模型之前，先尝试一些简单的模型，以此证明增加计算代价是有 意义的。有时简单模型就是你的最佳选择。\n",
    "* 如果时间顺序对数据很重要，那么循环网络是一种很适合的方法，与那些先将时间数据 展平的模型相比，其性能要更好。\n",
    "* 想要在循环网络中使用 dropout，你应该使用一个不随时间变化的 dropout 掩码与循环 dropout 掩码。这二者都内置于 Keras 的循环层中，所以你只需要使用循环层的 dropout 和 recurrent_dropout 参数即可。\n",
    "* 与单个 RNN 层相比，堆叠 RNN 的表示能力更加强大。但它的计算代价也更高，因此不 一定总是需要。虽然它在机器翻译等复杂问题上很有效，但在较小、较简单的问题上可能不一定有用。\n",
    " \n",
    "注意：有两个重要的概念我们这里没有详细介绍：循环注意（recurrent attention）和序列掩码（sequence masking）。这两个概念通常对自然语言处理特别有用，但并不适用于温度预测问题。你可以在学完本书后对其做进一步研究。\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}