Data Collection Cleaning and Merging.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:21:12.872991Z",
     "start_time": "2021-03-05T16:21:11.711594Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import datetime\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Sources #\n",
    "\n",
    "## London DataStore ##\n",
    "\n",
    "https://data.london.gov.uk/dataset/london-fire-brigade-incident-records \n",
    "\n",
    "2 files listing all LFB callouts for 2017-2020 (inclusive) and 2013-2016 (inclusive) - 2016-2020 used for the project.\n",
    "\n",
    "## Met Office Archives (CEDA Midas Open) ##\n",
    "\n",
    "https://catalogue.ceda.ac.uk/uuid/dbd451271eb04662beade68da43546e1\n",
    "\n",
    "1 file per year for hourly weather observations, with additional files per year for rain details and wind details for 2016-2019 (inclusive) - 12 files in total.\n",
    "\n",
    "## Meteostat ##\n",
    "\n",
    "https://github.com/meteostat Great resource (with python library and easy api) for historical weather data worldwide (initially was going to webscrape from the website - https://meteostat.net/en/station/03772) 1 File for 2020 - one observation (sun) missing from the dataset compared with met office records (and other observations to be feature engineered). Also used to backfill data missing from met office records.\n",
    "\n",
    "## Earth System Research Laboratories ##\n",
    "\n",
    "https://www.esrl.noaa.gov/gmd/grad/solcalc/calcdetails.html - excel spreadsheet to download to calculate sunrise/sunset times in London (needed for an 'is light' marker).\n",
    " \n",
    "## Met Office Forecast (London) ##\n",
    "\n",
    "https://www.metoffice.gov.uk/weather/forecast/gcpvj0v07#? - for current London forecast for the next 7 days (needed for current prediction).\n",
    "\n",
    "## Met Office Actual (Heathrow) ##\n",
    "\n",
    "https://www.metoffice.gov.uk/weather/observations/gcpsvg3nc - for current air pressure London (needed for current prediction)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# London Fire Brigade (LFB) Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:21:33.234688Z",
     "start_time": "2021-03-05T16:21:16.137304Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>IncidentNumber</th>\n",
       "      <th>DateOfCall</th>\n",
       "      <th>CalYear</th>\n",
       "      <th>TimeOfCall</th>\n",
       "      <th>HourOfCall</th>\n",
       "      <th>IncidentGroup</th>\n",
       "      <th>StopCodeDescription</th>\n",
       "      <th>SpecialServiceType</th>\n",
       "      <th>PropertyCategory</th>\n",
       "      <th>PropertyType</th>\n",
       "      <th>...</th>\n",
       "      <th>IncidentStationGround</th>\n",
       "      <th>FirstPumpArriving_AttendanceTime</th>\n",
       "      <th>FirstPumpArriving_DeployedFromStation</th>\n",
       "      <th>SecondPumpArriving_AttendanceTime</th>\n",
       "      <th>SecondPumpArriving_DeployedFromStation</th>\n",
       "      <th>NumStationsWithPumpsAttending</th>\n",
       "      <th>NumPumpsAttending</th>\n",
       "      <th>PumpCount</th>\n",
       "      <th>PumpHoursRoundUp</th>\n",
       "      <th>Notional Cost (£)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>413008</td>\n",
       "      <td>413008</td>\n",
       "      <td>413008.000000</td>\n",
       "      <td>413008</td>\n",
       "      <td>413008.000000</td>\n",
       "      <td>413008</td>\n",
       "      <td>413008</td>\n",
       "      <td>129372</td>\n",
       "      <td>413008</td>\n",
       "      <td>413008</td>\n",
       "      <td>...</td>\n",
       "      <td>413008</td>\n",
       "      <td>389188.000000</td>\n",
       "      <td>389181</td>\n",
       "      <td>157869.000000</td>\n",
       "      <td>157865</td>\n",
       "      <td>410795.000000</td>\n",
       "      <td>410795.000000</td>\n",
       "      <td>411209.000000</td>\n",
       "      <td>411085.000000</td>\n",
       "      <td>411085.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>413008</td>\n",
       "      <td>1461</td>\n",
       "      <td>NaN</td>\n",
       "      <td>83348</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3</td>\n",
       "      <td>10</td>\n",
       "      <td>21</td>\n",
       "      <td>9</td>\n",
       "      <td>284</td>\n",
       "      <td>...</td>\n",
       "      <td>102</td>\n",
       "      <td>NaN</td>\n",
       "      <td>105</td>\n",
       "      <td>NaN</td>\n",
       "      <td>102</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>008718-21012017</td>\n",
       "      <td>03/03/2018</td>\n",
       "      <td>NaN</td>\n",
       "      <td>18:38:33</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False Alarm</td>\n",
       "      <td>AFA</td>\n",
       "      <td>Flooding</td>\n",
       "      <td>Dwelling</td>\n",
       "      <td>Purpose Built Flats/Maisonettes - 4 to 9 storeys</td>\n",
       "      <td>...</td>\n",
       "      <td>Soho</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Soho</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Soho</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>1</td>\n",
       "      <td>835</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19</td>\n",
       "      <td>NaN</td>\n",
       "      <td>208732</td>\n",
       "      <td>158276</td>\n",
       "      <td>28246</td>\n",
       "      <td>207191</td>\n",
       "      <td>53870</td>\n",
       "      <td>...</td>\n",
       "      <td>15037</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14451</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4442</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2018.480664</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13.476175</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>310.483193</td>\n",
       "      <td>NaN</td>\n",
       "      <td>388.523554</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.383006</td>\n",
       "      <td>1.565881</td>\n",
       "      <td>1.619994</td>\n",
       "      <td>1.374375</td>\n",
       "      <td>460.746160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.108126</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.244883</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>132.087535</td>\n",
       "      <td>NaN</td>\n",
       "      <td>147.585822</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.719757</td>\n",
       "      <td>0.873267</td>\n",
       "      <td>1.661056</td>\n",
       "      <td>6.942830</td>\n",
       "      <td>2312.690899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2017.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>326.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2017.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>228.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>294.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>328.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2018.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>292.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>364.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>339.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2019.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>369.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>452.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>346.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2020.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1200.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1200.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>319.000000</td>\n",
       "      <td>2163.000000</td>\n",
       "      <td>709464.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>11 rows × 38 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         IncidentNumber  DateOfCall        CalYear TimeOfCall     HourOfCall  \\\n",
       "count            413008      413008  413008.000000     413008  413008.000000   \n",
       "unique           413008        1461            NaN      83348            NaN   \n",
       "top     008718-21012017  03/03/2018            NaN   18:38:33            NaN   \n",
       "freq                  1         835            NaN         19            NaN   \n",
       "mean                NaN         NaN    2018.480664        NaN      13.476175   \n",
       "std                 NaN         NaN       1.108126        NaN       6.244883   \n",
       "min                 NaN         NaN    2017.000000        NaN       0.000000   \n",
       "25%                 NaN         NaN    2017.000000        NaN       9.000000   \n",
       "50%                 NaN         NaN    2018.000000        NaN      14.000000   \n",
       "75%                 NaN         NaN    2019.000000        NaN      19.000000   \n",
       "max                 NaN         NaN    2020.000000        NaN      23.000000   \n",
       "\n",
       "       IncidentGroup StopCodeDescription SpecialServiceType PropertyCategory  \\\n",
       "count         413008              413008             129372           413008   \n",
       "unique             3                  10                 21                9   \n",
       "top      False Alarm                 AFA           Flooding         Dwelling   \n",
       "freq          208732              158276              28246           207191   \n",
       "mean             NaN                 NaN                NaN              NaN   \n",
       "std              NaN                 NaN                NaN              NaN   \n",
       "min              NaN                 NaN                NaN              NaN   \n",
       "25%              NaN                 NaN                NaN              NaN   \n",
       "50%              NaN                 NaN                NaN              NaN   \n",
       "75%              NaN                 NaN                NaN              NaN   \n",
       "max              NaN                 NaN                NaN              NaN   \n",
       "\n",
       "                                             PropertyType  ...  \\\n",
       "count                                              413008  ...   \n",
       "unique                                                284  ...   \n",
       "top     Purpose Built Flats/Maisonettes - 4 to 9 storeys   ...   \n",
       "freq                                                53870  ...   \n",
       "mean                                                  NaN  ...   \n",
       "std                                                   NaN  ...   \n",
       "min                                                   NaN  ...   \n",
       "25%                                                   NaN  ...   \n",
       "50%                                                   NaN  ...   \n",
       "75%                                                   NaN  ...   \n",
       "max                                                   NaN  ...   \n",
       "\n",
       "       IncidentStationGround FirstPumpArriving_AttendanceTime  \\\n",
       "count                 413008                    389188.000000   \n",
       "unique                   102                              NaN   \n",
       "top                     Soho                              NaN   \n",
       "freq                   15037                              NaN   \n",
       "mean                     NaN                       310.483193   \n",
       "std                      NaN                       132.087535   \n",
       "min                      NaN                         1.000000   \n",
       "25%                      NaN                       228.000000   \n",
       "50%                      NaN                       292.000000   \n",
       "75%                      NaN                       369.000000   \n",
       "max                      NaN                      1200.000000   \n",
       "\n",
       "       FirstPumpArriving_DeployedFromStation  \\\n",
       "count                                 389181   \n",
       "unique                                   105   \n",
       "top                                     Soho   \n",
       "freq                                   14451   \n",
       "mean                                     NaN   \n",
       "std                                      NaN   \n",
       "min                                      NaN   \n",
       "25%                                      NaN   \n",
       "50%                                      NaN   \n",
       "75%                                      NaN   \n",
       "max                                      NaN   \n",
       "\n",
       "        SecondPumpArriving_AttendanceTime  \\\n",
       "count                       157869.000000   \n",
       "unique                                NaN   \n",
       "top                                   NaN   \n",
       "freq                                  NaN   \n",
       "mean                           388.523554   \n",
       "std                            147.585822   \n",
       "min                              1.000000   \n",
       "25%                            294.000000   \n",
       "50%                            364.000000   \n",
       "75%                            452.000000   \n",
       "max                           1200.000000   \n",
       "\n",
       "        SecondPumpArriving_DeployedFromStation NumStationsWithPumpsAttending  \\\n",
       "count                                   157865                 410795.000000   \n",
       "unique                                     102                           NaN   \n",
       "top                                       Soho                           NaN   \n",
       "freq                                      4442                           NaN   \n",
       "mean                                       NaN                      1.383006   \n",
       "std                                        NaN                      0.719757   \n",
       "min                                        NaN                      1.000000   \n",
       "25%                                        NaN                      1.000000   \n",
       "50%                                        NaN                      1.000000   \n",
       "75%                                        NaN                      2.000000   \n",
       "max                                        NaN                     11.000000   \n",
       "\n",
       "       NumPumpsAttending      PumpCount PumpHoursRoundUp Notional Cost (£)  \n",
       "count      410795.000000  411209.000000    411085.000000     411085.000000  \n",
       "unique               NaN            NaN              NaN               NaN  \n",
       "top                  NaN            NaN              NaN               NaN  \n",
       "freq                 NaN            NaN              NaN               NaN  \n",
       "mean            1.565881       1.619994         1.374375        460.746160  \n",
       "std             0.873267       1.661056         6.942830       2312.690899  \n",
       "min             1.000000       1.000000         1.000000        326.000000  \n",
       "25%             1.000000       1.000000         1.000000        328.000000  \n",
       "50%             1.000000       1.000000         1.000000        339.000000  \n",
       "75%             2.000000       2.000000         1.000000        346.000000  \n",
       "max            14.000000     319.000000      2163.000000     709464.000000  \n",
       "\n",
       "[11 rows x 38 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#LFB database for inspection (most recent csv file - 2017-2020)\n",
    "LFB2017df = pd.read_csv('LFBIncident2017.csv', engine='python')\n",
    "LFB2017df.describe(include='all')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:21:33.927288Z",
     "start_time": "2021-03-05T16:21:33.237684Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 413008 entries, 0 to 413007\n",
      "Data columns (total 38 columns):\n",
      " #   Column                                  Non-Null Count   Dtype  \n",
      "---  ------                                  --------------   -----  \n",
      " 0   IncidentNumber                          413008 non-null  object \n",
      " 1   DateOfCall                              413008 non-null  object \n",
      " 2   CalYear                                 413008 non-null  int64  \n",
      " 3   TimeOfCall                              413008 non-null  object \n",
      " 4   HourOfCall                              413008 non-null  int64  \n",
      " 5   IncidentGroup                           413008 non-null  object \n",
      " 6   StopCodeDescription                     413008 non-null  object \n",
      " 7   SpecialServiceType                      129372 non-null  object \n",
      " 8   PropertyCategory                        413008 non-null  object \n",
      " 9   PropertyType                            413008 non-null  object \n",
      " 10  AddressQualifier                        413008 non-null  object \n",
      " 11  Postcode_full                           203965 non-null  object \n",
      " 12  Postcode_district                       413008 non-null  object \n",
      " 13  UPRN                                    413008 non-null  int64  \n",
      " 14  USRN                                    413008 non-null  int64  \n",
      " 15  IncGeo_BoroughCode                      413008 non-null  object \n",
      " 16  IncGeo_BoroughName                      413008 non-null  object \n",
      " 17  ProperCase                              413008 non-null  object \n",
      " 18  IncGeo_WardCode                         413008 non-null  object \n",
      " 19  IncGeo_WardName                         413008 non-null  object \n",
      " 20  IncGeo_WardNameNew                      413008 non-null  object \n",
      " 21  Easting_m                               203965 non-null  float64\n",
      " 22  Northing_m                              203965 non-null  float64\n",
      " 23  Easting_rounded                         413008 non-null  int64  \n",
      " 24  Northing_rounded                        413008 non-null  int64  \n",
      " 25  Latitude                                203965 non-null  float64\n",
      " 26  Longitude                               203965 non-null  float64\n",
      " 27  FRS                                     413008 non-null  object \n",
      " 28  IncidentStationGround                   413008 non-null  object \n",
      " 29  FirstPumpArriving_AttendanceTime        389188 non-null  float64\n",
      " 30  FirstPumpArriving_DeployedFromStation   389181 non-null  object \n",
      " 31  SecondPumpArriving_AttendanceTime       157869 non-null  float64\n",
      " 32  SecondPumpArriving_DeployedFromStation  157865 non-null  object \n",
      " 33  NumStationsWithPumpsAttending           410795 non-null  float64\n",
      " 34  NumPumpsAttending                       410795 non-null  float64\n",
      " 35  PumpCount                               411209 non-null  float64\n",
      " 36  PumpHoursRoundUp                        411085 non-null  float64\n",
      " 37  Notional Cost (£)                       411085 non-null  float64\n",
      "dtypes: float64(11), int64(6), object(21)\n",
      "memory usage: 119.7+ MB\n"
     ]
    }
   ],
   "source": [
    "LFB2017df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:21:56.643211Z",
     "start_time": "2021-03-05T16:21:56.600240Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>IncidentNumber</th>\n",
       "      <th>DateOfCall</th>\n",
       "      <th>CalYear</th>\n",
       "      <th>TimeOfCall</th>\n",
       "      <th>HourOfCall</th>\n",
       "      <th>IncidentGroup</th>\n",
       "      <th>StopCodeDescription</th>\n",
       "      <th>SpecialServiceType</th>\n",
       "      <th>PropertyCategory</th>\n",
       "      <th>PropertyType</th>\n",
       "      <th>...</th>\n",
       "      <th>IncidentStationGround</th>\n",
       "      <th>FirstPumpArriving_AttendanceTime</th>\n",
       "      <th>FirstPumpArriving_DeployedFromStation</th>\n",
       "      <th>SecondPumpArriving_AttendanceTime</th>\n",
       "      <th>SecondPumpArriving_DeployedFromStation</th>\n",
       "      <th>NumStationsWithPumpsAttending</th>\n",
       "      <th>NumPumpsAttending</th>\n",
       "      <th>PumpCount</th>\n",
       "      <th>PumpHoursRoundUp</th>\n",
       "      <th>Notional Cost (£)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>000226-01012017</td>\n",
       "      <td>01/01/2017</td>\n",
       "      <td>2017</td>\n",
       "      <td>10:15:32</td>\n",
       "      <td>10</td>\n",
       "      <td>Special Service</td>\n",
       "      <td>Special Service</td>\n",
       "      <td>Flooding</td>\n",
       "      <td>Dwelling</td>\n",
       "      <td>Purpose Built Flats/Maisonettes - 10 or more s...</td>\n",
       "      <td>...</td>\n",
       "      <td>Woodford</td>\n",
       "      <td>231.0</td>\n",
       "      <td>Woodford</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>326.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>000252-01012017</td>\n",
       "      <td>01/01/2017</td>\n",
       "      <td>2017</td>\n",
       "      <td>11:58:44</td>\n",
       "      <td>11</td>\n",
       "      <td>Special Service</td>\n",
       "      <td>Special Service</td>\n",
       "      <td>Lift Release</td>\n",
       "      <td>Dwelling</td>\n",
       "      <td>Purpose Built Flats/Maisonettes - 4 to 9 storeys</td>\n",
       "      <td>...</td>\n",
       "      <td>Stratford</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>652.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>000127-01012017</td>\n",
       "      <td>01/01/2017</td>\n",
       "      <td>2017</td>\n",
       "      <td>03:57:50</td>\n",
       "      <td>3</td>\n",
       "      <td>False Alarm</td>\n",
       "      <td>AFA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Dwelling</td>\n",
       "      <td>Purpose Built Flats/Maisonettes - 4 to 9 storeys</td>\n",
       "      <td>...</td>\n",
       "      <td>West Norwood</td>\n",
       "      <td>206.0</td>\n",
       "      <td>West Norwood</td>\n",
       "      <td>230.0</td>\n",
       "      <td>West Norwood</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>326.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>000378-01012017</td>\n",
       "      <td>01/01/2017</td>\n",
       "      <td>2017</td>\n",
       "      <td>19:05:55</td>\n",
       "      <td>19</td>\n",
       "      <td>Fire</td>\n",
       "      <td>Primary Fire</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Dwelling</td>\n",
       "      <td>Purpose Built Flats/Maisonettes - 4 to 9 storeys</td>\n",
       "      <td>...</td>\n",
       "      <td>Feltham</td>\n",
       "      <td>258.0</td>\n",
       "      <td>Feltham</td>\n",
       "      <td>279.0</td>\n",
       "      <td>Feltham</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>3586.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>000289-01012017</td>\n",
       "      <td>01/01/2017</td>\n",
       "      <td>2017</td>\n",
       "      <td>13:37:52</td>\n",
       "      <td>13</td>\n",
       "      <td>False Alarm</td>\n",
       "      <td>AFA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Dwelling</td>\n",
       "      <td>Self contained Sheltered Housing</td>\n",
       "      <td>...</td>\n",
       "      <td>Surbiton</td>\n",
       "      <td>490.0</td>\n",
       "      <td>Kingston</td>\n",
       "      <td>579.0</td>\n",
       "      <td>Kingston</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>326.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 38 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    IncidentNumber  DateOfCall  CalYear TimeOfCall  HourOfCall  \\\n",
       "0  000226-01012017  01/01/2017     2017   10:15:32          10   \n",
       "1  000252-01012017  01/01/2017     2017   11:58:44          11   \n",
       "2  000127-01012017  01/01/2017     2017   03:57:50           3   \n",
       "3  000378-01012017  01/01/2017     2017   19:05:55          19   \n",
       "4  000289-01012017  01/01/2017     2017   13:37:52          13   \n",
       "\n",
       "     IncidentGroup StopCodeDescription SpecialServiceType PropertyCategory  \\\n",
       "0  Special Service     Special Service           Flooding         Dwelling   \n",
       "1  Special Service     Special Service       Lift Release         Dwelling   \n",
       "2      False Alarm                 AFA                NaN         Dwelling   \n",
       "3             Fire        Primary Fire                NaN         Dwelling   \n",
       "4      False Alarm                 AFA                NaN         Dwelling   \n",
       "\n",
       "                                        PropertyType  ...  \\\n",
       "0  Purpose Built Flats/Maisonettes - 10 or more s...  ...   \n",
       "1  Purpose Built Flats/Maisonettes - 4 to 9 storeys   ...   \n",
       "2  Purpose Built Flats/Maisonettes - 4 to 9 storeys   ...   \n",
       "3  Purpose Built Flats/Maisonettes - 4 to 9 storeys   ...   \n",
       "4                  Self contained Sheltered Housing   ...   \n",
       "\n",
       "  IncidentStationGround FirstPumpArriving_AttendanceTime  \\\n",
       "0              Woodford                            231.0   \n",
       "1             Stratford                              NaN   \n",
       "2          West Norwood                            206.0   \n",
       "3               Feltham                            258.0   \n",
       "4              Surbiton                            490.0   \n",
       "\n",
       "  FirstPumpArriving_DeployedFromStation  SecondPumpArriving_AttendanceTime  \\\n",
       "0                              Woodford                                NaN   \n",
       "1                                   NaN                                NaN   \n",
       "2                          West Norwood                              230.0   \n",
       "3                               Feltham                              279.0   \n",
       "4                              Kingston                              579.0   \n",
       "\n",
       "   SecondPumpArriving_DeployedFromStation NumStationsWithPumpsAttending  \\\n",
       "0                                     NaN                           1.0   \n",
       "1                                     NaN                           1.0   \n",
       "2                            West Norwood                           1.0   \n",
       "3                                 Feltham                           1.0   \n",
       "4                                Kingston                           1.0   \n",
       "\n",
       "  NumPumpsAttending PumpCount PumpHoursRoundUp Notional Cost (£)  \n",
       "0               1.0       2.0              1.0             326.0  \n",
       "1               1.0       1.0              2.0             652.0  \n",
       "2               2.0       2.0              1.0             326.0  \n",
       "3               2.0       5.0             11.0            3586.0  \n",
       "4               2.0       2.0              1.0             326.0  \n",
       "\n",
       "[5 rows x 38 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LFB2017df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:22:14.914033Z",
     "start_time": "2021-03-05T16:22:03.276037Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 403985 entries, 0 to 403984\n",
      "Data columns (total 31 columns):\n",
      " #   Column                                  Non-Null Count   Dtype  \n",
      "---  ------                                  --------------   -----  \n",
      " 0   IncidentNumber                          403985 non-null  object \n",
      " 1   DateOfCall                              403985 non-null  object \n",
      " 2   CalYear                                 403985 non-null  int64  \n",
      " 3   TimeOfCall                              403985 non-null  object \n",
      " 4   HourOfCall                              403985 non-null  int64  \n",
      " 5   IncidentGroup                           403984 non-null  object \n",
      " 6   StopCodeDescription                     403984 non-null  object \n",
      " 7   SpecialServiceType                      123086 non-null  object \n",
      " 8   PropertyCategory                        403984 non-null  object \n",
      " 9   PropertyType                            403984 non-null  object \n",
      " 10  AddressQualifier                        403985 non-null  object \n",
      " 11  Postcode_full                           215034 non-null  object \n",
      " 12  Postcode_district                       403985 non-null  object \n",
      " 13  IncGeo_BoroughCode                      403985 non-null  object \n",
      " 14  IncGeo_BoroughName                      403985 non-null  object \n",
      " 15  ProperCase                              403985 non-null  object \n",
      " 16  IncGeo_WardCode                         403985 non-null  object \n",
      " 17  IncGeo_WardName                         403985 non-null  object \n",
      " 18  IncGeo_WardNameNew                      403985 non-null  object \n",
      " 19  Easting_m                               215034 non-null  float64\n",
      " 20  Northing_m                              215034 non-null  float64\n",
      " 21  Easting_rounded                         403985 non-null  int64  \n",
      " 22  Northing_rounded                        403985 non-null  int64  \n",
      " 23  FRS                                     403985 non-null  object \n",
      " 24  IncidentStationGround                   403985 non-null  object \n",
      " 25  FirstPumpArriving_AttendanceTime        369878 non-null  float64\n",
      " 26  FirstPumpArriving_DeployedFromStation   369877 non-null  object \n",
      " 27  SecondPumpArriving_AttendanceTime       5 non-null       float64\n",
      " 28  SecondPumpArriving_DeployedFromStation  5 non-null       object \n",
      " 29  NumStationsWithPumpsAttending           402015 non-null  float64\n",
      " 30  NumPumpsAttending                       402015 non-null  float64\n",
      "dtypes: float64(6), int64(4), object(21)\n",
      "memory usage: 95.5+ MB\n"
     ]
    }
   ],
   "source": [
    "#LFB database for inspection (next csv file - 2013-2016)\n",
    "LFB2016df = pd.read_csv('LFBIncident2016.csv', engine='python')\n",
    "LFB2016df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:22:19.037748Z",
     "start_time": "2021-03-05T16:22:18.998792Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>IncidentNumber</th>\n",
       "      <th>DateOfCall</th>\n",
       "      <th>CalYear</th>\n",
       "      <th>TimeOfCall</th>\n",
       "      <th>HourOfCall</th>\n",
       "      <th>IncidentGroup</th>\n",
       "      <th>StopCodeDescription</th>\n",
       "      <th>SpecialServiceType</th>\n",
       "      <th>PropertyCategory</th>\n",
       "      <th>PropertyType</th>\n",
       "      <th>...</th>\n",
       "      <th>Easting_rounded</th>\n",
       "      <th>Northing_rounded</th>\n",
       "      <th>FRS</th>\n",
       "      <th>IncidentStationGround</th>\n",
       "      <th>FirstPumpArriving_AttendanceTime</th>\n",
       "      <th>FirstPumpArriving_DeployedFromStation</th>\n",
       "      <th>SecondPumpArriving_AttendanceTime</th>\n",
       "      <th>SecondPumpArriving_DeployedFromStation</th>\n",
       "      <th>NumStationsWithPumpsAttending</th>\n",
       "      <th>NumPumpsAttending</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1131</td>\n",
       "      <td>01-Jan-13</td>\n",
       "      <td>2013</td>\n",
       "      <td>00:02:06</td>\n",
       "      <td>0</td>\n",
       "      <td>False Alarm</td>\n",
       "      <td>AFA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Other Residential</td>\n",
       "      <td>Boarding House/B&amp;B for homeless/asylum seekers</td>\n",
       "      <td>...</td>\n",
       "      <td>525450</td>\n",
       "      <td>184850</td>\n",
       "      <td>London</td>\n",
       "      <td>West Hampstead</td>\n",
       "      <td>167.0</td>\n",
       "      <td>West Hampstead</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4131</td>\n",
       "      <td>01-Jan-13</td>\n",
       "      <td>2013</td>\n",
       "      <td>00:02:09</td>\n",
       "      <td>0</td>\n",
       "      <td>False Alarm</td>\n",
       "      <td>AFA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Non Residential</td>\n",
       "      <td>Single shop</td>\n",
       "      <td>...</td>\n",
       "      <td>515450</td>\n",
       "      <td>185450</td>\n",
       "      <td>London</td>\n",
       "      <td>Northolt</td>\n",
       "      <td>236.0</td>\n",
       "      <td>Northolt</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5131</td>\n",
       "      <td>01-Jan-13</td>\n",
       "      <td>2013</td>\n",
       "      <td>00:02:54</td>\n",
       "      <td>0</td>\n",
       "      <td>False Alarm</td>\n",
       "      <td>AFA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Non Residential</td>\n",
       "      <td>Other cultural venue</td>\n",
       "      <td>...</td>\n",
       "      <td>522450</td>\n",
       "      <td>178650</td>\n",
       "      <td>London</td>\n",
       "      <td>Hammersmith</td>\n",
       "      <td>218.0</td>\n",
       "      <td>Hammersmith</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2131</td>\n",
       "      <td>01-Jan-13</td>\n",
       "      <td>2013</td>\n",
       "      <td>00:03:02</td>\n",
       "      <td>0</td>\n",
       "      <td>Fire</td>\n",
       "      <td>Secondary Fire</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Outdoor Structure</td>\n",
       "      <td>Small refuse/rubbish container</td>\n",
       "      <td>...</td>\n",
       "      <td>527850</td>\n",
       "      <td>181050</td>\n",
       "      <td>London</td>\n",
       "      <td>Soho</td>\n",
       "      <td>426.0</td>\n",
       "      <td>Knightsbridge</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3131</td>\n",
       "      <td>01-Jan-13</td>\n",
       "      <td>2013</td>\n",
       "      <td>00:03:03</td>\n",
       "      <td>0</td>\n",
       "      <td>False Alarm</td>\n",
       "      <td>AFA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Non Residential</td>\n",
       "      <td>Purpose built office</td>\n",
       "      <td>...</td>\n",
       "      <td>533350</td>\n",
       "      <td>180750</td>\n",
       "      <td>London</td>\n",
       "      <td>Whitechapel</td>\n",
       "      <td>346.0</td>\n",
       "      <td>Dowgate</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  IncidentNumber DateOfCall  CalYear TimeOfCall  HourOfCall IncidentGroup  \\\n",
       "0           1131  01-Jan-13     2013   00:02:06           0   False Alarm   \n",
       "1           4131  01-Jan-13     2013   00:02:09           0   False Alarm   \n",
       "2           5131  01-Jan-13     2013   00:02:54           0   False Alarm   \n",
       "3           2131  01-Jan-13     2013   00:03:02           0          Fire   \n",
       "4           3131  01-Jan-13     2013   00:03:03           0   False Alarm   \n",
       "\n",
       "  StopCodeDescription SpecialServiceType   PropertyCategory  \\\n",
       "0                 AFA                NaN  Other Residential   \n",
       "1                 AFA                NaN    Non Residential   \n",
       "2                 AFA                NaN    Non Residential   \n",
       "3      Secondary Fire                NaN  Outdoor Structure   \n",
       "4                 AFA                NaN    Non Residential   \n",
       "\n",
       "                                      PropertyType  ... Easting_rounded  \\\n",
       "0  Boarding House/B&B for homeless/asylum seekers   ...          525450   \n",
       "1                                     Single shop   ...          515450   \n",
       "2                            Other cultural venue   ...          522450   \n",
       "3                   Small refuse/rubbish container  ...          527850   \n",
       "4                            Purpose built office   ...          533350   \n",
       "\n",
       "  Northing_rounded     FRS IncidentStationGround  \\\n",
       "0           184850  London        West Hampstead   \n",
       "1           185450  London              Northolt   \n",
       "2           178650  London           Hammersmith   \n",
       "3           181050  London                  Soho   \n",
       "4           180750  London           Whitechapel   \n",
       "\n",
       "  FirstPumpArriving_AttendanceTime FirstPumpArriving_DeployedFromStation  \\\n",
       "0                            167.0                        West Hampstead   \n",
       "1                            236.0                              Northolt   \n",
       "2                            218.0                           Hammersmith   \n",
       "3                            426.0                         Knightsbridge   \n",
       "4                            346.0                               Dowgate   \n",
       "\n",
       "  SecondPumpArriving_AttendanceTime SecondPumpArriving_DeployedFromStation  \\\n",
       "0                               NaN                                    NaN   \n",
       "1                               NaN                                    NaN   \n",
       "2                               NaN                                    NaN   \n",
       "3                               NaN                                    NaN   \n",
       "4                               NaN                                    NaN   \n",
       "\n",
       "  NumStationsWithPumpsAttending  NumPumpsAttending  \n",
       "0                           1.0                1.0  \n",
       "1                           1.0                1.0  \n",
       "2                           1.0                1.0  \n",
       "3                           1.0                1.0  \n",
       "4                           2.0                2.0  \n",
       "\n",
       "[5 rows x 31 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LFB2016df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:24:40.302167Z",
     "start_time": "2021-03-05T16:24:38.381628Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-9-41d326f2b3da>:11: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  LFB2017df1['Datetime'] = pd.to_datetime(LFB2017df1.DateOfCall+' '+LFB2017df1.HourOfCall.astype(str), format='%d/%m/%Y %H')\n",
      "<ipython-input-9-41d326f2b3da>:21: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  LFB2016df1['Datetime'] = pd.to_datetime(LFB2016df1.DateOfCall+' '+LFB2016df1.HourOfCall.astype(str), format='%d-%b-%y %H')\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 616797 entries, 0 to 616796\n",
      "Data columns (total 8 columns):\n",
      " #   Column                 Non-Null Count   Dtype         \n",
      "---  ------                 --------------   -----         \n",
      " 0   DateOfCall             616797 non-null  object        \n",
      " 1   CalYear                616797 non-null  int64         \n",
      " 2   HourOfCall             616797 non-null  int64         \n",
      " 3   IncidentGroup          616796 non-null  object        \n",
      " 4   PropertyType           616796 non-null  object        \n",
      " 5   SpecialServiceType     192141 non-null  object        \n",
      " 6   IncidentStationGround  616797 non-null  object        \n",
      " 7   Datetime               616797 non-null  datetime64[ns]\n",
      "dtypes: datetime64[ns](1), int64(2), object(5)\n",
      "memory usage: 37.6+ MB\n"
     ]
    }
   ],
   "source": [
    "#Keeping only the relevant columns from both databases (two years from earlier csv) and combining.\n",
    "\n",
    "LFB2017df1 = LFB2017df[['DateOfCall',\n",
    " 'CalYear',\n",
    " 'HourOfCall',\n",
    " 'IncidentGroup',\n",
    " 'PropertyType',\n",
    " 'SpecialServiceType',\n",
    " 'IncidentStationGround']]\n",
    "\n",
    "LFB2017df1['Datetime'] = pd.to_datetime(LFB2017df1.DateOfCall+' '+LFB2017df1.HourOfCall.astype(str), format='%d/%m/%Y %H')\n",
    "\n",
    "LFB2016df1 = LFB2016df[['DateOfCall',\n",
    " 'CalYear',\n",
    " 'HourOfCall',\n",
    " 'IncidentGroup',\n",
    " 'PropertyType',\n",
    " 'SpecialServiceType',\n",
    " 'IncidentStationGround']]\n",
    "\n",
    "LFB2016df1['Datetime'] = pd.to_datetime(LFB2016df1.DateOfCall+' '+LFB2016df1.HourOfCall.astype(str), format='%d-%b-%y %H')\n",
    "LFB = pd.concat([LFB2017df1, LFB2016df1[(LFB2016df1['CalYear'] == 2015)|(LFB2016df1['CalYear'] == 2016)]], ignore_index=True)\n",
    "LFB.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:24:54.373180Z",
     "start_time": "2021-03-05T16:24:54.231264Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "286\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Purpose Built Flats/Maisonettes - 4 to 9 storeys         76862\n",
       "House - single occupancy                                 71512\n",
       "Purpose Built Flats/Maisonettes - Up to 3 storeys        57765\n",
       "Car                                                      29738\n",
       "Self contained Sheltered Housing                         28640\n",
       "Purpose built office                                     22444\n",
       "Converted Flat/Maisonettes - 3 or more storeys           22301\n",
       "Purpose Built Flats/Maisonettes - 10 or more storeys     19479\n",
       "Converted Flat/Maisonette - Up to 2 storeys              18023\n",
       "Single shop                                              15469\n",
       "Small refuse/rubbish container                           15093\n",
       "Retirement/Old Persons Home                              12535\n",
       "Domestic garden (vegetation not equipment)               10000\n",
       "Road surface/pavement                                     9971\n",
       "Loose refuse                                              8893\n",
       "Multiple Vehicles                                         8588\n",
       "Infant/Primary school                                     8086\n",
       "Hospital                                                  7664\n",
       "Restaurant/cafe                                           5990\n",
       "Secondary school                                          5476\n",
       "Nursing/Care Home/Hospice                                 5377\n",
       "Hostel (e.g. for homeless people)                         5232\n",
       "Tree scrub                                                5138\n",
       "Large refuse/rubbish container (eg skip/ paladin)         4803\n",
       "Hotel/motel                                               4628\n",
       "Motorcycle                                                4548\n",
       "Large supermarket                                         4522\n",
       "Warehouse                                                 4062\n",
       "Park                                                      3865\n",
       "College/University                                        3771\n",
       "Name: PropertyType, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#A closer look at the propertytype column - will be used for classification modelling\n",
    "print(LFB.PropertyType.nunique())\n",
    "LFB.PropertyType.value_counts()[:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:25:08.205691Z",
     "start_time": "2021-03-05T16:25:08.002521Z"
    }
   },
   "outputs": [],
   "source": [
    "# holiday marker\n",
    "import holidays\n",
    "hols = []\n",
    "for ptr in holidays.England(years = [2016,2017,2018,2019,2020]).items(): \n",
    "    hols.append(ptr[0]) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:25:09.761421Z",
     "start_time": "2021-03-05T16:25:09.750428Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[datetime.date(2016, 1, 1), datetime.date(2016, 3, 25), datetime.date(2016, 3, 28), datetime.date(2016, 5, 2), datetime.date(2016, 5, 30), datetime.date(2016, 8, 29), datetime.date(2016, 12, 25), datetime.date(2016, 12, 27), datetime.date(2016, 12, 26), datetime.date(2017, 1, 1), datetime.date(2017, 1, 2), datetime.date(2017, 4, 14), datetime.date(2017, 4, 17), datetime.date(2017, 5, 1), datetime.date(2017, 5, 29), datetime.date(2017, 8, 28), datetime.date(2017, 12, 25), datetime.date(2017, 12, 26), datetime.date(2018, 1, 1), datetime.date(2018, 3, 30), datetime.date(2018, 4, 2), datetime.date(2018, 5, 7), datetime.date(2018, 5, 28), datetime.date(2018, 8, 27), datetime.date(2018, 12, 25), datetime.date(2018, 12, 26), datetime.date(2019, 1, 1), datetime.date(2019, 4, 19), datetime.date(2019, 4, 22), datetime.date(2019, 5, 6), datetime.date(2019, 5, 27), datetime.date(2019, 8, 26), datetime.date(2019, 12, 25), datetime.date(2019, 12, 26), datetime.date(2020, 1, 1), datetime.date(2020, 4, 10), datetime.date(2020, 4, 13), datetime.date(2020, 5, 8), datetime.date(2020, 5, 25), datetime.date(2020, 8, 31), datetime.date(2020, 12, 25), datetime.date(2020, 12, 26), datetime.date(2020, 12, 28)]\n"
     ]
    }
   ],
   "source": [
    "print(hols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:25:15.539000Z",
     "start_time": "2021-03-05T16:25:13.034421Z"
    }
   },
   "outputs": [],
   "source": [
    "LFB['holiday'] = 0\n",
    "LFB['holiday'] = [1 if i in hols else 0 for i in LFB.Datetime]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:25:16.088407Z",
     "start_time": "2021-03-05T16:25:15.540722Z"
    }
   },
   "outputs": [],
   "source": [
    "# Weekend marker\n",
    "LFB['weekend'] = 0\n",
    "LFB['weekend'] = [1 if i in [5,6] else 0 for i in LFB.Datetime.dt.weekday]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:25:16.104399Z",
     "start_time": "2021-03-05T16:25:16.092405Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "179392"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LFB.weekend.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:26:12.329628Z",
     "start_time": "2021-03-05T16:25:19.676814Z"
    }
   },
   "outputs": [],
   "source": [
    "# lockdown marker\n",
    "series = pd.Series(['2020/03/23', '2020/06/15', '2020/11/05', '2020/12/02','2021/01/05'])\n",
    "lockdates = pd.to_datetime(series)\n",
    "\n",
    "LFB['lockdown'] = 0\n",
    "LFB['lockdown'] = [1 if (((i >= lockdates[0])&(i < lockdates[1]))|((i >= lockdates[2]) \n",
    "                                    & (i < lockdates[3]))|((i >= lockdates[4]) )) else 0 for i in LFB.Datetime]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:26:15.970685Z",
     "start_time": "2021-03-05T16:26:15.959693Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28220"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LFB.lockdown.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:26:19.003933Z",
     "start_time": "2021-03-05T16:26:18.796031Z"
    }
   },
   "outputs": [],
   "source": [
    "# Filling the special service null values.\n",
    "LFB = LFB.fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Met Office Data #"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2016 Weather ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:28:39.118530Z",
     "start_time": "2021-03-05T16:28:38.444902Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 8785 entries, 0 to 8784\n",
      "Columns: 104 entries, ob_time to drv_hr_sun_dur_q\n",
      "dtypes: float64(98), object(6)\n",
      "memory usage: 7.0+ MB\n"
     ]
    }
   ],
   "source": [
    "# Now for the weather - starting with the met office csv files (pre-stripped of some irrelevant header rows).\n",
    "Weath2016w = pd.read_csv('uk-hourly-weather-obs2016noleg.csv', engine='python')\n",
    "Weath2016w.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:28:39.927152Z",
     "start_time": "2021-03-05T16:28:39.909162Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ob_time',\n",
       " 'id',\n",
       " 'id_type',\n",
       " 'met_domain_name',\n",
       " 'version_num',\n",
       " 'src_id',\n",
       " 'rec_st_ind',\n",
       " 'wind_speed_unit_id',\n",
       " 'src_opr_type',\n",
       " 'wind_direction',\n",
       " 'wind_speed',\n",
       " 'prst_wx_id',\n",
       " 'past_wx_id_1',\n",
       " 'past_wx_id_2',\n",
       " 'cld_ttl_amt_id',\n",
       " 'low_cld_type_id',\n",
       " 'med_cld_type_id',\n",
       " 'hi_cld_type_id',\n",
       " 'cld_base_amt_id',\n",
       " 'cld_base_ht',\n",
       " 'visibility',\n",
       " 'msl_pressure',\n",
       " 'cld_amt_id_1',\n",
       " 'cloud_type_id_1',\n",
       " 'cld_base_ht_id_1',\n",
       " 'cld_amt_id_2',\n",
       " 'cloud_type_id_2',\n",
       " 'cld_base_ht_id_2',\n",
       " 'cld_amt_id_3',\n",
       " 'cloud_type_id_3',\n",
       " 'cld_base_ht_id_3',\n",
       " 'cld_amt_id_4',\n",
       " 'cloud_type_id_4',\n",
       " 'cld_base_ht_id_4',\n",
       " 'vert_vsby',\n",
       " 'air_temperature',\n",
       " 'dewpoint',\n",
       " 'wetb_temp',\n",
       " 'rltv_hum',\n",
       " 'stn_pres',\n",
       " 'alt_pres',\n",
       " 'ground_state_id',\n",
       " 'q10mnt_mxgst_spd',\n",
       " 'cavok_flag',\n",
       " 'cs_hr_sun_dur',\n",
       " 'wmo_hr_sun_dur',\n",
       " 'snow_depth',\n",
       " 'wind_direction_q',\n",
       " 'wind_speed_q',\n",
       " 'prst_wx_id_q',\n",
       " 'past_wx_id_1_q',\n",
       " 'past_wx_id_2_q',\n",
       " 'cld_ttl_amt_id_q',\n",
       " 'low_cld_type_id_q',\n",
       " 'med_cld_type_id_q',\n",
       " 'hi_cld_type_id_q',\n",
       " 'cld_base_amt_id_q',\n",
       " 'cld_base_ht_q',\n",
       " 'visibility_q',\n",
       " 'msl_pressure_q',\n",
       " 'air_temperature_q',\n",
       " 'dewpoint_q',\n",
       " 'wetb_temp_q',\n",
       " 'ground_state_id_q',\n",
       " 'cld_amt_id_1_q',\n",
       " 'cloud_type_id_1_q',\n",
       " 'cld_base_ht_id_1_q',\n",
       " 'cld_amt_id_2_q',\n",
       " 'cloud_type_id_2_q',\n",
       " 'cld_base_ht_id_2_q',\n",
       " 'cld_amt_id_3_q',\n",
       " 'cloud_type_id_3_q',\n",
       " 'cld_base_ht_id_3_q',\n",
       " 'cld_amt_id_4_q',\n",
       " 'cloud_type_id_4_q',\n",
       " 'cld_base_ht_id_4_q',\n",
       " 'vert_vsby_q',\n",
       " 'stn_pres_q',\n",
       " 'alt_pres_q',\n",
       " 'q10mnt_mxgst_spd_q',\n",
       " 'cs_hr_sun_dur_q',\n",
       " 'wmo_hr_sun_dur_q',\n",
       " 'snow_depth_q',\n",
       " 'wind_direction_j',\n",
       " 'wind_speed_j',\n",
       " 'prst_wx_id_j',\n",
       " 'past_wx_id_1_j',\n",
       " 'past_wx_id_2_j',\n",
       " 'cld_amt_id_j',\n",
       " 'cld_ht_j',\n",
       " 'visibility_j',\n",
       " 'msl_pressure_j',\n",
       " 'air_temperature_j',\n",
       " 'dewpoint_j',\n",
       " 'wetb_temp_j',\n",
       " 'rltv_hum_j',\n",
       " 'vert_vsby_j',\n",
       " 'stn_pres_j',\n",
       " 'alt_pres_j',\n",
       " 'q10mnt_mxgst_spd_j',\n",
       " 'meto_stmp_time',\n",
       " 'midas_stmp_etime',\n",
       " 'drv_hr_sun_dur',\n",
       " 'drv_hr_sun_dur_q']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(Weath2016w.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:28:46.928964Z",
     "start_time": "2021-03-05T16:28:46.918972Z"
    }
   },
   "outputs": [],
   "source": [
    "Weath2016w1 = Weath2016w[['ob_time',           \n",
    "   'wind_direction',   \n",
    "    'wind_speed',       \n",
    "    'cld_ttl_amt_id',   \n",
    "    'visibility',       \n",
    "    'air_temperature',  \n",
    "    'rltv_hum',         \n",
    "    'stn_pres',         \n",
    "    'wmo_hr_sun_dur']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:28:48.864818Z",
     "start_time": "2021-03-05T16:28:48.851826Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 8785 entries, 0 to 8784\n",
      "Data columns (total 9 columns):\n",
      " #   Column           Non-Null Count  Dtype  \n",
      "---  ------           --------------  -----  \n",
      " 0   ob_time          8785 non-null   object \n",
      " 1   wind_direction   8775 non-null   float64\n",
      " 2   wind_speed       8775 non-null   float64\n",
      " 3   cld_ttl_amt_id   8781 non-null   float64\n",
      " 4   visibility       8781 non-null   float64\n",
      " 5   air_temperature  8782 non-null   float64\n",
      " 6   rltv_hum         8782 non-null   float64\n",
      " 7   stn_pres         8782 non-null   float64\n",
      " 8   wmo_hr_sun_dur   8414 non-null   float64\n",
      "dtypes: float64(8), object(1)\n",
      "memory usage: 617.8+ KB\n"
     ]
    }
   ],
   "source": [
    "Weath2016w1.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:28:52.518086Z",
     "start_time": "2021-03-05T16:28:52.499096Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>01/01/2016 00:00</td>\n",
       "      <td>240.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>3.8</td>\n",
       "      <td>89.8</td>\n",
       "      <td>1017.8</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>01/01/2016 01:00</td>\n",
       "      <td>230.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1400.0</td>\n",
       "      <td>3.7</td>\n",
       "      <td>91.5</td>\n",
       "      <td>1018.5</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>01/01/2016 02:00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>700.0</td>\n",
       "      <td>2.6</td>\n",
       "      <td>94.6</td>\n",
       "      <td>1018.8</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>01/01/2016 03:00</td>\n",
       "      <td>170.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>800.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>94.3</td>\n",
       "      <td>1019.1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>01/01/2016 04:00</td>\n",
       "      <td>110.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>700.0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>96.4</td>\n",
       "      <td>1019.6</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            ob_time  wind_direction  wind_speed  cld_ttl_amt_id  visibility  \\\n",
       "0  01/01/2016 00:00           240.0         6.0             0.0      2000.0   \n",
       "1  01/01/2016 01:00           230.0         5.0             0.0      1400.0   \n",
       "2  01/01/2016 02:00             0.0         0.0             0.0       700.0   \n",
       "3  01/01/2016 03:00           170.0         3.0             0.0       800.0   \n",
       "4  01/01/2016 04:00           110.0         3.0             0.0       700.0   \n",
       "\n",
       "   air_temperature  rltv_hum  stn_pres  wmo_hr_sun_dur  \n",
       "0              3.8      89.8    1017.8             NaN  \n",
       "1              3.7      91.5    1018.5             0.0  \n",
       "2              2.6      94.6    1018.8             0.0  \n",
       "3              2.0      94.3    1019.1             0.0  \n",
       "4              2.3      96.4    1019.6             0.0  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2016w1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:28:54.862668Z",
     "start_time": "2021-03-05T16:28:54.620808Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 9490 entries, 0 to 9489\n",
      "Data columns (total 15 columns):\n",
      " #   Column            Non-Null Count  Dtype  \n",
      "---  ------            --------------  -----  \n",
      " 0   ob_end_time       9490 non-null   object \n",
      " 1   id                9489 non-null   float64\n",
      " 2   id_type           9489 non-null   object \n",
      " 3   ob_hour_count     9489 non-null   float64\n",
      " 4   version_num       9489 non-null   float64\n",
      " 5   met_domain_name   9489 non-null   object \n",
      " 6   src_id            9489 non-null   float64\n",
      " 7   rec_st_ind        9489 non-null   float64\n",
      " 8   prcp_amt          9488 non-null   float64\n",
      " 9   prcp_dur          0 non-null      float64\n",
      " 10  prcp_amt_q        9489 non-null   float64\n",
      " 11  prcp_dur_q        1 non-null      float64\n",
      " 12  prcp_amt_j        0 non-null      float64\n",
      " 13  meto_stmp_time    9489 non-null   object \n",
      " 14  midas_stmp_etime  9487 non-null   float64\n",
      "dtypes: float64(11), object(4)\n",
      "memory usage: 1.1+ MB\n"
     ]
    }
   ],
   "source": [
    "Weath2016r = pd.read_csv('uk-hourly-rain-obs_2016noleg.csv', engine='python')\n",
    "Weath2016r.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:28:58.765607Z",
     "start_time": "2021-03-05T16:28:58.734626Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_end_time</th>\n",
       "      <th>id</th>\n",
       "      <th>id_type</th>\n",
       "      <th>ob_hour_count</th>\n",
       "      <th>version_num</th>\n",
       "      <th>met_domain_name</th>\n",
       "      <th>src_id</th>\n",
       "      <th>rec_st_ind</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>prcp_dur</th>\n",
       "      <th>prcp_amt_q</th>\n",
       "      <th>prcp_dur_q</th>\n",
       "      <th>prcp_amt_j</th>\n",
       "      <th>meto_stmp_time</th>\n",
       "      <th>midas_stmp_etime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>01/01/2016 00:00</td>\n",
       "      <td>247539.0</td>\n",
       "      <td>RAIN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>SREW</td>\n",
       "      <td>708.0</td>\n",
       "      <td>1011.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31/12/2015 23:52</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>01/01/2016 01:00</td>\n",
       "      <td>247539.0</td>\n",
       "      <td>RAIN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>SREW</td>\n",
       "      <td>708.0</td>\n",
       "      <td>1011.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>01/01/2016 01:34</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>01/01/2016 02:00</td>\n",
       "      <td>247539.0</td>\n",
       "      <td>RAIN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>SREW</td>\n",
       "      <td>708.0</td>\n",
       "      <td>1011.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>01/01/2016 01:51</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>01/01/2016 03:00</td>\n",
       "      <td>247539.0</td>\n",
       "      <td>RAIN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>SREW</td>\n",
       "      <td>708.0</td>\n",
       "      <td>1011.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>01/01/2016 02:51</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>01/01/2016 04:00</td>\n",
       "      <td>247539.0</td>\n",
       "      <td>RAIN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>SREW</td>\n",
       "      <td>708.0</td>\n",
       "      <td>1011.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>01/01/2016 03:52</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        ob_end_time        id id_type  ob_hour_count  version_num  \\\n",
       "0  01/01/2016 00:00  247539.0    RAIN            1.0          1.0   \n",
       "1  01/01/2016 01:00  247539.0    RAIN            1.0          1.0   \n",
       "2  01/01/2016 02:00  247539.0    RAIN            1.0          1.0   \n",
       "3  01/01/2016 03:00  247539.0    RAIN            1.0          1.0   \n",
       "4  01/01/2016 04:00  247539.0    RAIN            1.0          1.0   \n",
       "\n",
       "  met_domain_name  src_id  rec_st_ind  prcp_amt  prcp_dur  prcp_amt_q  \\\n",
       "0            SREW   708.0      1011.0       0.0       NaN         1.0   \n",
       "1            SREW   708.0      1011.0       0.0       NaN         1.0   \n",
       "2            SREW   708.0      1011.0       0.0       NaN         1.0   \n",
       "3            SREW   708.0      1011.0       0.0       NaN         1.0   \n",
       "4            SREW   708.0      1011.0       0.0       NaN         1.0   \n",
       "\n",
       "   prcp_dur_q  prcp_amt_j    meto_stmp_time  midas_stmp_etime  \n",
       "0         NaN         NaN  31/12/2015 23:52               0.0  \n",
       "1         NaN         NaN  01/01/2016 01:34               0.0  \n",
       "2         NaN         NaN  01/01/2016 01:51               0.0  \n",
       "3         NaN         NaN  01/01/2016 02:51               0.0  \n",
       "4         NaN         NaN  01/01/2016 03:52               0.0  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2016r.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:29:00.587777Z",
     "start_time": "2021-03-05T16:29:00.563790Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0     8760\n",
       "12.0     726\n",
       "24.0       3\n",
       "Name: ob_hour_count, dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2016r.ob_hour_count.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:29:02.299624Z",
     "start_time": "2021-03-05T16:29:02.267624Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 8760 entries, 0 to 9488\n",
      "Data columns (total 15 columns):\n",
      " #   Column            Non-Null Count  Dtype  \n",
      "---  ------            --------------  -----  \n",
      " 0   ob_end_time       8760 non-null   object \n",
      " 1   id                8760 non-null   float64\n",
      " 2   id_type           8760 non-null   object \n",
      " 3   ob_hour_count     8760 non-null   float64\n",
      " 4   version_num       8760 non-null   float64\n",
      " 5   met_domain_name   8760 non-null   object \n",
      " 6   src_id            8760 non-null   float64\n",
      " 7   rec_st_ind        8760 non-null   float64\n",
      " 8   prcp_amt          8759 non-null   float64\n",
      " 9   prcp_dur          0 non-null      float64\n",
      " 10  prcp_amt_q        8760 non-null   float64\n",
      " 11  prcp_dur_q        0 non-null      float64\n",
      " 12  prcp_amt_j        0 non-null      float64\n",
      " 13  meto_stmp_time    8760 non-null   object \n",
      " 14  midas_stmp_etime  8758 non-null   float64\n",
      "dtypes: float64(11), object(4)\n",
      "memory usage: 1.1+ MB\n"
     ]
    }
   ],
   "source": [
    "Weath2016r1 = Weath2016r[Weath2016r['ob_hour_count'] == 1]\n",
    "Weath2016r1.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:29:15.669637Z",
     "start_time": "2021-03-05T16:29:07.384391Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['18/11/2016 10:00',\n",
       " '18/11/2016 11:00',\n",
       " '18/11/2016 12:00',\n",
       " '18/11/2016 13:00',\n",
       " '18/11/2016 14:00',\n",
       " '18/11/2016 15:00',\n",
       " '18/11/2016 16:00',\n",
       " '18/11/2016 17:00',\n",
       " '18/11/2016 18:00',\n",
       " '18/11/2016 19:00',\n",
       " '18/11/2016 20:00',\n",
       " '18/11/2016 21:00',\n",
       " '18/11/2016 22:00',\n",
       " '18/11/2016 23:00',\n",
       " '19/11/2016 00:00',\n",
       " '19/11/2016 01:00',\n",
       " '19/11/2016 02:00',\n",
       " '19/11/2016 03:00',\n",
       " '19/11/2016 04:00',\n",
       " '19/11/2016 05:00',\n",
       " '19/11/2016 06:00',\n",
       " '19/11/2016 07:00',\n",
       " '19/11/2016 08:00',\n",
       " '19/11/2016 09:00',\n",
       " 'end data']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Finding missing values (to be filled from meteostat if a large number of continuous missing, or backfilled for odd\n",
    "# missing hours).\n",
    "miss2016 = []\n",
    "for i in Weath2016w1['ob_time']:\n",
    "    if i not in list(Weath2016r1['ob_end_time']):\n",
    "        miss2016.append(i)\n",
    "miss2016"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:29:21.636759Z",
     "start_time": "2021-03-05T16:29:21.600786Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 8785 entries, 0 to 8784\n",
      "Data columns (total 10 columns):\n",
      " #   Column           Non-Null Count  Dtype  \n",
      "---  ------           --------------  -----  \n",
      " 0   ob_time          8785 non-null   object \n",
      " 1   wind_direction   8775 non-null   float64\n",
      " 2   wind_speed       8775 non-null   float64\n",
      " 3   cld_ttl_amt_id   8781 non-null   float64\n",
      " 4   visibility       8781 non-null   float64\n",
      " 5   air_temperature  8782 non-null   float64\n",
      " 6   rltv_hum         8782 non-null   float64\n",
      " 7   stn_pres         8782 non-null   float64\n",
      " 8   wmo_hr_sun_dur   8414 non-null   float64\n",
      " 9   prcp_amt         8759 non-null   float64\n",
      "dtypes: float64(9), object(1)\n",
      "memory usage: 755.0+ KB\n"
     ]
    }
   ],
   "source": [
    "Weath2016r1 = Weath2016r1.rename(columns={\"ob_end_time\": \"ob_time\"}, errors=\"raise\")\n",
    "df2=Weath2016r1[['ob_time','prcp_amt']]\n",
    "        \n",
    "Weath2016w1 = Weath2016w1.merge(df2,how = 'left', on = 'ob_time')\n",
    "Weath2016w1.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:29:25.177594Z",
     "start_time": "2021-03-05T16:29:24.877769Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 8782 entries, 0 to 8781\n",
      "Data columns (total 24 columns):\n",
      " #   Column             Non-Null Count  Dtype  \n",
      "---  ------             --------------  -----  \n",
      " 0   ob_end_time        8782 non-null   object \n",
      " 1   id_type            8781 non-null   object \n",
      " 2   id                 8781 non-null   float64\n",
      " 3   ob_hour_count      8781 non-null   float64\n",
      " 4   met_domain_name    8781 non-null   object \n",
      " 5   version_num        8781 non-null   float64\n",
      " 6   src_id             8781 non-null   float64\n",
      " 7   rec_st_ind         8781 non-null   float64\n",
      " 8   mean_wind_dir      8773 non-null   float64\n",
      " 9   mean_wind_speed    8773 non-null   float64\n",
      " 10  max_gust_dir       8781 non-null   float64\n",
      " 11  max_gust_speed     8781 non-null   float64\n",
      " 12  max_gust_ctime     8781 non-null   float64\n",
      " 13  mean_wind_dir_q    8773 non-null   float64\n",
      " 14  mean_wind_speed_q  8773 non-null   float64\n",
      " 15  max_gust_dir_q     8781 non-null   float64\n",
      " 16  max_gust_speed_q   8781 non-null   float64\n",
      " 17  max_gust_ctime_q   8781 non-null   float64\n",
      " 18  mean_wind_dir_j    0 non-null      float64\n",
      " 19  mean_wind_speed_j  0 non-null      float64\n",
      " 20  max_gust_dir_j     0 non-null      float64\n",
      " 21  max_gust_speed_j   0 non-null      float64\n",
      " 22  meto_stmp_time     8781 non-null   object \n",
      " 23  midas_stmp_etime   8779 non-null   float64\n",
      "dtypes: float64(20), object(4)\n",
      "memory usage: 1.6+ MB\n"
     ]
    }
   ],
   "source": [
    "Weath2016wind = pd.read_csv('uk-mean-wind-obs_2016noleg.csv', engine='python')\n",
    "Weath2016wind.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:29:29.552139Z",
     "start_time": "2021-03-05T16:29:29.499172Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_end_time</th>\n",
       "      <th>id_type</th>\n",
       "      <th>id</th>\n",
       "      <th>ob_hour_count</th>\n",
       "      <th>met_domain_name</th>\n",
       "      <th>version_num</th>\n",
       "      <th>src_id</th>\n",
       "      <th>rec_st_ind</th>\n",
       "      <th>mean_wind_dir</th>\n",
       "      <th>mean_wind_speed</th>\n",
       "      <th>...</th>\n",
       "      <th>mean_wind_speed_q</th>\n",
       "      <th>max_gust_dir_q</th>\n",
       "      <th>max_gust_speed_q</th>\n",
       "      <th>max_gust_ctime_q</th>\n",
       "      <th>mean_wind_dir_j</th>\n",
       "      <th>mean_wind_speed_j</th>\n",
       "      <th>max_gust_dir_j</th>\n",
       "      <th>max_gust_speed_j</th>\n",
       "      <th>meto_stmp_time</th>\n",
       "      <th>midas_stmp_etime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>01/01/2016 00:00</td>\n",
       "      <td>WIND</td>\n",
       "      <td>511305.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>HCM</td>\n",
       "      <td>1.0</td>\n",
       "      <td>708.0</td>\n",
       "      <td>1011.0</td>\n",
       "      <td>260.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31/12/2015 23:52</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>01/01/2016 01:00</td>\n",
       "      <td>WIND</td>\n",
       "      <td>511305.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>HCM</td>\n",
       "      <td>1.0</td>\n",
       "      <td>708.0</td>\n",
       "      <td>1011.0</td>\n",
       "      <td>230.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>01/01/2016 01:34</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>01/01/2016 02:00</td>\n",
       "      <td>WIND</td>\n",
       "      <td>511305.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>HCM</td>\n",
       "      <td>1.0</td>\n",
       "      <td>708.0</td>\n",
       "      <td>1011.0</td>\n",
       "      <td>220.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>01/01/2016 01:51</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>01/01/2016 03:00</td>\n",
       "      <td>WIND</td>\n",
       "      <td>511305.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>HCM</td>\n",
       "      <td>1.0</td>\n",
       "      <td>708.0</td>\n",
       "      <td>1011.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>01/01/2016 02:51</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>01/01/2016 04:00</td>\n",
       "      <td>WIND</td>\n",
       "      <td>511305.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>HCM</td>\n",
       "      <td>1.0</td>\n",
       "      <td>708.0</td>\n",
       "      <td>1011.0</td>\n",
       "      <td>150.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>01/01/2016 03:52</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        ob_end_time id_type        id  ob_hour_count met_domain_name  \\\n",
       "0  01/01/2016 00:00    WIND  511305.0            1.0             HCM   \n",
       "1  01/01/2016 01:00    WIND  511305.0            1.0             HCM   \n",
       "2  01/01/2016 02:00    WIND  511305.0            1.0             HCM   \n",
       "3  01/01/2016 03:00    WIND  511305.0            1.0             HCM   \n",
       "4  01/01/2016 04:00    WIND  511305.0            1.0             HCM   \n",
       "\n",
       "   version_num  src_id  rec_st_ind  mean_wind_dir  mean_wind_speed  ...  \\\n",
       "0          1.0   708.0      1011.0          260.0              8.0  ...   \n",
       "1          1.0   708.0      1011.0          230.0              5.0  ...   \n",
       "2          1.0   708.0      1011.0          220.0              2.0  ...   \n",
       "3          1.0   708.0      1011.0          170.0              3.0  ...   \n",
       "4          1.0   708.0      1011.0          150.0              4.0  ...   \n",
       "\n",
       "   mean_wind_speed_q  max_gust_dir_q  max_gust_speed_q  max_gust_ctime_q  \\\n",
       "0                6.0             6.0               6.0               6.0   \n",
       "1                6.0             6.0               6.0               6.0   \n",
       "2                6.0             6.0               6.0               6.0   \n",
       "3                6.0             6.0               6.0               6.0   \n",
       "4                6.0             6.0               6.0               6.0   \n",
       "\n",
       "   mean_wind_dir_j  mean_wind_speed_j  max_gust_dir_j  max_gust_speed_j  \\\n",
       "0              NaN                NaN             NaN               NaN   \n",
       "1              NaN                NaN             NaN               NaN   \n",
       "2              NaN                NaN             NaN               NaN   \n",
       "3              NaN                NaN             NaN               NaN   \n",
       "4              NaN                NaN             NaN               NaN   \n",
       "\n",
       "     meto_stmp_time  midas_stmp_etime  \n",
       "0  31/12/2015 23:52               0.0  \n",
       "1  01/01/2016 01:34               0.0  \n",
       "2  01/01/2016 01:51               0.0  \n",
       "3  01/01/2016 02:51               0.0  \n",
       "4  01/01/2016 03:52               0.0  \n",
       "\n",
       "[5 rows x 24 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2016wind.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:29:31.420739Z",
     "start_time": "2021-03-05T16:29:31.378760Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 8785 entries, 0 to 8784\n",
      "Data columns (total 11 columns):\n",
      " #   Column           Non-Null Count  Dtype  \n",
      "---  ------           --------------  -----  \n",
      " 0   ob_time          8785 non-null   object \n",
      " 1   wind_direction   8775 non-null   float64\n",
      " 2   wind_speed       8775 non-null   float64\n",
      " 3   cld_ttl_amt_id   8781 non-null   float64\n",
      " 4   visibility       8781 non-null   float64\n",
      " 5   air_temperature  8782 non-null   float64\n",
      " 6   rltv_hum         8782 non-null   float64\n",
      " 7   stn_pres         8782 non-null   float64\n",
      " 8   wmo_hr_sun_dur   8414 non-null   float64\n",
      " 9   prcp_amt         8759 non-null   float64\n",
      " 10  max_gust_speed   8781 non-null   float64\n",
      "dtypes: float64(10), object(1)\n",
      "memory usage: 823.6+ KB\n"
     ]
    }
   ],
   "source": [
    "Weath2016wind1 = Weath2016wind[['ob_end_time','max_gust_speed']]\n",
    "Weath2016wind1 = Weath2016wind1.rename(columns={\"ob_end_time\": \"ob_time\"}, errors=\"raise\")\n",
    "Weath2016w1 = Weath2016w1.merge(Weath2016wind1,how = 'left', on = 'ob_time')\n",
    "Weath2016w1.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:29:44.680886Z",
     "start_time": "2021-03-05T16:29:36.704443Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['22/03/2016 13:00', '20/06/2016 10:00', '20/06/2016 11:00']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluate missing values.\n",
    "miss2016wind = []\n",
    "for i in Weath2016w1['ob_time']:\n",
    "    if i not in list(Weath2016wind1['ob_time']):\n",
    "        miss2016wind.append(i)\n",
    "miss2016wind"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:29:46.018702Z",
     "start_time": "2021-03-05T16:29:45.998711Z"
    }
   },
   "outputs": [],
   "source": [
    "# Last row is NaN\n",
    "Weath2016w2 = Weath2016w1.drop(index = (len(Weath2016w1)-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:29:48.169112Z",
     "start_time": "2021-03-05T16:29:47.131709Z"
    }
   },
   "outputs": [],
   "source": [
    "Weath2016w2['ob_time'] = pd.to_datetime(Weath2016w2.ob_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:29:49.056814Z",
     "start_time": "2021-03-05T16:29:49.034827Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 8784 entries, 0 to 8783\n",
      "Data columns (total 11 columns):\n",
      " #   Column           Non-Null Count  Dtype         \n",
      "---  ------           --------------  -----         \n",
      " 0   ob_time          8784 non-null   datetime64[ns]\n",
      " 1   wind_direction   8775 non-null   float64       \n",
      " 2   wind_speed       8775 non-null   float64       \n",
      " 3   cld_ttl_amt_id   8781 non-null   float64       \n",
      " 4   visibility       8781 non-null   float64       \n",
      " 5   air_temperature  8782 non-null   float64       \n",
      " 6   rltv_hum         8782 non-null   float64       \n",
      " 7   stn_pres         8782 non-null   float64       \n",
      " 8   wmo_hr_sun_dur   8414 non-null   float64       \n",
      " 9   prcp_amt         8759 non-null   float64       \n",
      " 10  max_gust_speed   8781 non-null   float64       \n",
      "dtypes: datetime64[ns](1), float64(10)\n",
      "memory usage: 823.5 KB\n"
     ]
    }
   ],
   "source": [
    "Weath2016w2.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:29:54.451920Z",
     "start_time": "2021-03-05T16:29:51.111650Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\andre\\anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:670: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  iloc._setitem_with_indexer(indexer, value)\n"
     ]
    }
   ],
   "source": [
    "# Importing missing data\n",
    "from meteostat import Stations, Daily, Hourly\n",
    "from datetime import datetime\n",
    "# Set time period\n",
    "start = datetime(2016, 11, 18)\n",
    "end = datetime(2016, 11, 19, 9, 59)\n",
    "\n",
    "# Get hourly data\n",
    "data = Hourly('03772', start, end)\n",
    "data2016weathr = data.fetch()\n",
    "\n",
    "for i in data2016weathr.index:\n",
    "    Weath2016w2[['prcp_amt']].loc[i] = data2016weathr[['prcp']].loc[i]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:29:57.968545Z",
     "start_time": "2021-03-05T16:29:57.937560Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['NOAA Solar Calculations - Change any of the highlighted cells to get solar position data for that location and time-of-day for a year.',\n",
       "       'Unnamed: 1', 'Unnamed: 2', 'Date', 'Time (hrs past local midnight)',\n",
       "       'Julian Day', 'Julian Century', 'Unnamed: 7',\n",
       "       'Geom Mean Long Sun (deg)', 'Geom Mean Anom Sun (deg)',\n",
       "       'Eccent Earth Orbit', 'Sun Eq of Ctr', 'Sun True Long (deg)',\n",
       "       'Sun True Anom (deg)', 'Sun Rad Vector (AUs)', 'Sun App Long (deg)',\n",
       "       'Mean Obliq Ecliptic (deg)', 'Obliq Corr (deg)', 'Sun Rt Ascen (deg)',\n",
       "       'Sun Declin (deg)', 'var y', 'Eq of Time (minutes)', 'HA Sunrise (deg)',\n",
       "       'Solar Noon (LST)', 'Sunrise Time (LST)', 'Sunset Time (LST)',\n",
       "       'Sunlight Duration (minutes)', 'True Solar Time (min)',\n",
       "       'Hour Angle (deg)', 'Solar Zenith Angle (deg)',\n",
       "       'Solar Elevation Angle (deg)', 'Approx Atmospheric Refraction (deg)',\n",
       "       'Solar Elevation corrected for atm refraction (deg)',\n",
       "       'Solar Azimuth Angle (deg cw from N)', 'Unnamed: 34'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now for sunrise and sunset times (for 'islight' marker)\n",
    "Sunrise2016 = pd.read_csv('sunrise2016.csv', engine='python')\n",
    "Sunrise2016.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:29:59.595777Z",
     "start_time": "2021-03-05T16:29:59.530820Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\andre\\anaconda3\\lib\\site-packages\\pandas\\core\\generic.py:5168: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self[name] = value\n",
      "<ipython-input-39-35b5af636fad>:4: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  Sunrise20161['ob_time1'] = Sunrise20161['Date'].dt.date\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2016-01-01 00:00:00</td>\n",
       "      <td>240.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>3.8</td>\n",
       "      <td>89.8</td>\n",
       "      <td>1017.8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2016-01-01</td>\n",
       "      <td>2016-01-01</td>\n",
       "      <td>8:05:39</td>\n",
       "      <td>16:00:59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2016-01-01 01:00:00</td>\n",
       "      <td>230.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1400.0</td>\n",
       "      <td>3.7</td>\n",
       "      <td>91.5</td>\n",
       "      <td>1018.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2016-01-01</td>\n",
       "      <td>2016-01-01</td>\n",
       "      <td>8:05:39</td>\n",
       "      <td>16:00:59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2016-01-01 02:00:00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>700.0</td>\n",
       "      <td>2.6</td>\n",
       "      <td>94.6</td>\n",
       "      <td>1018.8</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2016-01-01</td>\n",
       "      <td>2016-01-01</td>\n",
       "      <td>8:05:39</td>\n",
       "      <td>16:00:59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2016-01-01 03:00:00</td>\n",
       "      <td>170.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>800.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>94.3</td>\n",
       "      <td>1019.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2016-01-01</td>\n",
       "      <td>2016-01-01</td>\n",
       "      <td>8:05:39</td>\n",
       "      <td>16:00:59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2016-01-01 04:00:00</td>\n",
       "      <td>110.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>700.0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>96.4</td>\n",
       "      <td>1019.6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2016-01-01</td>\n",
       "      <td>2016-01-01</td>\n",
       "      <td>8:05:39</td>\n",
       "      <td>16:00:59</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              ob_time  wind_direction  wind_speed  cld_ttl_amt_id  visibility  \\\n",
       "0 2016-01-01 00:00:00           240.0         6.0             0.0      2000.0   \n",
       "1 2016-01-01 01:00:00           230.0         5.0             0.0      1400.0   \n",
       "2 2016-01-01 02:00:00             0.0         0.0             0.0       700.0   \n",
       "3 2016-01-01 03:00:00           170.0         3.0             0.0       800.0   \n",
       "4 2016-01-01 04:00:00           110.0         3.0             0.0       700.0   \n",
       "\n",
       "   air_temperature  rltv_hum  stn_pres  wmo_hr_sun_dur  prcp_amt  \\\n",
       "0              3.8      89.8    1017.8             NaN       0.0   \n",
       "1              3.7      91.5    1018.5             0.0       0.0   \n",
       "2              2.6      94.6    1018.8             0.0       0.0   \n",
       "3              2.0      94.3    1019.1             0.0       0.0   \n",
       "4              2.3      96.4    1019.6             0.0       0.0   \n",
       "\n",
       "   max_gust_speed    ob_time1       Date Sunrise Time (LST) Sunset Time (LST)  \n",
       "0            14.0  2016-01-01 2016-01-01            8:05:39          16:00:59  \n",
       "1             8.0  2016-01-01 2016-01-01            8:05:39          16:00:59  \n",
       "2             5.0  2016-01-01 2016-01-01            8:05:39          16:00:59  \n",
       "3             6.0  2016-01-01 2016-01-01            8:05:39          16:00:59  \n",
       "4             6.0  2016-01-01 2016-01-01            8:05:39          16:00:59  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Sunrise20161 = Sunrise2016[['Date','Sunrise Time (LST)','Sunset Time (LST)']]\n",
    "Sunrise20161.Date = pd.to_datetime(Sunrise20161.Date)\n",
    "Weath2016w2['ob_time1'] = Weath2016w2.ob_time.dt.date\n",
    "Sunrise20161['ob_time1'] = Sunrise20161['Date'].dt.date\n",
    "Weath2016w2 = Weath2016w2.merge(Sunrise20161,how = 'left', on = 'ob_time1')\n",
    "Weath2016w2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:30:06.874376Z",
     "start_time": "2021-03-05T16:30:06.319592Z"
    }
   },
   "outputs": [],
   "source": [
    "# 'islight' marker\n",
    "Weath2016w2['sunrisehr']=Weath2016w2['Sunrise Time (LST)'].str.replace(r':\\d\\d:\\d\\d','').astype(int)\n",
    "Weath2016w2['sunsethr']=Weath2016w2['Sunset Time (LST)'].str.replace(r':\\d\\d:\\d\\d','').astype(int)\n",
    "\n",
    "Weath2016w2['islight'] = 0\n",
    "Weath2016w2['islight'] = [1 if ((Weath2016w2['sunrisehr'][i] <= Weath2016w2['ob_time'][i].hour) \n",
    "                                and (Weath2016w2['ob_time'][i].hour < Weath2016w2['sunsethr'][i])) \n",
    "                          else 0 for i in Weath2016w2.index]\n",
    "                           "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now for the UTC/GMT to summertime adjustment (Weather GMT only, LFB BST as well)\n",
    "Daylight saving time 2016 in United Kingdom began at 01:00 on\n",
    "Sunday, 27 March\n",
    "and ended at 02:00 on\n",
    "Sunday, 30 October"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:30:10.419713Z",
     "start_time": "2021-03-05T16:30:10.383729Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7274</th>\n",
       "      <td>2016-10-30 02:00:00</td>\n",
       "      <td>90.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>8.4</td>\n",
       "      <td>98.5</td>\n",
       "      <td>1028.6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>6:50:56</td>\n",
       "      <td>16:36:15</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time  wind_direction  wind_speed  cld_ttl_amt_id  \\\n",
       "7274 2016-10-30 02:00:00            90.0         2.0             0.0   \n",
       "\n",
       "      visibility  air_temperature  rltv_hum  stn_pres  wmo_hr_sun_dur  \\\n",
       "7274        20.0              8.4      98.5    1028.6             0.0   \n",
       "\n",
       "      prcp_amt  max_gust_speed    ob_time1       Date Sunrise Time (LST)  \\\n",
       "7274       0.0             3.0  2016-10-30 2016-10-30            6:50:56   \n",
       "\n",
       "     Sunset Time (LST)  sunrisehr  sunsethr  islight  \n",
       "7274          16:36:15          6        16        0  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2016w2[Weath2016w2['ob_time'] == '2016-10-30 02:00:00']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:30:11.952162Z",
     "start_time": "2021-03-05T16:30:11.898195Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7273</th>\n",
       "      <td>2016-10-30 01:00:00</td>\n",
       "      <td>50</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>70</td>\n",
       "      <td>9.1</td>\n",
       "      <td>98.6</td>\n",
       "      <td>1029</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>6:50:56</td>\n",
       "      <td>16:36:15</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7274</th>\n",
       "      <td>2016-10-30 02:00:00</td>\n",
       "      <td>90</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>8.4</td>\n",
       "      <td>98.5</td>\n",
       "      <td>1028.6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>6:50:56</td>\n",
       "      <td>16:36:15</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7274</th>\n",
       "      <td>2016-10-30 02:00:00</td>\n",
       "      <td>90</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>8.4</td>\n",
       "      <td>98.5</td>\n",
       "      <td>1028.6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>6:50:56</td>\n",
       "      <td>16:36:15</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7275</th>\n",
       "      <td>2016-10-30 03:00:00</td>\n",
       "      <td>250</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>10</td>\n",
       "      <td>9.6</td>\n",
       "      <td>100</td>\n",
       "      <td>1028.3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>6:50:56</td>\n",
       "      <td>16:36:15</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time wind_direction wind_speed cld_ttl_amt_id visibility  \\\n",
       "7273 2016-10-30 01:00:00             50          2              1         70   \n",
       "7274 2016-10-30 02:00:00             90          2              0         20   \n",
       "7274 2016-10-30 02:00:00             90          2              0         20   \n",
       "7275 2016-10-30 03:00:00            250          2              7         10   \n",
       "\n",
       "     air_temperature rltv_hum stn_pres wmo_hr_sun_dur prcp_amt max_gust_speed  \\\n",
       "7273             9.1     98.6     1029              0        0              3   \n",
       "7274             8.4     98.5   1028.6              0        0              3   \n",
       "7274             8.4     98.5   1028.6              0        0              3   \n",
       "7275             9.6      100   1028.3              0        0              3   \n",
       "\n",
       "        ob_time1       Date Sunrise Time (LST) Sunset Time (LST) sunrisehr  \\\n",
       "7273  2016-10-30 2016-10-30            6:50:56          16:36:15         6   \n",
       "7274  2016-10-30 2016-10-30            6:50:56          16:36:15         6   \n",
       "7274  2016-10-30 2016-10-30            6:50:56          16:36:15         6   \n",
       "7275  2016-10-30 2016-10-30            6:50:56          16:36:15         6   \n",
       "\n",
       "     sunsethr islight  \n",
       "7273       16       0  \n",
       "7274       16       0  \n",
       "7274       16       0  \n",
       "7275       16       0  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2016w3 = pd.concat([Weath2016w2.head(7274), pd.DataFrame(Weath2016w2.iloc[7274]).T, Weath2016w2.tail(len(Weath2016w2) - (7274))])\n",
    "Weath2016w3.iloc[[7273,7274,7275,7276]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:30:15.803749Z",
     "start_time": "2021-03-05T16:30:15.768772Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7273</th>\n",
       "      <td>2016-10-30 01:00:00</td>\n",
       "      <td>50</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>70</td>\n",
       "      <td>9.1</td>\n",
       "      <td>98.6</td>\n",
       "      <td>1029</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>6:50:56</td>\n",
       "      <td>16:36:15</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7274</th>\n",
       "      <td>2016-10-30 02:00:00</td>\n",
       "      <td>90</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>8.4</td>\n",
       "      <td>98.5</td>\n",
       "      <td>1028.6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>6:50:56</td>\n",
       "      <td>16:36:15</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7275</th>\n",
       "      <td>2016-10-30 02:00:00</td>\n",
       "      <td>90</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>8.4</td>\n",
       "      <td>98.5</td>\n",
       "      <td>1028.6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>6:50:56</td>\n",
       "      <td>16:36:15</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7276</th>\n",
       "      <td>2016-10-30 03:00:00</td>\n",
       "      <td>250</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>10</td>\n",
       "      <td>9.6</td>\n",
       "      <td>100</td>\n",
       "      <td>1028.3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>6:50:56</td>\n",
       "      <td>16:36:15</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time wind_direction wind_speed cld_ttl_amt_id visibility  \\\n",
       "7273 2016-10-30 01:00:00             50          2              1         70   \n",
       "7274 2016-10-30 02:00:00             90          2              0         20   \n",
       "7275 2016-10-30 02:00:00             90          2              0         20   \n",
       "7276 2016-10-30 03:00:00            250          2              7         10   \n",
       "\n",
       "     air_temperature rltv_hum stn_pres wmo_hr_sun_dur prcp_amt max_gust_speed  \\\n",
       "7273             9.1     98.6     1029              0        0              3   \n",
       "7274             8.4     98.5   1028.6              0        0              3   \n",
       "7275             8.4     98.5   1028.6              0        0              3   \n",
       "7276             9.6      100   1028.3              0        0              3   \n",
       "\n",
       "        ob_time1       Date Sunrise Time (LST) Sunset Time (LST) sunrisehr  \\\n",
       "7273  2016-10-30 2016-10-30            6:50:56          16:36:15         6   \n",
       "7274  2016-10-30 2016-10-30            6:50:56          16:36:15         6   \n",
       "7275  2016-10-30 2016-10-30            6:50:56          16:36:15         6   \n",
       "7276  2016-10-30 2016-10-30            6:50:56          16:36:15         6   \n",
       "\n",
       "     sunsethr islight  \n",
       "7273       16       0  \n",
       "7274       16       0  \n",
       "7275       16       0  \n",
       "7276       16       0  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2016w3.reset_index(inplace=True)\n",
    "Weath2016w3.drop('index', axis=1, inplace=True)\n",
    "Weath2016w3.iloc[[7273,7274,7275,7276]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:30:17.037955Z",
     "start_time": "2021-03-05T16:30:17.015987Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2065</th>\n",
       "      <td>2016-03-27 01:00:00</td>\n",
       "      <td>210.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>3500.0</td>\n",
       "      <td>7.1</td>\n",
       "      <td>81.1</td>\n",
       "      <td>997.8</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>5:45:21</td>\n",
       "      <td>18:25:04</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time  wind_direction  wind_speed  cld_ttl_amt_id  \\\n",
       "2065 2016-03-27 01:00:00           210.0        11.0             8.0   \n",
       "\n",
       "      visibility  air_temperature  rltv_hum  stn_pres  wmo_hr_sun_dur  \\\n",
       "2065      3500.0              7.1      81.1     997.8             0.0   \n",
       "\n",
       "      prcp_amt  max_gust_speed    ob_time1       Date Sunrise Time (LST)  \\\n",
       "2065       0.0            14.0  2016-03-27 2016-03-27            5:45:21   \n",
       "\n",
       "     Sunset Time (LST)  sunrisehr  sunsethr  islight  \n",
       "2065          18:25:04          5        18        0  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2016w2[Weath2016w2['ob_time'] == '2016-03-27 01:00:00']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:30:19.973305Z",
     "start_time": "2021-03-05T16:30:18.811953Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-45-0613b37d9fd7>:3: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  Weath2016w4['ob_time'][i+1] = Weath2016w3['ob_time'][i]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7271</th>\n",
       "      <td>2016-10-29 22:00:00</td>\n",
       "      <td>80</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>210</td>\n",
       "      <td>9.7</td>\n",
       "      <td>93.3</td>\n",
       "      <td>1029.6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>2016-10-29</td>\n",
       "      <td>2016-10-29</td>\n",
       "      <td>6:49:11</td>\n",
       "      <td>16:38:07</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7272</th>\n",
       "      <td>2016-10-29 23:00:00</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>90</td>\n",
       "      <td>9.8</td>\n",
       "      <td>98.7</td>\n",
       "      <td>1029.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>6:50:56</td>\n",
       "      <td>16:36:15</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7273</th>\n",
       "      <td>2016-10-30 00:00:00</td>\n",
       "      <td>50</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>70</td>\n",
       "      <td>9.1</td>\n",
       "      <td>98.6</td>\n",
       "      <td>1029</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>6:50:56</td>\n",
       "      <td>16:36:15</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7274</th>\n",
       "      <td>2016-10-30 01:00:00</td>\n",
       "      <td>90</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>8.4</td>\n",
       "      <td>98.5</td>\n",
       "      <td>1028.6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>6:50:56</td>\n",
       "      <td>16:36:15</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7275</th>\n",
       "      <td>2016-10-30 02:00:00</td>\n",
       "      <td>90</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>8.4</td>\n",
       "      <td>98.5</td>\n",
       "      <td>1028.6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>6:50:56</td>\n",
       "      <td>16:36:15</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7276</th>\n",
       "      <td>2016-10-30 03:00:00</td>\n",
       "      <td>250</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>10</td>\n",
       "      <td>9.6</td>\n",
       "      <td>100</td>\n",
       "      <td>1028.3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>2016-10-30</td>\n",
       "      <td>6:50:56</td>\n",
       "      <td>16:36:15</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time wind_direction wind_speed cld_ttl_amt_id visibility  \\\n",
       "7271 2016-10-29 22:00:00             80          2              0        210   \n",
       "7272 2016-10-29 23:00:00             30          1              0         90   \n",
       "7273 2016-10-30 00:00:00             50          2              1         70   \n",
       "7274 2016-10-30 01:00:00             90          2              0         20   \n",
       "7275 2016-10-30 02:00:00             90          2              0         20   \n",
       "7276 2016-10-30 03:00:00            250          2              7         10   \n",
       "\n",
       "     air_temperature rltv_hum stn_pres wmo_hr_sun_dur prcp_amt max_gust_speed  \\\n",
       "7271             9.7     93.3   1029.6              0        0              4   \n",
       "7272             9.8     98.7   1029.5            NaN        0              3   \n",
       "7273             9.1     98.6     1029              0        0              3   \n",
       "7274             8.4     98.5   1028.6              0        0              3   \n",
       "7275             8.4     98.5   1028.6              0        0              3   \n",
       "7276             9.6      100   1028.3              0        0              3   \n",
       "\n",
       "        ob_time1       Date Sunrise Time (LST) Sunset Time (LST) sunrisehr  \\\n",
       "7271  2016-10-29 2016-10-29            6:49:11          16:38:07         6   \n",
       "7272  2016-10-30 2016-10-30            6:50:56          16:36:15         6   \n",
       "7273  2016-10-30 2016-10-30            6:50:56          16:36:15         6   \n",
       "7274  2016-10-30 2016-10-30            6:50:56          16:36:15         6   \n",
       "7275  2016-10-30 2016-10-30            6:50:56          16:36:15         6   \n",
       "7276  2016-10-30 2016-10-30            6:50:56          16:36:15         6   \n",
       "\n",
       "     sunsethr islight  \n",
       "7271       16       0  \n",
       "7272       16       0  \n",
       "7273       16       0  \n",
       "7274       16       0  \n",
       "7275       16       0  \n",
       "7276       16       0  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2016w4 = Weath2016w3.copy()\n",
    "for i in range(2066,7274):\n",
    "    Weath2016w4['ob_time'][i+1] = Weath2016w3['ob_time'][i]\n",
    "Weath2016w4.iloc[[7271,7272,7273,7274,7275,7276]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:30:20.814677Z",
     "start_time": "2021-03-05T16:30:20.781697Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2065</th>\n",
       "      <td>2016-03-27 01:00:00</td>\n",
       "      <td>210</td>\n",
       "      <td>11</td>\n",
       "      <td>8</td>\n",
       "      <td>3500</td>\n",
       "      <td>7.1</td>\n",
       "      <td>81.1</td>\n",
       "      <td>997.8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>5:45:21</td>\n",
       "      <td>18:25:04</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2066</th>\n",
       "      <td>2016-03-27 02:00:00</td>\n",
       "      <td>200</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>3500</td>\n",
       "      <td>6.3</td>\n",
       "      <td>84.9</td>\n",
       "      <td>997.9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>5:45:21</td>\n",
       "      <td>18:25:04</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2067</th>\n",
       "      <td>2016-03-27 02:00:00</td>\n",
       "      <td>200</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>2200</td>\n",
       "      <td>5.8</td>\n",
       "      <td>89.3</td>\n",
       "      <td>997.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>5:45:21</td>\n",
       "      <td>18:25:04</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2068</th>\n",
       "      <td>2016-03-27 03:00:00</td>\n",
       "      <td>180</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>1800</td>\n",
       "      <td>5.8</td>\n",
       "      <td>90.8</td>\n",
       "      <td>997.7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>5:45:21</td>\n",
       "      <td>18:25:04</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time wind_direction wind_speed cld_ttl_amt_id visibility  \\\n",
       "2065 2016-03-27 01:00:00            210         11              8       3500   \n",
       "2066 2016-03-27 02:00:00            200          8              8       3500   \n",
       "2067 2016-03-27 02:00:00            200          5              4       2200   \n",
       "2068 2016-03-27 03:00:00            180          8              1       1800   \n",
       "\n",
       "     air_temperature rltv_hum stn_pres wmo_hr_sun_dur prcp_amt max_gust_speed  \\\n",
       "2065             7.1     81.1    997.8              0        0             14   \n",
       "2066             6.3     84.9    997.9              0        0             13   \n",
       "2067             5.8     89.3    997.5              0        0             10   \n",
       "2068             5.8     90.8    997.7              0        0             12   \n",
       "\n",
       "        ob_time1       Date Sunrise Time (LST) Sunset Time (LST) sunrisehr  \\\n",
       "2065  2016-03-27 2016-03-27            5:45:21          18:25:04         5   \n",
       "2066  2016-03-27 2016-03-27            5:45:21          18:25:04         5   \n",
       "2067  2016-03-27 2016-03-27            5:45:21          18:25:04         5   \n",
       "2068  2016-03-27 2016-03-27            5:45:21          18:25:04         5   \n",
       "\n",
       "     sunsethr islight  \n",
       "2065       18       0  \n",
       "2066       18       0  \n",
       "2067       18       0  \n",
       "2068       18       0  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2016w4.iloc[[2065,2066,2067,2068]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:30:22.670506Z",
     "start_time": "2021-03-05T16:30:22.612539Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2065</th>\n",
       "      <td>2016-03-27 01:00:00</td>\n",
       "      <td>210</td>\n",
       "      <td>11</td>\n",
       "      <td>8</td>\n",
       "      <td>3500</td>\n",
       "      <td>7.1</td>\n",
       "      <td>81.1</td>\n",
       "      <td>997.8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>5:45:21</td>\n",
       "      <td>18:25:04</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2067</th>\n",
       "      <td>2016-03-27 02:00:00</td>\n",
       "      <td>200</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>2200</td>\n",
       "      <td>5.8</td>\n",
       "      <td>89.3</td>\n",
       "      <td>997.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>5:45:21</td>\n",
       "      <td>18:25:04</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2068</th>\n",
       "      <td>2016-03-27 03:00:00</td>\n",
       "      <td>180</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>1800</td>\n",
       "      <td>5.8</td>\n",
       "      <td>90.8</td>\n",
       "      <td>997.7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>5:45:21</td>\n",
       "      <td>18:25:04</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2069</th>\n",
       "      <td>2016-03-27 04:00:00</td>\n",
       "      <td>180</td>\n",
       "      <td>9</td>\n",
       "      <td>2</td>\n",
       "      <td>2200</td>\n",
       "      <td>5.8</td>\n",
       "      <td>90.8</td>\n",
       "      <td>997.7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>5:45:21</td>\n",
       "      <td>18:25:04</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time wind_direction wind_speed cld_ttl_amt_id visibility  \\\n",
       "2065 2016-03-27 01:00:00            210         11              8       3500   \n",
       "2067 2016-03-27 02:00:00            200          5              4       2200   \n",
       "2068 2016-03-27 03:00:00            180          8              1       1800   \n",
       "2069 2016-03-27 04:00:00            180          9              2       2200   \n",
       "\n",
       "     air_temperature rltv_hum stn_pres wmo_hr_sun_dur prcp_amt max_gust_speed  \\\n",
       "2065             7.1     81.1    997.8              0        0             14   \n",
       "2067             5.8     89.3    997.5              0        0             10   \n",
       "2068             5.8     90.8    997.7              0        0             12   \n",
       "2069             5.8     90.8    997.7              0        0             13   \n",
       "\n",
       "        ob_time1       Date Sunrise Time (LST) Sunset Time (LST) sunrisehr  \\\n",
       "2065  2016-03-27 2016-03-27            5:45:21          18:25:04         5   \n",
       "2067  2016-03-27 2016-03-27            5:45:21          18:25:04         5   \n",
       "2068  2016-03-27 2016-03-27            5:45:21          18:25:04         5   \n",
       "2069  2016-03-27 2016-03-27            5:45:21          18:25:04         5   \n",
       "\n",
       "     sunsethr islight  \n",
       "2065       18       0  \n",
       "2067       18       0  \n",
       "2068       18       0  \n",
       "2069       18       1  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2016w5 = Weath2016w4.drop(2066)\n",
    "Weath2016w5.iloc[[2065,2066,2067,2068]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:30:26.183422Z",
     "start_time": "2021-03-05T16:30:26.132452Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2065</th>\n",
       "      <td>2016-03-27 01:00:00</td>\n",
       "      <td>210</td>\n",
       "      <td>11</td>\n",
       "      <td>8</td>\n",
       "      <td>3500</td>\n",
       "      <td>7.1</td>\n",
       "      <td>81.1</td>\n",
       "      <td>997.8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>5:45:21</td>\n",
       "      <td>18:25:04</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2066</th>\n",
       "      <td>2016-03-27 02:00:00</td>\n",
       "      <td>200</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>2200</td>\n",
       "      <td>5.8</td>\n",
       "      <td>89.3</td>\n",
       "      <td>997.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>5:45:21</td>\n",
       "      <td>18:25:04</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2067</th>\n",
       "      <td>2016-03-27 03:00:00</td>\n",
       "      <td>180</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>1800</td>\n",
       "      <td>5.8</td>\n",
       "      <td>90.8</td>\n",
       "      <td>997.7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>5:45:21</td>\n",
       "      <td>18:25:04</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2068</th>\n",
       "      <td>2016-03-27 04:00:00</td>\n",
       "      <td>180</td>\n",
       "      <td>9</td>\n",
       "      <td>2</td>\n",
       "      <td>2200</td>\n",
       "      <td>5.8</td>\n",
       "      <td>90.8</td>\n",
       "      <td>997.7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>2016-03-27</td>\n",
       "      <td>5:45:21</td>\n",
       "      <td>18:25:04</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time wind_direction wind_speed cld_ttl_amt_id visibility  \\\n",
       "2065 2016-03-27 01:00:00            210         11              8       3500   \n",
       "2066 2016-03-27 02:00:00            200          5              4       2200   \n",
       "2067 2016-03-27 03:00:00            180          8              1       1800   \n",
       "2068 2016-03-27 04:00:00            180          9              2       2200   \n",
       "\n",
       "     air_temperature rltv_hum stn_pres wmo_hr_sun_dur prcp_amt max_gust_speed  \\\n",
       "2065             7.1     81.1    997.8              0        0             14   \n",
       "2066             5.8     89.3    997.5              0        0             10   \n",
       "2067             5.8     90.8    997.7              0        0             12   \n",
       "2068             5.8     90.8    997.7              0        0             13   \n",
       "\n",
       "        ob_time1       Date Sunrise Time (LST) Sunset Time (LST) sunrisehr  \\\n",
       "2065  2016-03-27 2016-03-27            5:45:21          18:25:04         5   \n",
       "2066  2016-03-27 2016-03-27            5:45:21          18:25:04         5   \n",
       "2067  2016-03-27 2016-03-27            5:45:21          18:25:04         5   \n",
       "2068  2016-03-27 2016-03-27            5:45:21          18:25:04         5   \n",
       "\n",
       "     sunsethr islight  \n",
       "2065       18       0  \n",
       "2066       18       0  \n",
       "2067       18       0  \n",
       "2068       18       1  "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2016w5.reset_index(inplace=True)\n",
    "Weath2016w5.drop('index', axis=1, inplace=True)\n",
    "Weath2016w5.iloc[[2065,2066,2067,2068]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:30:27.696578Z",
     "start_time": "2021-03-05T16:30:27.691580Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['ob_time', 'wind_direction', 'wind_speed', 'cld_ttl_amt_id',\n",
       "       'visibility', 'air_temperature', 'rltv_hum', 'stn_pres',\n",
       "       'wmo_hr_sun_dur', 'prcp_amt', 'max_gust_speed', 'ob_time1', 'Date',\n",
       "       'Sunrise Time (LST)', 'Sunset Time (LST)', 'sunrisehr', 'sunsethr',\n",
       "       'islight'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2016w5.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:30:29.350983Z",
     "start_time": "2021-03-05T16:30:29.321022Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 8784 entries, 0 to 8783\n",
      "Data columns (total 12 columns):\n",
      " #   Column           Non-Null Count  Dtype         \n",
      "---  ------           --------------  -----         \n",
      " 0   ob_time          8784 non-null   datetime64[ns]\n",
      " 1   wind_direction   8775 non-null   object        \n",
      " 2   wind_speed       8775 non-null   object        \n",
      " 3   cld_ttl_amt_id   8781 non-null   object        \n",
      " 4   visibility       8781 non-null   object        \n",
      " 5   air_temperature  8782 non-null   object        \n",
      " 6   rltv_hum         8782 non-null   object        \n",
      " 7   stn_pres         8782 non-null   object        \n",
      " 8   wmo_hr_sun_dur   8414 non-null   object        \n",
      " 9   prcp_amt         8759 non-null   object        \n",
      " 10  max_gust_speed   8781 non-null   object        \n",
      " 11  islight          8784 non-null   object        \n",
      "dtypes: datetime64[ns](1), object(11)\n",
      "memory usage: 823.6+ KB\n"
     ]
    }
   ],
   "source": [
    "Weath2016w6 = Weath2016w5.drop(columns = ['ob_time1','Date','Sunrise Time (LST)', 'Sunset Time (LST)', 'sunrisehr', 'sunsethr'])\n",
    "Weath2016w6.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:30:31.763558Z",
     "start_time": "2021-03-05T16:30:31.717565Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 8784 entries, 0 to 8783\n",
      "Data columns (total 12 columns):\n",
      " #   Column           Non-Null Count  Dtype         \n",
      "---  ------           --------------  -----         \n",
      " 0   ob_time          8784 non-null   datetime64[ns]\n",
      " 1   wind_direction   8784 non-null   float64       \n",
      " 2   wind_speed       8784 non-null   float64       \n",
      " 3   cld_ttl_amt_id   8784 non-null   float64       \n",
      " 4   visibility       8784 non-null   float64       \n",
      " 5   air_temperature  8784 non-null   float64       \n",
      " 6   rltv_hum         8784 non-null   float64       \n",
      " 7   stn_pres         8784 non-null   float64       \n",
      " 8   wmo_hr_sun_dur   8784 non-null   float64       \n",
      " 9   prcp_amt         8784 non-null   float64       \n",
      " 10  max_gust_speed   8784 non-null   float64       \n",
      " 11  islight          8784 non-null   int64         \n",
      "dtypes: datetime64[ns](1), float64(10), int64(1)\n",
      "memory usage: 823.6 KB\n"
     ]
    }
   ],
   "source": [
    "# Filling empty values appropriately.\n",
    "Weath2016w6.fillna(method = 'backfill', inplace = True)\n",
    "Weath2016w6.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2017 Weather ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now for 2017, 2018, 2019. Function considered, but too many individual areas to adjust."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:30:46.648577Z",
     "start_time": "2021-03-05T16:30:37.990541Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['end data']"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2017w = pd.read_csv('uk-hourly-weather-obs2017noleg.csv', engine='python')\n",
    "Weath2017w1 = Weath2017w[['ob_time', 'wind_direction', 'wind_speed', 'cld_ttl_amt_id', 'visibility', 'air_temperature',  \n",
    "    'rltv_hum', 'stn_pres', 'wmo_hr_sun_dur']]\n",
    "Weath2017r = pd.read_csv('uk-hourly-rain-obs_2017noleg.csv', engine='python')\n",
    "Weath2017r1 = Weath2017r[Weath2017r['ob_hour_count'] == 1]\n",
    "miss2017 = []\n",
    "for i in Weath2017w1['ob_time']:\n",
    "    if i not in list(Weath2017r1['ob_end_time']):\n",
    "        miss2017.append(i)\n",
    "miss2017"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:30:57.719769Z",
     "start_time": "2021-03-05T16:30:47.948373Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['21/06/2017 12:00',\n",
       " '12/10/2017 11:00',\n",
       " '12/10/2017 12:00',\n",
       " '14/11/2017 13:00']"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2017r1 = Weath2017r1.rename(columns={\"ob_end_time\": \"ob_time\"}, errors=\"raise\")\n",
    "df3=Weath2017r1[['ob_time','prcp_amt']]\n",
    "        \n",
    "Weath2017w1 = Weath2017w1.merge(df3,how = 'left', on = 'ob_time')\n",
    "Weath2017wind = pd.read_csv('uk-mean-wind-obs_2017noleg.csv', engine='python')\n",
    "Weath2017wind1 = Weath2017wind[['ob_end_time','max_gust_speed']]\n",
    "Weath2017wind1 = Weath2017wind1.rename(columns={\"ob_end_time\": \"ob_time\"}, errors=\"raise\")\n",
    "Weath2017w1 = Weath2017w1.merge(Weath2017wind1,how = 'left', on = 'ob_time')\n",
    "miss2017wind = []\n",
    "for i in Weath2017w1['ob_time']:\n",
    "    if i not in list(Weath2017wind1['ob_time']):\n",
    "        miss2017wind.append(i)\n",
    "miss2017wind"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:31:03.447330Z",
     "start_time": "2021-03-05T16:31:01.951192Z"
    }
   },
   "outputs": [],
   "source": [
    "Weath2017w2 = Weath2017w1.drop(index = (len(Weath2017w1)-1))\n",
    "Weath2017w2['ob_time'] = pd.to_datetime(Weath2017w2.ob_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:31:03.747158Z",
     "start_time": "2021-03-05T16:31:03.450329Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\andre\\anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:670: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  iloc._setitem_with_indexer(indexer, value)\n"
     ]
    }
   ],
   "source": [
    "# Importing missing data (if needed)\n",
    "from meteostat import Stations, Daily, Hourly\n",
    "from datetime import datetime\n",
    "# Set time period\n",
    "start = datetime(2016, 11, 18)\n",
    "end = datetime(2016, 11, 19, 9, 59)\n",
    "\n",
    "# Get hourly data\n",
    "data = Hourly('03772', start, end)\n",
    "data2016weathr = data.fetch()\n",
    "\n",
    "for i in data2016weathr.index:\n",
    "    Weath2016w2[['prcp_amt']].loc[i] = data2016weathr[['prcp']].loc[i]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:31:08.395252Z",
     "start_time": "2021-03-05T16:31:07.657678Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\andre\\anaconda3\\lib\\site-packages\\pandas\\core\\generic.py:5168: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self[name] = value\n",
      "<ipython-input-56-0dded22c5f98>:5: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  Sunrise20171['ob_time1'] = Sunrise20171['Date'].dt.date\n"
     ]
    }
   ],
   "source": [
    "Sunrise2017 = pd.read_csv('sunrise2017.csv', engine='python')\n",
    "Sunrise20171 = Sunrise2017[['Date','Sunrise Time (LST)','Sunset Time (LST)']]\n",
    "Sunrise20171.Date = pd.to_datetime(Sunrise20171.Date)\n",
    "Weath2017w2['ob_time1'] = Weath2017w2.ob_time.dt.date\n",
    "Sunrise20171['ob_time1'] = Sunrise20171['Date'].dt.date\n",
    "Weath2017w2 = Weath2017w2.merge(Sunrise20171,how = 'left', on = 'ob_time1')\n",
    "Weath2017w2['sunrisehr']=Weath2017w2['Sunrise Time (LST)'].str.replace(r':\\d\\d:\\d\\d','').astype(int)\n",
    "Weath2017w2['sunsethr']=Weath2017w2['Sunset Time (LST)'].str.replace(r':\\d\\d:\\d\\d','').astype(int)\n",
    "\n",
    "Weath2017w2['islight'] = 0\n",
    "Weath2017w2['islight'] = [1 if ((Weath2017w2['sunrisehr'][i] <= Weath2017w2['ob_time'][i].hour) \n",
    "                                and (Weath2017w2['ob_time'][i].hour < Weath2017w2['sunsethr'][i])) \n",
    "                          else 0 for i in Weath2017w2.index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Daylight saving time 2017 in United Kingdom began at 01:00 on\n",
    "Sunday, 26 March\n",
    "and ended at 02:00 on\n",
    "Sunday, 29 October"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:31:17.217130Z",
     "start_time": "2021-03-05T16:31:17.174158Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7225</th>\n",
       "      <td>2017-10-29 02:00:00</td>\n",
       "      <td>260.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>5000.0</td>\n",
       "      <td>13.8</td>\n",
       "      <td>83.0</td>\n",
       "      <td>1015.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>2017-10-29</td>\n",
       "      <td>2017-10-29</td>\n",
       "      <td>6:48:45</td>\n",
       "      <td>16:38:35</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time  wind_direction  wind_speed  cld_ttl_amt_id  \\\n",
       "7225 2017-10-29 02:00:00           260.0        13.0             8.0   \n",
       "\n",
       "      visibility  air_temperature  rltv_hum  stn_pres  wmo_hr_sun_dur  \\\n",
       "7225      5000.0             13.8      83.0    1015.3             0.0   \n",
       "\n",
       "      prcp_amt  max_gust_speed    ob_time1       Date Sunrise Time (LST)  \\\n",
       "7225       0.0            19.0  2017-10-29 2017-10-29            6:48:45   \n",
       "\n",
       "     Sunset Time (LST)  sunrisehr  sunsethr  islight  \n",
       "7225          16:38:35          6        16        0  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2017w2[Weath2017w2['ob_time'] == '2017-10-29 02:00:00']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:31:19.354412Z",
     "start_time": "2021-03-05T16:31:19.291449Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2017</th>\n",
       "      <td>2017-03-26 01:00:00</td>\n",
       "      <td>50.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5000.0</td>\n",
       "      <td>6.2</td>\n",
       "      <td>48.0</td>\n",
       "      <td>1022.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2017-03-26</td>\n",
       "      <td>2017-03-26</td>\n",
       "      <td>5:48:11</td>\n",
       "      <td>18:22:59</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time  wind_direction  wind_speed  cld_ttl_amt_id  \\\n",
       "2017 2017-03-26 01:00:00            50.0        10.0             0.0   \n",
       "\n",
       "      visibility  air_temperature  rltv_hum  stn_pres  wmo_hr_sun_dur  \\\n",
       "2017      5000.0              6.2      48.0    1022.1             0.0   \n",
       "\n",
       "      prcp_amt  max_gust_speed    ob_time1       Date Sunrise Time (LST)  \\\n",
       "2017       0.0            16.0  2017-03-26 2017-03-26            5:48:11   \n",
       "\n",
       "     Sunset Time (LST)  sunrisehr  sunsethr  islight  \n",
       "2017          18:22:59          5        18        0  "
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2017w3 = pd.concat([Weath2017w2.head(7225), pd.DataFrame(Weath2017w2.iloc[7225]).T, Weath2017w2.tail(len(Weath2017w2) - (7225))])\n",
    "Weath2017w3.reset_index(inplace=True)\n",
    "Weath2017w3.drop('index', axis=1, inplace=True)\n",
    "Weath2017w2[Weath2017w2['ob_time'] == '2017-03-26 01:00:00']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:31:26.310892Z",
     "start_time": "2021-03-05T16:31:25.064608Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-59-7c82028e2e88>:3: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  Weath2017w4['ob_time'][i+1] = Weath2017w3['ob_time'][i]\n"
     ]
    }
   ],
   "source": [
    "Weath2017w4 = Weath2017w3.copy()\n",
    "for i in range(2018,7225):\n",
    "    Weath2017w4['ob_time'][i+1] = Weath2017w3['ob_time'][i]\n",
    "Weath2017w5 = Weath2017w4.drop(2018)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:31:30.521232Z",
     "start_time": "2021-03-05T16:31:30.474281Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2016</th>\n",
       "      <td>2017-03-26 00:00:00</td>\n",
       "      <td>60</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>5000</td>\n",
       "      <td>6.7</td>\n",
       "      <td>40.8</td>\n",
       "      <td>1022.8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>2017-03-26</td>\n",
       "      <td>2017-03-26</td>\n",
       "      <td>5:48:11</td>\n",
       "      <td>18:22:59</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2017</th>\n",
       "      <td>2017-03-26 01:00:00</td>\n",
       "      <td>50</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>5000</td>\n",
       "      <td>6.2</td>\n",
       "      <td>48</td>\n",
       "      <td>1022.1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "      <td>2017-03-26</td>\n",
       "      <td>2017-03-26</td>\n",
       "      <td>5:48:11</td>\n",
       "      <td>18:22:59</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2018</th>\n",
       "      <td>2017-03-26 02:00:00</td>\n",
       "      <td>40</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>3500</td>\n",
       "      <td>5.5</td>\n",
       "      <td>67.3</td>\n",
       "      <td>1021.6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>2017-03-26</td>\n",
       "      <td>2017-03-26</td>\n",
       "      <td>5:48:11</td>\n",
       "      <td>18:22:59</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019</th>\n",
       "      <td>2017-03-26 03:00:00</td>\n",
       "      <td>50</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>3000</td>\n",
       "      <td>5.6</td>\n",
       "      <td>74.9</td>\n",
       "      <td>1021.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2017-03-26</td>\n",
       "      <td>2017-03-26</td>\n",
       "      <td>5:48:11</td>\n",
       "      <td>18:22:59</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time wind_direction wind_speed cld_ttl_amt_id visibility  \\\n",
       "2016 2017-03-26 00:00:00             60          6              0       5000   \n",
       "2017 2017-03-26 01:00:00             50         10              0       5000   \n",
       "2018 2017-03-26 02:00:00             40          5              0       3500   \n",
       "2019 2017-03-26 03:00:00             50          7              0       3000   \n",
       "\n",
       "     air_temperature rltv_hum stn_pres wmo_hr_sun_dur prcp_amt max_gust_speed  \\\n",
       "2016             6.7     40.8   1022.8            NaN        0             16   \n",
       "2017             6.2       48   1022.1              0        0             16   \n",
       "2018             5.5     67.3   1021.6              0        0             10   \n",
       "2019             5.6     74.9   1021.5              0        0             12   \n",
       "\n",
       "        ob_time1       Date Sunrise Time (LST) Sunset Time (LST) sunrisehr  \\\n",
       "2016  2017-03-26 2017-03-26            5:48:11          18:22:59         5   \n",
       "2017  2017-03-26 2017-03-26            5:48:11          18:22:59         5   \n",
       "2018  2017-03-26 2017-03-26            5:48:11          18:22:59         5   \n",
       "2019  2017-03-26 2017-03-26            5:48:11          18:22:59         5   \n",
       "\n",
       "     sunsethr islight  \n",
       "2016       18       0  \n",
       "2017       18       0  \n",
       "2018       18       0  \n",
       "2019       18       0  "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2017w5.reset_index(inplace=True)\n",
    "Weath2017w5.drop('index', axis=1, inplace=True)\n",
    "Weath2017w5.iloc[[2016,2017,2018,2019]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:31:32.442403Z",
     "start_time": "2021-03-05T16:31:32.401425Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 8759 entries, 0 to 8758\n",
      "Data columns (total 12 columns):\n",
      " #   Column           Non-Null Count  Dtype         \n",
      "---  ------           --------------  -----         \n",
      " 0   ob_time          8759 non-null   datetime64[ns]\n",
      " 1   wind_direction   8759 non-null   float64       \n",
      " 2   wind_speed       8759 non-null   float64       \n",
      " 3   cld_ttl_amt_id   8759 non-null   float64       \n",
      " 4   visibility       8759 non-null   float64       \n",
      " 5   air_temperature  8759 non-null   float64       \n",
      " 6   rltv_hum         8759 non-null   float64       \n",
      " 7   stn_pres         8759 non-null   float64       \n",
      " 8   wmo_hr_sun_dur   8759 non-null   float64       \n",
      " 9   prcp_amt         8759 non-null   float64       \n",
      " 10  max_gust_speed   8759 non-null   float64       \n",
      " 11  islight          8759 non-null   int64         \n",
      "dtypes: datetime64[ns](1), float64(10), int64(1)\n",
      "memory usage: 821.3 KB\n"
     ]
    }
   ],
   "source": [
    "Weath2017w6 = Weath2017w5.drop(columns = ['ob_time1','Date','Sunrise Time (LST)', 'Sunset Time (LST)', 'sunrisehr', 'sunsethr'])\n",
    "Weath2017w6.fillna(method = 'backfill', inplace = True)\n",
    "Weath2017w6.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2018 Weather ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:31:46.260533Z",
     "start_time": "2021-03-05T16:31:38.073233Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['22/01/2018 15:00',\n",
       " '22/01/2018 16:00',\n",
       " '25/02/2018 10:00',\n",
       " '25/02/2018 11:00',\n",
       " '25/02/2018 12:00',\n",
       " '25/02/2018 13:00',\n",
       " '25/02/2018 14:00',\n",
       " '25/02/2018 15:00',\n",
       " '25/02/2018 16:00',\n",
       " '25/02/2018 17:00',\n",
       " '25/02/2018 18:00',\n",
       " '25/02/2018 19:00',\n",
       " '25/02/2018 20:00',\n",
       " '25/02/2018 21:00',\n",
       " '25/02/2018 22:00',\n",
       " '25/02/2018 23:00',\n",
       " '26/02/2018 00:00',\n",
       " '26/02/2018 01:00',\n",
       " '26/02/2018 02:00',\n",
       " '26/02/2018 03:00',\n",
       " '26/02/2018 04:00',\n",
       " '26/02/2018 05:00',\n",
       " '26/02/2018 06:00',\n",
       " '26/02/2018 07:00',\n",
       " '26/02/2018 08:00',\n",
       " '26/02/2018 09:00',\n",
       " '26/02/2018 10:00',\n",
       " '26/02/2018 11:00',\n",
       " '26/02/2018 12:00',\n",
       " '26/02/2018 13:00',\n",
       " '26/02/2018 14:00',\n",
       " '26/02/2018 15:00',\n",
       " '26/02/2018 16:00',\n",
       " '26/02/2018 17:00',\n",
       " '26/02/2018 18:00',\n",
       " '26/02/2018 19:00',\n",
       " '26/02/2018 20:00',\n",
       " '26/02/2018 21:00',\n",
       " '26/02/2018 22:00',\n",
       " '26/02/2018 23:00',\n",
       " '27/02/2018 00:00',\n",
       " '27/02/2018 01:00',\n",
       " '27/02/2018 02:00',\n",
       " '27/02/2018 03:00',\n",
       " '27/02/2018 04:00',\n",
       " '27/02/2018 05:00',\n",
       " '27/02/2018 06:00',\n",
       " '27/02/2018 07:00',\n",
       " '27/02/2018 08:00',\n",
       " '27/02/2018 09:00',\n",
       " '27/02/2018 10:00',\n",
       " '27/02/2018 11:00',\n",
       " '27/02/2018 12:00',\n",
       " '27/02/2018 13:00',\n",
       " '27/02/2018 14:00',\n",
       " '27/02/2018 15:00',\n",
       " '27/02/2018 16:00',\n",
       " '27/02/2018 17:00',\n",
       " '27/02/2018 18:00',\n",
       " '27/02/2018 19:00',\n",
       " '27/02/2018 20:00',\n",
       " '27/02/2018 21:00',\n",
       " '27/02/2018 22:00',\n",
       " '27/02/2018 23:00',\n",
       " '28/02/2018 00:00',\n",
       " '28/02/2018 01:00',\n",
       " '28/02/2018 02:00',\n",
       " '28/02/2018 03:00',\n",
       " '28/02/2018 04:00',\n",
       " '28/02/2018 05:00',\n",
       " '28/02/2018 06:00',\n",
       " '28/02/2018 07:00',\n",
       " '28/02/2018 08:00',\n",
       " '28/02/2018 09:00',\n",
       " '28/02/2018 10:00',\n",
       " '28/02/2018 11:00',\n",
       " '28/02/2018 12:00',\n",
       " '28/02/2018 13:00',\n",
       " '28/02/2018 14:00',\n",
       " '28/02/2018 15:00',\n",
       " '28/02/2018 16:00',\n",
       " '28/02/2018 17:00',\n",
       " '28/02/2018 18:00',\n",
       " '28/02/2018 19:00',\n",
       " '28/02/2018 20:00',\n",
       " '28/02/2018 21:00',\n",
       " '28/02/2018 22:00',\n",
       " '28/02/2018 23:00',\n",
       " '01/03/2018 00:00',\n",
       " '01/03/2018 01:00',\n",
       " '01/03/2018 02:00',\n",
       " '01/03/2018 03:00',\n",
       " '01/03/2018 04:00',\n",
       " '01/03/2018 05:00',\n",
       " '01/03/2018 06:00',\n",
       " '01/03/2018 07:00',\n",
       " '01/03/2018 08:00',\n",
       " '01/03/2018 09:00',\n",
       " '02/03/2018 10:00',\n",
       " '02/03/2018 11:00',\n",
       " '02/03/2018 12:00',\n",
       " '02/03/2018 13:00',\n",
       " '02/03/2018 14:00',\n",
       " '02/03/2018 15:00',\n",
       " '02/03/2018 16:00',\n",
       " '02/03/2018 17:00',\n",
       " '02/03/2018 18:00',\n",
       " '02/03/2018 19:00',\n",
       " '02/03/2018 20:00',\n",
       " '02/03/2018 21:00',\n",
       " '02/03/2018 22:00',\n",
       " '02/03/2018 23:00',\n",
       " '03/03/2018 00:00',\n",
       " '03/03/2018 01:00',\n",
       " '03/03/2018 02:00',\n",
       " '03/03/2018 03:00',\n",
       " '03/03/2018 04:00',\n",
       " '03/03/2018 05:00',\n",
       " '03/03/2018 06:00',\n",
       " '03/03/2018 07:00',\n",
       " '03/03/2018 08:00',\n",
       " '03/03/2018 09:00',\n",
       " '03/03/2018 10:00',\n",
       " '03/03/2018 11:00',\n",
       " '03/03/2018 12:00',\n",
       " '03/03/2018 13:00',\n",
       " '03/03/2018 14:00',\n",
       " '03/03/2018 15:00',\n",
       " '03/03/2018 16:00',\n",
       " '03/03/2018 17:00',\n",
       " '03/03/2018 18:00',\n",
       " '03/03/2018 19:00',\n",
       " '03/03/2018 20:00',\n",
       " '03/03/2018 21:00',\n",
       " '03/03/2018 22:00',\n",
       " '03/03/2018 23:00',\n",
       " '04/03/2018 00:00',\n",
       " '04/03/2018 01:00',\n",
       " '04/03/2018 02:00',\n",
       " '04/03/2018 03:00',\n",
       " '04/03/2018 04:00',\n",
       " '04/03/2018 05:00',\n",
       " '04/03/2018 06:00',\n",
       " '04/03/2018 07:00',\n",
       " '04/03/2018 08:00',\n",
       " '04/03/2018 09:00',\n",
       " '17/03/2018 10:00',\n",
       " '17/03/2018 11:00',\n",
       " '17/03/2018 12:00',\n",
       " '17/03/2018 13:00',\n",
       " '17/03/2018 14:00',\n",
       " '17/03/2018 15:00',\n",
       " '17/03/2018 16:00',\n",
       " '17/03/2018 17:00',\n",
       " '17/03/2018 18:00',\n",
       " '17/03/2018 19:00',\n",
       " '17/03/2018 20:00',\n",
       " '17/03/2018 21:00',\n",
       " '17/03/2018 22:00',\n",
       " '17/03/2018 23:00',\n",
       " '18/03/2018 00:00',\n",
       " '18/03/2018 01:00',\n",
       " '18/03/2018 02:00',\n",
       " '18/03/2018 03:00',\n",
       " '18/03/2018 04:00',\n",
       " '18/03/2018 05:00',\n",
       " '18/03/2018 06:00',\n",
       " '18/03/2018 07:00',\n",
       " '18/03/2018 08:00',\n",
       " '18/03/2018 09:00',\n",
       " '18/03/2018 10:00',\n",
       " '18/03/2018 11:00',\n",
       " '18/03/2018 12:00',\n",
       " '18/03/2018 13:00',\n",
       " '18/03/2018 14:00',\n",
       " '18/03/2018 15:00',\n",
       " '18/03/2018 16:00',\n",
       " '18/03/2018 17:00',\n",
       " '18/03/2018 18:00',\n",
       " '18/03/2018 19:00',\n",
       " '18/03/2018 20:00',\n",
       " '18/03/2018 21:00',\n",
       " '18/03/2018 22:00',\n",
       " '18/03/2018 23:00',\n",
       " '19/03/2018 00:00',\n",
       " '19/03/2018 01:00',\n",
       " '19/03/2018 02:00',\n",
       " '19/03/2018 03:00',\n",
       " '19/03/2018 04:00',\n",
       " '19/03/2018 05:00',\n",
       " '19/03/2018 06:00',\n",
       " '19/03/2018 07:00',\n",
       " '19/03/2018 08:00',\n",
       " '19/03/2018 09:00',\n",
       " '19/03/2018 10:00',\n",
       " '19/03/2018 11:00',\n",
       " '19/03/2018 12:00',\n",
       " '19/03/2018 13:00',\n",
       " '19/03/2018 14:00',\n",
       " '19/03/2018 15:00',\n",
       " '19/03/2018 16:00',\n",
       " '19/03/2018 17:00',\n",
       " '19/03/2018 18:00',\n",
       " '19/03/2018 19:00',\n",
       " '19/03/2018 20:00',\n",
       " '19/03/2018 21:00',\n",
       " '19/03/2018 22:00',\n",
       " '19/03/2018 23:00',\n",
       " '20/03/2018 00:00',\n",
       " '20/03/2018 01:00',\n",
       " '20/03/2018 02:00',\n",
       " '20/03/2018 03:00',\n",
       " '20/03/2018 04:00',\n",
       " '20/03/2018 05:00',\n",
       " '20/03/2018 06:00',\n",
       " '20/03/2018 07:00',\n",
       " '20/03/2018 08:00',\n",
       " '20/03/2018 09:00',\n",
       " 'end data']"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2018w = pd.read_csv('uk-hourly-weather-obs2018noleg.csv', engine='python')\n",
    "Weath2018w1 = Weath2018w[['ob_time', 'wind_direction', 'wind_speed', 'cld_ttl_amt_id', 'visibility', 'air_temperature',  \n",
    "    'rltv_hum', 'stn_pres', 'wmo_hr_sun_dur']]\n",
    "Weath2018r = pd.read_csv('uk-hourly-rain-obs_2018noleg.csv', engine='python')\n",
    "Weath2018r1 = Weath2018r[Weath2018r['ob_hour_count'] == 1]\n",
    "miss2018 = []\n",
    "for i in Weath2018w1['ob_time']:\n",
    "    if i not in list(Weath2018r1['ob_end_time']):\n",
    "        miss2018.append(i)\n",
    "miss2018"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:32:05.238826Z",
     "start_time": "2021-03-05T16:31:55.255554Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['22/01/2018 15:00',\n",
       " '22/01/2018 16:00',\n",
       " '08/02/2018 13:00',\n",
       " '15/02/2018 15:00',\n",
       " '29/06/2018 14:00',\n",
       " '29/06/2018 15:00']"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2018r1 = Weath2018r1.rename(columns={\"ob_end_time\": \"ob_time\"}, errors=\"raise\")\n",
    "df4=Weath2018r1[['ob_time','prcp_amt']]\n",
    "        \n",
    "Weath2018w1 = Weath2018w1.merge(df4,how = 'left', on = 'ob_time')\n",
    "Weath2018wind = pd.read_csv('uk-mean-wind-obs_2018noleg.csv', engine='python')\n",
    "Weath2018wind1 = Weath2018wind[['ob_end_time','max_gust_speed']]\n",
    "Weath2018wind1 = Weath2018wind1.rename(columns={\"ob_end_time\": \"ob_time\"}, errors=\"raise\")\n",
    "Weath2018w1 = Weath2018w1.merge(Weath2018wind1,how = 'left', on = 'ob_time')\n",
    "miss2018wind = []\n",
    "for i in Weath2018w1['ob_time']:\n",
    "    if i not in list(Weath2018wind1['ob_time']):\n",
    "        miss2018wind.append(i)\n",
    "miss2018wind"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:32:12.400111Z",
     "start_time": "2021-03-05T16:32:10.812022Z"
    }
   },
   "outputs": [],
   "source": [
    "Weath2018w2 = Weath2018w1.drop(index = (len(Weath2018w1)-1))\n",
    "Weath2018w2['ob_time'] = pd.to_datetime(Weath2018w2.ob_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:32:21.253122Z",
     "start_time": "2021-03-05T16:32:15.106564Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\andre\\anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:670: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  iloc._setitem_with_indexer(indexer, value)\n"
     ]
    }
   ],
   "source": [
    "# Importing missing data (if needed)\n",
    "from meteostat import Stations, Daily, Hourly\n",
    "from datetime import datetime\n",
    "# Set time period\n",
    "start = datetime(2018, 2, 25)\n",
    "end = datetime(2018, 3, 20, 9, 59)\n",
    "\n",
    "# Get hourly data\n",
    "data = Hourly('03772', start, end)\n",
    "data2018weathr = data.fetch()\n",
    "\n",
    "for i in data2018weathr.index:\n",
    "    Weath2018w2[['prcp_amt']].loc[i] = data2018weathr[['prcp']].loc[i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:32:24.481900Z",
     "start_time": "2021-03-05T16:32:23.777252Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\andre\\anaconda3\\lib\\site-packages\\pandas\\core\\generic.py:5168: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self[name] = value\n",
      "<ipython-input-66-99c49d08017c>:5: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  Sunrise20181['ob_time1'] = Sunrise20181['Date'].dt.date\n"
     ]
    }
   ],
   "source": [
    "Sunrise2018 = pd.read_csv('sunrise2018.csv', engine='python')\n",
    "Sunrise20181 = Sunrise2018[['Date','Sunrise Time (LST)','Sunset Time (LST)']]\n",
    "Sunrise20181.Date = pd.to_datetime(Sunrise20181.Date)\n",
    "Weath2018w2['ob_time1'] = Weath2018w2.ob_time.dt.date\n",
    "Sunrise20181['ob_time1'] = Sunrise20181['Date'].dt.date\n",
    "Weath2018w2 = Weath2018w2.merge(Sunrise20181,how = 'left', on = 'ob_time1')\n",
    "Weath2018w2['sunrisehr']=Weath2018w2['Sunrise Time (LST)'].str.replace(r':\\d\\d:\\d\\d','').astype(int)\n",
    "Weath2018w2['sunsethr']=Weath2018w2['Sunset Time (LST)'].str.replace(r':\\d\\d:\\d\\d','').astype(int)\n",
    "\n",
    "Weath2018w2['islight'] = 0\n",
    "Weath2018w2['islight'] = [1 if ((Weath2018w2['sunrisehr'][i] <= Weath2018w2['ob_time'][i].hour) \n",
    "                                and (Weath2018w2['ob_time'][i].hour < Weath2018w2['sunsethr'][i])) \n",
    "                          else 0 for i in Weath2018w2.index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Daylight saving time 2018 in United Kingdom began at 01:00 on\n",
    "Sunday, 25 March\n",
    "and ended at 02:00 on\n",
    "Sunday, 28 October"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:32:30.170231Z",
     "start_time": "2021-03-05T16:32:30.137252Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7191</th>\n",
       "      <td>2018-10-28 02:00:00</td>\n",
       "      <td>360.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3500.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>88.2</td>\n",
       "      <td>1013.6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>2018-10-28</td>\n",
       "      <td>2018-10-28</td>\n",
       "      <td>6:46:34</td>\n",
       "      <td>16:40:56</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time  wind_direction  wind_speed  cld_ttl_amt_id  \\\n",
       "7191 2018-10-28 02:00:00           360.0         8.0             0.0   \n",
       "\n",
       "      visibility  air_temperature  rltv_hum  stn_pres  wmo_hr_sun_dur  \\\n",
       "7191      3500.0              4.0      88.2    1013.6             0.0   \n",
       "\n",
       "      prcp_amt  max_gust_speed    ob_time1       Date Sunrise Time (LST)  \\\n",
       "7191       0.0            17.0  2018-10-28 2018-10-28            6:46:34   \n",
       "\n",
       "     Sunset Time (LST)  sunrisehr  sunsethr  islight  \n",
       "7191          16:40:56          6        16        0  "
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2018w2[Weath2018w2['ob_time'] == '2018-10-28 02:00:00']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:32:32.036224Z",
     "start_time": "2021-03-05T16:32:31.969249Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1990</th>\n",
       "      <td>2018-03-25 01:00:00</td>\n",
       "      <td>10.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>380.0</td>\n",
       "      <td>7.6</td>\n",
       "      <td>91.4</td>\n",
       "      <td>1002.8</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2018-03-25</td>\n",
       "      <td>2018-03-25</td>\n",
       "      <td>5:51:02</td>\n",
       "      <td>18:20:54</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time  wind_direction  wind_speed  cld_ttl_amt_id  \\\n",
       "1990 2018-03-25 01:00:00            10.0         3.0             8.0   \n",
       "\n",
       "      visibility  air_temperature  rltv_hum  stn_pres  wmo_hr_sun_dur  \\\n",
       "1990       380.0              7.6      91.4    1002.8             0.0   \n",
       "\n",
       "      prcp_amt  max_gust_speed    ob_time1       Date Sunrise Time (LST)  \\\n",
       "1990       0.0             7.0  2018-03-25 2018-03-25            5:51:02   \n",
       "\n",
       "     Sunset Time (LST)  sunrisehr  sunsethr  islight  \n",
       "1990          18:20:54          5        18        0  "
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2018w3 = pd.concat([Weath2018w2.head(7191), pd.DataFrame(Weath2018w2.iloc[7191]).T, Weath2018w2.tail(len(Weath2018w2) - (7191))])\n",
    "Weath2018w3.reset_index(inplace=True)\n",
    "Weath2018w3.drop('index', axis=1, inplace=True)\n",
    "Weath2018w2[Weath2018w2['ob_time'] == '2018-03-25 01:00:00']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:32:37.987151Z",
     "start_time": "2021-03-05T16:32:36.543195Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-69-be11a55ad7d3>:3: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  Weath2018w4['ob_time'][i+1] = Weath2018w3['ob_time'][i]\n"
     ]
    }
   ],
   "source": [
    "Weath2018w4 = Weath2018w3.copy()\n",
    "for i in range(1991,7191):\n",
    "    Weath2018w4['ob_time'][i+1] = Weath2018w3['ob_time'][i]\n",
    "Weath2018w5 = Weath2018w4.drop(1991)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:32:41.259215Z",
     "start_time": "2021-03-05T16:32:41.205247Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1990</th>\n",
       "      <td>2018-03-25 01:00:00</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>380</td>\n",
       "      <td>7.6</td>\n",
       "      <td>91.4</td>\n",
       "      <td>1002.8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>2018-03-25</td>\n",
       "      <td>2018-03-25</td>\n",
       "      <td>5:51:02</td>\n",
       "      <td>18:20:54</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1990</th>\n",
       "      <td>2018-03-25 01:00:00</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>380</td>\n",
       "      <td>7.6</td>\n",
       "      <td>91.4</td>\n",
       "      <td>1002.8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>2018-03-25</td>\n",
       "      <td>2018-03-25</td>\n",
       "      <td>5:51:02</td>\n",
       "      <td>18:20:54</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1992</th>\n",
       "      <td>2018-03-25 03:00:00</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>280</td>\n",
       "      <td>7.3</td>\n",
       "      <td>92.7</td>\n",
       "      <td>1003.9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>2018-03-25</td>\n",
       "      <td>2018-03-25</td>\n",
       "      <td>5:51:02</td>\n",
       "      <td>18:20:54</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time wind_direction wind_speed cld_ttl_amt_id visibility  \\\n",
       "1990 2018-03-25 01:00:00             10          3              8        380   \n",
       "1990 2018-03-25 01:00:00             10          3              8        380   \n",
       "1992 2018-03-25 03:00:00             10          4              8        280   \n",
       "\n",
       "     air_temperature rltv_hum stn_pres wmo_hr_sun_dur prcp_amt max_gust_speed  \\\n",
       "1990             7.6     91.4   1002.8              0        0              7   \n",
       "1990             7.6     91.4   1002.8              0        0              7   \n",
       "1992             7.3     92.7   1003.9              0        0              8   \n",
       "\n",
       "        ob_time1       Date Sunrise Time (LST) Sunset Time (LST) sunrisehr  \\\n",
       "1990  2018-03-25 2018-03-25            5:51:02          18:20:54         5   \n",
       "1990  2018-03-25 2018-03-25            5:51:02          18:20:54         5   \n",
       "1992  2018-03-25 2018-03-25            5:51:02          18:20:54         5   \n",
       "\n",
       "     sunsethr islight  \n",
       "1990       18       0  \n",
       "1990       18       0  \n",
       "1992       18       0  "
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2018w5.reset_index(inplace=True)\n",
    "Weath2018w5.drop('index', axis=1, inplace=True)\n",
    "Weath2018w5.iloc[[1990,1990,1992]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:32:44.233737Z",
     "start_time": "2021-03-05T16:32:44.197758Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 8749 entries, 0 to 8748\n",
      "Data columns (total 12 columns):\n",
      " #   Column           Non-Null Count  Dtype         \n",
      "---  ------           --------------  -----         \n",
      " 0   ob_time          8749 non-null   datetime64[ns]\n",
      " 1   wind_direction   8749 non-null   float64       \n",
      " 2   wind_speed       8749 non-null   float64       \n",
      " 3   cld_ttl_amt_id   8749 non-null   float64       \n",
      " 4   visibility       8749 non-null   float64       \n",
      " 5   air_temperature  8749 non-null   float64       \n",
      " 6   rltv_hum         8749 non-null   float64       \n",
      " 7   stn_pres         8749 non-null   float64       \n",
      " 8   wmo_hr_sun_dur   8749 non-null   float64       \n",
      " 9   prcp_amt         8749 non-null   float64       \n",
      " 10  max_gust_speed   8749 non-null   float64       \n",
      " 11  islight          8749 non-null   int64         \n",
      "dtypes: datetime64[ns](1), float64(10), int64(1)\n",
      "memory usage: 820.3 KB\n"
     ]
    }
   ],
   "source": [
    "Weath2018w6 = Weath2018w5.drop(columns = ['ob_time1','Date','Sunrise Time (LST)', 'Sunset Time (LST)', 'sunrisehr', 'sunsethr'])\n",
    "Weath2018w6.fillna(method = 'backfill', inplace = True)\n",
    "Weath2018w6.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2019 Weather ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:32:58.429852Z",
     "start_time": "2021-03-05T16:32:49.073202Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['24/04/2019 10:00',\n",
       " '24/04/2019 11:00',\n",
       " '24/04/2019 12:00',\n",
       " '24/04/2019 13:00',\n",
       " '24/04/2019 14:00',\n",
       " '24/04/2019 15:00',\n",
       " '24/04/2019 16:00',\n",
       " '24/04/2019 17:00',\n",
       " '24/04/2019 18:00',\n",
       " '24/04/2019 19:00',\n",
       " '24/04/2019 20:00',\n",
       " '24/04/2019 21:00',\n",
       " '24/04/2019 22:00',\n",
       " '24/04/2019 23:00',\n",
       " '25/04/2019 00:00',\n",
       " '25/04/2019 01:00',\n",
       " '25/04/2019 02:00',\n",
       " '25/04/2019 03:00',\n",
       " '25/04/2019 04:00',\n",
       " '25/04/2019 05:00',\n",
       " '25/04/2019 06:00',\n",
       " '25/04/2019 07:00',\n",
       " '25/04/2019 08:00',\n",
       " '25/04/2019 09:00',\n",
       " 'end data']"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2019w = pd.read_csv('uk-hourly-weather-obs2019noleg.csv', engine='python')\n",
    "Weath2019w1 = Weath2019w[['ob_time', 'wind_direction', 'wind_speed', 'cld_ttl_amt_id', 'visibility', 'air_temperature',  \n",
    "    'rltv_hum', 'stn_pres', 'wmo_hr_sun_dur']]\n",
    "Weath2019r = pd.read_csv('uk-hourly-rain-obs_2019noleg.csv', engine='python')\n",
    "Weath2019r1 = Weath2019r[Weath2019r['ob_hour_count'] == 1]\n",
    "miss2019 = []\n",
    "for i in Weath2019w1['ob_time']:\n",
    "    if i not in list(Weath2019r1['ob_end_time']):\n",
    "        miss2019.append(i)\n",
    "miss2019"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:33:22.139131Z",
     "start_time": "2021-03-05T16:33:12.434699Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['17/04/2019 22:00',\n",
       " '25/04/2019 01:00',\n",
       " '08/05/2019 22:00',\n",
       " '08/05/2019 23:00',\n",
       " '09/05/2019 00:00',\n",
       " '09/05/2019 01:00',\n",
       " '09/05/2019 02:00',\n",
       " '15/05/2019 22:00',\n",
       " '15/05/2019 23:00',\n",
       " '16/05/2019 02:00',\n",
       " '24/05/2019 12:00',\n",
       " '24/05/2019 14:00',\n",
       " '24/05/2019 15:00',\n",
       " '03/07/2019 16:00',\n",
       " '11/10/2019 07:00',\n",
       " '11/10/2019 08:00']"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2019r1 = Weath2019r1.rename(columns={\"ob_end_time\": \"ob_time\"}, errors=\"raise\")\n",
    "df5=Weath2019r1[['ob_time','prcp_amt']]\n",
    "        \n",
    "Weath2019w1 = Weath2019w1.merge(df5,how = 'left', on = 'ob_time')\n",
    "Weath2019wind = pd.read_csv('uk-mean-wind-obs_2019noleg.csv', engine='python')\n",
    "Weath2019wind1 = Weath2019wind[['ob_end_time','max_gust_speed']]\n",
    "Weath2019wind1 = Weath2019wind1.rename(columns={\"ob_end_time\": \"ob_time\"}, errors=\"raise\")\n",
    "Weath2019w1 = Weath2019w1.merge(Weath2019wind1,how = 'left', on = 'ob_time')\n",
    "miss2019wind = []\n",
    "for i in Weath2019w1['ob_time']:\n",
    "    if i not in list(Weath2019wind1['ob_time']):\n",
    "        miss2019wind.append(i)\n",
    "miss2019wind"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:33:26.989558Z",
     "start_time": "2021-03-05T16:33:25.949120Z"
    }
   },
   "outputs": [],
   "source": [
    "Weath2019w2 = Weath2019w1.drop(index = (len(Weath2019w1)-1))\n",
    "Weath2019w2['ob_time'] = pd.to_datetime(Weath2019w2.ob_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:33:32.847380Z",
     "start_time": "2021-03-05T16:33:28.731877Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\andre\\anaconda3\\lib\\site-packages\\pandas\\core\\indexing.py:670: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  iloc._setitem_with_indexer(indexer, value)\n"
     ]
    }
   ],
   "source": [
    "# Importing missing data (if needed)\n",
    "from meteostat import Stations, Daily, Hourly\n",
    "from datetime import datetime\n",
    "# Set time period\n",
    "start = datetime(2019, 4, 24)\n",
    "end = datetime(2019, 4, 25, 9, 59)\n",
    "\n",
    "# Get hourly data\n",
    "data = Hourly('03772', start, end)\n",
    "data2019weathr = data.fetch()\n",
    "\n",
    "for i in data2019weathr.index:\n",
    "    Weath2019w2[['prcp_amt']].loc[i] = data2019weathr[['prcp']].loc[i]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:33:36.988631Z",
     "start_time": "2021-03-05T16:33:36.327007Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\andre\\anaconda3\\lib\\site-packages\\pandas\\core\\generic.py:5168: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self[name] = value\n",
      "<ipython-input-76-1455fba9065f>:5: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  Sunrise20191['ob_time1'] = Sunrise20191['Date'].dt.date\n"
     ]
    }
   ],
   "source": [
    "Sunrise2019 = pd.read_csv('sunrise2019.csv', engine='python')\n",
    "Sunrise20191 = Sunrise2019[['Date','Sunrise Time (LST)','Sunset Time (LST)']]\n",
    "Sunrise20191.Date = pd.to_datetime(Sunrise20191.Date)\n",
    "Weath2019w2['ob_time1'] = Weath2019w2.ob_time.dt.date\n",
    "Sunrise20191['ob_time1'] = Sunrise20191['Date'].dt.date\n",
    "Weath2019w2 = Weath2019w2.merge(Sunrise20191,how = 'left', on = 'ob_time1')\n",
    "Weath2019w2['sunrisehr']=Weath2019w2['Sunrise Time (LST)'].str.replace(r':\\d\\d:\\d\\d','').astype(int)\n",
    "Weath2019w2['sunsethr']=Weath2019w2['Sunset Time (LST)'].str.replace(r':\\d\\d:\\d\\d','').astype(int)\n",
    "\n",
    "Weath2019w2['islight'] = 0\n",
    "Weath2019w2['islight'] = [1 if ((Weath2019w2['sunrisehr'][i] <= Weath2019w2['ob_time'][i].hour) \n",
    "                                and (Weath2019w2['ob_time'][i].hour < Weath2019w2['sunsethr'][i])) \n",
    "                          else 0 for i in Weath2019w2.index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Daylight saving time 2019 in United Kingdom began at 01:00 on\n",
    "Sunday, 31 March\n",
    "and ended at 02:00 on\n",
    "Sunday, 27 October"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:33:41.954014Z",
     "start_time": "2021-03-05T16:33:41.914043Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7175</th>\n",
       "      <td>2019-10-27 02:00:00</td>\n",
       "      <td>290.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1600.0</td>\n",
       "      <td>6.1</td>\n",
       "      <td>90.7</td>\n",
       "      <td>1016.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2019-10-27</td>\n",
       "      <td>2019-10-27</td>\n",
       "      <td>6:44:23</td>\n",
       "      <td>16:43:19</td>\n",
       "      <td>6</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time  wind_direction  wind_speed  cld_ttl_amt_id  \\\n",
       "7175 2019-10-27 02:00:00           290.0         7.0             3.0   \n",
       "\n",
       "      visibility  air_temperature  rltv_hum  stn_pres  wmo_hr_sun_dur  \\\n",
       "7175      1600.0              6.1      90.7    1016.4             0.0   \n",
       "\n",
       "      prcp_amt  max_gust_speed    ob_time1       Date Sunrise Time (LST)  \\\n",
       "7175       0.0            10.0  2019-10-27 2019-10-27            6:44:23   \n",
       "\n",
       "     Sunset Time (LST)  sunrisehr  sunsethr  islight  \n",
       "7175          16:43:19          6        16        0  "
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2019w2[Weath2019w2['ob_time'] == '2019-10-27 02:00:00']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:33:45.072347Z",
     "start_time": "2021-03-05T16:33:45.018361Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2137</th>\n",
       "      <td>2019-03-31 01:00:00</td>\n",
       "      <td>30.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>600.0</td>\n",
       "      <td>10.5</td>\n",
       "      <td>70.9</td>\n",
       "      <td>1018.9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2019-03-31</td>\n",
       "      <td>2019-03-31</td>\n",
       "      <td>5:37:54</td>\n",
       "      <td>18:30:33</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time  wind_direction  wind_speed  cld_ttl_amt_id  \\\n",
       "2137 2019-03-31 01:00:00            30.0         6.0             2.0   \n",
       "\n",
       "      visibility  air_temperature  rltv_hum  stn_pres  wmo_hr_sun_dur  \\\n",
       "2137       600.0             10.5      70.9    1018.9             0.0   \n",
       "\n",
       "      prcp_amt  max_gust_speed    ob_time1       Date Sunrise Time (LST)  \\\n",
       "2137       0.0            10.0  2019-03-31 2019-03-31            5:37:54   \n",
       "\n",
       "     Sunset Time (LST)  sunrisehr  sunsethr  islight  \n",
       "2137          18:30:33          5        18        0  "
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2019w3 = pd.concat([Weath2019w2.head(7175), pd.DataFrame(Weath2019w2.iloc[7175]).T, Weath2019w2.tail(len(Weath2019w2) - (7175))])\n",
    "Weath2019w3.reset_index(inplace=True)\n",
    "Weath2019w3.drop('index', axis=1, inplace=True)\n",
    "Weath2019w2[Weath2019w2['ob_time'] == '2019-03-31 01:00:00']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:33:47.825992Z",
     "start_time": "2021-03-05T16:33:46.780806Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-79-0a61460f5bfe>:3: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  Weath2019w4['ob_time'][i+1] = Weath2019w3['ob_time'][i]\n"
     ]
    }
   ],
   "source": [
    "Weath2019w4 = Weath2019w3.copy()\n",
    "for i in range(2138,7175):\n",
    "    Weath2019w4['ob_time'][i+1] = Weath2019w3['ob_time'][i]\n",
    "Weath2019w5 = Weath2019w4.drop(2138)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:33:48.919939Z",
     "start_time": "2021-03-05T16:33:48.875966Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ob_time</th>\n",
       "      <th>wind_direction</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>cld_ttl_amt_id</th>\n",
       "      <th>visibility</th>\n",
       "      <th>air_temperature</th>\n",
       "      <th>rltv_hum</th>\n",
       "      <th>stn_pres</th>\n",
       "      <th>wmo_hr_sun_dur</th>\n",
       "      <th>prcp_amt</th>\n",
       "      <th>max_gust_speed</th>\n",
       "      <th>ob_time1</th>\n",
       "      <th>Date</th>\n",
       "      <th>Sunrise Time (LST)</th>\n",
       "      <th>Sunset Time (LST)</th>\n",
       "      <th>sunrisehr</th>\n",
       "      <th>sunsethr</th>\n",
       "      <th>islight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2136</th>\n",
       "      <td>2019-03-31 00:00:00</td>\n",
       "      <td>10</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>700</td>\n",
       "      <td>11.3</td>\n",
       "      <td>65.8</td>\n",
       "      <td>1018.7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>14</td>\n",
       "      <td>2019-03-31</td>\n",
       "      <td>2019-03-31</td>\n",
       "      <td>5:37:54</td>\n",
       "      <td>18:30:33</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2137</th>\n",
       "      <td>2019-03-31 01:00:00</td>\n",
       "      <td>30</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>600</td>\n",
       "      <td>10.5</td>\n",
       "      <td>70.9</td>\n",
       "      <td>1018.9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>2019-03-31</td>\n",
       "      <td>2019-03-31</td>\n",
       "      <td>5:37:54</td>\n",
       "      <td>18:30:33</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2138</th>\n",
       "      <td>2019-03-31 02:00:00</td>\n",
       "      <td>40</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>280</td>\n",
       "      <td>9.6</td>\n",
       "      <td>80</td>\n",
       "      <td>1019.3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>2019-03-31</td>\n",
       "      <td>2019-03-31</td>\n",
       "      <td>5:37:54</td>\n",
       "      <td>18:30:33</td>\n",
       "      <td>5</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 ob_time wind_direction wind_speed cld_ttl_amt_id visibility  \\\n",
       "2136 2019-03-31 00:00:00             10          5              6        700   \n",
       "2137 2019-03-31 01:00:00             30          6              2        600   \n",
       "2138 2019-03-31 02:00:00             40          6              5        280   \n",
       "\n",
       "     air_temperature rltv_hum stn_pres wmo_hr_sun_dur prcp_amt max_gust_speed  \\\n",
       "2136            11.3     65.8   1018.7            NaN        0             14   \n",
       "2137            10.5     70.9   1018.9              0        0             10   \n",
       "2138             9.6       80   1019.3              0        0             11   \n",
       "\n",
       "        ob_time1       Date Sunrise Time (LST) Sunset Time (LST) sunrisehr  \\\n",
       "2136  2019-03-31 2019-03-31            5:37:54          18:30:33         5   \n",
       "2137  2019-03-31 2019-03-31            5:37:54          18:30:33         5   \n",
       "2138  2019-03-31 2019-03-31            5:37:54          18:30:33         5   \n",
       "\n",
       "     sunsethr islight  \n",
       "2136       18       0  \n",
       "2137       18       0  \n",
       "2138       18       0  "
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Weath2019w5.reset_index(inplace=True)\n",
    "Weath2019w5.drop('index', axis=1, inplace=True)\n",
    "Weath2019w5.iloc[[2136,2137,2138]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:33:50.856180Z",
     "start_time": "2021-03-05T16:33:50.814208Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 8757 entries, 0 to 8756\n",
      "Data columns (total 12 columns):\n",
      " #   Column           Non-Null Count  Dtype         \n",
      "---  ------           --------------  -----         \n",
      " 0   ob_time          8757 non-null   datetime64[ns]\n",
      " 1   wind_direction   8757 non-null   float64       \n",
      " 2   wind_speed       8757 non-null   float64       \n",
      " 3   cld_ttl_amt_id   8757 non-null   float64       \n",
      " 4   visibility       8757 non-null   float64       \n",
      " 5   air_temperature  8757 non-null   float64       \n",
      " 6   rltv_hum         8757 non-null   float64       \n",
      " 7   stn_pres         8757 non-null   float64       \n",
      " 8   wmo_hr_sun_dur   8757 non-null   float64       \n",
      " 9   prcp_amt         8757 non-null   float64       \n",
      " 10  max_gust_speed   8757 non-null   float64       \n",
      " 11  islight          8757 non-null   int64         \n",
      "dtypes: datetime64[ns](1), float64(10), int64(1)\n",
      "memory usage: 821.1 KB\n"
     ]
    }
   ],
   "source": [
    "Weath2019w6 = Weath2019w5.drop(columns = ['ob_time1','Date','Sunrise Time (LST)', 'Sunset Time (LST)', 'sunrisehr', 'sunsethr'])\n",
    "Weath2019w6.fillna(method = 'backfill', inplace = True)\n",
    "Weath2019w6.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2020 Weather ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:33:56.347336Z",
     "start_time": "2021-03-05T16:33:56.284373Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "DatetimeIndex: 8784 entries, 2020-01-01 00:00:00 to 2020-12-31 23:00:00\n",
      "Data columns (total 11 columns):\n",
      " #   Column  Non-Null Count  Dtype  \n",
      "---  ------  --------------  -----  \n",
      " 0   temp    8784 non-null   float64\n",
      " 1   dwpt    8784 non-null   float64\n",
      " 2   rhum    8784 non-null   float64\n",
      " 3   prcp    8784 non-null   float64\n",
      " 4   snow    0 non-null      float64\n",
      " 5   wdir    8784 non-null   float64\n",
      " 6   wspd    8784 non-null   float64\n",
      " 7   wpgt    8784 non-null   float64\n",
      " 8   pres    8784 non-null   float64\n",
      " 9   tsun    0 non-null      float64\n",
      " 10  coco    8784 non-null   float64\n",
      "dtypes: float64(11)\n",
      "memory usage: 823.5 KB\n"
     ]
    }
   ],
   "source": [
    "from meteostat import Hourly, Daily\n",
    "from meteostat import Stations\n",
    "from datetime import datetime\n",
    "# Set time period\n",
    "start = datetime(2020, 1, 1)\n",
    "end = datetime(2020, 12, 31, 23, 59)\n",
    "\n",
    "# Get hourly data\n",
    "data = Hourly('03772', start, end)\n",
    "data2020weath = data.fetch()\n",
    "\n",
    "# Print DataFrame\n",
    "data2020weath.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# table for converting weather codes to cloud cover and sun (to get closer to met office columns)\n",
    "\n",
    "Coco: Cloud     Sun\n",
    "1       0        1\n",
    "2\t2\n",
    "3\t6\n",
    "4\t8\n",
    "5\t8\n",
    "6\t8\n",
    "7\t8\n",
    "\n",
    "\n",
    "\n",
    "Code\tWeather Condition\n",
    "1\tClear\n",
    "2\tFair\n",
    "3\tCloudy\n",
    "4\tOvercast\n",
    "5\tFog\n",
    "6\tFreezing Fog\n",
    "7\tLight Rain\n",
    "8\tRain\n",
    "9\tHeavy Rain\n",
    "10\tFreezing Rain\n",
    "11\tHeavy Freezing Rain\n",
    "12\tSleet\n",
    "13\tHeavy Sleet\n",
    "14\tLight Snowfall\n",
    "15\tSnowfall\n",
    "16\tHeavy Snowfall\n",
    "17\tRain Shower\n",
    "18\tHeavy Rain Shower\n",
    "19\tSleet Shower\n",
    "20\tHeavy Sleet Shower\n",
    "21\tSnow Shower\n",
    "22\tHeavy Snow Shower\n",
    "23\tLightning\n",
    "24\tHail\n",
    "25\tThunderstorm\n",
    "26\tHeavy Thunderstorm\n",
    "27\tStorm\n",
    "\n",
    "Visibility measures\n",
    "\n",
    "VP (Very Poor)\tLess than 1,000 metres\n",
    "P (Poor)\tBetween 1,001 and 4,000 metres\n",
    "M (Medium)\tBetween 4,001 and 10,000 metres\n",
    "G (Good)\tBetween 10,001 and 20,000 metres\n",
    "VG (Very Good)\tBetween 20,001 and 40,000 metres\n",
    "E (Excellent)\tGreater than 40,000 metres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:34:04.825496Z",
     "start_time": "2021-03-05T16:34:01.677925Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-83-4a66c51af48f>:20: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['visibility'][i] = 100\n",
      "<ipython-input-83-4a66c51af48f>:21: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['cld_ttl_amt_id'][i] = 8\n",
      "<ipython-input-83-4a66c51af48f>:17: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['visibility'][i] = 4000\n",
      "<ipython-input-83-4a66c51af48f>:18: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['cld_ttl_amt_id'][i] = 8\n",
      "<ipython-input-83-4a66c51af48f>:35: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['visibility'][i] = 1000\n",
      "<ipython-input-83-4a66c51af48f>:36: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['cld_ttl_amt_id'][i] = 8\n",
      "<ipython-input-83-4a66c51af48f>:29: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['visibility'][i] = 2000\n",
      "<ipython-input-83-4a66c51af48f>:30: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['cld_ttl_amt_id'][i] = 8\n",
      "<ipython-input-83-4a66c51af48f>:26: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['visibility'][i] = 2000\n",
      "<ipython-input-83-4a66c51af48f>:27: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['cld_ttl_amt_id'][i] = 8\n",
      "<ipython-input-83-4a66c51af48f>:32: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['visibility'][i] = 2000\n",
      "<ipython-input-83-4a66c51af48f>:33: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['cld_ttl_amt_id'][i] = 8\n",
      "<ipython-input-83-4a66c51af48f>:14: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['visibility'][i] = 4000\n",
      "<ipython-input-83-4a66c51af48f>:15: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['cld_ttl_amt_id'][i] = 6\n",
      "<ipython-input-83-4a66c51af48f>:10: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['visibility'][i] = 4000\n",
      "<ipython-input-83-4a66c51af48f>:11: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['cld_ttl_amt_id'][i] = 2\n",
      "<ipython-input-83-4a66c51af48f>:12: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['sun'][i] = 0.5\n",
      "<ipython-input-83-4a66c51af48f>:6: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['visibility'][i] = 5000\n",
      "<ipython-input-83-4a66c51af48f>:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['cld_ttl_amt_id'][i] = 0\n",
      "<ipython-input-83-4a66c51af48f>:8: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data2020weath['sun'][i] = 1.0\n"
     ]
    }
   ],
   "source": [
    "data2020weath['sun'] = 0\n",
    "data2020weath['visibility'] = 0\n",
    "data2020weath['cld_ttl_amt_id'] = 0\n",
    "for i in range(len(data2020weath.coco)):\n",
    "    if data2020weath.coco[i] == 1.0:\n",
    "        data2020weath['visibility'][i] = 5000\n",
    "        data2020weath['cld_ttl_amt_id'][i] = 0\n",
    "        data2020weath['sun'][i] = 1.0\n",
    "    elif data2020weath.coco[i] == 2.0:\n",
    "        data2020weath['visibility'][i] = 4000\n",
    "        data2020weath['cld_ttl_amt_id'][i] = 2\n",
    "        data2020weath['sun'][i] = 0.5\n",
    "    elif data2020weath.coco[i] == 3.0:\n",
    "        data2020weath['visibility'][i] = 4000\n",
    "        data2020weath['cld_ttl_amt_id'][i] = 6\n",
    "    elif data2020weath.coco[i] == 4.0:\n",
    "        data2020weath['visibility'][i] = 4000\n",
    "        data2020weath['cld_ttl_amt_id'][i] = 8\n",
    "    elif data2020weath.coco[i] == 5.0:\n",
    "        data2020weath['visibility'][i] = 100\n",
    "        data2020weath['cld_ttl_amt_id'][i] = 8\n",
    "    elif data2020weath.coco[i] == 6.0:\n",
    "        data2020weath['visibility'][i] = 100\n",
    "        data2020weath['cld_ttl_amt_id'][i] = 8\n",
    "    elif data2020weath.coco[i] == 7.0:\n",
    "        data2020weath['visibility'][i] = 2000\n",
    "        data2020weath['cld_ttl_amt_id'][i] = 8\n",
    "    elif data2020weath.coco[i] == 8.0:\n",
    "        data2020weath['visibility'][i] = 2000\n",
    "        data2020weath['cld_ttl_amt_id'][i] = 8\n",
    "    elif data2020weath.coco[i] == 17.0:\n",
    "        data2020weath['visibility'][i] = 2000\n",
    "        data2020weath['cld_ttl_amt_id'][i] = 8\n",
    "    else: \n",
    "        data2020weath['visibility'][i] = 1000\n",
    "        data2020weath['cld_ttl_amt_id'][i] = 8\n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:34:09.385941Z",
     "start_time": "2021-03-05T16:34:09.364953Z"
    }
   },
   "outputs": [],
   "source": [
    "# Correct units for windspeed\n",
    "data2020weath['wind_speed'] = data2020weath['wspd']*0.539957"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:34:22.950884Z",
     "start_time": "2021-03-05T16:34:22.938890Z"
    }
   },
   "outputs": [],
   "source": [
    "# Renaming columns\n",
    "data2020weath = data2020weath.rename(columns={\"wdir\": \"wind_direction\",\n",
    "                                              \"temp\":\"air_temperature\", \"rhum\":\"rltv_hum\",\"pres\":\"stn_pres\", \n",
    "                                              \"sun\":\"wmo_hr_sun_dur\", \"prcp\":\"prcp_amt\",\n",
    "                                              \"wpgt\":\"max_gust_speed\"}, errors=\"raise\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:34:24.954824Z",
     "start_time": "2021-03-05T16:34:24.936836Z"
    }
   },
   "outputs": [],
   "source": [
    "# Dropping irrelevant columns\n",
    "data2020weatha = data2020weath.drop(columns = ['dwpt','snow','wspd','tsun','coco'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:34:27.097812Z",
     "start_time": "2021-03-05T16:34:27.083826Z"
    }
   },
   "outputs": [],
   "source": [
    "data2020weath1 = data2020weatha.reset_index()\n",
    "data2020weath1 = data2020weath1.rename(columns={\"time\": \"ob_time\"}, errors=\"raise\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:34:29.590236Z",
     "start_time": "2021-03-05T16:34:29.571246Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      2020-01-01 00:00:00\n",
       "1      2020-01-01 01:00:00\n",
       "2      2020-01-01 02:00:00\n",
       "3      2020-01-01 03:00:00\n",
       "4      2020-01-01 04:00:00\n",
       "               ...        \n",
       "8779   2020-12-31 19:00:00\n",
       "8780   2020-12-31 20:00:00\n",
       "8781   2020-12-31 21:00:00\n",
       "8782   2020-12-31 22:00:00\n",
       "8783   2020-12-31 23:00:00\n",
       "Name: ob_time, Length: 8784, dtype: datetime64[ns]"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data2020weath1.ob_time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:34:33.959650Z",
     "start_time": "2021-03-05T16:34:33.239063Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\andre\\anaconda3\\lib\\site-packages\\pandas\\core\\generic.py:5168: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self[name] = value\n",
      "<ipython-input-89-2d89038cf033>:5: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  Sunrise20201['ob_time1'] = Sunrise20201['Date'].dt.date\n"
     ]
    }
   ],
   "source": [
    "Sunrise2020 = pd.read_csv('sunrise2020.csv', engine='python')\n",
    "Sunrise20201 = Sunrise2020[['Date','Sunrise Time (LST)','Sunset Time (LST)']]\n",
    "Sunrise20201.Date = pd.to_datetime(Sunrise20201.Date)\n",
    "data2020weath1['ob_time1'] = data2020weath1.ob_time.dt.date\n",
    "Sunrise20201['ob_time1'] = Sunrise20201['Date'].dt.date\n",
    "Weath2020w2 = data2020weath1.merge(Sunrise20201,how = 'left', on = 'ob_time1')\n",
    "Weath2020w2['sunrisehr']=Weath2020w2['Sunrise Time (LST)'].str.replace(r':\\d\\d:\\d\\d','').astype(int)\n",
    "Weath2020w2['sunsethr']=Weath2020w2['Sunset Time (LST)'].str.replace(r':\\d\\d:\\d\\d','').astype(int)\n",
    "\n",
    "Weath2020w2['islight'] = 0\n",
    "Weath2020w2['islight'] = [1 if ((Weath2020w2['sunrisehr'][i] <= Weath2020w2['ob_time'][i].hour) \n",
    "                                and (Weath2020w2['ob_time'][i].hour < Weath2020w2['sunsethr'][i])) \n",
    "                          else 0 for i in Weath2020w2.index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:34:42.290349Z",
     "start_time": "2021-03-05T16:34:42.259367Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 8784 entries, 0 to 8783\n",
      "Data columns (total 12 columns):\n",
      " #   Column           Non-Null Count  Dtype         \n",
      "---  ------           --------------  -----         \n",
      " 0   ob_time          8784 non-null   datetime64[ns]\n",
      " 1   air_temperature  8784 non-null   float64       \n",
      " 2   rltv_hum         8784 non-null   float64       \n",
      " 3   prcp_amt         8784 non-null   float64       \n",
      " 4   wind_direction   8784 non-null   float64       \n",
      " 5   max_gust_speed   8784 non-null   float64       \n",
      " 6   stn_pres         8784 non-null   float64       \n",
      " 7   wmo_hr_sun_dur   8784 non-null   float64       \n",
      " 8   visibility       8784 non-null   float64       \n",
      " 9   cld_ttl_amt_id   8784 non-null   float64       \n",
      " 10  wind_speed       8784 non-null   float64       \n",
      " 11  islight          8784 non-null   int64         \n",
      "dtypes: datetime64[ns](1), float64(10), int64(1)\n",
      "memory usage: 1.2 MB\n"
     ]
    }
   ],
   "source": [
    "Weath2020w5 = Weath2020w2.drop(columns = ['ob_time1','Date','Sunrise Time (LST)', 'Sunset Time (LST)', 'sunrisehr', 'sunsethr'])\n",
    "Weath2020w6 = Weath2020w5.astype({'air_temperature': 'float64' ,\n",
    " 'rltv_hum': 'float64',\n",
    " 'prcp_amt': 'float64',\n",
    " 'wind_direction': 'float64',\n",
    " 'max_gust_speed': 'float64',\n",
    " 'stn_pres': 'float64',\n",
    " 'wmo_hr_sun_dur': 'float64',\n",
    " 'visibility': 'float64',\n",
    " 'cld_ttl_amt_id': 'float64',\n",
    " 'wind_speed': 'float64',\n",
    " 'islight' : 'int64'})\n",
    "Weath2020w6.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Merging Data #"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:34:47.554578Z",
     "start_time": "2021-03-05T16:34:47.282735Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 616797 entries, 0 to 616796\n",
      "Data columns (total 11 columns):\n",
      " #   Column                 Non-Null Count   Dtype         \n",
      "---  ------                 --------------   -----         \n",
      " 0   DateOfCall             616797 non-null  object        \n",
      " 1   CalYear                616797 non-null  int64         \n",
      " 2   HourOfCall             616797 non-null  int64         \n",
      " 3   IncidentGroup          616797 non-null  object        \n",
      " 4   PropertyType           616797 non-null  object        \n",
      " 5   SpecialServiceType     616797 non-null  object        \n",
      " 6   IncidentStationGround  616797 non-null  object        \n",
      " 7   Datetime               616797 non-null  datetime64[ns]\n",
      " 8   holiday                616797 non-null  int64         \n",
      " 9   weekend                616797 non-null  int64         \n",
      " 10  lockdown               616797 non-null  int64         \n",
      "dtypes: datetime64[ns](1), int64(5), object(5)\n",
      "memory usage: 51.8+ MB\n"
     ]
    }
   ],
   "source": [
    "LFB.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:34:50.083998Z",
     "start_time": "2021-03-05T16:34:50.045023Z"
    }
   },
   "outputs": [],
   "source": [
    "LFB = LFB.rename(columns={\"Datetime\": \"ob_time\"}, errors=\"raise\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:35:00.490573Z",
     "start_time": "2021-03-05T16:35:00.441601Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 43833 entries, 0 to 43832\n",
      "Data columns (total 12 columns):\n",
      " #   Column           Non-Null Count  Dtype         \n",
      "---  ------           --------------  -----         \n",
      " 0   air_temperature  43833 non-null  float64       \n",
      " 1   cld_ttl_amt_id   43833 non-null  float64       \n",
      " 2   islight          43833 non-null  int64         \n",
      " 3   max_gust_speed   43833 non-null  float64       \n",
      " 4   ob_time          43833 non-null  datetime64[ns]\n",
      " 5   prcp_amt         43833 non-null  float64       \n",
      " 6   rltv_hum         43833 non-null  float64       \n",
      " 7   stn_pres         43833 non-null  float64       \n",
      " 8   visibility       43833 non-null  float64       \n",
      " 9   wind_direction   43833 non-null  float64       \n",
      " 10  wind_speed       43833 non-null  float64       \n",
      " 11  wmo_hr_sun_dur   43833 non-null  float64       \n",
      "dtypes: datetime64[ns](1), float64(10), int64(1)\n",
      "memory usage: 4.0 MB\n"
     ]
    }
   ],
   "source": [
    "weather = pd.concat([Weath2016w6, Weath2017w6, Weath2018w6, Weath2019w6, Weath2020w6], ignore_index=True, sort=True)\n",
    "weather.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:35:09.337544Z",
     "start_time": "2021-03-05T16:35:08.715391Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 518227 entries, 0 to 518226\n",
      "Data columns (total 22 columns):\n",
      " #   Column                 Non-Null Count   Dtype         \n",
      "---  ------                 --------------   -----         \n",
      " 0   DateOfCall             518227 non-null  object        \n",
      " 1   CalYear                518227 non-null  int64         \n",
      " 2   HourOfCall             518227 non-null  int64         \n",
      " 3   IncidentGroup          518227 non-null  object        \n",
      " 4   PropertyType           518227 non-null  object        \n",
      " 5   SpecialServiceType     518227 non-null  object        \n",
      " 6   IncidentStationGround  518227 non-null  object        \n",
      " 7   ob_time                518227 non-null  datetime64[ns]\n",
      " 8   holiday                518227 non-null  int64         \n",
      " 9   weekend                518227 non-null  int64         \n",
      " 10  lockdown               518227 non-null  int64         \n",
      " 11  air_temperature        518227 non-null  float64       \n",
      " 12  cld_ttl_amt_id         518227 non-null  float64       \n",
      " 13  islight                518227 non-null  int64         \n",
      " 14  max_gust_speed         518227 non-null  float64       \n",
      " 15  prcp_amt               518227 non-null  float64       \n",
      " 16  rltv_hum               518227 non-null  float64       \n",
      " 17  stn_pres               518227 non-null  float64       \n",
      " 18  visibility             518227 non-null  float64       \n",
      " 19  wind_direction         518227 non-null  float64       \n",
      " 20  wind_speed             518227 non-null  float64       \n",
      " 21  wmo_hr_sun_dur         518227 non-null  float64       \n",
      "dtypes: datetime64[ns](1), float64(10), int64(6), object(5)\n",
      "memory usage: 90.9+ MB\n"
     ]
    }
   ],
   "source": [
    "LFBwkg = LFB.merge(weather, on = 'ob_time')\n",
    "LFBwkg.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:35:16.512631Z",
     "start_time": "2021-03-05T16:35:16.451666Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 98353 entries, 0 to 98352\n",
      "Data columns (total 11 columns):\n",
      " #   Column                 Non-Null Count  Dtype         \n",
      "---  ------                 --------------  -----         \n",
      " 0   DateOfCall             98353 non-null  object        \n",
      " 1   CalYear                98353 non-null  int64         \n",
      " 2   HourOfCall             98353 non-null  int64         \n",
      " 3   IncidentGroup          98353 non-null  object        \n",
      " 4   PropertyType           98353 non-null  object        \n",
      " 5   SpecialServiceType     98353 non-null  object        \n",
      " 6   IncidentStationGround  98353 non-null  object        \n",
      " 7   ob_time                98353 non-null  datetime64[ns]\n",
      " 8   holiday                98353 non-null  int64         \n",
      " 9   weekend                98353 non-null  int64         \n",
      " 10  lockdown               98353 non-null  int64         \n",
      "dtypes: datetime64[ns](1), int64(5), object(5)\n",
      "memory usage: 8.3+ MB\n"
     ]
    }
   ],
   "source": [
    "LFB2015 = LFB[LFB['CalYear'] == 2015].reset_index(drop = True)\n",
    "LFB2015.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-05T16:35:39.117345Z",
     "start_time": "2021-03-05T16:35:28.999087Z"
    }
   },
   "outputs": [],
   "source": [
    "LFBwkg.to_csv('Assets/LFBwkg.csv')\n",
    "LFB2015.to_csv('Assets/LFB2015.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "12px",
    "width": "160px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}