Assignment_clean.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment weather data\n",
    "\n",
    "In this assignment we will work with weatherdata from the KNMI. A subset of weatherdata is for you available in the file: `KNMI_20181231.csv`. The data consist of several stations with daily weather data of several years. Your task is to make a tidy dataframe ready for further processing. \n",
    "\n",
    "\n",
    "Learning outcomes\n",
    "\n",
    "- load, inspect and clean a dataset \n",
    "\n",
    "The assignment consists of 6 parts:\n",
    "\n",
    "- [part 1: load the data](#0)\n",
    "- [part 2: clean the data](#1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='0'></a>\n",
    "## Part 1: Load the data\n",
    "\n",
    "Either load the dataset `KNMI_20181231.csv` or `KNMI_20181231.txt.tsv`. The dataheaders contain spaces and are not very self explainable. Change this into more readable ones. Select data from station 270. Select only the mean, minimum and maximum temperature. The data should look something like this:\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "\tstation\tDate\tTmean\tTmin\tTmax\n",
    "97641\t270\t20000101\t42\t-4\t79\n",
    "97642\t270\t20000102\t55\t33\t74\n",
    "97643\t270\t20000103\t74\t49\t89\n",
    "97644\t270\t20000104\t46\t22\t75\n",
    "97645\t270\t20000105\t41\t14\t56\n",
    "...\t...\t...\t...\t...\t...\n",
    "104576\t270\t20181227\t57\t53\t62\n",
    "104577\t270\t20181228\t71\t58\t81\n",
    "104578\t270\t20181229\t85\t69\t102\n",
    "104579\t270\t20181230\t80\t68\t90\n",
    "104580\t270\t20181231\t87\t82\t97"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1'></a>\n",
    "## Part 2: Clean the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data ia not clean. There are empty cells in the dataframe which needs to be replaced with NaN's and the temperature is in centidegrees which needs to be transformed into degrees. The date field needs a datetime format. For visualization convience we would like to remove the leap year. Conduct the cleaning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#replace cells with spaces to NaN\n",
    "#change data formats\n",
    "#change temperatures to celcius degrees\n",
    "#remove leap year"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<ul><li>pd.to_datetime(df['Date'].astype(str), format='%Y%m%d')</li>\n",
    "    <li>regex for empty cells = `^\\s*$` </li>\n",
    "    <li>remove month == 2 & day == 29</li> \n",
    "</ul>\n",
    "</details>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Expected outcome"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "\n",
    "    station\tDate\t  Tmean\tTmin Tmax\n",
    "97641\t270\t2000-01-01\t4.2\t-0.4 7.9\n",
    "97642\t270\t2000-01-02\t5.5\t3.3\t 7.4\n",
    "97643\t270\t2000-01-03\t7.4\t4.9\t 8.9\n",
    "97644\t270\t2000-01-04\t4.6\t2.2\t 7.5\n",
    "97645\t270\t2000-01-05\t4.1\t1.4\t 5.6\n",
    "...\t...\t...\t...\t...\t...\n",
    "104576\t270\t2018-12-27\t5.7\t5.3\t 6.2\n",
    "104577\t270\t2018-12-28\t7.1\t5.8\t 8.1\n",
    "104578\t270\t2018-12-29\t8.5\t6.9\t 10.2\n",
    "104579\t270\t2018-12-30\t8.0\t6.8\t 9.0\n",
    "104580\t270\t2018-12-31\t8.7\t8.2\t 9.7\n",
    "\n",
    "\n",
    "DatetimeIndex: 24820 entries, 1951-01-01 to 2018-12-31\n",
    "Data columns (total 5 columns):\n",
    " #   Column   Non-Null Count  Dtype         \n",
    "---  ------   --------------  -----         \n",
    " 0   station  24820 non-null  int64         \n",
    " 1   Date     24820 non-null  datetime64[ns]\n",
    " 2   Tmean    24819 non-null  float64       \n",
    " 3   Tmin     24819 non-null  float64       \n",
    " 4   Tmax     24819 non-null  float64       \n",
    "dtypes: datetime64[ns](1), float64(3), int64(1)\n",
    "memory usage: 1.1 MB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}