Introduction_to_pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "srbIFiyxKUhF"
   },
   "source": [
    "## Introduction to Pandas \n",
    "\n",
    "Now that you know how Python works, we can start doing some data manipulation with Pandas. \n",
    "\n",
    "'Libraries' in Python are actually collections of code that enhance Python funtionality for specific purposes. Pandas is an open source Library in Python that provides easy data strcuture and analysis tools. \n",
    "\n",
    "Everytime you want to install a new library in python, you just need to install it and then import it in your command line. You can import it with its name or with an alias that will make it easier to call it later in your code.\n",
    "\n",
    "e.g. \n",
    "pip install pandas\n",
    "\n",
    "import pandas \n",
    "\n",
    "import pandas as pd \n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eJJdYrHUKUhJ"
   },
   "source": [
    "## Data types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "M5GvwM79KUhK"
   },
   "source": [
    "### Numerical\n",
    "\n",
    "Python has two numerical data types:\n",
    "- `int`, e.g. `10`\n",
    "- `float`, e.g. `10.12`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "khZATuDoKUhO"
   },
   "outputs": [],
   "source": [
    "i = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "colab_type": "code",
    "id": "PZ75lnckKUhg",
    "outputId": "e128f336-848c-4e49-d38d-60c090a279d8"
   },
   "outputs": [],
   "source": [
    "type(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Fl63taQOKUhs"
   },
   "outputs": [],
   "source": [
    "f = 10.12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "APeUvkCNKUh5",
    "outputId": "2e64e275-9ea0-45da-dc4a-5f8a62adae0d"
   },
   "outputs": [],
   "source": [
    "isinstance(i, int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UazSLx6aKUiB",
    "outputId": "2290cf3a-c0f5-4a95-d7f1-69698ccf0ac9"
   },
   "outputs": [],
   "source": [
    "isinstance(f, int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "g78PQvBpKUiL",
    "outputId": "23b60d23-6cff-4756-c8dc-35aa9ed21fd7"
   },
   "outputs": [],
   "source": [
    "type(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zOBTQlOVKUiY"
   },
   "source": [
    "Python has two signs for division, which produce different results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qUgJE8ngKUiZ",
    "outputId": "f348bd73-3b8c-48d4-927d-a97f4d6d6ffd"
   },
   "outputs": [],
   "source": [
    "i // 3 == i / 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "--p2gDhJKUid",
    "outputId": "eb3f8f28-a94c-4d98-c577-e6d41941829d"
   },
   "outputs": [],
   "source": [
    "i // 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bIijmscPKUio",
    "outputId": "8824d2fc-54ef-461b-b7c7-9d8f36285450"
   },
   "outputs": [],
   "source": [
    "i / 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tT_EzCX_KUi5",
    "outputId": "ee8469a0-9bc2-4622-c11a-ef5159a510d5"
   },
   "outputs": [],
   "source": [
    "type(i // 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RPzjUDO6KUi9",
    "outputId": "35718fd4-9aa3-4421-aeb3-7ad069d6f085"
   },
   "outputs": [],
   "source": [
    "type(i / 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OUF0Cy2sKUjG"
   },
   "source": [
    "### Strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fnyYl1HdKUjI"
   },
   "outputs": [],
   "source": [
    "mystring = \"A string of text\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HRsWMesvKUjO"
   },
   "source": [
    "As of Python 3, strings are by default encoded in Unicode. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "L1t7SjioKUjP",
    "outputId": "5d113158-6c95-44a2-ea6b-03d815459abf"
   },
   "outputs": [],
   "source": [
    "type(mystring.encode('utf-8'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4R61N5vqKUjZ"
   },
   "source": [
    "Strings in Python are **list** of characters, thus they can be manipulated as any other *iterable*. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sha3wjkcKUjb",
    "outputId": "98aed9a5-a93f-4625-9645-5a0f07004128"
   },
   "outputs": [],
   "source": [
    "# we can iterate through the characters\n",
    "# of a string\n",
    "\n",
    "for char in mystring:\n",
    "    print(char)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Z_aKZ58QKUjf",
    "outputId": "26d1c804-bf55-4cd1-80f5-e2909299661e"
   },
   "outputs": [],
   "source": [
    "# slicing by means of indices works as expected\n",
    "\n",
    "mystring[2:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UpN5LkfIKUjn",
    "outputId": "8f0a6911-5648-4790-b0c5-0a14bde0c3a1"
   },
   "outputs": [],
   "source": [
    "mystring[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SsTA_geWKUju"
   },
   "source": [
    "#### Concatenation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SEJSdKbuKUjv"
   },
   "outputs": [],
   "source": [
    "newstring = \"This is \" + mystring.lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "t9lqX7ffKUj1",
    "outputId": "0a4457a8-7c99-49b0-d788-bf6abe3d15fc"
   },
   "outputs": [],
   "source": [
    "newstring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nQrrn0miKUj_"
   },
   "source": [
    "A very handy feature introduced in Python 3.6.x are f-strings:\n",
    "- they are declared by prepending the character `f` to the quote signs containing the text\n",
    "- they use curly brackets `{variable_name}` to specify the position in a string where the content of an existing variable should be inserted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "N5DxEdNbKUkB",
    "outputId": "17736fb2-420e-4146-e797-2ddde36cb20f"
   },
   "outputs": [],
   "source": [
    "f'## {mystring} ##'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5YjoO_OnKUkG"
   },
   "source": [
    "The curly brackets can contain *any* Python expression (except assignment of variables); the expression will be executed and its returned output interpolated within the string template."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bYAvuM50KUkI",
    "outputId": "0c2f0ba6-3397-4aca-c5bf-535bf259cdf6"
   },
   "outputs": [],
   "source": [
    "f'The length of `mystring` is {len(mystring)} characters.'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0LgeUHpfKUkN"
   },
   "source": [
    "**Q**: Can you explain what's going on in the cell below?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "V9eA3oJTKUkP",
    "outputId": "c69f1747-365d-4453-962d-95a590d7677e"
   },
   "outputs": [],
   "source": [
    "s = \"repetita iuvant\"\n",
    "print(f'{\", \".join([s for i in range(0, 10)])}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PyfyewN6KUka"
   },
   "source": [
    "Can you rewrite the cell above in an alternative way?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zLubyhUZKUkb"
   },
   "source": [
    "#### Transformation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NW3glZRUKUkd",
    "outputId": "102e331d-639d-4676-c8c3-439e578e8711"
   },
   "outputs": [],
   "source": [
    "mystring.lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_86HHAtnKUkp",
    "outputId": "eb3bb094-6017-4865-c24b-2bba8326594d"
   },
   "outputs": [],
   "source": [
    "mystring.upper()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "p6JChzyCKUk0",
    "outputId": "3576c06f-2523-4b88-d4e7-37d3b9572204"
   },
   "outputs": [],
   "source": [
    "mystring.replace(\"string\", \"list\").replace(\"text\", \"characters\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cOAKQ4J_KUk4"
   },
   "source": [
    "### Date and time\n",
    "\n",
    " Limit of this data type when working with historical data (timestamps failed before a certain date around 1700)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "krAQs2ZsKUk6"
   },
   "source": [
    "#### `datetime.date`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "palGMhd7KUk7"
   },
   "outputs": [],
   "source": [
    "from datetime import date, datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sh_rsxcPKUlB"
   },
   "outputs": [],
   "source": [
    "# `date` takes three arguments:\n",
    "# 1. year, 2. month, 3. day\n",
    "\n",
    "d = date(1982, 7, 17)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XSq5m3-lKUlH",
    "outputId": "ed795dce-82c1-47c8-f9a0-85e3a19785ff"
   },
   "outputs": [],
   "source": [
    "type(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1r0dStoHKUlN"
   },
   "source": [
    "**NB**: When creating a date, order matters! Try this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "s0wEwcjjKUlP",
    "outputId": "0447b5b8-69f4-4cc7-f115-f3c3e60dfb69"
   },
   "outputs": [],
   "source": [
    "d = date(19, 7, 1782)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3LzIyP7uKUlT",
    "outputId": "bbab5d0b-dbe1-461c-9613-0cad39df807b"
   },
   "outputs": [],
   "source": [
    "d.today()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aIwoePKfKUlY",
    "outputId": "32ec58e3-bff2-4255-a2ac-9a84c5c7698c"
   },
   "outputs": [],
   "source": [
    "f'{d.day}.{d.month}.{d.year}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WRn8iCAZKUlb",
    "outputId": "b864bace-0d25-4815-8ac7-0bd611d98f94"
   },
   "outputs": [],
   "source": [
    "f'{d.year}/{str(d.month)}/{d.day}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XrRDBS-ZKUlk",
    "outputId": "5eb97b15-132b-453b-c819-20bbfddba720"
   },
   "outputs": [],
   "source": [
    "f'{d.year}/{str(d.month).zfill(2)}/{d.day}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rFWYsAgiKUlw"
   },
   "source": [
    "#### `datetime.datetime`\n",
    "\n",
    "`datetime` adds information about hour/minute/second/micro second to a date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nPXkiR99KUlx"
   },
   "outputs": [],
   "source": [
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-N7ZnPkfKUl0"
   },
   "outputs": [],
   "source": [
    "dt = datetime.utcnow()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1x4340udKUl4",
    "outputId": "034982cb-4008-4478-db80-0a425cbdf1e8"
   },
   "outputs": [],
   "source": [
    "dt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ggJIImxLKUl8",
    "outputId": "2d40687f-81f9-4fa0-c79d-ae63478f7614"
   },
   "outputs": [],
   "source": [
    "dt.isoformat()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cHuw7c-9KUmF",
    "outputId": "5c16b4bf-c09e-491b-99e2-8aabdc2c5659"
   },
   "outputs": [],
   "source": [
    "dt.date()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4mNBaWxvKUmK",
    "outputId": "91cb495e-b7c9-4a8e-eec6-488460087393"
   },
   "outputs": [],
   "source": [
    "datetime.now().strftime(\"%m/%d/%Y, %H:%M:%S\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AyVHihOBKUmP"
   },
   "source": [
    "## Python data structures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tSJs0n-4KUmQ"
   },
   "source": [
    "### Lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hoH_s2z3KUmQ"
   },
   "outputs": [],
   "source": [
    "l = list(range(0, 5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CMfU2l9_KUmU",
    "outputId": "e33d090a-9aa6-45d6-d952-311ade40c177"
   },
   "outputs": [],
   "source": [
    "l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "95hX716vKUmZ"
   },
   "source": [
    "The `extend()` method can be used to append elements to an existing list.\n",
    "\n",
    "**NB**: `extend` operates directly on the list, modifying it in place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "t_P3OOz5KUmd"
   },
   "outputs": [],
   "source": [
    "l.extend(range(1, 10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ah7muEUVKUmi",
    "outputId": "6fd5a6d0-04d6-413f-cfd2-50c91ca75456"
   },
   "outputs": [],
   "source": [
    "l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qMCTp1d3KUmk",
    "outputId": "101bd473-0c30-4ec2-d1eb-45283f955a32"
   },
   "outputs": [],
   "source": [
    "l + list(range(5, 10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Sjp5PXo9KUmq",
    "outputId": "9c383177-31fd-473d-f068-ba5f59e34c9f"
   },
   "outputs": [],
   "source": [
    "l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jygAxwOOKUmu"
   },
   "source": [
    "`count()` can be used to count the number of times a given value is found within a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NoznvLcVKUmw",
    "outputId": "0c7383e0-1f5a-48d2-b18f-6f3cff8f4140"
   },
   "outputs": [],
   "source": [
    "for n in range(0, 10):\n",
    "    print(f'{n} occurs {l.count(n)} times in list `l`')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AoFinmJNKUm7",
    "outputId": "59d4bb03-d9bb-4144-9e91-e028edce671a"
   },
   "outputs": [],
   "source": [
    "l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VgCudLaNKUnC",
    "outputId": "d0f896af-9f66-44ad-8770-3e2543623b4d"
   },
   "outputs": [],
   "source": [
    "l.index(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Cw6Y5dBCKUnG"
   },
   "source": [
    "`pop()` remove the last item of a list and, as `extend()`, operates directly on the variable, modifying its value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7H6Uj3d_KUnH",
    "outputId": "1cb46aca-7c4c-494f-b467-ed70049df2a9"
   },
   "outputs": [],
   "source": [
    "while(len(l) > 0):\n",
    "    print(f'Removing {l.pop()} from my list')\n",
    "    print(f'Size of `l` is {len(l)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9tD8gIkvKUnK",
    "outputId": "cd4297a7-30b3-4881-e530-791a58f04148"
   },
   "outputs": [],
   "source": [
    "# you cannot remove an element from an empty list\n",
    "\n",
    "l.pop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xfzk6520KUnO"
   },
   "source": [
    "### Dictionaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CAHOClZsKUnP"
   },
   "outputs": [],
   "source": [
    "d = {\n",
    "    \"count\": 0,\n",
    "    \"type\": \"child\",\n",
    "    \"average\": 1.2\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0UJHQYU1KUnS",
    "outputId": "6ea49c1f-0425-4cef-c11d-d476535dc139"
   },
   "outputs": [],
   "source": [
    "d.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "64ktavTvKUna",
    "outputId": "6c3a20e3-d667-476a-c58e-edd45433e0f1"
   },
   "outputs": [],
   "source": [
    "d.values()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ewVbzdZkKUne",
    "outputId": "f1851eab-36d0-4bc8-c4b5-1d29c8fc0688"
   },
   "outputs": [],
   "source": [
    "d['count']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MvhdtSQaKUnk"
   },
   "outputs": [],
   "source": [
    "d1 = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tElZxh0eKUns",
    "outputId": "f0ef26a5-4342-4b6c-c77a-763b2edb6134"
   },
   "outputs": [],
   "source": [
    "assert d1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bGSY6Hu3KUn0",
    "outputId": "1f46ff8d-4009-451e-8541-dc03eac56d33"
   },
   "outputs": [],
   "source": [
    "if d:\n",
    "    print(\"hello\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "i8k7gKm2KUn8"
   },
   "outputs": [],
   "source": [
    "if d1:\n",
    "    print(\"hello\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rI_XYOFrKUoB"
   },
   "source": [
    "**Q**: Why? Can you explain what's going on?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4VVL_FKGKUoC"
   },
   "source": [
    "### Tuples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Tq-iMQQvKUoD"
   },
   "source": [
    "Tuples are similar to lists, as they are both iterables. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ioO3szVLKUoE"
   },
   "outputs": [],
   "source": [
    "t = tuple((0, \"child\", 1.2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5vOIcLDVKUoG",
    "outputId": "d8fb3655-c357-4882-c11e-94c18c1d0b40"
   },
   "outputs": [],
   "source": [
    "t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fKDmaX5bKUoK"
   },
   "source": [
    "As any interable, you can iterate over it (as one would expect):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "colab": {},
    "colab_type": "code",
    "id": "_LnvQVcPKUoL",
    "outputId": "8d28f5be-18fe-4aeb-cbc7-198d897aa9c8"
   },
   "outputs": [],
   "source": [
    "for value in t:\n",
    "    print(value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rPeXhdiLKUoQ"
   },
   "source": [
    "The main difference between the two is that tuples do no support slicing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EG7mSuNAKUoR",
    "outputId": "765cdfaa-e0cf-4a17-d05a-c200e03ffabf"
   },
   "outputs": [],
   "source": [
    "t[1] = 'adult'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fliNOGxFKUoT"
   },
   "source": [
    "## Data structures (`pandas`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NO6a646aKUoU"
   },
   "source": [
    "### `Series`\n",
    "\n",
    "In `pandas`, series are the building blocks of dataframes.\n",
    "\n",
    "Think of a series as a column in a table. A series collects *observations* about a given *variable*. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wdPAp-KiKUoV"
   },
   "outputs": [],
   "source": [
    "from random import random\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from pandas import Series, DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0rHUb6UUKUoY"
   },
   "source": [
    "#### Numerical series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KyMUMR71KUoZ"
   },
   "outputs": [],
   "source": [
    "# let's create a series containing 100 random numbers\n",
    "# ranging between 0 and 1\n",
    "\n",
    "s = pd.Series([random() for n in range(0, 100)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DFaqRrkSKUob"
   },
   "source": [
    "Each observation in the series has an **index** as well as a set of **values**: they can be accessed via the omonymous properties:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uCC6WyeZKUob",
    "outputId": "259e5c6d-cbe6-4ad3-dee4-10978b39b523"
   },
   "outputs": [],
   "source": [
    "s.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "V3AQHiq4KUof",
    "outputId": "f04cc4c9-294d-4b70-e0bc-805de6998884"
   },
   "outputs": [],
   "source": [
    "list(s.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "y37qF7SuKUok",
    "outputId": "f9228ef5-b158-410d-e52f-f90482826841"
   },
   "outputs": [],
   "source": [
    "s.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7IxcfbevKUon"
   },
   "source": [
    "The `head()` and `tail()` methods allows for looking at the begininning and end of a series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gCuax-CaKUoo",
    "outputId": "f6efea40-f397-47ca-fc38-4c36e72a4ca3"
   },
   "outputs": [],
   "source": [
    "s.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "R7_JxbXyKUor",
    "outputId": "36928187-4592-4170-dc82-7492117a50f3"
   },
   "outputs": [],
   "source": [
    "s.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "t4GEW8jjKUot"
   },
   "source": [
    "The `value_counts()` method returns a count of distinct values within a series."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1tCitZk4KUox"
   },
   "source": [
    "Is there any number in `s` that occurs twice?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Q9mY11hkKUo2",
    "outputId": "23c1c3b5-6f82-4c89-8577-1633b16ff62e"
   },
   "outputs": [],
   "source": [
    "# a `Series` can be easily cast into a list\n",
    "\n",
    "list(s.value_counts()).count(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "I37IU7TMKUo6"
   },
   "source": [
    "Another way of verifying this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-4nCuZ6HKUo6",
    "outputId": "adb81dcf-0263-48f8-dd24-a548cbc5be99"
   },
   "outputs": [],
   "source": [
    "s.is_unique"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WYDcLtH2KUo9",
    "outputId": "9259c13b-bc03-4be7-b9ea-bdbb94d80c3e"
   },
   "outputs": [],
   "source": [
    "s.min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cnYnENOOKUpB",
    "outputId": "976bb7ce-9b0d-4d0b-f72e-c927eb0df102"
   },
   "outputs": [],
   "source": [
    "s.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UfQCHcfPKUpF",
    "outputId": "6e2118d8-1c6f-4c94-b495-f61db707b22d"
   },
   "outputs": [],
   "source": [
    "s.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "moLsp1bZKUpL",
    "outputId": "4f3d5a1e-8acb-46be-bb48-f240f72f767d"
   },
   "outputs": [],
   "source": [
    "s.median()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gvsLNPPwKUpO"
   },
   "source": [
    "#### Datetime series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kx6Uo4mpKUpO"
   },
   "outputs": [],
   "source": [
    "from random import randint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "colab": {},
    "colab_type": "code",
    "id": "B08JV2ARKUpR"
   },
   "outputs": [],
   "source": [
    "# let's generate a list of random dates\n",
    "# in the range 1900-1950\n",
    "\n",
    "dates = [\n",
    "    date(\n",
    "        year,\n",
    "        randint(1, 12),\n",
    "        randint(1, 28) # try replacing with 31 and see what happens\n",
    "    )\n",
    "    for year in range(1900,1950)\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2KxBhaVFKUpV"
   },
   "outputs": [],
   "source": [
    "s1 = pd.Series(dates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9ZW5wMilKUpY",
    "outputId": "1d4199ce-5929-45a5-af95-a30dddd22f0b"
   },
   "outputs": [],
   "source": [
    "s1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uPpYhnMaKUpb",
    "outputId": "ad0a653f-b982-4ee7-f08b-960ffb3482a9"
   },
   "outputs": [],
   "source": [
    "type(s1[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_UNlFvKYKUpd"
   },
   "outputs": [],
   "source": [
    "s1 = Series(pd.to_datetime(dates))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YUC6T_lmKUpj",
    "outputId": "4af65a5c-8408-47c4-c268-02c5f2d8e89a"
   },
   "outputs": [],
   "source": [
    "type(s1[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7OyKubDqKUpp",
    "outputId": "77458657-e6e1-44f6-c01f-d1e50ad52429"
   },
   "outputs": [],
   "source": [
    "s1[1].day_name()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8SjKQQu6KUpu",
    "outputId": "61f7b734-740c-4ca6-ef30-97063cb2ac0f"
   },
   "outputs": [],
   "source": [
    "s1.min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WspZ91MYKUpy",
    "outputId": "615cd8f2-a900-49b7-b2de-ccb206dc2a9b"
   },
   "outputs": [],
   "source": [
    "s1.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SYG0UsaVKUp2",
    "outputId": "89215f21-59b6-46ef-95f9-d4d10ce7d51d"
   },
   "outputs": [],
   "source": [
    "s1.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TotogdtXKUp4"
   },
   "source": [
    "### `DataFrame`\n",
    "\n",
    "\n",
    "What is a `pandas.DataFrame`? Think of it as an in-memory spreadsheet that you can analyse and manipulate programmatically.\n",
    "\n",
    "A `DataFrame` is a collection of `Series` having the same length and whose indexes are in sync. A *collection* means that each column of a dataframe is a series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tU6otLTNKUp5"
   },
   "source": [
    "Let's create a toy `DataFrame` by hand. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "alfELPIeKUp5"
   },
   "outputs": [],
   "source": [
    "dates = [\n",
    "    date(\n",
    "        year,\n",
    "        randint(1, 12),\n",
    "        randint(1, 28) # try replacing with 31 and see what happens\n",
    "    )\n",
    "    for year in range(1980,1990)\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kncSPBrdKUp7"
   },
   "outputs": [],
   "source": [
    "counts = [\n",
    "    randint(0, 10000)\n",
    "    for i in range(0, 10)\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nOMkIIJMKUp-"
   },
   "outputs": [],
   "source": [
    "event_types = [\"fire\", \"flood\", \"car_crash\", \"plane_crash\"]\n",
    "events = [\n",
    "    np.random.choice(event_types)\n",
    "    for i in range(0, 10)\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "m8ZoxAksKUqB"
   },
   "outputs": [],
   "source": [
    "assert len(events) == len(counts) == len(dates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yxO84KjjKUqE"
   },
   "outputs": [],
   "source": [
    "toy_df = pd.DataFrame({\n",
    "    \"date\": dates,\n",
    "    \"count\": counts,\n",
    "    \"event\": events\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "intUlBRXKUqI",
    "outputId": "70fd64c7-b9a3-49fe-8f9b-b1a9c408ea9e"
   },
   "outputs": [],
   "source": [
    "toy_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "i4DV5X_bKUqK"
   },
   "source": [
    "**Try out**: what happens if you change the length of either of the two lists? Try e.g. passing 20 dates instead of 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zWnDAuT-KUqL"
   },
   "outputs": [],
   "source": [
    "# instead of a dictionary of lists, you can pass\n",
    "# directly a dictionary of `pandas.Series`. The result is the same.\n",
    "\n",
    "toy_df = pd.DataFrame(\n",
    "    {\n",
    "        \"date\": pd.to_datetime(dates),\n",
    "        \"count\": counts,\n",
    "        \"event\": Series(events)\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "J1ANf5TaKUqN"
   },
   "outputs": [],
   "source": [
    "toy_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1U3WSfbRKUqU"
   },
   "outputs": [],
   "source": [
    "# a df is a collection of series\n",
    "# each column is a series\n",
    "\n",
    "type(toy_df.date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-bMIOXYmKUqZ"
   },
   "outputs": [],
   "source": [
    "toy_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QEBbeGR-KUqb"
   },
   "source": [
    "## Data manipulation in `pandas`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DGXmB0yjKUqi"
   },
   "source": [
    "### Data types\n",
    "\n",
    "String, datetimes (see above), categorical data.\n",
    "\n",
    "In `pandas`, categories behave very much like string, yet they lead to better performances (faster operations, optimized storage)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QHnMH3m3KUqk",
    "outputId": "67551d1d-249b-4046-d935-7de7af09f8b8"
   },
   "outputs": [],
   "source": [
    "# transforms a Series with strings into categories\n",
    "\n",
    "toy_df.event = toy_df.event.astype('category')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qyaRPyDlKUqs",
    "outputId": "2491068a-9da0-4c58-a221-18dbf3b82e21"
   },
   "outputs": [],
   "source": [
    "toy_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "b-iY9jSAKUqz"
   },
   "source": [
    "##### How are categories represented?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Kne2xUUFKUq0",
    "outputId": "fa085031-24e7-4b83-fdc1-bd09e3aaddb1"
   },
   "outputs": [],
   "source": [
    "toy_df.event.cat.codes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "l7erPGqTKUq2",
    "outputId": "17429ac2-2f26-4c74-8b74-a60404dea5c8"
   },
   "outputs": [],
   "source": [
    "toy_df.event.cat.categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6LX-_adQKUq5"
   },
   "outputs": [],
   "source": [
    "toy_df.event.cat.rename_categories({\"plane_crash\": \"airplane_crash\"}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4ZcaDvQmKUq8",
    "outputId": "4111a4f2-3d6c-4475-ab80-0cabb44787b5"
   },
   "outputs": [],
   "source": [
    "toy_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "raYgCMhXKUq-"
   },
   "outputs": [],
   "source": [
    "toy_df.event.cat.rename_categories({\"plane_crash\": \"plane_crash\"}, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n0Y2izF1KUrA",
    "outputId": "c748e653-7fb0-49d2-bb5c-162868d30435"
   },
   "outputs": [],
   "source": [
    "toy_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EBZtieSMKUrC",
    "outputId": "f9bc35e8-6da7-457c-ffe3-250d16619d63"
   },
   "outputs": [],
   "source": [
    "# back to the original type\n",
    "\n",
    "toy_df.event.astype(str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "znySndGwKUrD"
   },
   "source": [
    "### Accessor properties\n",
    "\n",
    "For certain data types (string, datetime), `pandas` provides a number of common methods that can be called on any series containing values of that type. These methods become available as methods of the series itself within a property — called *accessor* — named after the data type:\n",
    "\n",
    "- the `.dt.*` accessor contains methods to operate on `datetime` series\n",
    "- the `str.` accessor contains methods to operate on `str` (string) series.\n",
    "\n",
    "As you will see in a moment, these methods are very convenient when filtering rows of a dataset based on the value of a certain column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NP02_xERKUrE"
   },
   "source": [
    "#### `datetime` accessor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hvkKZxeuKUrF"
   },
   "source": [
    "To work with datetime series `pandas` provide a bunch of useful methods to operate on a series: they can be called from the `.dt` property of a datetime series.\n",
    "\n",
    "They can be used to:\n",
    "- convert from one timezone to another\n",
    "- get the day/day name/month/year information from each date\n",
    "- and much more (see the [documentation]())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FhrhxCk7KUrK",
    "outputId": "4b78a789-7e2a-4833-ee06-c7cb6186fcbd"
   },
   "outputs": [],
   "source": [
    "s1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NGtZZ8ARKUrN",
    "outputId": "feb309ea-1da8-4b3d-880c-cc9633078fe0"
   },
   "outputs": [],
   "source": [
    "s1.dt.day_name().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7diDmhcpKUrT"
   },
   "source": [
    "#### `str` accessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tjcLp3UzKUrU"
   },
   "outputs": [],
   "source": [
    "s = Series([\"uno\", \"due\", \"tre\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aq1Lg0EDKUra",
    "outputId": "22506cfe-f247-4062-94df-5c455b3dced0"
   },
   "outputs": [],
   "source": [
    "s.str.contains('o')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rimIqcr4KUrd"
   },
   "source": [
    "### Exploring a dataframe\n",
    "\n",
    "Exploring a dataframe: df.head(), df.tail(), df.info()."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Zjk6ctWEKUrf"
   },
   "source": [
    "The method `info()` gives you information about a dataframe:\n",
    "- how much space does it take in memory?\n",
    "- what is the datatype of each column?\n",
    "- how many records are there?\n",
    "- how many `null` values does each column contain (!)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Dz5xpO-AKUrg",
    "outputId": "34256f9d-8159-42c2-fd06-13b1cda542ea"
   },
   "outputs": [],
   "source": [
    "toy_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "q0KqbKC2KUri"
   },
   "source": [
    "Alternatively, if you need to know only the number of columns and rows you can use the `.shape` property.\n",
    "\n",
    "It returns a tuple with 1) number of rows, 2) number of columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3nLbJr7OKUrj",
    "outputId": "f30b9a70-5779-4462-bcc9-51ffaadff46d"
   },
   "outputs": [],
   "source": [
    "toy_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Rx6MENN5KUrn"
   },
   "source": [
    "`head()` prints by first five rows of a dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dkcJhsWCKUro",
    "outputId": "516bfe38-b876-41ab-f3cb-3a2caef7a6dc"
   },
   "outputs": [],
   "source": [
    "toy_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Am_xAvjHKUrw"
   },
   "source": [
    "But the number of lines displayed is a parameter that can be changed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dDNtdES7KUrw",
    "outputId": "8cf1f961-98ef-4269-f97b-5818fb294414"
   },
   "outputs": [],
   "source": [
    "toy_df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-0Xed03iKUry"
   },
   "source": [
    "`tail()` does the opposite, i.e. prints the last n rows in the dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Via9f9gMKUry",
    "outputId": "c487469e-d0b2-4471-960c-a595df80b54a"
   },
   "outputs": [],
   "source": [
    "toy_df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VVL_6TCxKUsj"
   },
   "source": [
    "### Working with columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dg7fj15sKUsk"
   },
   "source": [
    "#### Casting\n",
    "\n",
    "We call *casting* the operation of changing the act of changing the data type of one or more variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xYZII1r4KUsl"
   },
   "outputs": [],
   "source": [
    "# we define a string with value \"10\"\n",
    "number_str = \"10\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pCYtnuKhKUsr"
   },
   "outputs": [],
   "source": [
    "# we change its type from string (`str`)\n",
    "# to integeer (`int`). This is call casting\n",
    "\n",
    "number_int = int(number_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JwLEIsM_KUsx",
    "outputId": "62d15461-c5df-4dca-fab0-eef125fc6000"
   },
   "outputs": [],
   "source": [
    "# the types of the two variable are different indeed\n",
    "\n",
    "type(number_str) == type(number_int)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6Eq6ll56KUs0"
   },
   "source": [
    "`pandas` objects like `Series` and `DataFrame` provide the method `astype()` to apply casting on their contents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zYlMBex-KUs_"
   },
   "source": [
    "To cast the type of the `profession_cat` column, we can use directly the `astype()` method of the Series: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mipt1jpaKUtA"
   },
   "outputs": [],
   "source": [
    "toy_df.event = toy_df.event.astype('category')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wbK1jRRxKUtB",
    "outputId": "ead334e4-726b-446e-9468-28286f7ce359"
   },
   "outputs": [],
   "source": [
    "toy_df.event.cat.categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8xWPJzP6KUtW"
   },
   "source": [
    "#### Adding columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qrl7m6u-KUtf"
   },
   "source": [
    "Let's go back to our toy dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-2xZpFAcKUtg",
    "outputId": "6fec6fce-bbc9-4ebb-96c6-84dc66ee68f5"
   },
   "outputs": [],
   "source": [
    "toy_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8XGbAr3VKUtj"
   },
   "source": [
    "Using the column selector with the name of a column that does not exist yet will add the effect of setting the values of all rows in that column to the value specified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iewWFpF7KUtm"
   },
   "outputs": [],
   "source": [
    "toy_df['country'] = \"UK\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0pBfU0cYKUto",
    "outputId": "619c08d6-c41e-4e76-ab4e-f482b1af1fae"
   },
   "outputs": [],
   "source": [
    "toy_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ohRzO821KUtp"
   },
   "source": [
    "But if the column already exists, its value is reset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "d2DQKcB0KUtq"
   },
   "outputs": [],
   "source": [
    "toy_df['country'] = \"USA\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "P9wQwLDjKUtt",
    "outputId": "94cf2d74-1b68-45d8-9c70-202acc852e95"
   },
   "outputs": [],
   "source": [
    "toy_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "O7x5NxWnKUtu"
   },
   "source": [
    "#### Removing columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7DOs1PusKUtv"
   },
   "source": [
    "The double square bracket notation ``[[...]]`` returns a dataframe having only the columns specified inside the inner brackets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cJogmg8OKUtv"
   },
   "source": [
    "This said, removing a column is done by unselecting it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ln0W5H5mKUtv"
   },
   "outputs": [],
   "source": [
    "# here we removed the column country \n",
    "\n",
    "toy_df2 = toy_df[['date', 'count', 'event']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sNvk3bd_KUty",
    "outputId": "0a3b8287-4e8e-403d-d7d3-d963276b38c9"
   },
   "outputs": [],
   "source": [
    "# it worked!\n",
    "\n",
    "toy_df2.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lYG3UQlyKUt0"
   },
   "source": [
    "#### Setting a column as index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "lQMlVkgiKUt1",
    "outputId": "0052df68-7b5c-460f-f399-bb8d71855d34"
   },
   "outputs": [],
   "source": [
    "toy_df.set_index('date')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8ers0gLYKUt5",
    "outputId": "7bd348b3-5908-45f3-e991-8c266b83d352"
   },
   "outputs": [],
   "source": [
    "toy_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mZAbgRIfKUt7"
   },
   "outputs": [],
   "source": [
    "toy_df.set_index('date', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7dV16hN9KUt8",
    "outputId": "aa40e458-bd7a-443a-c8b2-73f32c9f3d23"
   },
   "outputs": [],
   "source": [
    "toy_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OGpw89rxKUt-"
   },
   "source": [
    "**Q**: can you explain the effect of the `inplace` parameter by looking at the cells above?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1Wtoh3ArKUt_"
   },
   "source": [
    "### Accessing data\n",
    "\n",
    " .loc, .iloc, slicing, iteration over rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PJhrRMAaKUt_",
    "outputId": "d9dd8517-5e0a-4a60-e8d2-7c97f3bb54eb"
   },
   "outputs": [],
   "source": [
    "toy_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "azSr0Mo1KUuD"
   },
   "source": [
    "#### Label-based indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PjLMTw0WKUuF"
   },
   "outputs": [],
   "source": [
    "toy_df.loc['1902':'1904']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ipc2EuHAKUuI"
   },
   "source": [
    "#### Integer-based indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DdMqM9ZAKUuJ",
    "outputId": "47a1c0cd-1547-4303-8994-2793ed9cd9c8"
   },
   "outputs": [],
   "source": [
    "# select a single row, the first one\n",
    "\n",
    "toy_df.iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MFwkuwliKUuN",
    "outputId": "7bf707fa-e2aa-4f34-a643-94378bf4d439"
   },
   "outputs": [],
   "source": [
    "# select  a range of rows by index\n",
    "\n",
    "toy_df.iloc[[1,3,-1]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LAv7n8LPKUuQ",
    "outputId": "badacd47-8fc7-4298-dee1-fb959a452c95"
   },
   "outputs": [],
   "source": [
    "# select  a range of rows with slicing\n",
    "\n",
    "toy_df.iloc[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OQCOulT1KUuT",
    "outputId": "58caaaec-4507-45ad-d899-3c624835384d"
   },
   "outputs": [],
   "source": [
    "toy_df.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lQC_wE9qKUuZ"
   },
   "source": [
    "#### Iterating over rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xGcRgE6uKUua",
    "outputId": "33feba82-b25b-4439-a27f-648d5151c2ba"
   },
   "outputs": [],
   "source": [
    "for n, row in toy_df.iterrows():\n",
    "    print(n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "I3VYgYTyKUud",
    "outputId": "fab8e40d-d5cb-4f6a-d730-d5bb176a7667"
   },
   "outputs": [],
   "source": [
    "for n, row in toy_df.iterrows():\n",
    "    print(n, row.event)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "Copia de Introduction_to_pandas.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "320px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}