unit01_welcome.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# Unit 1: Interactive notebooks & tabular data objects\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "## Goals of this lecture:\n",
    "* Understand what Interactive computing means\n",
    "* Understand how Jupyter Notebook works\n",
    "* Tabular data building blocks: Numpy arrays, Pandas series and Pandas dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "\n",
    "1. [Introducing Notebooks and Markdown](#section1)\n",
    "2. [NumPy arrays](#section2)\n",
    "3. [Introducing Pandas](#section3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "source": [
    "<a id='section1'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 1. Interactive computing: Jupyter Notebooks\n",
    "\n",
    "* Interactive computing is a computer program with a \"human in the loop.\"\n",
    "* A combination of text (written in Markdown) & code that you can run (written in Python). \n",
    "* Run code, display and explain results.\n",
    "* **Tell a story**.  \n",
    "\n",
    "<div>\n",
    "<center><img src=\"https://github.com/nlihin/data-analytics/blob/main/images/tell%20a%20story.jpg?raw=true\" width=\"500\"/></center> \n",
    "</div>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### ipynb\n",
    "\n",
    "This notebook is in ipynb format.  \n",
    "> ipynb stands for: Interactive Python NoteBook\n",
    "\n",
    "We'll run ipynb through JupyterLabs. You can also run it using [Colab](https://colab.research.google.com/)\n",
    "\n",
    "Advanced content:\n",
    "* ipynb files are stored as JSON and can be opened with any text editor\n",
    "* If you want to read more, [look here](https://jupyterlab.readthedocs.io/en/latest/user/notebook.html#notebook).\n",
    "* Learn more here: [Jupyter: Thinking and Storytelling With Code and Data](https://ieeexplore.ieee.org/abstract/document/9387490)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Markdown\n",
    "\n",
    "> Markdown is an easy markup language - a language for formatting text. \n",
    "\n",
    "Double click on the cell to enter **edit mode**\n",
    "Shift + Enter to run\n",
    "\n",
    "Advanced content: \n",
    "* [More on Markdown](http://daringfireball.net/projects/markdown/syntax)\n",
    "* When Markdown is not enough, use html\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Keyboard shortcuts - command mode\n",
    "\n",
    "* Scroll up and down your cells with your Up and Down keys.  \n",
    "* Press A or B to insert a new cell above or below the active cell.  \n",
    "* M will transform the active cell to a Markdown cell.  \n",
    "* Y will set the active cell to a code cell.  \n",
    "* D + D (D twice) will delete the active cell.  \n",
    "* Z will undo cell deletion.  \n",
    "* Hold Shift and press Up or Down to select multiple cells at once.\n",
    "* With multiple cells selected, Shift + M will merge your selection. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Keyboard shortcuts - edit mode\n",
    "\n",
    "* Ctrl + Enter to run the current cell.  \n",
    "* Shift + Enter to run the current cell and move to the next cell (or create a new one if there isn’t a next cell)  \n",
    "* Alt + Enter to run the current cell and insert a new cell below.  \n",
    "* Ctrl + Shift + - will split the active cell at the cursor.  \n",
    "* Ctrl + Click to create multiple cursors within a cell.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Markdown syntax\n",
    "\n",
    "#  level 1 heading\n",
    "\n",
    "##  level 2 heading\n",
    "\n",
    "### level 3 heading\n",
    "\n",
    "---\n",
    "\n",
    "> a blockquote\n",
    ">> nested"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "###Markdown syntax\n",
    "**bold emphasis**  *italic emphasis* \n",
    "\n",
    "1. Numbered list.\n",
    "2. Another item.\n",
    "\n",
    "[hyperlinks](https://www.dataquest.io)\n",
    "\n",
    "\n",
    "Easy way to add an image: ![Alt text](https://github.com/nlihin/data-analytics/blob/main/images/LogoArielUniversity.jpg?raw=true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Code cells\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello world\n"
     ]
    }
   ],
   "source": [
    "print ('hello world')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "The last line of every code cell is displayed   \n",
    "Even if you don't print it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = 2 + 2 # The result of this line will not be displayed\n",
    "b = 3 + 3 # The result of this line will be displayed, because it is the last line of the cell\n",
    "a\n",
    "b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "### <span style=\"color:blue\"> Exercise:</span>\n",
    "> How do you suggest to print both a and b? Try it:\n",
    ">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Notebook Kernal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class_name = \"Intro to Data Analysis\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "message = class_name + \" is great!!!!!!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Intro to Data Analysis is great!!!!!!'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "message"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "### <span style=\"color:blue\"> Exercise:</span>\n",
    "> Change the `class_name` (remove the # sign first)\n",
    ">\n",
    "> What does message display now?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intro to Data Analysis\n",
      "Intro to Data Analysis is great!!!!!!\n"
     ]
    }
   ],
   "source": [
    "# class_name = \n",
    "print(class_name)\n",
    "print(message)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "The Kernal is a computational engine that runs code in Jupyter Notebook.\n",
    "\n",
    "- **Key Features:**\n",
    "  - **Run Code**: Executes one cell at a time.\n",
    "  - **Keeps Memory**: Remembers variables/functions until you restart.\n",
    "  - **Execution Numbers**: The number next to the cell (e.g., `[3]`) shows the order it was run."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "* Often, after changing a cell, you'll want to rerun all the cells below it\n",
    "  * \"Run -> Run Selected Cell and All Below\".\n",
    "\n",
    "**Before submitting your exam/project/homework**  \n",
    "**\"Kernal -> Restart kernal and run all cells\"** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 2. NumPy arrays \n",
    "\n",
    "<div>\n",
    "<center><img src=\"https://github.com/nlihin/data-analytics/blob/main/images/escher.PNG?raw=true\" width=\"300\"/>\n",
    "    <p style=\"text-align: center;\"><em>Learning to do the impossible. \"Waterfall\" by M.C. Escher, 1961.</em></p></center>\n",
    "</div>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Our data is mostly numerical data, e.g., stock prices, sales figures, sensor measurements, sports scores, database tables, etc.  \n",
    "The Numpy library provides specialized data structures, functions, and other tools for numerical computing in Python.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "### The difference between a list and a np.array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "student1 = [100, 88, 97]\n",
    "student2 = [100, 88, 60]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "raw_mimetype": "",
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "type(student1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "raw_mimetype": "",
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "student1 == student2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "raw_mimetype": "",
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "student1 + student2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Same data, but in a np.array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "student1_np = np.array([100, 88, 97])\n",
    "student2_np = np.array([100, 88, 60])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "raw_mimetype": "",
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "type(student1_np)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "raw_mimetype": "",
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "student1_np == student2_np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Numerical operations with np.array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Numpy arrays are better than list for operating on numerical data:\n",
    "\n",
    "- **Ease of use:**  small, concise, and intuitive mathematical expressions rather than using loops & custom functions.\n",
    "\n",
    "- **Performance:** Numpy operations and functions are implemented internally in C++, which makes them much faster than using Python statements & loops that are interpreted at runtime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Compute the average score in each exam:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "raw_mimetype": "",
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "(student1_np + student2_np)/2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Compare two arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "x = np.array([1,2,\"cat\"])\n",
    "y = np.array([1,3,\"cat\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "raw_mimetype": "",
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "x == y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<a id='section3'></a>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 3. Introducing Pandas\n",
    "---\n",
    "\n",
    "<div>\n",
    "<center><img src=\"https://github.com/nlihin/data-analytics/blob/main/images/pandas.JPG?raw=true\" width=\"400\"/></center>\n",
    "</div>\n",
    "\n",
    "\n",
    "If you want to read some more:\n",
    "* [Panda's documentation](https://pandas.pydata.org/pandas-docs/stable/)  \n",
    "* [Intro to data structures](https://pandas.pydata.org/docs/user_guide/dsintro.html)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### The pandas library \n",
    "Pandas is a popular Python library used for working in tabular data (similar to the data stored in a spreadsheet).\n",
    "\n",
    "There are two main data structure used by pandas\n",
    "- Series: a one-dimensional labeled array\n",
    "- Dataframe: a two-dimensional labeled data structure (think of a spreadsheet, table, or a dictionary of Series objects)\n",
    "\n",
    "Pandas is efficient for **data manipulation and analysis**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### The pandas Dataframe object\n",
    "First, import pandas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Reading a Dataframe from an online file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Date</th>\n",
       "      <th>AveragePrice</th>\n",
       "      <th>total_sold</th>\n",
       "      <th>small_sold</th>\n",
       "      <th>large_sold</th>\n",
       "      <th>sma</th>\n",
       "      <th>Total Bags</th>\n",
       "      <th>Small Bags</th>\n",
       "      <th>Large Bags</th>\n",
       "      <th>XLarge Bags</th>\n",
       "      <th>type</th>\n",
       "      <th>year</th>\n",
       "      <th>region</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>27/12/2015</td>\n",
       "      <td>1.33</td>\n",
       "      <td>64236.62</td>\n",
       "      <td>1036.74</td>\n",
       "      <td>54454.85</td>\n",
       "      <td>48.16</td>\n",
       "      <td>8696.87</td>\n",
       "      <td>8603.62</td>\n",
       "      <td>93.25</td>\n",
       "      <td>0.0</td>\n",
       "      <td>conventional</td>\n",
       "      <td>2015</td>\n",
       "      <td>Albany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>20/12/2015</td>\n",
       "      <td>1.35</td>\n",
       "      <td>54876.98</td>\n",
       "      <td>674.28</td>\n",
       "      <td>44638.81</td>\n",
       "      <td>58.33</td>\n",
       "      <td>9505.56</td>\n",
       "      <td>9408.07</td>\n",
       "      <td>97.49</td>\n",
       "      <td>0.0</td>\n",
       "      <td>conventional</td>\n",
       "      <td>2015</td>\n",
       "      <td>Albany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>13/12/2015</td>\n",
       "      <td>0.93</td>\n",
       "      <td>118220.22</td>\n",
       "      <td>794.70</td>\n",
       "      <td>109149.67</td>\n",
       "      <td>130.50</td>\n",
       "      <td>8145.35</td>\n",
       "      <td>8042.21</td>\n",
       "      <td>103.14</td>\n",
       "      <td>0.0</td>\n",
       "      <td>conventional</td>\n",
       "      <td>2015</td>\n",
       "      <td>Albany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>06/12/2015</td>\n",
       "      <td>1.08</td>\n",
       "      <td>78992.15</td>\n",
       "      <td>1132.00</td>\n",
       "      <td>71976.41</td>\n",
       "      <td>72.58</td>\n",
       "      <td>5811.16</td>\n",
       "      <td>5677.40</td>\n",
       "      <td>133.76</td>\n",
       "      <td>0.0</td>\n",
       "      <td>conventional</td>\n",
       "      <td>2015</td>\n",
       "      <td>Albany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>29/11/2015</td>\n",
       "      <td>1.28</td>\n",
       "      <td>51039.60</td>\n",
       "      <td>941.48</td>\n",
       "      <td>43838.39</td>\n",
       "      <td>75.78</td>\n",
       "      <td>6183.95</td>\n",
       "      <td>5986.26</td>\n",
       "      <td>197.69</td>\n",
       "      <td>0.0</td>\n",
       "      <td>conventional</td>\n",
       "      <td>2015</td>\n",
       "      <td>Albany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18244</th>\n",
       "      <td>7</td>\n",
       "      <td>04/02/2018</td>\n",
       "      <td>1.63</td>\n",
       "      <td>17074.83</td>\n",
       "      <td>2046.96</td>\n",
       "      <td>1529.20</td>\n",
       "      <td>0.00</td>\n",
       "      <td>13498.67</td>\n",
       "      <td>13066.82</td>\n",
       "      <td>431.85</td>\n",
       "      <td>0.0</td>\n",
       "      <td>organic</td>\n",
       "      <td>2018</td>\n",
       "      <td>WestTexNewMexico</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18245</th>\n",
       "      <td>8</td>\n",
       "      <td>28/01/2018</td>\n",
       "      <td>1.71</td>\n",
       "      <td>13888.04</td>\n",
       "      <td>1191.70</td>\n",
       "      <td>3431.50</td>\n",
       "      <td>0.00</td>\n",
       "      <td>9264.84</td>\n",
       "      <td>8940.04</td>\n",
       "      <td>324.80</td>\n",
       "      <td>0.0</td>\n",
       "      <td>organic</td>\n",
       "      <td>2018</td>\n",
       "      <td>WestTexNewMexico</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18246</th>\n",
       "      <td>9</td>\n",
       "      <td>21/01/2018</td>\n",
       "      <td>1.87</td>\n",
       "      <td>13766.76</td>\n",
       "      <td>1191.92</td>\n",
       "      <td>2452.79</td>\n",
       "      <td>727.94</td>\n",
       "      <td>9394.11</td>\n",
       "      <td>9351.80</td>\n",
       "      <td>42.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>organic</td>\n",
       "      <td>2018</td>\n",
       "      <td>WestTexNewMexico</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18247</th>\n",
       "      <td>10</td>\n",
       "      <td>14/01/2018</td>\n",
       "      <td>1.93</td>\n",
       "      <td>16205.22</td>\n",
       "      <td>1527.63</td>\n",
       "      <td>2981.04</td>\n",
       "      <td>727.01</td>\n",
       "      <td>10969.54</td>\n",
       "      <td>10919.54</td>\n",
       "      <td>50.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>organic</td>\n",
       "      <td>2018</td>\n",
       "      <td>WestTexNewMexico</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18248</th>\n",
       "      <td>11</td>\n",
       "      <td>07/01/2018</td>\n",
       "      <td>1.62</td>\n",
       "      <td>17489.58</td>\n",
       "      <td>2894.77</td>\n",
       "      <td>2356.13</td>\n",
       "      <td>224.53</td>\n",
       "      <td>12014.15</td>\n",
       "      <td>11988.14</td>\n",
       "      <td>26.01</td>\n",
       "      <td>0.0</td>\n",
       "      <td>organic</td>\n",
       "      <td>2018</td>\n",
       "      <td>WestTexNewMexico</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>18249 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0        Date  AveragePrice  total_sold  small_sold  \\\n",
       "0               0  27/12/2015          1.33    64236.62     1036.74   \n",
       "1               1  20/12/2015          1.35    54876.98      674.28   \n",
       "2               2  13/12/2015          0.93   118220.22      794.70   \n",
       "3               3  06/12/2015          1.08    78992.15     1132.00   \n",
       "4               4  29/11/2015          1.28    51039.60      941.48   \n",
       "...           ...         ...           ...         ...         ...   \n",
       "18244           7  04/02/2018          1.63    17074.83     2046.96   \n",
       "18245           8  28/01/2018          1.71    13888.04     1191.70   \n",
       "18246           9  21/01/2018          1.87    13766.76     1191.92   \n",
       "18247          10  14/01/2018          1.93    16205.22     1527.63   \n",
       "18248          11  07/01/2018          1.62    17489.58     2894.77   \n",
       "\n",
       "       large_sold     sma  Total Bags  Small Bags  Large Bags  XLarge Bags  \\\n",
       "0        54454.85   48.16     8696.87     8603.62       93.25          0.0   \n",
       "1        44638.81   58.33     9505.56     9408.07       97.49          0.0   \n",
       "2       109149.67  130.50     8145.35     8042.21      103.14          0.0   \n",
       "3        71976.41   72.58     5811.16     5677.40      133.76          0.0   \n",
       "4        43838.39   75.78     6183.95     5986.26      197.69          0.0   \n",
       "...           ...     ...         ...         ...         ...          ...   \n",
       "18244     1529.20    0.00    13498.67    13066.82      431.85          0.0   \n",
       "18245     3431.50    0.00     9264.84     8940.04      324.80          0.0   \n",
       "18246     2452.79  727.94     9394.11     9351.80       42.31          0.0   \n",
       "18247     2981.04  727.01    10969.54    10919.54       50.00          0.0   \n",
       "18248     2356.13  224.53    12014.15    11988.14       26.01          0.0   \n",
       "\n",
       "               type  year            region  \n",
       "0      conventional  2015            Albany  \n",
       "1      conventional  2015            Albany  \n",
       "2      conventional  2015            Albany  \n",
       "3      conventional  2015            Albany  \n",
       "4      conventional  2015            Albany  \n",
       "...             ...   ...               ...  \n",
       "18244       organic  2018  WestTexNewMexico  \n",
       "18245       organic  2018  WestTexNewMexico  \n",
       "18246       organic  2018  WestTexNewMexico  \n",
       "18247       organic  2018  WestTexNewMexico  \n",
       "18248       organic  2018  WestTexNewMexico  \n",
       "\n",
       "[18249 rows x 14 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "file_name = \"https://raw.githubusercontent.com/nlihin/data-analytics/main/datasets/avocado.csv\"\n",
    "df = pd.read_csv(file_name)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "This file contains data on Avocado sales. For example - the number of bags sold. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### The pandas Series object"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Each column in the `dataframe` object is a `series`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "We can select a single column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                  Albany\n",
       "1                  Albany\n",
       "2                  Albany\n",
       "3                  Albany\n",
       "4                  Albany\n",
       "               ...       \n",
       "18244    WestTexNewMexico\n",
       "18245    WestTexNewMexico\n",
       "18246    WestTexNewMexico\n",
       "18247    WestTexNewMexico\n",
       "18248    WestTexNewMexico\n",
       "Name: region, Length: 18249, dtype: object"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"region\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Ask for its type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(df[\"region\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "### <span style=\"color:blue\"> Exercise:</span>\n",
    "> Display the `Small Bags` column\n",
    ">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Pandas Series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "A series logic is similar to that of a `np.array`, not that of a `list`  \n",
    "For example, if we want to check if `Total Bags` is correct:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         True\n",
       "1         True\n",
       "2         True\n",
       "3         True\n",
       "4         True\n",
       "         ...  \n",
       "18244     True\n",
       "18245     True\n",
       "18246    False\n",
       "18247     True\n",
       "18248     True\n",
       "Length: 18249, dtype: bool"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"Small Bags\"]+ df[\"Large Bags\"] + df[\"XLarge Bags\"] == df[\"Total Bags\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "We see that it is not always correct. How many times is it correct? We use `sum()` to find out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14213"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(df[\"Small Bags\"]+ df[\"Large Bags\"] + df[\"XLarge Bags\"] == df[\"Total Bags\"]).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## End of lesson - what you need to know:\n",
    "\n",
    "* How to work with a Jupyter notebook\n",
    "* What are Numpay array, Pandas series, Pandas dataframes\n",
    "* Read a file and some simple manipulations (see Exercise 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "\n",
    "<div>\n",
    "<center><img src=\"https://github.com/nlihin/data-analytics/blob/main/images/hawaii.PNG?raw=true\" width=\"700\"/> </center>\n",
    "</div>\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}