01. Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "Being able to easily load and process data is a crucial task that can make any data science more pleasant. In this notebook, we will cover most common types often encountered in data science tasks, and we will be using this data throughout the rest of this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "using BenchmarkTools\n",
    "using DataFrames\n",
    "using DelimitedFiles\n",
    "using CSV\n",
    "using XLSX\n",
    "using Downloads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 🗃️ Get some data\n",
    "In Julia, it's pretty easy to dowload a file from the web using the `download` (https://docs.julialang.org/en/v1/stdlib/Downloads/) function. But also, you can use your favorite command line commad to download files by easily switching from Julia via the `;` key. Let's try both.\n",
    "\n",
    "Note: `download` depends on external tools such as curl, wget or fetch. So you must have one of these."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"programming_languages.csv\""
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "P = Downloads.download(\"https://raw.githubusercontent.com/nassarhuda/easy_data/master/programming_languages.csv\",\n",
    "    \"programming_languages.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way would be to use a shell command to get the same file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 📂 Read your data from text files.\n",
    "The key question here is to load data from files such as `csv` files, `xlsx` files, or just raw text files. We will go over some Julia packages that will allow us to read such files very easily.\n",
    "\n",
    "Let's start with the package `DelimitedFiles` which is in the standard library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#=\n",
    "readdlm(source, \n",
    "    delim::AbstractChar, \n",
    "    T::Type, \n",
    "    eol::AbstractChar; \n",
    "    header=false, \n",
    "    skipstart=0, \n",
    "    skipblanks=true, \n",
    "    use_mmap, \n",
    "    quotes=true, \n",
    "    dims, \n",
    "    comments=false, \n",
    "    comment_char='#')\n",
    "=#\n",
    "P,H = readdlm(\"programming_languages.csv\",',';header=true);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1×2 Matrix{AbstractString}:\n",
       " \"year\"  \"language\""
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "H"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To write to a text file, you can:\n",
    "writedlm(\"programminglanguages_dlm.txt\", P, '-')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A more powerful package to use here is the `CSV` package. By default, the CSV package imports the data to a DataFrame, which can have several advantages as we will see below.\n",
    "\n",
    "In general,[`CSV.jl`](https://juliadata.github.io/CSV.jl/stable/) is the recommended way to load CSVs in Julia. Only use `DelimitedFiles` when you have a more complicated file where you want to specify several things."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "C = CSV.read(\"programming_languages.csv\", DataFrame);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "typeof(C) = DataFrame\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>year</th><th>language</th></tr><tr><th></th><th>Int64</th><th>String</th></tr></thead><tbody><p>10 rows × 2 columns</p><tr><th>1</th><td>1951</td><td>Regional Assembly Language</td></tr><tr><th>2</th><td>1952</td><td>Autocode</td></tr><tr><th>3</th><td>1954</td><td>IPL</td></tr><tr><th>4</th><td>1955</td><td>FLOW-MATIC</td></tr><tr><th>5</th><td>1957</td><td>FORTRAN</td></tr><tr><th>6</th><td>1957</td><td>COMTRAN</td></tr><tr><th>7</th><td>1958</td><td>LISP</td></tr><tr><th>8</th><td>1958</td><td>ALGOL 58</td></tr><tr><th>9</th><td>1959</td><td>FACT</td></tr><tr><th>10</th><td>1959</td><td>COBOL</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& year & language\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & String\\\\\n",
       "\t\\hline\n",
       "\t1 & 1951 & Regional Assembly Language \\\\\n",
       "\t2 & 1952 & Autocode \\\\\n",
       "\t3 & 1954 & IPL \\\\\n",
       "\t4 & 1955 & FLOW-MATIC \\\\\n",
       "\t5 & 1957 & FORTRAN \\\\\n",
       "\t6 & 1957 & COMTRAN \\\\\n",
       "\t7 & 1958 & LISP \\\\\n",
       "\t8 & 1958 & ALGOL 58 \\\\\n",
       "\t9 & 1959 & FACT \\\\\n",
       "\t10 & 1959 & COBOL \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m10×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m year  \u001b[0m\u001b[1m language                   \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64 \u001b[0m\u001b[90m String                     \u001b[0m\n",
       "─────┼───────────────────────────────────\n",
       "   1 │  1951  Regional Assembly Language\n",
       "   2 │  1952  Autocode\n",
       "   3 │  1954  IPL\n",
       "   4 │  1955  FLOW-MATIC\n",
       "   5 │  1957  FORTRAN\n",
       "   6 │  1957  COMTRAN\n",
       "   7 │  1958  LISP\n",
       "   8 │  1958  ALGOL 58\n",
       "   9 │  1959  FACT\n",
       "  10 │  1959  COBOL"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@show typeof(C)\n",
    "C[1:10,:]\n",
    "# C.year #[!,:year]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "typeof(P) = Matrix{Any}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "10×2 Matrix{Any}:\n",
       " 1951  \"Regional Assembly Language\"\n",
       " 1952  \"Autocode\"\n",
       " 1954  \"IPL\"\n",
       " 1955  \"FLOW-MATIC\"\n",
       " 1957  \"FORTRAN\"\n",
       " 1957  \"COMTRAN\"\n",
       " 1958  \"LISP\"\n",
       " 1958  \"ALGOL 58\"\n",
       " 1959  \"FACT\"\n",
       " 1959  \"COBOL\""
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "@show typeof(P)\n",
    "P[1:10,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2-element Vector{String}:\n",
       " \"year\"\n",
       " \"language\""
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names(C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>variable</th><th>mean</th><th>min</th><th>median</th><th>max</th><th>nmissing</th><th>eltype</th></tr><tr><th></th><th>Symbol</th><th>Union…</th><th>Any</th><th>Union…</th><th>Any</th><th>Int64</th><th>DataType</th></tr></thead><tbody><p>2 rows × 7 columns</p><tr><th>1</th><td>year</td><td>1982.99</td><td>1951</td><td>1986.0</td><td>2014</td><td>0</td><td>Int64</td></tr><tr><th>2</th><td>language</td><td></td><td>ALGOL 58</td><td></td><td>dBase III</td><td>0</td><td>String</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccccc}\n",
       "\t& variable & mean & min & median & max & nmissing & eltype\\\\\n",
       "\t\\hline\n",
       "\t& Symbol & Union… & Any & Union… & Any & Int64 & DataType\\\\\n",
       "\t\\hline\n",
       "\t1 & year & 1982.99 & 1951 & 1986.0 & 2014 & 0 & Int64 \\\\\n",
       "\t2 & language &  & ALGOL 58 &  & dBase III & 0 & String \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m2×7 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m variable \u001b[0m\u001b[1m mean    \u001b[0m\u001b[1m min      \u001b[0m\u001b[1m median \u001b[0m\u001b[1m max       \u001b[0m\u001b[1m nmissing \u001b[0m\u001b[1m eltype   \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Symbol   \u001b[0m\u001b[90m Union…  \u001b[0m\u001b[90m Any      \u001b[0m\u001b[90m Union… \u001b[0m\u001b[90m Any       \u001b[0m\u001b[90m Int64    \u001b[0m\u001b[90m DataType \u001b[0m\n",
       "─────┼────────────────────────────────────────────────────────────────────\n",
       "   1 │ year      1982.99  1951      1986.0  2014              0  Int64\n",
       "   2 │ language \u001b[90m         \u001b[0m ALGOL 58 \u001b[90m        \u001b[0m dBase III         0  String"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names(C)\n",
    "C.year\n",
    "C.language\n",
    "describe(C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  87.708 μs (325 allocations: 51.19 KiB)\n",
      "  35.417 μs (227 allocations: 22.02 KiB)\n"
     ]
    }
   ],
   "source": [
    "@btime P,H = readdlm(\"programming_languages.csv\",',';header=true);\n",
    "@btime C = CSV.read(\"programming_languages.csv\", DataFrame);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"programminglanguages_CSV.csv\""
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# To write to a *.csv file using the CSV package\n",
    "CSV.write(\"programminglanguages_CSV.csv\", DataFrame(P, :auto))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another type of files that we may often need to read is `XLSX` files. Let's try to read a new file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9×6 Matrix{Any}:\n",
       "      \"RegionID\"  \"RegionName\"    …      \"2008-03\"      \"2008-04\"\n",
       "  6181            \"New York\"             missing        missing\n",
       " 12447            \"Los Angeles\"      1446           1705\n",
       " 39051            \"Houston\"          2926           3121\n",
       " 17426            \"Chicago\"          2910           3022\n",
       "  6915            \"San Antonio\"   …  1479           1529\n",
       " 13271            \"Philadelphia\"     1609           1795\n",
       " 40326            \"Phoenix\"          1310           1519\n",
       " 18959            \"Las Vegas\"        1618           1856"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "T = XLSX.readdata(\"data/zillow_data_download_april2020.xlsx\", #file name\n",
    "    \"Sale_counts_city\", #sheet name\n",
    "    \"A1:F9\" #cell range\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you don't want to specify cell ranges... though this will take a little longer..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "G = XLSX.readtable(\"data/zillow_data_download_april2020.xlsx\",\"Sale_counts_city\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, `G` is a tuple of two items. The first is an vector of vectors where each vector corresponds to a column in the excel file. And the second is the header with the column names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "148-element Vector{Any}:\n",
       " Any[6181, 12447, 39051, 17426, 6915, 13271, 40326, 18959, 54296, 38128  …  396952, 397236, 398030, 398104, 398357, 398712, 398716, 399081, 737789, 760882]\n",
       " Any[\"New York\", \"Los Angeles\", \"Houston\", \"Chicago\", \"San Antonio\", \"Philadelphia\", \"Phoenix\", \"Las Vegas\", \"San Diego\", \"Dallas\"  …  \"Barnard Plantation\", \"Windsor Place\", \"Stockbridge\", \"Mattamiscontis\", \"Chase Stream\", \"Bowdoin College Grant West\", \"Summerset\", \"Long Pond\", \"Hideout\", \"Ebeemee\"]\n",
       " Any[\"New York\", \"California\", \"Texas\", \"Illinois\", \"Texas\", \"Pennsylvania\", \"Arizona\", \"Nevada\", \"California\", \"Texas\"  …  \"Maine\", \"Missouri\", \"Wisconsin\", \"Maine\", \"Maine\", \"Maine\", \"South Dakota\", \"Maine\", \"Utah\", \"Maine\"]\n",
       " Any[1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  28750, 28751, 28752, 28753, 28754, 28755, 28756, 28757, 28758, 28759]\n",
       " Any[missing, 1446, 2926, 2910, 1479, 1609, 1310, 1618, 772, 1158  …  0, 0, 0, 0, 0, 0, 0, 0, 1, 0]\n",
       " Any[missing, 1705, 3121, 3022, 1529, 1795, 1519, 1856, 1057, 1232  …  0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
       " Any[missing, 1795, 3220, 2937, 1582, 1709, 1654, 1961, 1195, 1240  …  0, 0, 0, 0, 0, 0, 2, 0, 0, 0]\n",
       " Any[missing, 1890, 3405, 3224, 1761, 1914, 1743, 2022, 1115, 1236  …  0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
       " Any[missing, 2072, 3352, 3464, 1678, 1984, 1975, 2219, 1272, 1392  …  0, 0, 0, 0, 0, 1, 1, 0, 0, 0]\n",
       " Any[missing, 1876, 2962, 3371, 1468, 1739, 1809, 2248, 1171, 1138  …  0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
       " Any[missing, 1981, 2017, 2979, 1480, 1635, 1965, 2207, 1193, 1042  …  0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
       " Any[missing, 2195, 2595, 2869, 1475, 1598, 2074, 2372, 1319, 1089  …  0, 1, 0, 0, 0, 0, 0, 0, 0, 0]\n",
       " Any[missing, 1579, 1865, 1915, 949, 1169, 1435, 1880, 873, 772  …  0, 0, 0, 0, 1, 0, 0, 0, 0, 0]\n",
       " ⋮\n",
       " Any[3456, 2150, 3283, 2735, 1989, 1958, 3378, 2748, 1207, 1058  …  0, 1, 0, 1, 4, 0, 6, 0, 0, 0]\n",
       " Any[3652, 2537, 3620, 3197, 2137, 2177, 3728, 2959, 1384, 1221  …  0, 0, 0, 0, 0, 0, 4, 0, 0, 0]\n",
       " Any[3473, 2651, 3894, 3702, 2384, 2682, 3898, 3284, 1542, 1407  …  0, 0, 0, 0, 0, 2, 3, 0, 0, 0]\n",
       " Any[3900, 2504, 3511, 3549, 2242, 2270, 3570, 3058, 1433, 1235  …  0, 0, 2, 0, 0, 2, 6, 0, 1, 0]\n",
       " Any[4493, 2675, 3687, 3622, 2300, 2479, 3616, 3235, 1529, 1423  …  0, 0, 0, 0, 0, 3, 10, 0, 1, 0]\n",
       " Any[4266, 2759, 3414, 3619, 2233, 2503, 3581, 3254, 1479, 1266  …  0, missing, 0, 0, 0, 3, 5, 0, 2, 0]\n",
       " Any[4183, 2334, 2803, 2938, 1788, 2184, 3199, 2821, 1307, 1143  …  0, missing, 0, 0, 1, 1, 4, 0, 1, 0]\n",
       " Any[4232, 2594, 2969, 2930, 1917, 2337, 3189, 3097, 1356, 1239  …  0, missing, 0, 0, 0, missing, 8, 0, 2, 0]\n",
       " Any[3869, 2272, 2699, 2176, 1520, 1931, 2867, 2702, 1179, 1117  …  0, missing, 0, 0, 0, missing, 8, 0, missing, 0]\n",
       " Any[3938, 2394, 3338, 2866, 1876, 2192, 2934, 2810, 1276, 1315  …  0, missing, 0, 0, 0, missing, 5, 0, missing, 0]\n",
       " Any[4306, 2006, 2407, 2240, 1460, 2381, 2226, 2512, 1020, 1056  …  0, missing, 0, 0, 0, missing, 1, 0, missing, 1]\n",
       " Any[4054, 1522, 2682, 2100, 1626, 1620, 2325, 2547, missing, 1292  …  missing, missing, missing, missing, missing, missing, 2, 0, missing, 0]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "G[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10-element Vector{Any}:\n",
       "  6181\n",
       " 12447\n",
       " 39051\n",
       " 17426\n",
       "  6915\n",
       " 13271\n",
       " 40326\n",
       " 18959\n",
       " 54296\n",
       " 38128"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "G[1][1][1:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10-element Vector{Symbol}:\n",
       " :RegionID\n",
       " :RegionName\n",
       " :StateName\n",
       " :SizeRank\n",
       " Symbol(\"2008-03\")\n",
       " Symbol(\"2008-04\")\n",
       " Symbol(\"2008-05\")\n",
       " Symbol(\"2008-06\")\n",
       " Symbol(\"2008-07\")\n",
       " Symbol(\"2008-08\")"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "G[2][1:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can easily store this data in a DataFrame. `DataFrame(G...)` uses the \"splat\" operator to _unwrap_ these arrays and pass them to the DataFrame constructor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>RegionID</th><th>RegionName</th><th>StateName</th><th>SizeRank</th><th>2008-03</th><th>2008-04</th><th>2008-05</th></tr><tr><th></th><th>Any</th><th>Any</th><th>Any</th><th>Any</th><th>Any</th><th>Any</th><th>Any</th></tr></thead><tbody><p>28,759 rows × 148 columns (omitted printing of 141 columns)</p><tr><th>1</th><td>6181</td><td>New York</td><td>New York</td><td>1</td><td><em>missing</em></td><td><em>missing</em></td><td><em>missing</em></td></tr><tr><th>2</th><td>12447</td><td>Los Angeles</td><td>California</td><td>2</td><td>1446</td><td>1705</td><td>1795</td></tr><tr><th>3</th><td>39051</td><td>Houston</td><td>Texas</td><td>3</td><td>2926</td><td>3121</td><td>3220</td></tr><tr><th>4</th><td>17426</td><td>Chicago</td><td>Illinois</td><td>4</td><td>2910</td><td>3022</td><td>2937</td></tr><tr><th>5</th><td>6915</td><td>San Antonio</td><td>Texas</td><td>5</td><td>1479</td><td>1529</td><td>1582</td></tr><tr><th>6</th><td>13271</td><td>Philadelphia</td><td>Pennsylvania</td><td>6</td><td>1609</td><td>1795</td><td>1709</td></tr><tr><th>7</th><td>40326</td><td>Phoenix</td><td>Arizona</td><td>7</td><td>1310</td><td>1519</td><td>1654</td></tr><tr><th>8</th><td>18959</td><td>Las Vegas</td><td>Nevada</td><td>8</td><td>1618</td><td>1856</td><td>1961</td></tr><tr><th>9</th><td>54296</td><td>San Diego</td><td>California</td><td>9</td><td>772</td><td>1057</td><td>1195</td></tr><tr><th>10</th><td>38128</td><td>Dallas</td><td>Texas</td><td>10</td><td>1158</td><td>1232</td><td>1240</td></tr><tr><th>11</th><td>10221</td><td>Austin</td><td>Texas</td><td>11</td><td>997</td><td>1140</td><td>1168</td></tr><tr><th>12</th><td>33839</td><td>San Jose</td><td>California</td><td>12</td><td>455</td><td>555</td><td>583</td></tr><tr><th>13</th><td>25290</td><td>Jacksonville</td><td>Florida</td><td>13</td><td>613</td><td>730</td><td>738</td></tr><tr><th>14</th><td>32149</td><td>Indianapolis</td><td>Indiana</td><td>14</td><td><em>missing</em></td><td><em>missing</em></td><td><em>missing</em></td></tr><tr><th>15</th><td>20330</td><td>San Francisco</td><td>California</td><td>15</td><td>419</td><td>485</td><td>553</td></tr><tr><th>16</th><td>24043</td><td>Charlotte</td><td>North Carolina</td><td>16</td><td>1101</td><td>1201</td><td>1380</td></tr><tr><th>17</th><td>18172</td><td>Fort Worth</td><td>Texas</td><td>17</td><td>862</td><td>1130</td><td>1000</td></tr><tr><th>18</th><td>7481</td><td>Tucson</td><td>Arizona</td><td>18</td><td>737</td><td>841</td><td>842</td></tr><tr><th>19</th><td>10920</td><td>Columbus</td><td>Ohio</td><td>19</td><td>832</td><td>975</td><td>939</td></tr><tr><th>20</th><td>12455</td><td>Louisville</td><td>Kentucky</td><td>20</td><td>881</td><td>1017</td><td>967</td></tr><tr><th>21</th><td>13121</td><td>Orlando</td><td>Florida</td><td>21</td><td>550</td><td>507</td><td>489</td></tr><tr><th>22</th><td>17933</td><td>El Paso</td><td>Texas</td><td>22</td><td><em>missing</em></td><td><em>missing</em></td><td><em>missing</em></td></tr><tr><th>23</th><td>17762</td><td>Detroit</td><td>Michigan</td><td>23</td><td><em>missing</em></td><td><em>missing</em></td><td><em>missing</em></td></tr><tr><th>24</th><td>11093</td><td>Denver</td><td>Colorado</td><td>24</td><td>975</td><td>1076</td><td>1241</td></tr><tr><th>25</th><td>16037</td><td>Seattle</td><td>Washington</td><td>25</td><td>615</td><td>641</td><td>689</td></tr><tr><th>26</th><td>32811</td><td>Memphis</td><td>Tennessee</td><td>26</td><td>872</td><td>802</td><td>775</td></tr><tr><th>27</th><td>44269</td><td>Boston</td><td>Massachusetts</td><td>27</td><td><em>missing</em></td><td><em>missing</em></td><td><em>missing</em></td></tr><tr><th>28</th><td>41568</td><td>Washington</td><td>District of Columbia</td><td>28</td><td>479</td><td>429</td><td>503</td></tr><tr><th>29</th><td>13373</td><td>Portland</td><td>Oregon</td><td>29</td><td>787</td><td>745</td><td>926</td></tr><tr><th>30</th><td>6118</td><td>Nashville</td><td>Tennessee</td><td>30</td><td>671</td><td>778</td><td>779</td></tr><tr><th>&vellip;</th><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td><td>&vellip;</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccccccc}\n",
       "\t& RegionID & RegionName & StateName & SizeRank & 2008-03 & 2008-04 & 2008-05 & \\\\\n",
       "\t\\hline\n",
       "\t& Any & Any & Any & Any & Any & Any & Any & \\\\\n",
       "\t\\hline\n",
       "\t1 & 6181 & New York & New York & 1 & \\emph{missing} & \\emph{missing} & \\emph{missing} & $\\dots$ \\\\\n",
       "\t2 & 12447 & Los Angeles & California & 2 & 1446 & 1705 & 1795 & $\\dots$ \\\\\n",
       "\t3 & 39051 & Houston & Texas & 3 & 2926 & 3121 & 3220 & $\\dots$ \\\\\n",
       "\t4 & 17426 & Chicago & Illinois & 4 & 2910 & 3022 & 2937 & $\\dots$ \\\\\n",
       "\t5 & 6915 & San Antonio & Texas & 5 & 1479 & 1529 & 1582 & $\\dots$ \\\\\n",
       "\t6 & 13271 & Philadelphia & Pennsylvania & 6 & 1609 & 1795 & 1709 & $\\dots$ \\\\\n",
       "\t7 & 40326 & Phoenix & Arizona & 7 & 1310 & 1519 & 1654 & $\\dots$ \\\\\n",
       "\t8 & 18959 & Las Vegas & Nevada & 8 & 1618 & 1856 & 1961 & $\\dots$ \\\\\n",
       "\t9 & 54296 & San Diego & California & 9 & 772 & 1057 & 1195 & $\\dots$ \\\\\n",
       "\t10 & 38128 & Dallas & Texas & 10 & 1158 & 1232 & 1240 & $\\dots$ \\\\\n",
       "\t11 & 10221 & Austin & Texas & 11 & 997 & 1140 & 1168 & $\\dots$ \\\\\n",
       "\t12 & 33839 & San Jose & California & 12 & 455 & 555 & 583 & $\\dots$ \\\\\n",
       "\t13 & 25290 & Jacksonville & Florida & 13 & 613 & 730 & 738 & $\\dots$ \\\\\n",
       "\t14 & 32149 & Indianapolis & Indiana & 14 & \\emph{missing} & \\emph{missing} & \\emph{missing} & $\\dots$ \\\\\n",
       "\t15 & 20330 & San Francisco & California & 15 & 419 & 485 & 553 & $\\dots$ \\\\\n",
       "\t16 & 24043 & Charlotte & North Carolina & 16 & 1101 & 1201 & 1380 & $\\dots$ \\\\\n",
       "\t17 & 18172 & Fort Worth & Texas & 17 & 862 & 1130 & 1000 & $\\dots$ \\\\\n",
       "\t18 & 7481 & Tucson & Arizona & 18 & 737 & 841 & 842 & $\\dots$ \\\\\n",
       "\t19 & 10920 & Columbus & Ohio & 19 & 832 & 975 & 939 & $\\dots$ \\\\\n",
       "\t20 & 12455 & Louisville & Kentucky & 20 & 881 & 1017 & 967 & $\\dots$ \\\\\n",
       "\t21 & 13121 & Orlando & Florida & 21 & 550 & 507 & 489 & $\\dots$ \\\\\n",
       "\t22 & 17933 & El Paso & Texas & 22 & \\emph{missing} & \\emph{missing} & \\emph{missing} & $\\dots$ \\\\\n",
       "\t23 & 17762 & Detroit & Michigan & 23 & \\emph{missing} & \\emph{missing} & \\emph{missing} & $\\dots$ \\\\\n",
       "\t24 & 11093 & Denver & Colorado & 24 & 975 & 1076 & 1241 & $\\dots$ \\\\\n",
       "\t25 & 16037 & Seattle & Washington & 25 & 615 & 641 & 689 & $\\dots$ \\\\\n",
       "\t26 & 32811 & Memphis & Tennessee & 26 & 872 & 802 & 775 & $\\dots$ \\\\\n",
       "\t27 & 44269 & Boston & Massachusetts & 27 & \\emph{missing} & \\emph{missing} & \\emph{missing} & $\\dots$ \\\\\n",
       "\t28 & 41568 & Washington & District of Columbia & 28 & 479 & 429 & 503 & $\\dots$ \\\\\n",
       "\t29 & 13373 & Portland & Oregon & 29 & 787 & 745 & 926 & $\\dots$ \\\\\n",
       "\t30 & 6118 & Nashville & Tennessee & 30 & 671 & 778 & 779 & $\\dots$ \\\\\n",
       "\t$\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ &  \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m28759×148 DataFrame\u001b[0m\n",
       "\u001b[1m   Row \u001b[0m│\u001b[1m RegionID \u001b[0m\u001b[1m RegionName                 \u001b[0m\u001b[1m StateName     \u001b[0m\u001b[1m SizeRank \u001b[0m\u001b[1m 2008-0\u001b[0m ⋯\n",
       "\u001b[1m       \u001b[0m│\u001b[90m Any      \u001b[0m\u001b[90m Any                        \u001b[0m\u001b[90m Any           \u001b[0m\u001b[90m Any      \u001b[0m\u001b[90m Any   \u001b[0m ⋯\n",
       "───────┼────────────────────────────────────────────────────────────────────────\n",
       "     1 │ 6181      New York                    New York       1        \u001b[90m missin\u001b[0m ⋯\n",
       "     2 │ 12447     Los Angeles                 California     2         1446\n",
       "     3 │ 39051     Houston                     Texas          3         2926\n",
       "     4 │ 17426     Chicago                     Illinois       4         2910\n",
       "     5 │ 6915      San Antonio                 Texas          5         1479   ⋯\n",
       "     6 │ 13271     Philadelphia                Pennsylvania   6         1609\n",
       "     7 │ 40326     Phoenix                     Arizona        7         1310\n",
       "     8 │ 18959     Las Vegas                   Nevada         8         1618\n",
       "     9 │ 54296     San Diego                   California     9         772    ⋯\n",
       "    10 │ 38128     Dallas                      Texas          10        1158\n",
       "    11 │ 10221     Austin                      Texas          11        997\n",
       "   ⋮   │    ⋮                  ⋮                     ⋮           ⋮         ⋮   ⋱\n",
       " 28750 │ 396952    Barnard Plantation          Maine          28750     0\n",
       " 28751 │ 397236    Windsor Place               Missouri       28751     0      ⋯\n",
       " 28752 │ 398030    Stockbridge                 Wisconsin      28752     0\n",
       " 28753 │ 398104    Mattamiscontis              Maine          28753     0\n",
       " 28754 │ 398357    Chase Stream                Maine          28754     0\n",
       " 28755 │ 398712    Bowdoin College Grant West  Maine          28755     0      ⋯\n",
       " 28756 │ 398716    Summerset                   South Dakota   28756     0\n",
       " 28757 │ 399081    Long Pond                   Maine          28757     0\n",
       " 28758 │ 737789    Hideout                     Utah           28758     1\n",
       " 28759 │ 760882    Ebeemee                     Maine          28759     0      ⋯\n",
       "\u001b[36m                                              144 columns and 28738 rows omitted\u001b[0m"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "D = DataFrame(G...) # equivalent to DataFrame(G[1],G[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>item</th><th>price</th></tr><tr><th></th><th>String</th><th>Float64</th></tr></thead><tbody><p>4 rows × 2 columns</p><tr><th>1</th><td>apple</td><td>0.85</td></tr><tr><th>2</th><td>cucumber</td><td>1.6</td></tr><tr><th>3</th><td>tomato</td><td>0.8</td></tr><tr><th>4</th><td>banana</td><td>0.6</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& item & price\\\\\n",
       "\t\\hline\n",
       "\t& String & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & apple & 0.85 \\\\\n",
       "\t2 & cucumber & 1.6 \\\\\n",
       "\t3 & tomato & 0.8 \\\\\n",
       "\t4 & banana & 0.6 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m4×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m item     \u001b[0m\u001b[1m price   \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m String   \u001b[0m\u001b[90m Float64 \u001b[0m\n",
       "─────┼───────────────────\n",
       "   1 │ apple        0.85\n",
       "   2 │ cucumber     1.6\n",
       "   3 │ tomato       0.8\n",
       "   4 │ banana       0.6"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "foods = [\"apple\", \"cucumber\", \"tomato\", \"banana\"]\n",
    "calories = [105,47,22,105]\n",
    "prices = [0.85,1.6,0.8,0.6,]\n",
    "dataframe_calories = DataFrame(item=foods,calories=calories)\n",
    "dataframe_prices = DataFrame(item=foods,price=prices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>item</th><th>calories</th><th>price</th></tr><tr><th></th><th>String</th><th>Int64</th><th>Float64</th></tr></thead><tbody><p>4 rows × 3 columns</p><tr><th>1</th><td>apple</td><td>105</td><td>0.85</td></tr><tr><th>2</th><td>cucumber</td><td>47</td><td>1.6</td></tr><tr><th>3</th><td>tomato</td><td>22</td><td>0.8</td></tr><tr><th>4</th><td>banana</td><td>105</td><td>0.6</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccc}\n",
       "\t& item & calories & price\\\\\n",
       "\t\\hline\n",
       "\t& String & Int64 & Float64\\\\\n",
       "\t\\hline\n",
       "\t1 & apple & 105 & 0.85 \\\\\n",
       "\t2 & cucumber & 47 & 1.6 \\\\\n",
       "\t3 & tomato & 22 & 0.8 \\\\\n",
       "\t4 & banana & 105 & 0.6 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m4×3 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m item     \u001b[0m\u001b[1m calories \u001b[0m\u001b[1m price   \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m String   \u001b[0m\u001b[90m Int64    \u001b[0m\u001b[90m Float64 \u001b[0m\n",
       "─────┼─────────────────────────────\n",
       "   1 │ apple          105     0.85\n",
       "   2 │ cucumber        47     1.6\n",
       "   3 │ tomato          22     0.8\n",
       "   4 │ banana         105     0.6"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DF = innerjoin(dataframe_calories,dataframe_prices,on=:item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>x1</th><th>x2</th><th>x3</th><th>x4</th><th>x5</th><th>x6</th></tr><tr><th></th><th>Any</th><th>Any</th><th>Any</th><th>Any</th><th>Any</th><th>Any</th></tr></thead><tbody><p>9 rows × 6 columns</p><tr><th>1</th><td>RegionID</td><td>RegionName</td><td>StateName</td><td>SizeRank</td><td>2008-03</td><td>2008-04</td></tr><tr><th>2</th><td>6181</td><td>New York</td><td>New York</td><td>1</td><td><em>missing</em></td><td><em>missing</em></td></tr><tr><th>3</th><td>12447</td><td>Los Angeles</td><td>California</td><td>2</td><td>1446</td><td>1705</td></tr><tr><th>4</th><td>39051</td><td>Houston</td><td>Texas</td><td>3</td><td>2926</td><td>3121</td></tr><tr><th>5</th><td>17426</td><td>Chicago</td><td>Illinois</td><td>4</td><td>2910</td><td>3022</td></tr><tr><th>6</th><td>6915</td><td>San Antonio</td><td>Texas</td><td>5</td><td>1479</td><td>1529</td></tr><tr><th>7</th><td>13271</td><td>Philadelphia</td><td>Pennsylvania</td><td>6</td><td>1609</td><td>1795</td></tr><tr><th>8</th><td>40326</td><td>Phoenix</td><td>Arizona</td><td>7</td><td>1310</td><td>1519</td></tr><tr><th>9</th><td>18959</td><td>Las Vegas</td><td>Nevada</td><td>8</td><td>1618</td><td>1856</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cccccc}\n",
       "\t& x1 & x2 & x3 & x4 & x5 & x6\\\\\n",
       "\t\\hline\n",
       "\t& Any & Any & Any & Any & Any & Any\\\\\n",
       "\t\\hline\n",
       "\t1 & RegionID & RegionName & StateName & SizeRank & 2008-03 & 2008-04 \\\\\n",
       "\t2 & 6181 & New York & New York & 1 & \\emph{missing} & \\emph{missing} \\\\\n",
       "\t3 & 12447 & Los Angeles & California & 2 & 1446 & 1705 \\\\\n",
       "\t4 & 39051 & Houston & Texas & 3 & 2926 & 3121 \\\\\n",
       "\t5 & 17426 & Chicago & Illinois & 4 & 2910 & 3022 \\\\\n",
       "\t6 & 6915 & San Antonio & Texas & 5 & 1479 & 1529 \\\\\n",
       "\t7 & 13271 & Philadelphia & Pennsylvania & 6 & 1609 & 1795 \\\\\n",
       "\t8 & 40326 & Phoenix & Arizona & 7 & 1310 & 1519 \\\\\n",
       "\t9 & 18959 & Las Vegas & Nevada & 8 & 1618 & 1856 \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m9×6 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m x1       \u001b[0m\u001b[1m x2           \u001b[0m\u001b[1m x3           \u001b[0m\u001b[1m x4       \u001b[0m\u001b[1m x5      \u001b[0m\u001b[1m x6      \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Any      \u001b[0m\u001b[90m Any          \u001b[0m\u001b[90m Any          \u001b[0m\u001b[90m Any      \u001b[0m\u001b[90m Any     \u001b[0m\u001b[90m Any     \u001b[0m\n",
       "─────┼──────────────────────────────────────────────────────────────────\n",
       "   1 │ RegionID  RegionName    StateName     SizeRank  2008-03  2008-04\n",
       "   2 │ 6181      New York      New York      1        \u001b[90m missing \u001b[0m\u001b[90m missing \u001b[0m\n",
       "   3 │ 12447     Los Angeles   California    2         1446     1705\n",
       "   4 │ 39051     Houston       Texas         3         2926     3121\n",
       "   5 │ 17426     Chicago       Illinois      4         2910     3022\n",
       "   6 │ 6915      San Antonio   Texas         5         1479     1529\n",
       "   7 │ 13271     Philadelphia  Pennsylvania  6         1609     1795\n",
       "   8 │ 40326     Phoenix       Arizona       7         1310     1519\n",
       "   9 │ 18959     Las Vegas     Nevada        8         1618     1856"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we can also use the DataFrame constructor on a Matrix\n",
    "DataFrame(T, :auto)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also easily write data to an XLSX file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "ename": "LoadError",
     "evalue": "AssertionError: writefile_using_XLSX.xlsx already exists.",
     "output_type": "error",
     "traceback": [
      "AssertionError: writefile_using_XLSX.xlsx already exists.",
      "",
      "Stacktrace:",
      " [1] writetable(filename::String, data::Vector{Any}, columnnames::Vector{Symbol}; overwrite::Bool, sheetname::String, anchor_cell::XLSX.CellRef)",
      "   @ XLSX ~/.julia/packages/XLSX/cRAtd/src/write.jl:594",
      " [2] writetable(filename::String, data::Vector{Any}, columnnames::Vector{Symbol})",
      "   @ XLSX ~/.julia/packages/XLSX/cRAtd/src/write.jl:593",
      " [3] top-level scope",
      "   @ In[22]:3",
      " [4] eval",
      "   @ ./boot.jl:360 [inlined]",
      " [5] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)",
      "   @ Base ./loading.jl:1094"
     ]
    }
   ],
   "source": [
    "# if you already have a dataframe: \n",
    "# XLSX.writetable(\"filename.xlsx\", collect(DataFrames.eachcol(df)), DataFrames.names(df))\n",
    "XLSX.writetable(\"writefile_using_XLSX.xlsx\",G[1],G[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ⬇️ Importing your data\n",
    "\n",
    "Often, the data you want to import is not stored in plain text, and you might want to import different kinds of types. Here we will go over importing `jld`, `npz`, `rda`, and `mat` files. Hopefully, these four will capture the types from four common programming languages used in Data Science (Julia, Python, R, Matlab).\n",
    "\n",
    "We will use a toy example here of a very small matrix. But the same syntax will hold for bigger files.\n",
    "\n",
    "```\n",
    "4×5 Array{Int64,2}:\n",
    " 2  1446  1705  1795  1890\n",
    " 3  2926  3121  3220  3405\n",
    " 4  2910  3022  2937  3224\n",
    " 5  1479  1529  1582  1761\n",
    " ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "using JLD\n",
    "jld_data = JLD.load(\"data/mytempdata.jld\")\n",
    "save(\"mywrite.jld\", \"A\", jld_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "using NPZ\n",
    "npz_data = npzread(\"data/mytempdata.npz\")\n",
    "npzwrite(\"mywrite.npz\", npz_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RObject{NilSxp}\n",
       "NULL\n"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "using RData\n",
    "R_data = RData.load(\"data/mytempdata.rda\")\n",
    "# We'll need RCall to save here. https://github.com/JuliaData/RData.jl/issues/56\n",
    "using RCall\n",
    "@rput R_data\n",
    "R\"save(R_data, file=\\\"mywrite.rda\\\")\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "using MAT\n",
    "Matlab_data = matread(\"data/mytempdata.mat\")\n",
    "matwrite(\"mywrite.mat\",Matlab_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "typeof(jld_data) = Dict{String, Any}\n",
      "typeof(npz_data) = Matrix{Int64}\n",
      "typeof(R_data) = Dict{String, Any}\n",
      "typeof(Matlab_data) = Dict{String, Any}\n"
     ]
    }
   ],
   "source": [
    "@show typeof(jld_data)\n",
    "@show typeof(npz_data)\n",
    "@show typeof(R_data)\n",
    "@show typeof(Matlab_data)\n",
    ";"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{String, Any} with 1 entry:\n",
       "  \"tempdata\" => [2 1446 … 1795 1890; 3 2926 … 3220 3405; 4 2910 … 2937 3224; 5 …"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Matlab_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 🔢 Time to process the data from Julia\n",
    "We will mainly cover `Matrix` (or `Vector`), `DataFrame`s, and `dict`s (or dictionaries). Let's bring back our programming languages dataset and start playing it the matrix it's stored in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "73×2 Matrix{Any}:\n",
       " 1951  \"Regional Assembly Language\"\n",
       " 1952  \"Autocode\"\n",
       " 1954  \"IPL\"\n",
       " 1955  \"FLOW-MATIC\"\n",
       " 1957  \"FORTRAN\"\n",
       " 1957  \"COMTRAN\"\n",
       " 1958  \"LISP\"\n",
       " 1958  \"ALGOL 58\"\n",
       " 1959  \"FACT\"\n",
       " 1959  \"COBOL\"\n",
       " 1959  \"RPG\"\n",
       " 1962  \"APL\"\n",
       " 1962  \"Simula\"\n",
       "    ⋮  \n",
       " 2003  \"Scala\"\n",
       " 2005  \"F#\"\n",
       " 2006  \"PowerShell\"\n",
       " 2007  \"Clojure\"\n",
       " 2009  \"Go\"\n",
       " 2010  \"Rust\"\n",
       " 2011  \"Dart\"\n",
       " 2011  \"Kotlin\"\n",
       " 2011  \"Red\"\n",
       " 2011  \"Elixir\"\n",
       " 2012  \"Julia\"\n",
       " 2014  \"Swift\""
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "P"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some quick questions we might want to ask about this simple data.\n",
    "- Which year was was a given language invented?\n",
    "- How many languages were created in a given year?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2012"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Q1: Which year was was a given language invented?\n",
    "function year_created(P,language::String)\n",
    "    loc = findfirst(P[:,2] .== language)\n",
    "    return P[loc,1]\n",
    "end\n",
    "year_created(P,\"Julia\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "ename": "LoadError",
     "evalue": "ArgumentError: invalid index: nothing of type Nothing",
     "output_type": "error",
     "traceback": [
      "ArgumentError: invalid index: nothing of type Nothing",
      "",
      "Stacktrace:",
      " [1] to_index(i::Nothing)",
      "   @ Base ./indices.jl:300",
      " [2] to_index(A::Matrix{Any}, i::Nothing)",
      "   @ Base ./indices.jl:277",
      " [3] to_indices",
      "   @ ./indices.jl:333 [inlined]",
      " [4] to_indices",
      "   @ ./indices.jl:324 [inlined]",
      " [5] getindex(::Matrix{Any}, ::Nothing, ::Int64)",
      "   @ Base ./abstractarray.jl:1170",
      " [6] year_created(P::Matrix{Any}, language::String)",
      "   @ Main ./In[30]:4",
      " [7] top-level scope",
      "   @ In[31]:1",
      " [8] eval",
      "   @ ./boot.jl:360 [inlined]",
      " [9] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)",
      "   @ Base ./loading.jl:1094"
     ]
    }
   ],
   "source": [
    "year_created(P,\"W\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "ename": "LoadError",
     "evalue": "Error: Language not found.",
     "output_type": "error",
     "traceback": [
      "Error: Language not found.",
      "",
      "Stacktrace:",
      " [1] error(s::String)",
      "   @ Base ./error.jl:33",
      " [2] year_created_handle_error(P::Matrix{Any}, language::String)",
      "   @ Main ./In[32]:4",
      " [3] top-level scope",
      "   @ In[32]:6",
      " [4] eval",
      "   @ ./boot.jl:360 [inlined]",
      " [5] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)",
      "   @ Base ./loading.jl:1094"
     ]
    }
   ],
   "source": [
    "function year_created_handle_error(P,language::String)\n",
    "    loc = findfirst(P[:,2] .== language)\n",
    "    !isnothing(loc) && return P[loc,1]\n",
    "    error(\"Error: Language not found.\")\n",
    "end\n",
    "year_created_handle_error(P,\"W\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Q2: How many languages were created in a given year?\n",
    "function how_many_per_year(P,year::Int64)\n",
    "    year_count = length(findall(P[:,1].==year))\n",
    "    return year_count\n",
    "end\n",
    "how_many_per_year(P,2011)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try to store this data in a DataFrame..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>year</th><th>language</th></tr><tr><th></th><th>Int64</th><th>String</th></tr></thead><tbody><p>73 rows × 2 columns</p><tr><th>1</th><td>1951</td><td>Regional Assembly Language</td></tr><tr><th>2</th><td>1952</td><td>Autocode</td></tr><tr><th>3</th><td>1954</td><td>IPL</td></tr><tr><th>4</th><td>1955</td><td>FLOW-MATIC</td></tr><tr><th>5</th><td>1957</td><td>FORTRAN</td></tr><tr><th>6</th><td>1957</td><td>COMTRAN</td></tr><tr><th>7</th><td>1958</td><td>LISP</td></tr><tr><th>8</th><td>1958</td><td>ALGOL 58</td></tr><tr><th>9</th><td>1959</td><td>FACT</td></tr><tr><th>10</th><td>1959</td><td>COBOL</td></tr><tr><th>11</th><td>1959</td><td>RPG</td></tr><tr><th>12</th><td>1962</td><td>APL</td></tr><tr><th>13</th><td>1962</td><td>Simula</td></tr><tr><th>14</th><td>1962</td><td>SNOBOL</td></tr><tr><th>15</th><td>1963</td><td>CPL</td></tr><tr><th>16</th><td>1964</td><td>Speakeasy</td></tr><tr><th>17</th><td>1964</td><td>BASIC</td></tr><tr><th>18</th><td>1964</td><td>PL/I</td></tr><tr><th>19</th><td>1966</td><td>JOSS</td></tr><tr><th>20</th><td>1967</td><td>BCPL</td></tr><tr><th>21</th><td>1968</td><td>Logo</td></tr><tr><th>22</th><td>1969</td><td>B</td></tr><tr><th>23</th><td>1970</td><td>Pascal</td></tr><tr><th>24</th><td>1970</td><td>Forth</td></tr><tr><th>25</th><td>1972</td><td>C</td></tr><tr><th>26</th><td>1972</td><td>Smalltalk</td></tr><tr><th>27</th><td>1972</td><td>Prolog</td></tr><tr><th>28</th><td>1973</td><td>ML</td></tr><tr><th>29</th><td>1975</td><td>Scheme</td></tr><tr><th>30</th><td>1978</td><td>SQL </td></tr><tr><th>&vellip;</th><td>&vellip;</td><td>&vellip;</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& year & language\\\\\n",
       "\t\\hline\n",
       "\t& Int64 & String\\\\\n",
       "\t\\hline\n",
       "\t1 & 1951 & Regional Assembly Language \\\\\n",
       "\t2 & 1952 & Autocode \\\\\n",
       "\t3 & 1954 & IPL \\\\\n",
       "\t4 & 1955 & FLOW-MATIC \\\\\n",
       "\t5 & 1957 & FORTRAN \\\\\n",
       "\t6 & 1957 & COMTRAN \\\\\n",
       "\t7 & 1958 & LISP \\\\\n",
       "\t8 & 1958 & ALGOL 58 \\\\\n",
       "\t9 & 1959 & FACT \\\\\n",
       "\t10 & 1959 & COBOL \\\\\n",
       "\t11 & 1959 & RPG \\\\\n",
       "\t12 & 1962 & APL \\\\\n",
       "\t13 & 1962 & Simula \\\\\n",
       "\t14 & 1962 & SNOBOL \\\\\n",
       "\t15 & 1963 & CPL \\\\\n",
       "\t16 & 1964 & Speakeasy \\\\\n",
       "\t17 & 1964 & BASIC \\\\\n",
       "\t18 & 1964 & PL/I \\\\\n",
       "\t19 & 1966 & JOSS \\\\\n",
       "\t20 & 1967 & BCPL \\\\\n",
       "\t21 & 1968 & Logo \\\\\n",
       "\t22 & 1969 & B \\\\\n",
       "\t23 & 1970 & Pascal \\\\\n",
       "\t24 & 1970 & Forth \\\\\n",
       "\t25 & 1972 & C \\\\\n",
       "\t26 & 1972 & Smalltalk \\\\\n",
       "\t27 & 1972 & Prolog \\\\\n",
       "\t28 & 1973 & ML \\\\\n",
       "\t29 & 1975 & Scheme \\\\\n",
       "\t30 & 1978 & SQL  \\\\\n",
       "\t$\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m73×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m year  \u001b[0m\u001b[1m language                   \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Int64 \u001b[0m\u001b[90m String                     \u001b[0m\n",
       "─────┼───────────────────────────────────\n",
       "   1 │  1951  Regional Assembly Language\n",
       "   2 │  1952  Autocode\n",
       "   3 │  1954  IPL\n",
       "   4 │  1955  FLOW-MATIC\n",
       "   5 │  1957  FORTRAN\n",
       "   6 │  1957  COMTRAN\n",
       "   7 │  1958  LISP\n",
       "   8 │  1958  ALGOL 58\n",
       "   9 │  1959  FACT\n",
       "  10 │  1959  COBOL\n",
       "  11 │  1959  RPG\n",
       "  ⋮  │   ⋮                ⋮\n",
       "  64 │  2006  PowerShell\n",
       "  65 │  2007  Clojure\n",
       "  66 │  2009  Go\n",
       "  67 │  2010  Rust\n",
       "  68 │  2011  Dart\n",
       "  69 │  2011  Kotlin\n",
       "  70 │  2011  Red\n",
       "  71 │  2011  Elixir\n",
       "  72 │  2012  Julia\n",
       "  73 │  2014  Swift\n",
       "\u001b[36m                          52 rows omitted\u001b[0m"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "P_df = C #DataFrame(year = P[:,1], language = P[:,2]) # or DataFrame(P)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Even better, since we know the types of each column, we can create the DataFrame as follows:\n",
    "# P_df = DataFrame(year = Int.(P[:,1]), language = string.(P[:,2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now let's answer the same questions we just answered..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2012"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Q1: Which year was was a given language invented?\n",
    "# it's a little more intuitive and you don't need to remember the column ids\n",
    "function year_created(P_df,language::String)\n",
    "    loc = findfirst(P_df.language .== language)\n",
    "    return P_df.year[loc]\n",
    "end\n",
    "year_created(P_df,\"Julia\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "ename": "LoadError",
     "evalue": "ArgumentError: invalid index: nothing of type Nothing",
     "output_type": "error",
     "traceback": [
      "ArgumentError: invalid index: nothing of type Nothing",
      "",
      "Stacktrace:",
      " [1] to_index(i::Nothing)",
      "   @ Base ./indices.jl:300",
      " [2] to_index(A::Vector{Int64}, i::Nothing)",
      "   @ Base ./indices.jl:277",
      " [3] to_indices",
      "   @ ./indices.jl:333 [inlined]",
      " [4] to_indices",
      "   @ ./indices.jl:325 [inlined]",
      " [5] getindex(A::Vector{Int64}, I::Nothing)",
      "   @ Base ./abstractarray.jl:1170",
      " [6] year_created(P_df::DataFrame, language::String)",
      "   @ Main ./In[36]:5",
      " [7] top-level scope",
      "   @ In[37]:1",
      " [8] eval",
      "   @ ./boot.jl:360 [inlined]",
      " [9] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)",
      "   @ Base ./loading.jl:1094"
     ]
    }
   ],
   "source": [
    "year_created(P_df,\"W\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "ename": "LoadError",
     "evalue": "Error: Language not found.",
     "output_type": "error",
     "traceback": [
      "Error: Language not found.",
      "",
      "Stacktrace:",
      " [1] error(s::String)",
      "   @ Base ./error.jl:33",
      " [2] year_created_handle_error(P_df::DataFrame, language::String)",
      "   @ Main ./In[38]:4",
      " [3] top-level scope",
      "   @ In[38]:6",
      " [4] eval",
      "   @ ./boot.jl:360 [inlined]",
      " [5] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)",
      "   @ Base ./loading.jl:1094"
     ]
    }
   ],
   "source": [
    "function year_created_handle_error(P_df,language::String)\n",
    "    loc = findfirst(P_df.language .== language)\n",
    "    !isnothing(loc) && return P_df.year[loc]\n",
    "    error(\"Error: Language not found.\")\n",
    "end\n",
    "year_created_handle_error(P_df,\"W\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Q2: How many languages were created in a given year?\n",
    "function how_many_per_year(P_df,year::Int64)\n",
    "    year_count = length(findall(P_df.year.==year))\n",
    "    return year_count\n",
    "end\n",
    "how_many_per_year(P_df,2011)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll use dictionaries. A quick way to create a dictionary is with the `Dict()` command. But this creates a dictionary without types. Here, we will specify the types of this dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{Any, Any} with 3 entries:\n",
       "  \"B\" => 2\n",
       "  \"A\" => 1\n",
       "  1   => [1, 2]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A quick example to show how to build a dictionary\n",
    "Dict([(\"A\", 1), (\"B\", 2),(1,[1,2])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dict{Integer, Vector{String}}()"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "P_dictionary = Dict{Integer,Vector{String}}()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "#P_dictionary[67] = [\"julia\",\"programming\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "ename": "LoadError",
     "evalue": "MethodError: \u001b[0mCannot `convert` an object of type \u001b[92mString\u001b[39m\u001b[0m to an object of type \u001b[91mInteger\u001b[39m\n\u001b[0mClosest candidates are:\n\u001b[0m  convert(::Type{T}, \u001b[91m::Base.TwicePrecision\u001b[39m) where T<:Number at twiceprecision.jl:250\n\u001b[0m  convert(::Type{T}, \u001b[91m::AbstractChar\u001b[39m) where T<:Number at char.jl:180\n\u001b[0m  convert(::Type{T}, \u001b[91m::CartesianIndex{1}\u001b[39m) where T<:Number at multidimensional.jl:136\n\u001b[0m  ...",
     "output_type": "error",
     "traceback": [
      "MethodError: \u001b[0mCannot `convert` an object of type \u001b[92mString\u001b[39m\u001b[0m to an object of type \u001b[91mInteger\u001b[39m\n\u001b[0mClosest candidates are:\n\u001b[0m  convert(::Type{T}, \u001b[91m::Base.TwicePrecision\u001b[39m) where T<:Number at twiceprecision.jl:250\n\u001b[0m  convert(::Type{T}, \u001b[91m::AbstractChar\u001b[39m) where T<:Number at char.jl:180\n\u001b[0m  convert(::Type{T}, \u001b[91m::CartesianIndex{1}\u001b[39m) where T<:Number at multidimensional.jl:136\n\u001b[0m  ...",
      "",
      "Stacktrace:",
      " [1] setindex!(h::Dict{Integer, Vector{String}}, v0::Int64, key0::String)",
      "   @ Base ./dict.jl:374",
      " [2] top-level scope",
      "   @ In[43]:2",
      " [3] eval",
      "   @ ./boot.jl:360 [inlined]",
      " [4] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)",
      "   @ Base ./loading.jl:1094"
     ]
    }
   ],
   "source": [
    "# this is not going to work.\n",
    "P_dictionary[\"julia\"] = 7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's populate the dictionary with years as keys and vectors that hold all the programming languages created in each year as their values. Even though this looks like more work, we often need to do it just once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "dict = Dict{Integer,Vector{String}}()\n",
    "for i = 1:size(P,1)\n",
    "    year,lang = P[i,:]\n",
    "    if year in keys(dict)\n",
    "        dict[year] = push!(dict[year],lang) \n",
    "        # note that push! is not our favorite thing to do in Julia, \n",
    "        # but we're focusing on correctness rather than speed here\n",
    "    else\n",
    "        dict[year] = [lang]\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Though a smarter way to do this is:\n",
    "curyear = P_df.year[1]\n",
    "P_dictionary[curyear] = [P_df.language[1]]\n",
    "for (i,nextyear) in enumerate(P_df.year[2:end])\n",
    "    if nextyear == curyear\n",
    "        #same key\n",
    "        P_dictionary[curyear] = push!(P_dictionary[curyear],P_df.language[i+1])\n",
    "        # note that push! is not our favorite thing to do in Julia, \n",
    "        # but we're focusing on correctness rather than speed here\n",
    "    else\n",
    "        curyear = nextyear\n",
    "        P_dictionary[curyear] = [P_df.language[i+1]]\n",
    "    end\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "45"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "length(keys(P_dictionary))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "45"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "length(unique(P[:,1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2012"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Q1: Which year was was a given language invented?\n",
    "# now instead of looking in one long vector, we will look in many small vectors\n",
    "function year_created(P_dictionary,language::String)\n",
    "    keys_vec = collect(keys(P_dictionary))\n",
    "    lookup = map(keyid -> findfirst(P_dictionary[keyid].==language),keys_vec)\n",
    "    # now the lookup vector has `nothing` or a numeric value. We want to find the index of the numeric value.\n",
    "    return keys_vec[findfirst((!isnothing).(lookup))]\n",
    "end\n",
    "year_created(P_dictionary,\"Julia\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Q2: How many languages were created in a given year?\n",
    "how_many_per_year(P_dictionary,year::Int64) = length(P_dictionary[year])\n",
    "how_many_per_year(P_dictionary,2011)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 📝 A note about missing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>year</th><th>language</th></tr><tr><th></th><th>Any</th><th>Any</th></tr></thead><tbody><p>73 rows × 2 columns</p><tr><th>1</th><td><em>missing</em></td><td>Regional Assembly Language</td></tr><tr><th>2</th><td>1952</td><td>Autocode</td></tr><tr><th>3</th><td>1954</td><td>IPL</td></tr><tr><th>4</th><td>1955</td><td>FLOW-MATIC</td></tr><tr><th>5</th><td>1957</td><td>FORTRAN</td></tr><tr><th>6</th><td>1957</td><td>COMTRAN</td></tr><tr><th>7</th><td>1958</td><td>LISP</td></tr><tr><th>8</th><td>1958</td><td>ALGOL 58</td></tr><tr><th>9</th><td>1959</td><td>FACT</td></tr><tr><th>10</th><td>1959</td><td>COBOL</td></tr><tr><th>11</th><td>1959</td><td>RPG</td></tr><tr><th>12</th><td>1962</td><td>APL</td></tr><tr><th>13</th><td>1962</td><td>Simula</td></tr><tr><th>14</th><td>1962</td><td>SNOBOL</td></tr><tr><th>15</th><td>1963</td><td>CPL</td></tr><tr><th>16</th><td>1964</td><td>Speakeasy</td></tr><tr><th>17</th><td>1964</td><td>BASIC</td></tr><tr><th>18</th><td>1964</td><td>PL/I</td></tr><tr><th>19</th><td>1966</td><td>JOSS</td></tr><tr><th>20</th><td>1967</td><td>BCPL</td></tr><tr><th>21</th><td>1968</td><td>Logo</td></tr><tr><th>22</th><td>1969</td><td>B</td></tr><tr><th>23</th><td>1970</td><td>Pascal</td></tr><tr><th>24</th><td>1970</td><td>Forth</td></tr><tr><th>25</th><td>1972</td><td>C</td></tr><tr><th>26</th><td>1972</td><td>Smalltalk</td></tr><tr><th>27</th><td>1972</td><td>Prolog</td></tr><tr><th>28</th><td>1973</td><td>ML</td></tr><tr><th>29</th><td>1975</td><td>Scheme</td></tr><tr><th>30</th><td>1978</td><td>SQL </td></tr><tr><th>&vellip;</th><td>&vellip;</td><td>&vellip;</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& year & language\\\\\n",
       "\t\\hline\n",
       "\t& Any & Any\\\\\n",
       "\t\\hline\n",
       "\t1 & \\emph{missing} & Regional Assembly Language \\\\\n",
       "\t2 & 1952 & Autocode \\\\\n",
       "\t3 & 1954 & IPL \\\\\n",
       "\t4 & 1955 & FLOW-MATIC \\\\\n",
       "\t5 & 1957 & FORTRAN \\\\\n",
       "\t6 & 1957 & COMTRAN \\\\\n",
       "\t7 & 1958 & LISP \\\\\n",
       "\t8 & 1958 & ALGOL 58 \\\\\n",
       "\t9 & 1959 & FACT \\\\\n",
       "\t10 & 1959 & COBOL \\\\\n",
       "\t11 & 1959 & RPG \\\\\n",
       "\t12 & 1962 & APL \\\\\n",
       "\t13 & 1962 & Simula \\\\\n",
       "\t14 & 1962 & SNOBOL \\\\\n",
       "\t15 & 1963 & CPL \\\\\n",
       "\t16 & 1964 & Speakeasy \\\\\n",
       "\t17 & 1964 & BASIC \\\\\n",
       "\t18 & 1964 & PL/I \\\\\n",
       "\t19 & 1966 & JOSS \\\\\n",
       "\t20 & 1967 & BCPL \\\\\n",
       "\t21 & 1968 & Logo \\\\\n",
       "\t22 & 1969 & B \\\\\n",
       "\t23 & 1970 & Pascal \\\\\n",
       "\t24 & 1970 & Forth \\\\\n",
       "\t25 & 1972 & C \\\\\n",
       "\t26 & 1972 & Smalltalk \\\\\n",
       "\t27 & 1972 & Prolog \\\\\n",
       "\t28 & 1973 & ML \\\\\n",
       "\t29 & 1975 & Scheme \\\\\n",
       "\t30 & 1978 & SQL  \\\\\n",
       "\t$\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m73×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m year    \u001b[0m\u001b[1m language                   \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Any     \u001b[0m\u001b[90m Any                        \u001b[0m\n",
       "─────┼─────────────────────────────────────\n",
       "   1 │\u001b[90m missing \u001b[0m Regional Assembly Language\n",
       "   2 │ 1952     Autocode\n",
       "   3 │ 1954     IPL\n",
       "   4 │ 1955     FLOW-MATIC\n",
       "   5 │ 1957     FORTRAN\n",
       "   6 │ 1957     COMTRAN\n",
       "   7 │ 1958     LISP\n",
       "   8 │ 1958     ALGOL 58\n",
       "   9 │ 1959     FACT\n",
       "  10 │ 1959     COBOL\n",
       "  11 │ 1959     RPG\n",
       "  ⋮  │    ⋮                 ⋮\n",
       "  64 │ 2006     PowerShell\n",
       "  65 │ 2007     Clojure\n",
       "  66 │ 2009     Go\n",
       "  67 │ 2010     Rust\n",
       "  68 │ 2011     Dart\n",
       "  69 │ 2011     Kotlin\n",
       "  70 │ 2011     Red\n",
       "  71 │ 2011     Elixir\n",
       "  72 │ 2012     Julia\n",
       "  73 │ 2014     Swift\n",
       "\u001b[36m                            52 rows omitted\u001b[0m"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# assume there were missing values in our dataframe\n",
    "P[1,1] = missing\n",
    "P_df = DataFrame(year = P[:,1], language = P[:,2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"data-frame\"><thead><tr><th></th><th>year</th><th>language</th></tr><tr><th></th><th>Any</th><th>Any</th></tr></thead><tbody><p>72 rows × 2 columns</p><tr><th>1</th><td>1952</td><td>Autocode</td></tr><tr><th>2</th><td>1954</td><td>IPL</td></tr><tr><th>3</th><td>1955</td><td>FLOW-MATIC</td></tr><tr><th>4</th><td>1957</td><td>FORTRAN</td></tr><tr><th>5</th><td>1957</td><td>COMTRAN</td></tr><tr><th>6</th><td>1958</td><td>LISP</td></tr><tr><th>7</th><td>1958</td><td>ALGOL 58</td></tr><tr><th>8</th><td>1959</td><td>FACT</td></tr><tr><th>9</th><td>1959</td><td>COBOL</td></tr><tr><th>10</th><td>1959</td><td>RPG</td></tr><tr><th>11</th><td>1962</td><td>APL</td></tr><tr><th>12</th><td>1962</td><td>Simula</td></tr><tr><th>13</th><td>1962</td><td>SNOBOL</td></tr><tr><th>14</th><td>1963</td><td>CPL</td></tr><tr><th>15</th><td>1964</td><td>Speakeasy</td></tr><tr><th>16</th><td>1964</td><td>BASIC</td></tr><tr><th>17</th><td>1964</td><td>PL/I</td></tr><tr><th>18</th><td>1966</td><td>JOSS</td></tr><tr><th>19</th><td>1967</td><td>BCPL</td></tr><tr><th>20</th><td>1968</td><td>Logo</td></tr><tr><th>21</th><td>1969</td><td>B</td></tr><tr><th>22</th><td>1970</td><td>Pascal</td></tr><tr><th>23</th><td>1970</td><td>Forth</td></tr><tr><th>24</th><td>1972</td><td>C</td></tr><tr><th>25</th><td>1972</td><td>Smalltalk</td></tr><tr><th>26</th><td>1972</td><td>Prolog</td></tr><tr><th>27</th><td>1973</td><td>ML</td></tr><tr><th>28</th><td>1975</td><td>Scheme</td></tr><tr><th>29</th><td>1978</td><td>SQL </td></tr><tr><th>30</th><td>1980</td><td>C++ </td></tr><tr><th>&vellip;</th><td>&vellip;</td><td>&vellip;</td></tr></tbody></table>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|cc}\n",
       "\t& year & language\\\\\n",
       "\t\\hline\n",
       "\t& Any & Any\\\\\n",
       "\t\\hline\n",
       "\t1 & 1952 & Autocode \\\\\n",
       "\t2 & 1954 & IPL \\\\\n",
       "\t3 & 1955 & FLOW-MATIC \\\\\n",
       "\t4 & 1957 & FORTRAN \\\\\n",
       "\t5 & 1957 & COMTRAN \\\\\n",
       "\t6 & 1958 & LISP \\\\\n",
       "\t7 & 1958 & ALGOL 58 \\\\\n",
       "\t8 & 1959 & FACT \\\\\n",
       "\t9 & 1959 & COBOL \\\\\n",
       "\t10 & 1959 & RPG \\\\\n",
       "\t11 & 1962 & APL \\\\\n",
       "\t12 & 1962 & Simula \\\\\n",
       "\t13 & 1962 & SNOBOL \\\\\n",
       "\t14 & 1963 & CPL \\\\\n",
       "\t15 & 1964 & Speakeasy \\\\\n",
       "\t16 & 1964 & BASIC \\\\\n",
       "\t17 & 1964 & PL/I \\\\\n",
       "\t18 & 1966 & JOSS \\\\\n",
       "\t19 & 1967 & BCPL \\\\\n",
       "\t20 & 1968 & Logo \\\\\n",
       "\t21 & 1969 & B \\\\\n",
       "\t22 & 1970 & Pascal \\\\\n",
       "\t23 & 1970 & Forth \\\\\n",
       "\t24 & 1972 & C \\\\\n",
       "\t25 & 1972 & Smalltalk \\\\\n",
       "\t26 & 1972 & Prolog \\\\\n",
       "\t27 & 1973 & ML \\\\\n",
       "\t28 & 1975 & Scheme \\\\\n",
       "\t29 & 1978 & SQL  \\\\\n",
       "\t30 & 1980 & C++  \\\\\n",
       "\t$\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m72×2 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m year \u001b[0m\u001b[1m language   \u001b[0m\n",
       "\u001b[1m     \u001b[0m│\u001b[90m Any  \u001b[0m\u001b[90m Any        \u001b[0m\n",
       "─────┼──────────────────\n",
       "   1 │ 1952  Autocode\n",
       "   2 │ 1954  IPL\n",
       "   3 │ 1955  FLOW-MATIC\n",
       "   4 │ 1957  FORTRAN\n",
       "   5 │ 1957  COMTRAN\n",
       "   6 │ 1958  LISP\n",
       "   7 │ 1958  ALGOL 58\n",
       "   8 │ 1959  FACT\n",
       "   9 │ 1959  COBOL\n",
       "  10 │ 1959  RPG\n",
       "  11 │ 1962  APL\n",
       "  ⋮  │  ⋮        ⋮\n",
       "  63 │ 2006  PowerShell\n",
       "  64 │ 2007  Clojure\n",
       "  65 │ 2009  Go\n",
       "  66 │ 2010  Rust\n",
       "  67 │ 2011  Dart\n",
       "  68 │ 2011  Kotlin\n",
       "  69 │ 2011  Red\n",
       "  70 │ 2011  Elixir\n",
       "  71 │ 2012  Julia\n",
       "  72 │ 2014  Swift\n",
       "\u001b[36m         51 rows omitted\u001b[0m"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dropmissing(P_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finally...\n",
    "After finishing this notebook, you should be able to:\n",
    "- [ ] dowload a data file from the web given a url\n",
    "- [ ] load data from a file from a text file via DelimitedFiles or CSV\n",
    "- [ ] write your data to a text file or csv file\n",
    "- [ ] load data from file types xlsx, jld, npz, mat, rda\n",
    "- [ ] write your data to an xlsx file, jld, npz, mat, rda\n",
    "- [ ] store data in a 2D array (`Matrix`), or `DataFrame` or `Dict`\n",
    "- [ ] write functions to perform basic lookups on `Matrix`, `DataFrame`, and `Dict` types\n",
    "- [ ] use some of the basic functions on `DataFrame`s such as: `dropmissing`, `describe`, `combine(groupby)`, and `innerjoin`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 🥳 One cool finding\n",
    "\n",
    "Julia was created in 2012"
   ]
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "kernelspec": {
   "display_name": "Julia 1.6.0",
   "language": "julia",
   "name": "julia-1.6"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}