2 - Extracting Stock Data Using WebScraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <img src=\"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/Logos/organization_logo/organization_logo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\"  />\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Extracting Stock Data Using a Web Scraping</h1>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not all stock data is available via API in this assignment; we will use web-scraping to obtain financial data.\n",
    "\n",
    "Using beautiful soup we will extract historical share data from a web-page.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: plotly==5.3.1 in c:\\users\\alvaro\\anaconda3\\lib\\site-packages (5.3.1)\n",
      "Requirement already satisfied: six in c:\\users\\alvaro\\anaconda3\\lib\\site-packages (from plotly==5.3.1) (1.16.0)\n",
      "Requirement already satisfied: tenacity>=6.2.0 in c:\\users\\alvaro\\anaconda3\\lib\\site-packages (from plotly==5.3.1) (8.0.1)\n"
     ]
    }
   ],
   "source": [
    "#!pip install pandas==1.3.3\n",
    "#!pip install requests==2.26.0\n",
    "#!mamba install bs4==4.10.0 -y\n",
    "#!mamba install html5lib==1.1 -y\n",
    "#!pip install lxml==4.6.4\n",
    "!pip install plotly==5.3.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Webscraping to Extract Netflix Stock Data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we must use the `request` library to downlaod the webpage, and extract the text. We will extract Netflix stock data <https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/netflix_data_webpage.html>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/netflix_data_webpage.html\"\n",
    "\n",
    "data  = requests.get(url).text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we must parse the text into html using `beautiful_soup`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(data, 'html5lib')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<td class=\"Py(10px) Ta(start) Pend(10px)\" data-reactid=\"52\"><span data-reactid=\"53\">Jun 01, 2021</span></td>,\n",
       " <td class=\"Py(10px) Pstart(10px)\" data-reactid=\"54\"><span data-reactid=\"55\">504.01</span></td>,\n",
       " <td class=\"Py(10px) Pstart(10px)\" data-reactid=\"56\"><span data-reactid=\"57\">536.13</span></td>,\n",
       " <td class=\"Py(10px) Pstart(10px)\" data-reactid=\"58\"><span data-reactid=\"59\">482.14</span></td>,\n",
       " <td class=\"Py(10px) Pstart(10px)\" data-reactid=\"60\"><span data-reactid=\"61\">528.21</span></td>,\n",
       " <td class=\"Py(10px) Pstart(10px)\" data-reactid=\"62\"><span data-reactid=\"63\">528.21</span></td>,\n",
       " <td class=\"Py(10px) Pstart(10px)\" data-reactid=\"64\"><span data-reactid=\"65\">78,560,600</span></td>]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find(\"tbody\").find('tr').find_all('td')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can turn the html table into a pandas dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "netflix_data = pd.DataFrame(columns=[\"Date\", \"Open\", \"High\", \"Low\", \"Close\", \"Volume\"])\n",
    "\n",
    "# First we isolate the body of the table which contains all the information\n",
    "# Then we loop through each row and find all the column values for each row\n",
    "for row in soup.find(\"tbody\").find_all('tr'):\n",
    "    col = row.find_all(\"td\")\n",
    "    date = col[0].text\n",
    "    Open = col[1].text\n",
    "    high = col[2].text\n",
    "    low = col[3].text\n",
    "    close = col[4].text\n",
    "    adj_close = col[5].text\n",
    "    volume = col[6].text\n",
    "    \n",
    "    # Finally we append the data of each row to the table\n",
    "    netflix_data = netflix_data.append({\"Date\":date, \"Open\":Open, \"High\":high, \"Low\":low, \"Close\":close, \"Adj Close\":adj_close, \"Volume\":volume}, ignore_index=True)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now print out the dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Adj Close</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Jun 01, 2021</td>\n",
       "      <td>504.01</td>\n",
       "      <td>536.13</td>\n",
       "      <td>482.14</td>\n",
       "      <td>528.21</td>\n",
       "      <td>78,560,600</td>\n",
       "      <td>528.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>May 01, 2021</td>\n",
       "      <td>512.65</td>\n",
       "      <td>518.95</td>\n",
       "      <td>478.54</td>\n",
       "      <td>502.81</td>\n",
       "      <td>66,927,600</td>\n",
       "      <td>502.81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Apr 01, 2021</td>\n",
       "      <td>529.93</td>\n",
       "      <td>563.56</td>\n",
       "      <td>499.00</td>\n",
       "      <td>513.47</td>\n",
       "      <td>111,573,300</td>\n",
       "      <td>513.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Mar 01, 2021</td>\n",
       "      <td>545.57</td>\n",
       "      <td>556.99</td>\n",
       "      <td>492.85</td>\n",
       "      <td>521.66</td>\n",
       "      <td>90,183,900</td>\n",
       "      <td>521.66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Feb 01, 2021</td>\n",
       "      <td>536.79</td>\n",
       "      <td>566.65</td>\n",
       "      <td>518.28</td>\n",
       "      <td>538.85</td>\n",
       "      <td>61,902,300</td>\n",
       "      <td>538.85</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Date    Open    High     Low   Close       Volume Adj Close\n",
       "0  Jun 01, 2021  504.01  536.13  482.14  528.21   78,560,600    528.21\n",
       "1  May 01, 2021  512.65  518.95  478.54  502.81   66,927,600    502.81\n",
       "2  Apr 01, 2021  529.93  563.56  499.00  513.47  111,573,300    513.47\n",
       "3  Mar 01, 2021  545.57  556.99  492.85  521.66   90,183,900    521.66\n",
       "4  Feb 01, 2021  536.79  566.65  518.28  538.85   61,902,300    538.85"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "netflix_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the pandas `read_html` function using the url\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[                                                 Date  \\\n",
       " 0                                        Jun 01, 2021   \n",
       " 1                                        May 01, 2021   \n",
       " 2                                        Apr 01, 2021   \n",
       " 3                                        Mar 01, 2021   \n",
       " 4                                        Feb 01, 2021   \n",
       " ..                                                ...   \n",
       " 66                                       Dec 01, 2015   \n",
       " 67                                       Nov 01, 2015   \n",
       " 68                                       Oct 01, 2015   \n",
       " 69                                       Sep 01, 2015   \n",
       " 70  *Close price adjusted for splits.**Adjusted cl...   \n",
       " \n",
       "                                                  Open  \\\n",
       " 0                                              504.01   \n",
       " 1                                              512.65   \n",
       " 2                                              529.93   \n",
       " 3                                              545.57   \n",
       " 4                                              536.79   \n",
       " ..                                                ...   \n",
       " 66                                             124.47   \n",
       " 67                                             109.20   \n",
       " 68                                             102.91   \n",
       " 69                                             109.35   \n",
       " 70  *Close price adjusted for splits.**Adjusted cl...   \n",
       " \n",
       "                                                  High  \\\n",
       " 0                                              536.13   \n",
       " 1                                              518.95   \n",
       " 2                                              563.56   \n",
       " 3                                              556.99   \n",
       " 4                                              566.65   \n",
       " ..                                                ...   \n",
       " 66                                             133.27   \n",
       " 67                                             126.60   \n",
       " 68                                             115.83   \n",
       " 69                                             111.24   \n",
       " 70  *Close price adjusted for splits.**Adjusted cl...   \n",
       " \n",
       "                                                   Low  \\\n",
       " 0                                              482.14   \n",
       " 1                                              478.54   \n",
       " 2                                              499.00   \n",
       " 3                                              492.85   \n",
       " 4                                              518.28   \n",
       " ..                                                ...   \n",
       " 66                                             113.85   \n",
       " 67                                             101.86   \n",
       " 68                                              96.26   \n",
       " 69                                              93.55   \n",
       " 70  *Close price adjusted for splits.**Adjusted cl...   \n",
       " \n",
       "                                                Close*  \\\n",
       " 0                                              528.21   \n",
       " 1                                              502.81   \n",
       " 2                                              513.47   \n",
       " 3                                              521.66   \n",
       " 4                                              538.85   \n",
       " ..                                                ...   \n",
       " 66                                             114.38   \n",
       " 67                                             123.33   \n",
       " 68                                             108.38   \n",
       " 69                                             103.26   \n",
       " 70  *Close price adjusted for splits.**Adjusted cl...   \n",
       " \n",
       "                                           Adj Close**  \\\n",
       " 0                                              528.21   \n",
       " 1                                              502.81   \n",
       " 2                                              513.47   \n",
       " 3                                              521.66   \n",
       " 4                                              538.85   \n",
       " ..                                                ...   \n",
       " 66                                             114.38   \n",
       " 67                                             123.33   \n",
       " 68                                             108.38   \n",
       " 69                                             103.26   \n",
       " 70  *Close price adjusted for splits.**Adjusted cl...   \n",
       " \n",
       "                                                Volume  \n",
       " 0                                            78560600  \n",
       " 1                                            66927600  \n",
       " 2                                           111573300  \n",
       " 3                                            90183900  \n",
       " 4                                            61902300  \n",
       " ..                                                ...  \n",
       " 66                                          319939200  \n",
       " 67                                          320321800  \n",
       " 68                                          446204400  \n",
       " 69                                          497401200  \n",
       " 70  *Close price adjusted for splits.**Adjusted cl...  \n",
       " \n",
       " [71 rows x 7 columns]]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "read_html_pandas_data = pd.read_html(url)\n",
    "read_html_pandas_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can convert the BeautifulSoup object to a string\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[                                                 Date  \\\n",
       " 0                                        Jun 01, 2021   \n",
       " 1                                        May 01, 2021   \n",
       " 2                                        Apr 01, 2021   \n",
       " 3                                        Mar 01, 2021   \n",
       " 4                                        Feb 01, 2021   \n",
       " ..                                                ...   \n",
       " 66                                       Dec 01, 2015   \n",
       " 67                                       Nov 01, 2015   \n",
       " 68                                       Oct 01, 2015   \n",
       " 69                                       Sep 01, 2015   \n",
       " 70  *Close price adjusted for splits.**Adjusted cl...   \n",
       " \n",
       "                                                  Open  \\\n",
       " 0                                              504.01   \n",
       " 1                                              512.65   \n",
       " 2                                              529.93   \n",
       " 3                                              545.57   \n",
       " 4                                              536.79   \n",
       " ..                                                ...   \n",
       " 66                                             124.47   \n",
       " 67                                             109.20   \n",
       " 68                                             102.91   \n",
       " 69                                             109.35   \n",
       " 70  *Close price adjusted for splits.**Adjusted cl...   \n",
       " \n",
       "                                                  High  \\\n",
       " 0                                              536.13   \n",
       " 1                                              518.95   \n",
       " 2                                              563.56   \n",
       " 3                                              556.99   \n",
       " 4                                              566.65   \n",
       " ..                                                ...   \n",
       " 66                                             133.27   \n",
       " 67                                             126.60   \n",
       " 68                                             115.83   \n",
       " 69                                             111.24   \n",
       " 70  *Close price adjusted for splits.**Adjusted cl...   \n",
       " \n",
       "                                                   Low  \\\n",
       " 0                                              482.14   \n",
       " 1                                              478.54   \n",
       " 2                                              499.00   \n",
       " 3                                              492.85   \n",
       " 4                                              518.28   \n",
       " ..                                                ...   \n",
       " 66                                             113.85   \n",
       " 67                                             101.86   \n",
       " 68                                              96.26   \n",
       " 69                                              93.55   \n",
       " 70  *Close price adjusted for splits.**Adjusted cl...   \n",
       " \n",
       "                                                Close*  \\\n",
       " 0                                              528.21   \n",
       " 1                                              502.81   \n",
       " 2                                              513.47   \n",
       " 3                                              521.66   \n",
       " 4                                              538.85   \n",
       " ..                                                ...   \n",
       " 66                                             114.38   \n",
       " 67                                             123.33   \n",
       " 68                                             108.38   \n",
       " 69                                             103.26   \n",
       " 70  *Close price adjusted for splits.**Adjusted cl...   \n",
       " \n",
       "                                           Adj Close**  \\\n",
       " 0                                              528.21   \n",
       " 1                                              502.81   \n",
       " 2                                              513.47   \n",
       " 3                                              521.66   \n",
       " 4                                              538.85   \n",
       " ..                                                ...   \n",
       " 66                                             114.38   \n",
       " 67                                             123.33   \n",
       " 68                                             108.38   \n",
       " 69                                             103.26   \n",
       " 70  *Close price adjusted for splits.**Adjusted cl...   \n",
       " \n",
       "                                                Volume  \n",
       " 0                                            78560600  \n",
       " 1                                            66927600  \n",
       " 2                                           111573300  \n",
       " 3                                            90183900  \n",
       " 4                                            61902300  \n",
       " ..                                                ...  \n",
       " 66                                          319939200  \n",
       " 67                                          320321800  \n",
       " 68                                          446204400  \n",
       " 69                                          497401200  \n",
       " 70  *Close price adjusted for splits.**Adjusted cl...  \n",
       " \n",
       " [71 rows x 7 columns]]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "read_html_pandas_data = pd.read_html(str(soup))\n",
    "read_html_pandas_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beacause there is only one table on the page, we just take the first table in the list returned\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close*</th>\n",
       "      <th>Adj Close**</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Jun 01, 2021</td>\n",
       "      <td>504.01</td>\n",
       "      <td>536.13</td>\n",
       "      <td>482.14</td>\n",
       "      <td>528.21</td>\n",
       "      <td>528.21</td>\n",
       "      <td>78560600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>May 01, 2021</td>\n",
       "      <td>512.65</td>\n",
       "      <td>518.95</td>\n",
       "      <td>478.54</td>\n",
       "      <td>502.81</td>\n",
       "      <td>502.81</td>\n",
       "      <td>66927600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Apr 01, 2021</td>\n",
       "      <td>529.93</td>\n",
       "      <td>563.56</td>\n",
       "      <td>499.00</td>\n",
       "      <td>513.47</td>\n",
       "      <td>513.47</td>\n",
       "      <td>111573300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Mar 01, 2021</td>\n",
       "      <td>545.57</td>\n",
       "      <td>556.99</td>\n",
       "      <td>492.85</td>\n",
       "      <td>521.66</td>\n",
       "      <td>521.66</td>\n",
       "      <td>90183900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Feb 01, 2021</td>\n",
       "      <td>536.79</td>\n",
       "      <td>566.65</td>\n",
       "      <td>518.28</td>\n",
       "      <td>538.85</td>\n",
       "      <td>538.85</td>\n",
       "      <td>61902300</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Date    Open    High     Low  Close* Adj Close**     Volume\n",
       "0  Jun 01, 2021  504.01  536.13  482.14  528.21      528.21   78560600\n",
       "1  May 01, 2021  512.65  518.95  478.54  502.81      502.81   66927600\n",
       "2  Apr 01, 2021  529.93  563.56  499.00  513.47      513.47  111573300\n",
       "3  Mar 01, 2021  545.57  556.99  492.85  521.66      521.66   90183900\n",
       "4  Feb 01, 2021  536.79  566.65  518.28  538.85      538.85   61902300"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "netflix_dataframe = read_html_pandas_data[0]\n",
    "\n",
    "netflix_dataframe.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Webscraping to Extract Amazon Stock Data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the `requests` library to download the webpage <https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/amazon_data_webpage.html>. Save the text of the response as a variable named `html_data`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = ' https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/amazon_data_webpage.html'\n",
    "html_data = requests.get(url).text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we parse the html data using `beautiful_soup`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "beautiful_soup = BeautifulSoup(html_data, 'html5lib')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get the content of the title attribute:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<title>Amazon.com, Inc. (AMZN) Stock Historical Prices &amp; Data - Yahoo Finance</title>"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "beautiful_soup.title"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using beautiful soup we extract the table with historical share prices and store it into a dataframe named `amazon_data`. The dataframe should have columns Date, Open, High, Low, Close, Adj Close, and Volume. We fill in each variable with the correct data from the list `col`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "amazon_data = pd.DataFrame(columns=[\"Date\", \"Open\", \"High\", \"Low\", \"Close\", \"Volume\"])\n",
    "\n",
    "for row in beautiful_soup.find(\"tbody\").find_all(\"tr\"):\n",
    "    col = row.find_all(\"td\")\n",
    "    date = col[0].text\n",
    "    Open = col[1].text\n",
    "    high = col[2].text\n",
    "    low = col[3].text\n",
    "    close = col[4].text\n",
    "    adj_close = col[5].text\n",
    "    volume = col[6].text\n",
    "    \n",
    "    amazon_data = amazon_data.append({\"Date\":date, \"Open\":Open, \"High\":high, \"Low\":low, \"Close\":close, \"Adj Close\":adj_close, \"Volume\":volume}, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can print out the first five rows of the `amazon_data` dataframe you created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Adj Close</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Jan 01, 2021</td>\n",
       "      <td>3,270.00</td>\n",
       "      <td>3,363.89</td>\n",
       "      <td>3,086.00</td>\n",
       "      <td>3,206.20</td>\n",
       "      <td>71,528,900</td>\n",
       "      <td>3,206.20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dec 01, 2020</td>\n",
       "      <td>3,188.50</td>\n",
       "      <td>3,350.65</td>\n",
       "      <td>3,072.82</td>\n",
       "      <td>3,256.93</td>\n",
       "      <td>77,556,200</td>\n",
       "      <td>3,256.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Nov 01, 2020</td>\n",
       "      <td>3,061.74</td>\n",
       "      <td>3,366.80</td>\n",
       "      <td>2,950.12</td>\n",
       "      <td>3,168.04</td>\n",
       "      <td>90,810,500</td>\n",
       "      <td>3,168.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Oct 01, 2020</td>\n",
       "      <td>3,208.00</td>\n",
       "      <td>3,496.24</td>\n",
       "      <td>3,019.00</td>\n",
       "      <td>3,036.15</td>\n",
       "      <td>116,226,100</td>\n",
       "      <td>3,036.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sep 01, 2020</td>\n",
       "      <td>3,489.58</td>\n",
       "      <td>3,552.25</td>\n",
       "      <td>2,871.00</td>\n",
       "      <td>3,148.73</td>\n",
       "      <td>115,899,300</td>\n",
       "      <td>3,148.73</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Date      Open      High       Low     Close       Volume Adj Close\n",
       "0  Jan 01, 2021  3,270.00  3,363.89  3,086.00  3,206.20   71,528,900  3,206.20\n",
       "1  Dec 01, 2020  3,188.50  3,350.65  3,072.82  3,256.93   77,556,200  3,256.93\n",
       "2  Nov 01, 2020  3,061.74  3,366.80  2,950.12  3,168.04   90,810,500  3,168.04\n",
       "3  Oct 01, 2020  3,208.00  3,496.24  3,019.00  3,036.15  116,226,100  3,036.15\n",
       "4  Sep 01, 2020  3,489.58  3,552.25  2,871.00  3,148.73  115,899,300  3,148.73"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amazon_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get the name of the columns of the dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close'], dtype='object')"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amazon_data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get the `Open` of the last row of the amazon_data dataframe?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(61, 7)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amazon_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'656.29'"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amazon_data.loc[60,'Open']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Adj Close</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Jan 01, 2021</td>\n",
       "      <td>3,270.00</td>\n",
       "      <td>3,363.89</td>\n",
       "      <td>3,086.00</td>\n",
       "      <td>3,206.20</td>\n",
       "      <td>71,528,900</td>\n",
       "      <td>3,206.20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dec 01, 2020</td>\n",
       "      <td>3,188.50</td>\n",
       "      <td>3,350.65</td>\n",
       "      <td>3,072.82</td>\n",
       "      <td>3,256.93</td>\n",
       "      <td>77,556,200</td>\n",
       "      <td>3,256.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Nov 01, 2020</td>\n",
       "      <td>3,061.74</td>\n",
       "      <td>3,366.80</td>\n",
       "      <td>2,950.12</td>\n",
       "      <td>3,168.04</td>\n",
       "      <td>90,810,500</td>\n",
       "      <td>3,168.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Oct 01, 2020</td>\n",
       "      <td>3,208.00</td>\n",
       "      <td>3,496.24</td>\n",
       "      <td>3,019.00</td>\n",
       "      <td>3,036.15</td>\n",
       "      <td>116,226,100</td>\n",
       "      <td>3,036.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sep 01, 2020</td>\n",
       "      <td>3,489.58</td>\n",
       "      <td>3,552.25</td>\n",
       "      <td>2,871.00</td>\n",
       "      <td>3,148.73</td>\n",
       "      <td>115,899,300</td>\n",
       "      <td>3,148.73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>May 01, 2016</td>\n",
       "      <td>663.92</td>\n",
       "      <td>724.23</td>\n",
       "      <td>656.00</td>\n",
       "      <td>722.79</td>\n",
       "      <td>90,614,500</td>\n",
       "      <td>722.79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>Apr 01, 2016</td>\n",
       "      <td>590.49</td>\n",
       "      <td>669.98</td>\n",
       "      <td>585.25</td>\n",
       "      <td>659.59</td>\n",
       "      <td>78,464,200</td>\n",
       "      <td>659.59</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>Mar 01, 2016</td>\n",
       "      <td>556.29</td>\n",
       "      <td>603.24</td>\n",
       "      <td>538.58</td>\n",
       "      <td>593.64</td>\n",
       "      <td>94,009,500</td>\n",
       "      <td>593.64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>Feb 01, 2016</td>\n",
       "      <td>578.15</td>\n",
       "      <td>581.80</td>\n",
       "      <td>474.00</td>\n",
       "      <td>552.52</td>\n",
       "      <td>124,144,800</td>\n",
       "      <td>552.52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>Jan 01, 2016</td>\n",
       "      <td>656.29</td>\n",
       "      <td>657.72</td>\n",
       "      <td>547.18</td>\n",
       "      <td>587.00</td>\n",
       "      <td>130,200,900</td>\n",
       "      <td>587.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>61 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            Date      Open      High       Low     Close       Volume  \\\n",
       "0   Jan 01, 2021  3,270.00  3,363.89  3,086.00  3,206.20   71,528,900   \n",
       "1   Dec 01, 2020  3,188.50  3,350.65  3,072.82  3,256.93   77,556,200   \n",
       "2   Nov 01, 2020  3,061.74  3,366.80  2,950.12  3,168.04   90,810,500   \n",
       "3   Oct 01, 2020  3,208.00  3,496.24  3,019.00  3,036.15  116,226,100   \n",
       "4   Sep 01, 2020  3,489.58  3,552.25  2,871.00  3,148.73  115,899,300   \n",
       "..           ...       ...       ...       ...       ...          ...   \n",
       "56  May 01, 2016    663.92    724.23    656.00    722.79   90,614,500   \n",
       "57  Apr 01, 2016    590.49    669.98    585.25    659.59   78,464,200   \n",
       "58  Mar 01, 2016    556.29    603.24    538.58    593.64   94,009,500   \n",
       "59  Feb 01, 2016    578.15    581.80    474.00    552.52  124,144,800   \n",
       "60  Jan 01, 2016    656.29    657.72    547.18    587.00  130,200,900   \n",
       "\n",
       "   Adj Close  \n",
       "0   3,206.20  \n",
       "1   3,256.93  \n",
       "2   3,168.04  \n",
       "3   3,036.15  \n",
       "4   3,148.73  \n",
       "..       ...  \n",
       "56    722.79  \n",
       "57    659.59  \n",
       "58    593.64  \n",
       "59    552.52  \n",
       "60    587.00  \n",
       "\n",
       "[61 rows x 7 columns]"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amazon_data"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "vscode": {
   "interpreter": {
    "hash": "83ef6e50cc29290c96a43b2b28e951fa9950334aea30833e486848d425c5f395"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}