Final Assignment_Webscraping_Extracting Stock Data Using a Web Scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <img src=\"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/Logos/organization_logo/organization_logo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\"  />\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Extracting Stock Data Using a Web Scraping</h1>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not all stock data is available via API in this assignment; you will use web-scraping to obtain financial data. You will be quizzed on your results.\\\n",
    "Using beautiful soup we will extract historical share data from a web-page.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Table of Contents</h2>\n",
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "    <ul>\n",
    "        <li>Downloading the Webpage Using Requests Library</li>\n",
    "        <li>Parsing Webpage HTML Using BeautifulSoup</li>\n",
    "        <li>Extracting Data and Building DataFrame</li>\n",
    "    </ul>\n",
    "<p>\n",
    "    Estimated Time Needed: <strong>30 min</strong></p>\n",
    "</div>\n",
    "\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "                  __    __    __    __\n",
      "                 /  \\  /  \\  /  \\  /  \\\n",
      "                /    \\/    \\/    \\/    \\\n",
      "███████████████/  /██/  /██/  /██/  /████████████████████████\n",
      "              /  / \\   / \\   / \\   / \\  \\____\n",
      "             /  /   \\_/   \\_/   \\_/   \\    o \\__,\n",
      "            / _/                       \\_____/  `\n",
      "            |/\n",
      "        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗\n",
      "        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗\n",
      "        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║\n",
      "        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║\n",
      "        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║\n",
      "        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝\n",
      "\n",
      "        mamba (0.15.3) supported by @QuantStack\n",
      "\n",
      "        GitHub:  https://github.com/mamba-org/mamba\n",
      "        Twitter: https://twitter.com/QuantStack\n",
      "\n",
      "█████████████████████████████████████████████████████████████\n",
      "\n",
      "\n",
      "Looking for: ['bs4==4.10.0']\n",
      "\n",
      "pkgs/r/noarch            [>                   ] (--:--) Finalizing...\n",
      "pkgs/r/noarch            [>                   ] (--:--) Done\n",
      "pkgs/r/noarch            [====================] (00m:00s) Done\n",
      "pkgs/main/linux-64       [<=>                 ] (00m:00s) \n",
      "pkgs/main/linux-64       [=>                ] (00m:00s) 572 KB / ?? (1.85 MB/s)\n",
      "pkgs/main/linux-64       [=>                ] (00m:00s) 572 KB / ?? (1.85 MB/s)\n",
      "pkgs/main/noarch         [<=>                 ] (00m:00s) \n",
      "pkgs/main/linux-64       [=>                ] (00m:00s) 572 KB / ?? (1.85 MB/s)\n",
      "pkgs/main/noarch         [=>                ] (00m:00s) 468 KB / ?? (1.51 MB/s)\n",
      "pkgs/main/linux-64       [=>                ] (00m:00s) 572 KB / ?? (1.85 MB/s)\n",
      "pkgs/main/noarch         [=>                ] (00m:00s) 468 KB / ?? (1.51 MB/s)\n",
      "pkgs/r/linux-64          [<=>                 ] (00m:00s) \n",
      "pkgs/main/linux-64       [=>                ] (00m:00s) 572 KB / ?? (1.85 MB/s)\n",
      "pkgs/main/noarch         [=>                ] (00m:00s) 468 KB / ?? (1.51 MB/s)\n",
      "pkgs/r/linux-64          [=>                ] (00m:00s) 580 KB / ?? (1.87 MB/s)\n",
      "pkgs/main/linux-64       [=>                ] (00m:00s) 572 KB / ?? (1.85 MB/s)\n",
      "pkgs/main/noarch         [<=>                 ] (00m:00s) Finalizing...\n",
      "pkgs/r/linux-64          [=>                ] (00m:00s) 580 KB / ?? (1.87 MB/s)\n",
      "pkgs/main/linux-64       [=>                ] (00m:00s) 572 KB / ?? (1.85 MB/s)\n",
      "pkgs/main/noarch         [<=>                 ] (00m:00s) Done\n",
      "pkgs/r/linux-64          [=>                ] (00m:00s) 580 KB / ?? (1.87 MB/s)\n",
      "pkgs/main/noarch         [====================] (00m:00s) Done\n",
      "pkgs/main/linux-64       [=>                ] (00m:00s) 572 KB / ?? (1.85 MB/s)\n",
      "pkgs/r/linux-64          [=>                ] (00m:00s) 580 KB / ?? (1.87 MB/s)\n",
      "pkgs/main/linux-64       [=>                ] (00m:00s) 572 KB / ?? (1.85 MB/s)\n",
      "pkgs/r/linux-64          [<=>                 ] (00m:00s) Finalizing...\n",
      "pkgs/main/linux-64       [=>                ] (00m:00s) 572 KB / ?? (1.85 MB/s)\n",
      "pkgs/r/linux-64          [<=>                 ] (00m:00s) Done\n",
      "pkgs/r/linux-64          [====================] (00m:00s) Done\n",
      "pkgs/main/linux-64       [=>                ] (00m:00s) 572 KB / ?? (1.85 MB/s)\n",
      "pkgs/main/linux-64       [<=>               ] (00m:00s) 572 KB / ?? (1.85 MB/s)\n",
      "pkgs/main/linux-64       [<=>              ] (00m:00s) 1016 KB / ?? (2.11 MB/s)\n",
      "pkgs/main/linux-64       [ <=>             ] (00m:00s) 1016 KB / ?? (2.11 MB/s)\n",
      "pkgs/main/linux-64       [  <=>               ] (00m:00s) 2 MB / ?? (3.44 MB/s)\n",
      "pkgs/main/linux-64       [   <=>              ] (00m:00s) 2 MB / ?? (3.44 MB/s)\n",
      "pkgs/main/linux-64       [   <=>              ] (00m:00s) 3 MB / ?? (3.73 MB/s)\n",
      "pkgs/main/linux-64       [    <=>             ] (00m:00s) 3 MB / ?? (3.73 MB/s)\n",
      "pkgs/main/linux-64       [    <=>             ] (00m:00s) 4 MB / ?? (3.94 MB/s)\n",
      "pkgs/main/linux-64       [    <=>             ] (00m:00s) Finalizing...\n",
      "pkgs/main/linux-64       [    <=>             ] (00m:00s) Done\n",
      "pkgs/main/linux-64       [====================] (00m:00s) Done\n",
      "\n",
      "Pinned packages:\n",
      "  - python 3.7.*\n",
      "\n",
      "\n",
      "Transaction\n",
      "\n",
      "  Prefix: /home/jupyterlab/conda/envs/python\n",
      "\n",
      "  Updating specs:\n",
      "\n",
      "   - bs4==4.10.0\n",
      "   - ca-certificates\n",
      "   - certifi\n",
      "   - openssl\n",
      "\n",
      "\n",
      "  Package               Version  Build           Channel                  Size\n",
      "────────────────────────────────────────────────────────────────────────────────\n",
      "  Install:\n",
      "────────────────────────────────────────────────────────────────────────────────\n",
      "\n",
      "\u001b[32m  + beautifulsoup4 \u001b[00m      4.10.0  pyh06a4308_0    pkgs/main/noarch        85 KB\n",
      "\u001b[32m  + bs4            \u001b[00m      4.10.0  hd3eb1b0_0      pkgs/main/noarch        10 KB\n",
      "\u001b[32m  + soupsieve      \u001b[00m       2.3.1  pyhd3eb1b0_0    pkgs/main/noarch        34 KB\n",
      "\n",
      "  Change:\n",
      "────────────────────────────────────────────────────────────────────────────────\n",
      "\n",
      "\u001b[31m  - certifi        \u001b[00m   2021.10.8  py37h89c1867_1  installed                    \n",
      "\u001b[32m  + certifi        \u001b[00m   2021.10.8  py37h06a4308_2  pkgs/main/linux-64     151 KB\n",
      "\n",
      "  Upgrade:\n",
      "────────────────────────────────────────────────────────────────────────────────\n",
      "\n",
      "\u001b[31m  - ca-certificates\u001b[00m   2021.10.8  ha878542_0      installed                    \n",
      "\u001b[32m  + ca-certificates\u001b[00m  2021.10.26  h06a4308_2      pkgs/main/linux-64     115 KB\n",
      "\u001b[31m  - openssl        \u001b[00m      1.1.1l  h7f98852_0      installed                    \n",
      "\u001b[32m  + openssl        \u001b[00m      1.1.1m  h7f8727e_0      pkgs/main/linux-64       3 MB\n",
      "\n",
      "  Summary:\n",
      "\n",
      "  Install: 3 packages\n",
      "  Change: 1 packages\n",
      "  Upgrade: 2 packages\n",
      "\n",
      "  Total download: 3 MB\n",
      "\n",
      "────────────────────────────────────────────────────────────────────────────────\n",
      "\n",
      "Downloading  [>                                        ] (00m:00s)  183.81 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "\u001b[2A\u001b[0KFinished soupsieve                            (00m:00s)              34 KB    191 KB/s\n",
      "Downloading  [>                                        ] (00m:00s)  183.81 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [>                                        ] (00m:00s)  183.81 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [>                                        ] (00m:00s)  183.81 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [=>                                       ] (00m:00s)  634.84 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "\u001b[2A\u001b[0KFinished beautifulsoup4                       (00m:00s)              85 KB    468 KB/s\n",
      "Downloading  [=>                                       ] (00m:00s)  634.84 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [=>                                       ] (00m:00s)  634.84 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [==>                                      ] (00m:00s)    1.19 MB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [==>                                      ] (00m:00s)    1.19 MB/s\n",
      "Extracting   [======>                                  ] (00m:00s)        1 / 6\n",
      "Downloading  [==>                                      ] (00m:00s)    1.19 MB/s\n",
      "Extracting   [======>                                  ] (00m:00s)        1 / 6\n",
      "\u001b[2A\u001b[0KFinished ca-certificates                      (00m:00s)             115 KB    596 KB/s\n",
      "Downloading  [==>                                      ] (00m:00s)    1.19 MB/s\n",
      "Extracting   [======>                                  ] (00m:00s)        1 / 6\n",
      "Downloading  [==>                                      ] (00m:00s)    1.19 MB/s\n",
      "Extracting   [======>                                  ] (00m:00s)        1 / 6\n",
      "Downloading  [====>                                    ] (00m:00s)    1.92 MB/s\n",
      "Extracting   [======>                                  ] (00m:00s)        1 / 6\n",
      "\u001b[2A\u001b[0KFinished certifi                              (00m:00s)             151 KB    793 KB/s\n",
      "Downloading  [====>                                    ] (00m:00s)    1.92 MB/s\n",
      "Extracting   [======>                                  ] (00m:00s)        1 / 6\n",
      "Downloading  [====>                                    ] (00m:00s)    1.92 MB/s\n",
      "Extracting   [======>                                  ] (00m:00s)        1 / 6\n",
      "Downloading  [====>                                    ] (00m:00s)    1.92 MB/s\n",
      "Extracting   [=============>                           ] (00m:00s)        2 / 6\n",
      "Downloading  [====>                                    ] (00m:00s)    1.92 MB/s\n",
      "Extracting   [=============>                           ] (00m:00s)        2 / 6\n",
      "Downloading  [====>                                    ] (00m:00s)    1.92 MB/s\n",
      "Extracting   [====================>                    ] (00m:00s)        3 / 6\n",
      "Downloading  [====>                                    ] (00m:00s)    1.92 MB/s\n",
      "Extracting   [====================>                    ] (00m:00s)        3 / 6\n",
      "Downloading  [=====>                                   ] (00m:00s)    1.70 MB/s\n",
      "Extracting   [====================>                    ] (00m:00s)        3 / 6\n",
      "\u001b[2A\u001b[0KFinished bs4                                  (00m:00s)              10 KB     46 KB/s\n",
      "Downloading  [=====>                                   ] (00m:00s)    1.70 MB/s\n",
      "Extracting   [====================>                    ] (00m:00s)        3 / 6\n",
      "Downloading  [=====>                                   ] (00m:00s)    1.70 MB/s\n",
      "Extracting   [====================>                    ] (00m:00s)        3 / 6\n",
      "Downloading  [=========================================] (00m:00s)   12.55 MB/s\n",
      "Extracting   [====================>                    ] (00m:00s)        3 / 6\n",
      "Downloading  [=========================================] (00m:00s)   12.55 MB/s\n",
      "Extracting   [===========================>             ] (00m:00s)        4 / 6\n",
      "Downloading  [=========================================] (00m:00s)   12.55 MB/s\n",
      "Extracting   [===========================>             ] (00m:00s)        4 / 6\n",
      "Downloading  [=========================================] (00m:00s)   12.55 MB/s\n",
      "Extracting   [==================================>      ] (00m:00s)        5 / 6\n",
      "\u001b[2A\u001b[0KFinished openssl                              (00m:00s)               3 MB     11 MB/s\n",
      "Downloading  [=========================================] (00m:00s)   12.55 MB/s\n",
      "Extracting   [==================================>      ] (00m:00s)        5 / 6\n",
      "Downloading  [=========================================] (00m:00s)   12.55 MB/s\n",
      "Extracting   [==================================>      ] (00m:00s)        5 / 6\n",
      "Downloading  [=========================================] (00m:00s)   12.55 MB/s\n",
      "Extracting   [==================================>      ] (00m:00s)        5 / 6\n",
      "Downloading  [=========================================] (00m:00s)   12.55 MB/s\n",
      "Extracting   [=========================================] (00m:00s)        6 / 6\n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n",
      "\n",
      "                  __    __    __    __\n",
      "                 /  \\  /  \\  /  \\  /  \\\n",
      "                /    \\/    \\/    \\/    \\\n",
      "███████████████/  /██/  /██/  /██/  /████████████████████████\n",
      "              /  / \\   / \\   / \\   / \\  \\____\n",
      "             /  /   \\_/   \\_/   \\_/   \\    o \\__,\n",
      "            / _/                       \\_____/  `\n",
      "            |/\n",
      "        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗\n",
      "        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗\n",
      "        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║\n",
      "        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║\n",
      "        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║\n",
      "        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝\n",
      "\n",
      "        mamba (0.15.3) supported by @QuantStack\n",
      "\n",
      "        GitHub:  https://github.com/mamba-org/mamba\n",
      "        Twitter: https://twitter.com/QuantStack\n",
      "\n",
      "█████████████████████████████████████████████████████████████\n",
      "\n",
      "\n",
      "Looking for: ['html5lib==1.1']\n",
      "\n",
      "pkgs/main/linux-64       Using cache\n",
      "pkgs/main/noarch         Using cache\n",
      "pkgs/r/linux-64          Using cache\n",
      "pkgs/r/noarch            Using cache\n",
      "\n",
      "Pinned packages:\n",
      "  - python 3.7.*\n",
      "\n",
      "\n",
      "Transaction\n",
      "\n",
      "  Prefix: /home/jupyterlab/conda/envs/python\n",
      "\n",
      "  Updating specs:\n",
      "\n",
      "   - html5lib==1.1\n",
      "   - ca-certificates\n",
      "   - certifi\n",
      "   - openssl\n",
      "\n",
      "\n",
      "  Package         Version  Build         Channel                 Size\n",
      "───────────────────────────────────────────────────────────────────────\n",
      "  Install:\n",
      "───────────────────────────────────────────────────────────────────────\n",
      "\n",
      "\u001b[32m  + html5lib    \u001b[00m      1.1  pyhd3eb1b0_0  pkgs/main/noarch       91 KB\n",
      "\u001b[32m  + webencodings\u001b[00m    0.5.1  py37_1        pkgs/main/linux-64     19 KB\n",
      "\n",
      "  Summary:\n",
      "\n",
      "  Install: 2 packages\n",
      "\n",
      "  Total download: 110 KB\n",
      "\n",
      "───────────────────────────────────────────────────────────────────────\n",
      "\n",
      "Downloading  [======>                                  ] (00m:00s)   71.78 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "\u001b[2A\u001b[0KFinished webencodings                         (00m:00s)              19 KB     72 KB/s\n",
      "Downloading  [======>                                  ] (00m:00s)   71.78 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [======>                                  ] (00m:00s)   71.78 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [=========================================] (00m:00s)  410.14 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [=========================================] (00m:00s)  410.14 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "\u001b[2A\u001b[0KFinished html5lib                             (00m:00s)              91 KB    339 KB/s\n",
      "Downloading  [=========================================] (00m:00s)  410.14 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [=========================================] (00m:00s)  410.14 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [=========================================] (00m:00s)  410.14 KB/s\n",
      "Extracting   [====================>                    ] (00m:00s)        1 / 2\n",
      "Downloading  [=========================================] (00m:00s)  410.14 KB/s\n",
      "Extracting   [====================>                    ] (00m:00s)        1 / 2\n",
      "Downloading  [=========================================] (00m:00s)  410.14 KB/s\n",
      "Extracting   [=========================================] (00m:00s)        2 / 2\n"
     ]
    }
   ],
   "source": [
    "#!pip install pandas==1.3.3\n",
    "#!pip install requests==2.26.0\n",
    "!mamba install bs4==4.10.0 -y\n",
    "!mamba install html5lib==1.1 -y\n",
    "!pip install lxml==4.6.4\n",
    "#!pip install plotly==5.3.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Webscraping to Extract Stock Data Example\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we must use the `request` library to downlaod the webpage, and extract the text. We will extract Netflix stock data <https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/netflix_data_webpage.html>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/netflix_data_webpage.html\"\n",
    "\n",
    "data  = requests.get(url).text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we must parse the text into html using `beautiful_soup`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(data, 'html5lib')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can turn the html table into a pandas dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "netflix_data = pd.DataFrame(columns=[\"Date\", \"Open\", \"High\", \"Low\", \"Close\", \"Volume\"])\n",
    "\n",
    "# First we isolate the body of the table which contains all the information\n",
    "# Then we loop through each row and find all the column values for each row\n",
    "for row in soup.find(\"tbody\").find_all('tr'):\n",
    "    col = row.find_all(\"td\")\n",
    "    date = col[0].text\n",
    "    Open = col[1].text\n",
    "    high = col[2].text\n",
    "    low = col[3].text\n",
    "    close = col[4].text\n",
    "    adj_close = col[5].text\n",
    "    volume = col[6].text\n",
    "    \n",
    "    # Finally we append the data of each row to the table\n",
    "    netflix_data = netflix_data.append({\"Date\":date, \"Open\":Open, \"High\":high, \"Low\":low, \"Close\":close, \"Adj Close\":adj_close, \"Volume\":volume}, ignore_index=True)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now print out the dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "netflix_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the pandas `read_html` function using the url\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "read_html_pandas_data = pd.read_html(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can convert the BeautifulSoup object to a string\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "read_html_pandas_data = pd.read_html(str(soup))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beacause there is only one table on the page, we just take the first table in the list returned\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "netflix_dataframe = read_html_pandas_data[0]\n",
    "\n",
    "netflix_dataframe.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Webscraping to Extract Stock Data Exercise\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `requests` library to download the webpage <https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/amazon_data_webpage.html>. Save the text of the response as a variable named `html_data`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url2 = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/amazon_data_webpage.html\"\n",
    "html_data = requests.get(url2).text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Parse the html data using `beautiful_soup`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "html_soup = BeautifulSoup(html_data, 'html5lib')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Question 1</b> What is the content of the title attribute:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "html_soup.title"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using beautiful soup extract the table with historical share prices and store it into a dataframe named `amazon_data`. The dataframe should have columns Date, Open, High, Low, Close, Adj Close, and Volume. Fill in each variable with the correct data from the list `col`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "amazon_data = pd.DataFrame(columns=[\"Date\", \"Open\", \"High\", \"Low\", \"Close\", \"Adj Close\", \"Volume\"])\n",
    "\n",
    "for row in html_soup.find(\"tbody\").find_all(\"tr\"):\n",
    "    col = row.find_all(\"td\")\n",
    "    date = col[0].text\n",
    "    Open = col[1].text\n",
    "    high = col[2].text\n",
    "    low = col[3].text\n",
    "    close = col[4].text\n",
    "    adj_close = col[5].text\n",
    "    volume = col[6].text\n",
    "    \n",
    "    amazon_data = amazon_data.append({\"Date\":date, \"Open\":Open, \"High\":high, \"Low\":low, \"Close\":close, \"Adj Close\":adj_close, \"Volume\":volume}, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print out the first five rows of the `amazon_data` dataframe you created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "amazon_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Question 2</b> What is the name of the columns of the dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(amazon_data.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Question 3</b> What is the `Open` of the last row of the amazon_data dataframe?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "amazon_data.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>About the Authors:</h2> \n",
    "\n",
    "<a href=\"https://www.linkedin.com/in/joseph-s-50398b136/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0220ENSkillsNetwork23455606-2021-01-01\">Joseph Santarcangelo</a> has a PhD in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD.\n",
    "\n",
    "Azim Hirjani\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n",
    "\n",
    "| Date (YYYY-MM-DD) | Version | Changed By | Change Description |\n",
    "| ----------------- | ------- | ---------- | ------------------ |\n",
    "\n",
    "```\n",
    "| 2021-06-09       | 1.2     | Lakshmi Holla|Added URL in question 3 |\n",
    "```\n",
    "\n",
    "\\| 2020-11-10        | 1.1     | Malika Singla | Deleted the Optional part |\n",
    "\\| 2020-08-27        | 1.0     | Malika Singla | Added lab to GitLab       |\n",
    "\n",
    "<hr>\n",
    "\n",
    "## <h3 align=\"center\"> © IBM Corporation 2020. All rights reserved. <h3/>\n",
    "\n",
    "<p>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}