github-top-repositories-scrapping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "22d2c6f1-34ef-4467-b998-a0816b640e7a",
   "metadata": {},
   "source": [
    "# TOP REPOSITORIES TOPIC FOR GITHUB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7daa8ced-3626-4448-840a-a89e5c0a8a49",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "69b6b783-bdbe-46a9-9191-d2815bef3a91",
   "metadata": {},
   "source": [
    "## Process i'm following to build a Python Web Scraping Project From Scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76555733-69f9-4203-835b-64c2da2997a5",
   "metadata": {
    "tags": []
   },
   "source": [
    "![](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcS3bNvHFcfFCkH9TXpLkjZNt5Fq_oDbKyRmqg&usqp=CAU)\n",
    "\n",
    "Web scraping is the process of extracting and parsing data from websites in an automated fashion using a computer program. It's a useful technique for creating datasets for research and learning. Follow these steps to build a web scraping project from scratch using Python and its ecosystem of libraries:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89658877-394d-4e0a-839e-d2ea9928105d",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d9a1c7f7-3a5d-43dd-837c-7853f22e0a10",
   "metadata": {},
   "source": [
    "1. **Pick a website and describe your objective**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dab184e5-fa4f-4440-b401-09d1a11cf0c5",
   "metadata": {
    "tags": []
   },
   "source": [
    "\n",
    "- Browse through different sites and pick on to scrape. Check the \"Project Ideas\" section for inspiration.\n",
    "- Identify the information you'd like to scrape from the site. Decide the format of the output CSV file.\n",
    "- Summarize your project idea and outline your strategy in a Juptyer notebook. Use the \"New\" button above.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55818505-2877-47c7-9174-46c259da2cdb",
   "metadata": {},
   "source": [
    "2. **Use the requests library to download web pages**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "554b600a-3623-4ec9-b813-57d33e0423b9",
   "metadata": {},
   "source": [
    "-  Inspect the website's HTML source and identify the right URLs to download. \n",
    "- Download and save web pages locally using the `requests` library.\n",
    "- Create a function to automate downloading for different topics/search queries."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c9045d4-9089-4913-8cf1-5c3174cd09b6",
   "metadata": {},
   "source": [
    "3. **Use Beautiful Soup to parse and extract information**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b13cda8-be8b-4903-9b86-058c0218d781",
   "metadata": {},
   "source": [
    "- Parse and explore the structure of downloaded web pages using Beautiful soup.\n",
    "- Use the right properties and methods to extract the required information.\n",
    "- Create functions to extract from the page into lists and dictionaries.\n",
    "- (Optional) Use a REST API to acquire additional information if required.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a71d9999-2f6c-4699-8ff0-10132b0ebd93",
   "metadata": {},
   "source": [
    "4. **Create CSV file(s) with the extracted information**\n",
    "\n",
    " - Create functions for the end-to-end process of downloading, parsing, and saving CSVs.\n",
    " - Execute the function with different inputs to create a dataset of CSV files.\n",
    " - Verify the information in the CSV files by reading them back using [Pandas](https://pandas.pydata.org).\n",
    "\n",
    "\n",
    "5. **Document and share my  work**\n",
    "\n",
    " - Add proper headings and documentation in your Jupyter notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d66f93d-2db0-40da-8940-b74e49523fed",
   "metadata": {},
   "source": [
    "## Project Outlines\n",
    "- Going to scrape https://github.com/topics.\n",
    "- Get a list of topics. For each topic, will get topic title, topic page URL and topic description.\n",
    "- For each topic, will get top 20 repositories in the topic from the topic page\n",
    "- For each repository, will get the repo name, username, stars and repo URL\n",
    "- For each topic, will create a CSV file in the following format:\n",
    "\n",
    "```\n",
    "Repo Name,Username,Stars,Repo URL\n",
    "three.js,mardoob,92200,https://github.com/mrdoob/three.js\n",
    "react-three-fibre,pmndrs,22700,https://github.com/pmndrs/react-three-fiber\n",
    "libgdx,libgdx,21500,https://github.com/libgdx/libgdx\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abbd4936-1086-46a7-8907-37f495265fd7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f3dc4918-0c0c-4235-9f41-ac7324879b7b",
   "metadata": {},
   "source": [
    "### Use the `requests` library to download the web pages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "71de66ee-a04b-433f-885f-2e8d64baf785",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install requests --upgrade --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "2870e002-c797-42ac-8440-c110d8613177",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "f9982202-9d4b-4192-a87b-bd3bbe979efc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "topics_url = 'https://github.com/topics'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "d3c23fa6-d145-43f6-a4e7-f9768d39953e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "response_url = requests.get(topics_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "54eaabae-9f06-475b-92f9-d874a6445c96",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response_url.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dbe825f-9bab-4375-bc12-f7a73b04f1fe",
   "metadata": {},
   "source": [
    "**200** is the status code that the response code is successfull. If you wanna learn about status code more visit https://developer.mozilla.org/en-US/docs/Web/HTTP/Status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "7e76bce0-032d-4681-96a0-929d452929ff",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "155352"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(response_url.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "4f63a835-1ded-445d-94ea-39413df1d0d1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "topics_page_contents = response_url.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "d6968af9-b438-4e3e-823b-3cba76fe50b6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n\\n<!DOCTYPE html>\\n<html lang=\"en\" data-color-mode=\"auto\" data-light-theme=\"light\" data-dark-theme=\"dark\" data-a11y-animated-images=\"system\">\\n  <head>\\n    <meta charset=\"utf-8\">\\n  <link rel=\"dns-prefetch\" href=\"https://github.githubassets.com\">\\n  <link rel=\"dns-prefetch\" href=\"https://avatars.githubusercontent.com\">\\n  <link rel=\"dns-prefetch\" href=\"https://github-cloud.s3.amazonaws.com\">\\n  <link rel=\"dns-prefetch\" href=\"https://user-images.githubusercontent.com/\">\\n  <link rel=\"preconnect\" href=\"https://github.githubassets.com\" crossorigin>\\n  <link rel=\"preconnect\" href=\"https://avatars.githubusercontent.com\">\\n\\n  \\n\\n  <link crossorigin=\"anonymous\" media=\"all\" rel=\"stylesheet\" href=\"https://github.githubassets.com/assets/light-0946cdc16f15.css\" /><link crossorigin=\"anonymous\" media=\"all\" rel=\"stylesheet\" href=\"https://github.githubassets.com/assets/dark-3946c959759a.css\" /><link data-color-theme=\"dark_dimmed\" crossorigin=\"anonymous\" media=\"all\" rel=\"stylesheet\" data-href=\"https://github.gi'"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topics_page_contents[:1000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "6cf407ae-80f8-4d63-8186-1914e5a90799",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open('webpage.html', 'w') as f:\n",
    "    f.write(topics_page_contents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "905b2b16-d6f0-4be8-8378-4de85368d4a7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install beautifulsoup4 --upgrade --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "4394e023-85be-43f0-be12-0d975f3d8883",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "d314281f-0291-4b41-ab2a-dec00eaa8702",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "doc = BeautifulSoup(topics_page_contents, 'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "1bf5af70-029f-4f2d-a3fe-a524478c0328",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bs4.BeautifulSoup"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "cb66a81c-8c91-487a-9a5c-896e56578139",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "topic_title_p_tags = doc.find_all('p', {'class': 'f3 lh-condensed mb-0 mt-1 Link--primary'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "3c659f8a-3c05-42e9-9436-3cb986b20975",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(topic_title_p_tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "c29a223e-c3fc-4672-9e41-a5ec5a9e0148",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">3D</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Ajax</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Algorithm</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Amp</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Android</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Angular</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Ansible</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">API</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Arduino</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">ASP.NET</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Atom</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Awesome Lists</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Amazon Web Services</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Azure</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Babel</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Bash</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Bitcoin</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Bootstrap</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Bot</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">C</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Chrome</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Chrome extension</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Command line interface</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Clojure</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Code quality</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Code review</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Compiler</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">Continuous integration</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">COVID-19</p>,\n",
       " <p class=\"f3 lh-condensed mb-0 mt-1 Link--primary\">C++</p>]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_title_p_tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "3580808e-7334-4e7f-9414-bc949f09fa30",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['3D', 'Ajax', 'Algorithm', 'Amp', 'Android', 'Angular', 'Ansible', 'API', 'Arduino', 'ASP.NET', 'Atom', 'Awesome Lists', 'Amazon Web Services', 'Azure', 'Babel', 'Bash', 'Bitcoin', 'Bootstrap', 'Bot', 'C', 'Chrome', 'Chrome extension', 'Command line interface', 'Clojure', 'Code quality', 'Code review', 'Compiler', 'Continuous integration', 'COVID-19', 'C++']\n"
     ]
    }
   ],
   "source": [
    "topic_titles = []\n",
    "\n",
    "for tag in topic_title_p_tags:\n",
    "    topic_titles.append(tag.text)\n",
    "\n",
    "print(topic_titles)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "8161b487-3345-47ab-a917-c919e58d81f3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "topic_desc_tags = doc.find_all('p', {'class': 'f5 color-fg-muted mb-0 mt-1'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "3c8ee51a-f442-4ab5-995f-18bf6b3ed503",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(topic_desc_tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "5d20f865-ef21-4035-bc1d-a5b3f9e9de72",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<p class=\"f5 color-fg-muted mb-0 mt-1\">\n",
       "           3D refers to the use of three-dimensional graphics, modeling, and animation in various industries.\n",
       "         </p>,\n",
       " <p class=\"f5 color-fg-muted mb-0 mt-1\">\n",
       "           Ajax is a technique for creating interactive web applications.\n",
       "         </p>,\n",
       " <p class=\"f5 color-fg-muted mb-0 mt-1\">\n",
       "           Algorithms are self-contained sequences that carry out a variety of tasks.\n",
       "         </p>,\n",
       " <p class=\"f5 color-fg-muted mb-0 mt-1\">\n",
       "           Amp is a non-blocking concurrency library for PHP.\n",
       "         </p>,\n",
       " <p class=\"f5 color-fg-muted mb-0 mt-1\">\n",
       "           Android is an operating system built by Google designed for mobile devices.\n",
       "         </p>]"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_desc_tags[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "e932a832-6a91-4133-a36b-70fca11c985b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "topic_link_tags = doc.find_all('a', {'class': 'no-underline flex-grow-0'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98b40534",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "00140cc9-ad4c-4d23-9887-6767def21fb8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['3D refers to the use of three-dimensional graphics, modeling, and animation in various industries.',\n",
       " 'Ajax is a technique for creating interactive web applications.',\n",
       " 'Algorithms are self-contained sequences that carry out a variety of tasks.',\n",
       " 'Amp is a non-blocking concurrency library for PHP.',\n",
       " 'Android is an operating system built by Google designed for mobile devices.']"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_descs = []\n",
    "\n",
    "for tag in topic_desc_tags:\n",
    "    topic_descs.append(tag.text.strip())\n",
    "    \n",
    "topic_descs[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df8a9592-3040-48f4-9581-165edd5e04df",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "7a30d51f-5668-4f83-a7b0-4bcb52916ca1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://github.com/topics/3d',\n",
       " 'https://github.com/topics/ajax',\n",
       " 'https://github.com/topics/algorithm',\n",
       " 'https://github.com/topics/amphp',\n",
       " 'https://github.com/topics/android',\n",
       " 'https://github.com/topics/angular',\n",
       " 'https://github.com/topics/ansible',\n",
       " 'https://github.com/topics/api',\n",
       " 'https://github.com/topics/arduino',\n",
       " 'https://github.com/topics/aspnet',\n",
       " 'https://github.com/topics/atom',\n",
       " 'https://github.com/topics/awesome',\n",
       " 'https://github.com/topics/aws',\n",
       " 'https://github.com/topics/azure',\n",
       " 'https://github.com/topics/babel',\n",
       " 'https://github.com/topics/bash',\n",
       " 'https://github.com/topics/bitcoin',\n",
       " 'https://github.com/topics/bootstrap',\n",
       " 'https://github.com/topics/bot',\n",
       " 'https://github.com/topics/c',\n",
       " 'https://github.com/topics/chrome',\n",
       " 'https://github.com/topics/chrome-extension',\n",
       " 'https://github.com/topics/cli',\n",
       " 'https://github.com/topics/clojure',\n",
       " 'https://github.com/topics/code-quality',\n",
       " 'https://github.com/topics/code-review',\n",
       " 'https://github.com/topics/compiler',\n",
       " 'https://github.com/topics/continuous-integration',\n",
       " 'https://github.com/topics/covid-19',\n",
       " 'https://github.com/topics/cpp']"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_urls = []\n",
    "base_url = 'https://github.com'\n",
    "\n",
    "for tag in topic_link_tags:\n",
    "    topic_urls.append(base_url + tag['href'])\n",
    "    \n",
    "topic_urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "8a5a895d-fa7a-4e92-a6a8-a1d544afb1d2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install pandas --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "0fdf38a1-c69d-4ab4-b75b-a9c9a43934d3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "2e353064-fe6d-49f6-8fae-fe2695d6e5f1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "topics_dict = {\n",
    "    'title': topic_titles,\n",
    "    'description': topic_descs,\n",
    "    'url': topic_urls\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "96f6bc9f-b002-4e01-809b-0e4db1934305",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'title': ['3D',\n",
       "  'Ajax',\n",
       "  'Algorithm',\n",
       "  'Amp',\n",
       "  'Android',\n",
       "  'Angular',\n",
       "  'Ansible',\n",
       "  'API',\n",
       "  'Arduino',\n",
       "  'ASP.NET',\n",
       "  'Atom',\n",
       "  'Awesome Lists',\n",
       "  'Amazon Web Services',\n",
       "  'Azure',\n",
       "  'Babel',\n",
       "  'Bash',\n",
       "  'Bitcoin',\n",
       "  'Bootstrap',\n",
       "  'Bot',\n",
       "  'C',\n",
       "  'Chrome',\n",
       "  'Chrome extension',\n",
       "  'Command line interface',\n",
       "  'Clojure',\n",
       "  'Code quality',\n",
       "  'Code review',\n",
       "  'Compiler',\n",
       "  'Continuous integration',\n",
       "  'COVID-19',\n",
       "  'C++'],\n",
       " 'description': ['3D refers to the use of three-dimensional graphics, modeling, and animation in various industries.',\n",
       "  'Ajax is a technique for creating interactive web applications.',\n",
       "  'Algorithms are self-contained sequences that carry out a variety of tasks.',\n",
       "  'Amp is a non-blocking concurrency library for PHP.',\n",
       "  'Android is an operating system built by Google designed for mobile devices.',\n",
       "  'Angular is an open source web application platform.',\n",
       "  'Ansible is a simple and powerful automation engine.',\n",
       "  'An API (Application Programming Interface) is a collection of protocols and subroutines for building software.',\n",
       "  'Arduino is an open source platform for building electronic devices.',\n",
       "  'ASP.NET is a web framework for building modern web apps and services.',\n",
       "  'Atom is a open source text editor built with web technologies.',\n",
       "  'An awesome list is a list of awesome things curated by the community.',\n",
       "  'Amazon Web Services provides on-demand cloud computing platforms on a subscription basis.',\n",
       "  'Azure is a cloud computing service created by Microsoft.',\n",
       "  'Babel is a compiler for writing next generation JavaScript, today.',\n",
       "  'Bash is a shell and command language interpreter for the GNU operating system.',\n",
       "  'Bitcoin is a cryptocurrency developed by Satoshi Nakamoto.',\n",
       "  'Bootstrap is an HTML, CSS, and JavaScript framework.',\n",
       "  'A bot is an application that runs automated tasks over the Internet.',\n",
       "  'C is a general purpose programming language that first appeared in 1972.',\n",
       "  'Chrome is a web browser from the tech company Google.',\n",
       "  'Chrome extensions enable users to customize the Chrome browsing experience.',\n",
       "  'A CLI, or command-line interface, is a console that helps users issue commands to a program.',\n",
       "  'Clojure is a dynamic, general-purpose programming language.',\n",
       "  'Automate your code review with style, quality, security, and test‑coverage checks when you need them.',\n",
       "  'Ensure your code meets quality standards and ship with confidence.',\n",
       "  'Compilers are software that translate higher-level programming languages to lower-level languages (e.g. machine code).',\n",
       "  'Automatically build and test your code as you push it upstream, preventing bugs from being deployed to production.',\n",
       "  'The coronavirus disease 2019 (COVID-19) is an infectious disease caused by SARS-CoV-2.',\n",
       "  'C++ is a general purpose and object-oriented programming language.'],\n",
       " 'url': ['https://github.com/topics/3d',\n",
       "  'https://github.com/topics/ajax',\n",
       "  'https://github.com/topics/algorithm',\n",
       "  'https://github.com/topics/amphp',\n",
       "  'https://github.com/topics/android',\n",
       "  'https://github.com/topics/angular',\n",
       "  'https://github.com/topics/ansible',\n",
       "  'https://github.com/topics/api',\n",
       "  'https://github.com/topics/arduino',\n",
       "  'https://github.com/topics/aspnet',\n",
       "  'https://github.com/topics/atom',\n",
       "  'https://github.com/topics/awesome',\n",
       "  'https://github.com/topics/aws',\n",
       "  'https://github.com/topics/azure',\n",
       "  'https://github.com/topics/babel',\n",
       "  'https://github.com/topics/bash',\n",
       "  'https://github.com/topics/bitcoin',\n",
       "  'https://github.com/topics/bootstrap',\n",
       "  'https://github.com/topics/bot',\n",
       "  'https://github.com/topics/c',\n",
       "  'https://github.com/topics/chrome',\n",
       "  'https://github.com/topics/chrome-extension',\n",
       "  'https://github.com/topics/cli',\n",
       "  'https://github.com/topics/clojure',\n",
       "  'https://github.com/topics/code-quality',\n",
       "  'https://github.com/topics/code-review',\n",
       "  'https://github.com/topics/compiler',\n",
       "  'https://github.com/topics/continuous-integration',\n",
       "  'https://github.com/topics/covid-19',\n",
       "  'https://github.com/topics/cpp']}"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topics_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "cdf4ed40-2060-40bd-a3b9-cc79c5b69c57",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "topics_df = pd.DataFrame(topics_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "ba10c168-c3a9-4a83-aea5-9e1c0ccf8ae2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>description</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3D</td>\n",
       "      <td>3D refers to the use of three-dimensional grap...</td>\n",
       "      <td>https://github.com/topics/3d</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ajax</td>\n",
       "      <td>Ajax is a technique for creating interactive w...</td>\n",
       "      <td>https://github.com/topics/ajax</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Algorithm</td>\n",
       "      <td>Algorithms are self-contained sequences that c...</td>\n",
       "      <td>https://github.com/topics/algorithm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Amp</td>\n",
       "      <td>Amp is a non-blocking concurrency library for ...</td>\n",
       "      <td>https://github.com/topics/amphp</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Android</td>\n",
       "      <td>Android is an operating system built by Google...</td>\n",
       "      <td>https://github.com/topics/android</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Angular</td>\n",
       "      <td>Angular is an open source web application plat...</td>\n",
       "      <td>https://github.com/topics/angular</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Ansible</td>\n",
       "      <td>Ansible is a simple and powerful automation en...</td>\n",
       "      <td>https://github.com/topics/ansible</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>API</td>\n",
       "      <td>An API (Application Programming Interface) is ...</td>\n",
       "      <td>https://github.com/topics/api</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Arduino</td>\n",
       "      <td>Arduino is an open source platform for buildin...</td>\n",
       "      <td>https://github.com/topics/arduino</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>ASP.NET</td>\n",
       "      <td>ASP.NET is a web framework for building modern...</td>\n",
       "      <td>https://github.com/topics/aspnet</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Atom</td>\n",
       "      <td>Atom is a open source text editor built with w...</td>\n",
       "      <td>https://github.com/topics/atom</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Awesome Lists</td>\n",
       "      <td>An awesome list is a list of awesome things cu...</td>\n",
       "      <td>https://github.com/topics/awesome</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Amazon Web Services</td>\n",
       "      <td>Amazon Web Services provides on-demand cloud c...</td>\n",
       "      <td>https://github.com/topics/aws</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Azure</td>\n",
       "      <td>Azure is a cloud computing service created by ...</td>\n",
       "      <td>https://github.com/topics/azure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Babel</td>\n",
       "      <td>Babel is a compiler for writing next generatio...</td>\n",
       "      <td>https://github.com/topics/babel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Bash</td>\n",
       "      <td>Bash is a shell and command language interpret...</td>\n",
       "      <td>https://github.com/topics/bash</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Bitcoin</td>\n",
       "      <td>Bitcoin is a cryptocurrency developed by Satos...</td>\n",
       "      <td>https://github.com/topics/bitcoin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Bootstrap</td>\n",
       "      <td>Bootstrap is an HTML, CSS, and JavaScript fram...</td>\n",
       "      <td>https://github.com/topics/bootstrap</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Bot</td>\n",
       "      <td>A bot is an application that runs automated ta...</td>\n",
       "      <td>https://github.com/topics/bot</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>C</td>\n",
       "      <td>C is a general purpose programming language th...</td>\n",
       "      <td>https://github.com/topics/c</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Chrome</td>\n",
       "      <td>Chrome is a web browser from the tech company ...</td>\n",
       "      <td>https://github.com/topics/chrome</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Chrome extension</td>\n",
       "      <td>Chrome extensions enable users to customize th...</td>\n",
       "      <td>https://github.com/topics/chrome-extension</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Command line interface</td>\n",
       "      <td>A CLI, or command-line interface, is a console...</td>\n",
       "      <td>https://github.com/topics/cli</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Clojure</td>\n",
       "      <td>Clojure is a dynamic, general-purpose programm...</td>\n",
       "      <td>https://github.com/topics/clojure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Code quality</td>\n",
       "      <td>Automate your code review with style, quality,...</td>\n",
       "      <td>https://github.com/topics/code-quality</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Code review</td>\n",
       "      <td>Ensure your code meets quality standards and s...</td>\n",
       "      <td>https://github.com/topics/code-review</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Compiler</td>\n",
       "      <td>Compilers are software that translate higher-l...</td>\n",
       "      <td>https://github.com/topics/compiler</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Continuous integration</td>\n",
       "      <td>Automatically build and test your code as you ...</td>\n",
       "      <td>https://github.com/topics/continuous-integration</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>COVID-19</td>\n",
       "      <td>The coronavirus disease 2019 (COVID-19) is an ...</td>\n",
       "      <td>https://github.com/topics/covid-19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>C++</td>\n",
       "      <td>C++ is a general purpose and object-oriented p...</td>\n",
       "      <td>https://github.com/topics/cpp</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     title                                        description  \\\n",
       "0                       3D  3D refers to the use of three-dimensional grap...   \n",
       "1                     Ajax  Ajax is a technique for creating interactive w...   \n",
       "2                Algorithm  Algorithms are self-contained sequences that c...   \n",
       "3                      Amp  Amp is a non-blocking concurrency library for ...   \n",
       "4                  Android  Android is an operating system built by Google...   \n",
       "5                  Angular  Angular is an open source web application plat...   \n",
       "6                  Ansible  Ansible is a simple and powerful automation en...   \n",
       "7                      API  An API (Application Programming Interface) is ...   \n",
       "8                  Arduino  Arduino is an open source platform for buildin...   \n",
       "9                  ASP.NET  ASP.NET is a web framework for building modern...   \n",
       "10                    Atom  Atom is a open source text editor built with w...   \n",
       "11           Awesome Lists  An awesome list is a list of awesome things cu...   \n",
       "12     Amazon Web Services  Amazon Web Services provides on-demand cloud c...   \n",
       "13                   Azure  Azure is a cloud computing service created by ...   \n",
       "14                   Babel  Babel is a compiler for writing next generatio...   \n",
       "15                    Bash  Bash is a shell and command language interpret...   \n",
       "16                 Bitcoin  Bitcoin is a cryptocurrency developed by Satos...   \n",
       "17               Bootstrap  Bootstrap is an HTML, CSS, and JavaScript fram...   \n",
       "18                     Bot  A bot is an application that runs automated ta...   \n",
       "19                       C  C is a general purpose programming language th...   \n",
       "20                  Chrome  Chrome is a web browser from the tech company ...   \n",
       "21        Chrome extension  Chrome extensions enable users to customize th...   \n",
       "22  Command line interface  A CLI, or command-line interface, is a console...   \n",
       "23                 Clojure  Clojure is a dynamic, general-purpose programm...   \n",
       "24            Code quality  Automate your code review with style, quality,...   \n",
       "25             Code review  Ensure your code meets quality standards and s...   \n",
       "26                Compiler  Compilers are software that translate higher-l...   \n",
       "27  Continuous integration  Automatically build and test your code as you ...   \n",
       "28                COVID-19  The coronavirus disease 2019 (COVID-19) is an ...   \n",
       "29                     C++  C++ is a general purpose and object-oriented p...   \n",
       "\n",
       "                                                 url  \n",
       "0                       https://github.com/topics/3d  \n",
       "1                     https://github.com/topics/ajax  \n",
       "2                https://github.com/topics/algorithm  \n",
       "3                    https://github.com/topics/amphp  \n",
       "4                  https://github.com/topics/android  \n",
       "5                  https://github.com/topics/angular  \n",
       "6                  https://github.com/topics/ansible  \n",
       "7                      https://github.com/topics/api  \n",
       "8                  https://github.com/topics/arduino  \n",
       "9                   https://github.com/topics/aspnet  \n",
       "10                    https://github.com/topics/atom  \n",
       "11                 https://github.com/topics/awesome  \n",
       "12                     https://github.com/topics/aws  \n",
       "13                   https://github.com/topics/azure  \n",
       "14                   https://github.com/topics/babel  \n",
       "15                    https://github.com/topics/bash  \n",
       "16                 https://github.com/topics/bitcoin  \n",
       "17               https://github.com/topics/bootstrap  \n",
       "18                     https://github.com/topics/bot  \n",
       "19                       https://github.com/topics/c  \n",
       "20                  https://github.com/topics/chrome  \n",
       "21        https://github.com/topics/chrome-extension  \n",
       "22                     https://github.com/topics/cli  \n",
       "23                 https://github.com/topics/clojure  \n",
       "24            https://github.com/topics/code-quality  \n",
       "25             https://github.com/topics/code-review  \n",
       "26                https://github.com/topics/compiler  \n",
       "27  https://github.com/topics/continuous-integration  \n",
       "28                https://github.com/topics/covid-19  \n",
       "29                     https://github.com/topics/cpp  "
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topics_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "815b4e87-c8e0-4767-89a8-4f6f573e95e0",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Create CSV files with the extracted information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "c7ff4c5f-8cb4-4a37-a1b5-285a3d69abba",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "topics_df.to_csv('topics.csv', index = None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02256cee-d63d-4632-9c1b-bf9c339d67b7",
   "metadata": {},
   "source": [
    "### Getting information out of a topic page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "2b581ec0-3925-485f-a56a-9c0070d6f71c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "topic_page_url = topic_urls[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "d55e0372-0d3b-4365-9737-fdb4be4e9f51",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://github.com/topics/3d'"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_page_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "ecc2d111-2f40-4095-9080-a481f1754596",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(topic_page_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "32d1e89d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "d7a47729",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "463455"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(response.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "132ceb02",
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_doc = BeautifulSoup(response.text, 'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "36597856",
   "metadata": {},
   "outputs": [],
   "source": [
    "h3_selection_class = 'f3 color-fg-muted text-normal lh-condensed'\n",
    "repo_tags = topic_doc.find_all('h3', {'class': h3_selection_class})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "bdb85ad9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(repo_tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "0e3f8990",
   "metadata": {},
   "outputs": [],
   "source": [
    "a_tags = repo_tags[0].find_all('a')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "6ebd7e9f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mrdoob'"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a_tags[0].text.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "5976a6d6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'three.js'"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a_tags[1].text.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "99431c1a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://github.com/mrdoob/three.js\n"
     ]
    }
   ],
   "source": [
    "repo_url = base_url + a_tags[1]['href']\n",
    "print(repo_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b958f93",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "589b3a3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "star_tags = topic_doc.find_all('span', {'class': 'Counter js-social-count'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "4e44158c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(star_tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "843752db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'15.4k'"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "star_tags[6].text.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "2142573f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_star_count(stars_str):\n",
    "    stars_str = stars_str.strip()\n",
    "    if stars_str[-1] == 'k':\n",
    "        return int(float(stars_str[:-1]) * 1000)\n",
    "    return int(stars_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "91d58134",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15400"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parse_star_count(star_tags[6].text.strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4365fdad",
   "metadata": {},
   "source": [
    "### divide and conquer of def `parse_star_count(stars_str)"
   ]
  },
  {
   "cell_type": "raw",
   "id": "79fd29d4",
   "metadata": {},
   "source": [
    "stars_str = '889'"
   ]
  },
  {
   "cell_type": "raw",
   "id": "5a13ab44",
   "metadata": {},
   "source": [
    "int(stars_str)"
   ]
  },
  {
   "cell_type": "raw",
   "id": "a1437bb0",
   "metadata": {},
   "source": [
    "stars_str = '15.4k'"
   ]
  },
  {
   "cell_type": "raw",
   "id": "91961a7f",
   "metadata": {},
   "source": [
    "stars_str[-1]"
   ]
  },
  {
   "cell_type": "raw",
   "id": "328dfaf1",
   "metadata": {},
   "source": [
    "int(float(stars_str[:-1]) * 1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "5bfa829b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_repo_info(h3_tag, star_tag):\n",
    "    # returns all necessary info about a repository\n",
    "    a_tags = h3_tag.find_all('a')\n",
    "    username = a_tags[0].text.strip()\n",
    "    repo_name = a_tags[1].text.strip()\n",
    "    repo_url = base_url + a_tags[0]['href']\n",
    "    stars = parse_star_count(star_tag.text.strip())\n",
    "    return username, repo_name, stars, repo_url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "68f459fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('mrdoob', 'three.js', 92300, 'https://github.com/mrdoob')"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_repo_info(repo_tags[0], star_tags[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "97eaa055",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('pmndrs', 'react-three-fiber', 22700, 'https://github.com/pmndrs')"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_repo_info(repo_tags[1], star_tags[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75cef5e9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "d1393209",
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_repos_dict = {\n",
    "    'username': [], \n",
    "    'repo_name': [], \n",
    "    'stars': [],\n",
    "    'repo_url': []\n",
    "}\n",
    "\n",
    "\n",
    "for i in range(len(repo_tags)):\n",
    "    repo_info = get_repo_info(repo_tags[i], star_tags[i])\n",
    "    topic_repos_dict['username'].append(repo_info[0])\n",
    "    topic_repos_dict['repo_name'].append(repo_info[1])\n",
    "    topic_repos_dict['stars'].append(repo_info[2])\n",
    "    topic_repos_dict['repo_url'].append(repo_info[3])\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "c75ab8b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_repos_df = pd.DataFrame(topic_repos_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "99122e0b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>repo_name</th>\n",
       "      <th>stars</th>\n",
       "      <th>repo_url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>mrdoob</td>\n",
       "      <td>three.js</td>\n",
       "      <td>92300</td>\n",
       "      <td>https://github.com/mrdoob</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>pmndrs</td>\n",
       "      <td>react-three-fiber</td>\n",
       "      <td>22700</td>\n",
       "      <td>https://github.com/pmndrs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>libgdx</td>\n",
       "      <td>libgdx</td>\n",
       "      <td>21500</td>\n",
       "      <td>https://github.com/libgdx</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BabylonJS</td>\n",
       "      <td>Babylon.js</td>\n",
       "      <td>20700</td>\n",
       "      <td>https://github.com/BabylonJS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ssloy</td>\n",
       "      <td>tinyrenderer</td>\n",
       "      <td>17000</td>\n",
       "      <td>https://github.com/ssloy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>lettier</td>\n",
       "      <td>3d-game-shaders-for-beginners</td>\n",
       "      <td>15500</td>\n",
       "      <td>https://github.com/lettier</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>aframevr</td>\n",
       "      <td>aframe</td>\n",
       "      <td>15400</td>\n",
       "      <td>https://github.com/aframevr</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>FreeCAD</td>\n",
       "      <td>FreeCAD</td>\n",
       "      <td>14200</td>\n",
       "      <td>https://github.com/FreeCAD</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>CesiumGS</td>\n",
       "      <td>cesium</td>\n",
       "      <td>10500</td>\n",
       "      <td>https://github.com/CesiumGS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>metafizzy</td>\n",
       "      <td>zdog</td>\n",
       "      <td>9700</td>\n",
       "      <td>https://github.com/metafizzy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>isl-org</td>\n",
       "      <td>Open3D</td>\n",
       "      <td>8900</td>\n",
       "      <td>https://github.com/isl-org</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>timzhang642</td>\n",
       "      <td>3D-Machine-Learning</td>\n",
       "      <td>8900</td>\n",
       "      <td>https://github.com/timzhang642</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>blender</td>\n",
       "      <td>blender</td>\n",
       "      <td>8600</td>\n",
       "      <td>https://github.com/blender</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>a1studmuffin</td>\n",
       "      <td>SpaceshipGenerator</td>\n",
       "      <td>7400</td>\n",
       "      <td>https://github.com/a1studmuffin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>domlysz</td>\n",
       "      <td>BlenderGIS</td>\n",
       "      <td>6400</td>\n",
       "      <td>https://github.com/domlysz</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>FyroxEngine</td>\n",
       "      <td>Fyrox</td>\n",
       "      <td>6200</td>\n",
       "      <td>https://github.com/FyroxEngine</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>google</td>\n",
       "      <td>model-viewer</td>\n",
       "      <td>5600</td>\n",
       "      <td>https://github.com/google</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>openscad</td>\n",
       "      <td>openscad</td>\n",
       "      <td>5600</td>\n",
       "      <td>https://github.com/openscad</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>nerfstudio-project</td>\n",
       "      <td>nerfstudio</td>\n",
       "      <td>5200</td>\n",
       "      <td>https://github.com/nerfstudio-project</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>spritejs</td>\n",
       "      <td>spritejs</td>\n",
       "      <td>5200</td>\n",
       "      <td>https://github.com/spritejs</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              username                      repo_name  stars  \\\n",
       "0               mrdoob                       three.js  92300   \n",
       "1               pmndrs              react-three-fiber  22700   \n",
       "2               libgdx                         libgdx  21500   \n",
       "3            BabylonJS                     Babylon.js  20700   \n",
       "4                ssloy                   tinyrenderer  17000   \n",
       "5              lettier  3d-game-shaders-for-beginners  15500   \n",
       "6             aframevr                         aframe  15400   \n",
       "7              FreeCAD                        FreeCAD  14200   \n",
       "8             CesiumGS                         cesium  10500   \n",
       "9            metafizzy                           zdog   9700   \n",
       "10             isl-org                         Open3D   8900   \n",
       "11         timzhang642            3D-Machine-Learning   8900   \n",
       "12             blender                        blender   8600   \n",
       "13        a1studmuffin             SpaceshipGenerator   7400   \n",
       "14             domlysz                     BlenderGIS   6400   \n",
       "15         FyroxEngine                          Fyrox   6200   \n",
       "16              google                   model-viewer   5600   \n",
       "17            openscad                       openscad   5600   \n",
       "18  nerfstudio-project                     nerfstudio   5200   \n",
       "19            spritejs                       spritejs   5200   \n",
       "\n",
       "                                 repo_url  \n",
       "0               https://github.com/mrdoob  \n",
       "1               https://github.com/pmndrs  \n",
       "2               https://github.com/libgdx  \n",
       "3            https://github.com/BabylonJS  \n",
       "4                https://github.com/ssloy  \n",
       "5              https://github.com/lettier  \n",
       "6             https://github.com/aframevr  \n",
       "7              https://github.com/FreeCAD  \n",
       "8             https://github.com/CesiumGS  \n",
       "9            https://github.com/metafizzy  \n",
       "10             https://github.com/isl-org  \n",
       "11         https://github.com/timzhang642  \n",
       "12             https://github.com/blender  \n",
       "13        https://github.com/a1studmuffin  \n",
       "14             https://github.com/domlysz  \n",
       "15         https://github.com/FyroxEngine  \n",
       "16              https://github.com/google  \n",
       "17            https://github.com/openscad  \n",
       "18  https://github.com/nerfstudio-project  \n",
       "19            https://github.com/spritejs  "
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_repos_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "336f8119",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "range(0, 20)"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "range(len(repo_tags))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78d416e4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "171dcca5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3c4532ab",
   "metadata": {},
   "source": [
    "## Final Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "id": "093df7aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "def get_topic_page(topic_url):\n",
    "       # Download the page\n",
    "    response = requests.get(topic_url)\n",
    "\n",
    "    # Check the successfull response\n",
    "    if response.status_code != 200:\n",
    "        raise Exception('Failed to load page {}'.format(topic_url))\n",
    "    # Parse using BeautifulSoup\n",
    "    topic_doc = BeautifulSoup(response.text, 'html.parser')\n",
    "\n",
    "    return topic_doc\n",
    "    \n",
    "def get_repo_info(h3_tag, star_tag):\n",
    "    # returns all necessary info about a repository\n",
    "    a_tags = h3_tag.find_all('a')\n",
    "    username = a_tags[0].text.strip()\n",
    "    repo_name = a_tags[1].text.strip()\n",
    "    repo_url = base_url + a_tags[0]['href']\n",
    "    stars = parse_star_count(star_tag.text.strip())\n",
    "    return username, repo_name, stars, repo_url\n",
    "\n",
    "def get_topic_repos(topic_doc):\n",
    "    # Get the h3 tags containing username, repo title and repo URL\n",
    "    h3_selection_class = 'f3 color-fg-muted text-normal lh-condensed'\n",
    "    repo_tags = topic_doc.find_all('h3', {'class': h3_selection_class})\n",
    "    # Get star tags\n",
    "    star_tags = topic_doc.find_all('span', {'class': 'Counter js-social-count'})\n",
    "    \n",
    "    # Get repo info\n",
    "    topic_repos_dict = {\n",
    "        'username': [], \n",
    "         'repo_name': [], \n",
    "         'stars': [],\n",
    "         'repo_url': []\n",
    "     }\n",
    "    for i in range(len(repo_tags)):\n",
    "        repo_info = get_repo_info(repo_tags[i], star_tags[i])\n",
    "        topic_repos_dict['username'].append(repo_info[0])\n",
    "        topic_repos_dict['repo_name'].append(repo_info[1])\n",
    "        topic_repos_dict['stars'].append(repo_info[2])\n",
    "        topic_repos_dict['repo_url'].append(repo_info[3])\n",
    "\n",
    "    return pd.DataFrame(topic_repos_dict)\n",
    "\n",
    "def scrape_topic(topic_url, path):\n",
    "    if os.path.exists(path):\n",
    "        print(\"The file {} already exists. Skipping...\".format(fname))\n",
    "    topic_df = get_topic_repos(get_topic_page(topic_url))\n",
    "    topic_df.to_csv(path, index = None)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "id": "08ec05cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://github.com/topics/ajax'"
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url4 = topic_urls[1]\n",
    "\n",
    "url4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "id": "ade9552b",
   "metadata": {},
   "outputs": [],
   "source": [
    "topic4_doc = get_topic_page(url4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "id": "f4d88f08",
   "metadata": {},
   "outputs": [],
   "source": [
    "topic4_repos = get_topic_repos(topic4_doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "id": "976f85d8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>username</th>\n",
       "      <th>repo_name</th>\n",
       "      <th>stars</th>\n",
       "      <th>repo_url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ljianshu</td>\n",
       "      <td>Blog</td>\n",
       "      <td>7600</td>\n",
       "      <td>https://github.com/ljianshu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>metafizzy</td>\n",
       "      <td>infinite-scroll</td>\n",
       "      <td>7300</td>\n",
       "      <td>https://github.com/metafizzy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>developit</td>\n",
       "      <td>unfetch</td>\n",
       "      <td>5600</td>\n",
       "      <td>https://github.com/developit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>olifolkerd</td>\n",
       "      <td>tabulator</td>\n",
       "      <td>5500</td>\n",
       "      <td>https://github.com/olifolkerd</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>jquery-form</td>\n",
       "      <td>form</td>\n",
       "      <td>5200</td>\n",
       "      <td>https://github.com/jquery-form</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Studio-42</td>\n",
       "      <td>elFinder</td>\n",
       "      <td>4400</td>\n",
       "      <td>https://github.com/Studio-42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>elbywan</td>\n",
       "      <td>wretch</td>\n",
       "      <td>3900</td>\n",
       "      <td>https://github.com/elbywan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>dwyl</td>\n",
       "      <td>learn-to-send-email-via-google-script-html-no-...</td>\n",
       "      <td>3000</td>\n",
       "      <td>https://github.com/dwyl</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>ded</td>\n",
       "      <td>reqwest</td>\n",
       "      <td>2900</td>\n",
       "      <td>https://github.com/ded</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>LeaVerou</td>\n",
       "      <td>bliss</td>\n",
       "      <td>2400</td>\n",
       "      <td>https://github.com/LeaVerou</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>wendux</td>\n",
       "      <td>ajax-hook</td>\n",
       "      <td>2300</td>\n",
       "      <td>https://github.com/wendux</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>noelboss</td>\n",
       "      <td>featherlight</td>\n",
       "      <td>2100</td>\n",
       "      <td>https://github.com/noelboss</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>craftpip</td>\n",
       "      <td>jquery-confirm</td>\n",
       "      <td>1800</td>\n",
       "      <td>https://github.com/craftpip</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>ptaoussanis</td>\n",
       "      <td>sente</td>\n",
       "      <td>1700</td>\n",
       "      <td>https://github.com/ptaoussanis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>nette</td>\n",
       "      <td>tracy</td>\n",
       "      <td>1600</td>\n",
       "      <td>https://github.com/nette</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>loadingio</td>\n",
       "      <td>css-spinner</td>\n",
       "      <td>1600</td>\n",
       "      <td>https://github.com/loadingio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>joaomilho</td>\n",
       "      <td>Enterprise</td>\n",
       "      <td>1600</td>\n",
       "      <td>https://github.com/joaomilho</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>k8w</td>\n",
       "      <td>tsrpc</td>\n",
       "      <td>1500</td>\n",
       "      <td>https://github.com/k8w</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>codingforentrepreneurs</td>\n",
       "      <td>eCommerce</td>\n",
       "      <td>1400</td>\n",
       "      <td>https://github.com/codingforentrepreneurs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>u014427391</td>\n",
       "      <td>jeeplatform</td>\n",
       "      <td>1400</td>\n",
       "      <td>https://github.com/u014427391</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  username                                          repo_name  \\\n",
       "0                 ljianshu                                               Blog   \n",
       "1                metafizzy                                    infinite-scroll   \n",
       "2                developit                                            unfetch   \n",
       "3               olifolkerd                                          tabulator   \n",
       "4              jquery-form                                               form   \n",
       "5                Studio-42                                           elFinder   \n",
       "6                  elbywan                                             wretch   \n",
       "7                     dwyl  learn-to-send-email-via-google-script-html-no-...   \n",
       "8                      ded                                            reqwest   \n",
       "9                 LeaVerou                                              bliss   \n",
       "10                  wendux                                          ajax-hook   \n",
       "11                noelboss                                       featherlight   \n",
       "12                craftpip                                     jquery-confirm   \n",
       "13             ptaoussanis                                              sente   \n",
       "14                   nette                                              tracy   \n",
       "15               loadingio                                        css-spinner   \n",
       "16               joaomilho                                         Enterprise   \n",
       "17                     k8w                                              tsrpc   \n",
       "18  codingforentrepreneurs                                          eCommerce   \n",
       "19              u014427391                                        jeeplatform   \n",
       "\n",
       "    stars                                   repo_url  \n",
       "0    7600                https://github.com/ljianshu  \n",
       "1    7300               https://github.com/metafizzy  \n",
       "2    5600               https://github.com/developit  \n",
       "3    5500              https://github.com/olifolkerd  \n",
       "4    5200             https://github.com/jquery-form  \n",
       "5    4400               https://github.com/Studio-42  \n",
       "6    3900                 https://github.com/elbywan  \n",
       "7    3000                    https://github.com/dwyl  \n",
       "8    2900                     https://github.com/ded  \n",
       "9    2400                https://github.com/LeaVerou  \n",
       "10   2300                  https://github.com/wendux  \n",
       "11   2100                https://github.com/noelboss  \n",
       "12   1800                https://github.com/craftpip  \n",
       "13   1700             https://github.com/ptaoussanis  \n",
       "14   1600                   https://github.com/nette  \n",
       "15   1600               https://github.com/loadingio  \n",
       "16   1600               https://github.com/joaomilho  \n",
       "17   1500                     https://github.com/k8w  \n",
       "18   1400  https://github.com/codingforentrepreneurs  \n",
       "19   1400              https://github.com/u014427391  "
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic4_repos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "id": "9d51e9fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "### let's do all above 4 lines of code in a single line of code\n",
    "\n",
    "get_topic_repos(get_topic_page(topic_urls[1])).to_csv('ajax.csv', index = None)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27c60430",
   "metadata": {},
   "source": [
    "##  Now, Write a single function to:\n",
    "\n",
    "- Get the list of topics from the topics page\n",
    "- Get the list of top repos from the individual topic pages\n",
    "- For each topic, create a CSV of the top repos for the topic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "id": "3c0da7e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_topic_titles(doc):\n",
    "    topic_title_p_tags = doc.find_all('p', {'class': 'f3 lh-condensed mb-0 mt-1 Link--primary'})\n",
    "    topic_titles = []\n",
    "    for tag in topic_title_p_tags:\n",
    "        topic_titles.append(tag.text)\n",
    "    return topic_titles\n",
    "    \n",
    "def get_topic_descs(doc):\n",
    "    topic_desc_tags = doc.find_all('p', {'class': 'f5 color-fg-muted mb-0 mt-1'})\n",
    "    topic_descs = []\n",
    "    for tag in topic_desc_tags:\n",
    "        topic_descs.append(tag.text.strip())\n",
    "    return topic_descs\n",
    "    \n",
    "def get_topic_urls(doc):\n",
    "    topic_link_tags = doc.find_all('a', {'class': 'no-underline flex-grow-0'})\n",
    "    topic_urls = []\n",
    "    base_url = 'https://github.com'\n",
    "\n",
    "    for tag in topic_link_tags:\n",
    "        topic_urls.append(base_url + tag['href'])\n",
    "\n",
    "    return topic_urls\n",
    "    \n",
    "    \n",
    "def scrape_topics():\n",
    "    url = 'https://github.com/topics'\n",
    "    response = requests.get(topics_url)\n",
    "    if response.status_code != 200:\n",
    "        raise Exception('Failed to load page {}'.format(topic_url))\n",
    "        \n",
    "    doc = BeautifulSoup(response.text, 'html.parser')\n",
    "    topics_dict = {\n",
    "        'title': get_topic_titles(doc),\n",
    "        'description': get_topic_descs(doc),\n",
    "        'url': get_topic_urls(doc)\n",
    "    \n",
    "    }\n",
    "        \n",
    "    return pd.DataFrame(topics_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "id": "99ecabd1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function makedirs in module os:\n",
      "\n",
      "makedirs(name, mode=511, exist_ok=False)\n",
      "    makedirs(name [, mode=0o777][, exist_ok=False])\n",
      "    \n",
      "    Super-mkdir; create a leaf directory and all intermediate ones.  Works like\n",
      "    mkdir, except that any intermediate path segment (not just the rightmost)\n",
      "    will be created if it does not exist. If the target directory already\n",
      "    exists, raise an OSError if exist_ok is False. Otherwise no exception is\n",
      "    raised.  This is recursive.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "help(os.makedirs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "id": "68c94afe",
   "metadata": {},
   "outputs": [],
   "source": [
    "def scrape_topics_repos():\n",
    "    print('Scrapping list of topics')\n",
    "    topics_df = scrape_topics()\n",
    "    \n",
    "    os.makedirs('data', exist_ok=True)\n",
    "    \n",
    "    for index, row in topics_df.iterrows():\n",
    "        print('Scrapping top repositories for \"{}\"'.format(row['title']))\n",
    "        scrape_topic(row['url'], 'data/{}.csv'.format(row['title']))\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "id": "4783ea4a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scrapping list of topics\n",
      "Scrapping top repositories for \"3D\"\n",
      "Scrapping top repositories for \"Ajax\"\n",
      "Scrapping top repositories for \"Algorithm\"\n",
      "Scrapping top repositories for \"Amp\"\n",
      "Scrapping top repositories for \"Android\"\n",
      "Scrapping top repositories for \"Angular\"\n",
      "Scrapping top repositories for \"Ansible\"\n",
      "Scrapping top repositories for \"API\"\n",
      "Scrapping top repositories for \"Arduino\"\n",
      "Scrapping top repositories for \"ASP.NET\"\n",
      "Scrapping top repositories for \"Atom\"\n",
      "Scrapping top repositories for \"Awesome Lists\"\n",
      "Scrapping top repositories for \"Amazon Web Services\"\n",
      "Scrapping top repositories for \"Azure\"\n",
      "Scrapping top repositories for \"Babel\"\n",
      "Scrapping top repositories for \"Bash\"\n",
      "Scrapping top repositories for \"Bitcoin\"\n",
      "Scrapping top repositories for \"Bootstrap\"\n",
      "Scrapping top repositories for \"Bot\"\n",
      "Scrapping top repositories for \"C\"\n",
      "Scrapping top repositories for \"Chrome\"\n",
      "Scrapping top repositories for \"Chrome extension\"\n",
      "Scrapping top repositories for \"Command line interface\"\n",
      "Scrapping top repositories for \"Clojure\"\n",
      "Scrapping top repositories for \"Code quality\"\n",
      "Scrapping top repositories for \"Code review\"\n",
      "Scrapping top repositories for \"Compiler\"\n",
      "Scrapping top repositories for \"Continuous integration\"\n",
      "Scrapping top repositories for \"COVID-19\"\n",
      "Scrapping top repositories for \"C++\"\n"
     ]
    }
   ],
   "source": [
    "scrape_topics_repos()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "id": "dcdb265c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "942cd626",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  },
  "toc-autonumbering": true,
  "toc-showcode": false,
  "toc-showmarkdowntxt": true,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}