Web_scrapping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b081f728",
   "metadata": {},
   "source": [
    "# Web Scrapping with Beautiful Soup and Wptool API\n",
    "## Introduction\n",
    "This project is about extracting data from different websites. The aim is to list the best 100 universities in Nigeria. Python package, [BeautifulSoup](https://pypi.org/project/beautifulsoup4/), will be used to get the list from the [Webometrics Ranking table of 2020](https://www.theabusites.com/webometrics-ranking-2019/), found on the THEABUSITE website.\n",
    "\n",
    "The Webometrics Ranking of World Universities, also known as Ranking Web of Universities, is a ranking system for the world’s universities based on a composite indicator that takes into account both the volume of the Web content (number of web pages and files) and the visibility and impact of these web publications according to the number of external links (site citations) they received.\n",
    "\n",
    "Additional data about each university, such as a motto, year of establishment, chancellor, vice-chancellor, students, e.t.c, will be extracted from Wikipedia using the API, [wptools](https://pypi.org/project/wptools/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e6b08ab",
   "metadata": {},
   "source": [
    "## Importing Required Libraries/Package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "688db792",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests as r\n",
    "import pandas as pd\n",
    "from bs4 import BeautifulSoup\n",
    "import os\n",
    "import wptools as wp\n",
    "import time\n",
    "import json\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f321e17",
   "metadata": {},
   "source": [
    "## Accessing the website using request library and saving the content to a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0fb584aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# url of the website where the table of top 100 universities are.\n",
    "url = 'https://www.theabusites.com/webometrics-ranking-2019/'\n",
    "html = r.get(url)\n",
    "\n",
    "# Creatiing a folder on the file directory to save the content of the url\n",
    "folder = 'top_100_universities_in_ng'\n",
    "if not os.path.exists(folder):\n",
    "    os.makedirs(folder)\n",
    "with open(os.path.join(folder, \"./webometrics_ranking_2019.html\"), mode='wb') as file:\n",
    "    file.write(html.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3ae616d",
   "metadata": {},
   "source": [
    "## Accessing the saved content for extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6ea1ec67",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"top_100_universities_in_ng/webometrics_ranking_2019.html\", 'r') as file:\n",
    "    soup = BeautifulSoup(file, 'lxml')\n",
    "    table = soup.find('table')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "69e89286",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filter all row in the table\n",
    "rows = table.find_all('tr')\n",
    "sub = rows[1:] # remove first item in the list, it contains table headings.\n",
    "\n",
    "# Empty list to save extracted information\n",
    "top = []\n",
    "for i in range(len(sub)):\n",
    "    rank = sub[i].find_all('td')[0].text\n",
    "    world_rank = sub[i].find_all('td')[1].text\n",
    "    universities = sub[i].find_all('td')[2].text\n",
    "    url = sub[i].find_all('a')[0]['href']\n",
    "    presence_rank = sub[i].find_all('td')[4].text\n",
    "    impact_rank = sub[i].find_all('td')[5].text\n",
    "    openness_rank = sub[i].find_all('td')[6].text\n",
    "    excellence_rank = sub[i].find_all('td')[7].text\n",
    "    \n",
    "    top.append({\n",
    "        'ranking':int(rank),\n",
    "        'world_rank':int(world_rank),\n",
    "        'universities':universities,\n",
    "        'website':url,\n",
    "        'presence_rank':presence_rank,\n",
    "        'impact_rank':impact_rank,\n",
    "        'openness_rank':openness_rank,\n",
    "        'excellence_rank':excellence_rank\n",
    "    })\n",
    "\n",
    "# Creating column names as seen on web\n",
    "col=['ranking', 'world_rank', 'universities', 'website', 'presence_rank',\n",
    "    'impact_rank', 'openness_rank', 'excellence_rank']\n",
    "\n",
    "# Converting extracted information to pandas dataframe\n",
    "df_1 = pd.DataFrame(top, columns=col)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6e32f7f",
   "metadata": {},
   "source": [
    "## First extraction result in a data frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fa5ca8dd",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ranking</th>\n",
       "      <th>world_rank</th>\n",
       "      <th>universities</th>\n",
       "      <th>website</th>\n",
       "      <th>presence_rank</th>\n",
       "      <th>impact_rank</th>\n",
       "      <th>openness_rank</th>\n",
       "      <th>excellence_rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1322</td>\n",
       "      <td>University of Ibadan</td>\n",
       "      <td>https://www.ui.edu.ng/</td>\n",
       "      <td>2113</td>\n",
       "      <td>2088</td>\n",
       "      <td>1057</td>\n",
       "      <td>1561</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1742</td>\n",
       "      <td>Covenant University Ota</td>\n",
       "      <td>http://covenantuniversity.edu.ng/</td>\n",
       "      <td>1169</td>\n",
       "      <td>3884</td>\n",
       "      <td>1356</td>\n",
       "      <td>1797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1805</td>\n",
       "      <td>University of Nigeria</td>\n",
       "      <td>http://www.unn.edu.ng/</td>\n",
       "      <td>1311</td>\n",
       "      <td>3279</td>\n",
       "      <td>1038</td>\n",
       "      <td>2243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1984</td>\n",
       "      <td>University of Lagos</td>\n",
       "      <td>https://unilag.edu.ng/</td>\n",
       "      <td>161</td>\n",
       "      <td>4143</td>\n",
       "      <td>1521</td>\n",
       "      <td>2312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>2053</td>\n",
       "      <td>Obafemi Awolowo University</td>\n",
       "      <td>http://oauife.edu.ng/</td>\n",
       "      <td>2916</td>\n",
       "      <td>4560</td>\n",
       "      <td>1616</td>\n",
       "      <td>2025</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>96</td>\n",
       "      <td>14054</td>\n",
       "      <td>Yobe State University (Bukar Abba Ibrahim Univ...</td>\n",
       "      <td>https://www.ysu.edu.ng/</td>\n",
       "      <td>17040</td>\n",
       "      <td>15459</td>\n",
       "      <td>6490</td>\n",
       "      <td>6084</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>97</td>\n",
       "      <td>14091</td>\n",
       "      <td>Taraba State University Jalingo</td>\n",
       "      <td>http://www.tsuniversity.edu.ng/</td>\n",
       "      <td>18418</td>\n",
       "      <td>14883</td>\n",
       "      <td>6690</td>\n",
       "      <td>6084</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>98</td>\n",
       "      <td>14110</td>\n",
       "      <td>Ondo State University of Science &amp; Technology ...</td>\n",
       "      <td>http://www.osustech.edu.ng/</td>\n",
       "      <td>23528</td>\n",
       "      <td>12868</td>\n",
       "      <td>7168</td>\n",
       "      <td>6084</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>99</td>\n",
       "      <td>14347</td>\n",
       "      <td>Anchor University Lagos</td>\n",
       "      <td>https://aul.edu.ng/</td>\n",
       "      <td>9304</td>\n",
       "      <td>17841</td>\n",
       "      <td>5779</td>\n",
       "      <td>6084</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>100</td>\n",
       "      <td>14446</td>\n",
       "      <td>Federal University Birnin Kebbi</td>\n",
       "      <td>http://www.fubk.edu.ng/</td>\n",
       "      <td>26140</td>\n",
       "      <td>15000</td>\n",
       "      <td>6456</td>\n",
       "      <td>6084</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    ranking  world_rank                                       universities  \\\n",
       "0         1        1322                               University of Ibadan   \n",
       "1         2        1742                            Covenant University Ota   \n",
       "2         3        1805                              University of Nigeria   \n",
       "3         4        1984                                University of Lagos   \n",
       "4         5        2053                         Obafemi Awolowo University   \n",
       "..      ...         ...                                                ...   \n",
       "95       96       14054  Yobe State University (Bukar Abba Ibrahim Univ...   \n",
       "96       97       14091                    Taraba State University Jalingo   \n",
       "97       98       14110  Ondo State University of Science & Technology ...   \n",
       "98       99       14347                            Anchor University Lagos   \n",
       "99      100       14446                    Federal University Birnin Kebbi   \n",
       "\n",
       "                              website presence_rank impact_rank openness_rank  \\\n",
       "0              https://www.ui.edu.ng/          2113        2088          1057   \n",
       "1   http://covenantuniversity.edu.ng/          1169        3884          1356   \n",
       "2              http://www.unn.edu.ng/          1311        3279          1038   \n",
       "3              https://unilag.edu.ng/           161        4143          1521   \n",
       "4               http://oauife.edu.ng/          2916        4560          1616   \n",
       "..                                ...           ...         ...           ...   \n",
       "95            https://www.ysu.edu.ng/         17040       15459          6490   \n",
       "96    http://www.tsuniversity.edu.ng/         18418       14883          6690   \n",
       "97        http://www.osustech.edu.ng/         23528       12868          7168   \n",
       "98                https://aul.edu.ng/          9304       17841          5779   \n",
       "99            http://www.fubk.edu.ng/         26140       15000          6456   \n",
       "\n",
       "   excellence_rank  \n",
       "0             1561  \n",
       "1             1797  \n",
       "2             2243  \n",
       "3             2312  \n",
       "4             2025  \n",
       "..             ...  \n",
       "95            6084  \n",
       "96            6084  \n",
       "97            6084  \n",
       "98            6084  \n",
       "99            6084  \n",
       "\n",
       "[100 rows x 8 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Preview of extracted information\n",
    "df_1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ed16ceb",
   "metadata": {},
   "source": [
    "## List of the school wikipedia urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "4bd45875",
   "metadata": {},
   "outputs": [],
   "source": [
    "# List of wikipedia urls for each school\n",
    "top_100_urls = [\n",
    "    'https://en.wikipedia.org/wiki/University_of_Ibadan',\n",
    "    'https://en.wikipedia.org/wiki/Covenant_University',\n",
    "    'https://en.wikipedia.org/wiki/University_of_Nigeria',\n",
    "    'https://en.wikipedia.org/wiki/University_of_Lagos',\n",
    "    'https://en.wikipedia.org/wiki/Obafemi_Awolowo_University',\n",
    "    'https://en.wikipedia.org/wiki/Ahmadu_Bello_University',\n",
    "    'https://en.wikipedia.org/wiki/University_of_Ilorin',\n",
    "    'https://en.wikipedia.org/wiki/Federal_University_of_Technology_Akure',\n",
    "    'https://en.wikipedia.org/wiki/University_of_Port_Harcourt',\n",
    "    'https://en.wikipedia.org/wiki/Adekunle_Ajasin_University',\n",
    "    'https://en.wikipedia.org/wiki/University_of_Benin_(Nigeria)',\n",
    "    'https://en.wikipedia.org/wiki/Federal_University_of_Technology,_Minna',\n",
    "    'https://en.wikipedia.org/wiki/Ladoke_Akintola_University_of_Technology',\n",
    "    'https://en.wikipedia.org/wiki/Rivers_State_University',\n",
    "    'https://en.wikipedia.org/wiki/University_of_Calabar',\n",
    "    'https://en.wikipedia.org/wiki/Bayero_University_Kano',\n",
    "    'https://en.wikipedia.org/wiki/Lagos_State_University',\n",
    "    'https://en.wikipedia.org/wiki/University_of_Jos',\n",
    "    'https://en.wikipedia.org/wiki/Federal_University_of_Technology_Owerri',\n",
    "    'https://en.wikipedia.org/wiki/University_of_Uyo',\n",
    "    'https://en.wikipedia.org/wiki/Nnamdi_Azikiwe_University',\n",
    "    'https://en.wikipedia.org/wiki/Olabisi_Onabanjo_University',\n",
    "    'https://en.wikipedia.org/wiki/Federal_University_of_Agriculture,_Abeokuta',\n",
    "    'https://en.wikipedia.org/wiki/University_of_Abuja',\n",
    "    'https://en.wikipedia.org/wiki/University_of_Maiduguri',\n",
    "    'https://en.wikipedia.org/wiki/Usmanu_Danfodiyo_University',\n",
    "    'https://en.wikipedia.org/wiki/Abubakar_Tafawa_Balewa_University',\n",
    "    'https://en.wikipedia.org/wiki/Ebonyi_State_University',\n",
    "    'https://en.wikipedia.org/wiki/Federal_University_of_Petroleum_Resources_Effurun',\n",
    "    'https://en.wikipedia.org/wiki/Benue_State_University',\n",
    "    'https://en.wikipedia.org/wiki/American_University_of_Nigeria',\n",
    "    'https://en.wikipedia.org/wiki/Federal_University_Oye_Ekiti',\n",
    "    'https://en.wikipedia.org/wiki/University_of_Agriculture,_Makurdihttps://en.wikipedia.org/wiki/Niger_Delta_University',\n",
    "    'https://en.wikipedia.org/wiki/African_University_of_Science_and_Technology',\n",
    "    'https://en.wikipedia.org/wiki/Skyline_University',\n",
    "    'https://en.wikipedia.org/wiki/Landmark_University',\n",
    "    'https://en.wikipedia.org/wiki/Delta_State_University,_Abraka',\n",
    "    'https://en.wikipedia.org/wiki/Ekiti_State_University',\n",
    "    'https://en.wikipedia.org/wiki/Babcock_University',\n",
    "    'https://en.wikipedia.org/wiki/Michael_Okpara_University_of_Agriculture',\n",
    "    'https://en.wikipedia.org/wiki/Alex_Ekwueme_Federal_University_Ndufu_Alike_Ikwo',\n",
    "    'https://en.wikipedia.org/wiki/Osun_State_University',\n",
    "    'https://en.wikipedia.org/wiki/Cross_River_University_of_Technology',\n",
    "    'https://en.wikipedia.org/wiki/Redeemer\\'s_University_Nigeria',\n",
    "    'https://en.wikipedia.org/wiki/Kwara_State_University',\n",
    "    'https://en.wikipedia.org/wiki/Michael_Okpara_University_of_Agriculture',\n",
    "    'https://en.wikipedia.org/wiki/Abia_State_University_Uturu',\n",
    "    'https://en.wikipedia.org/wiki/Federal_University,_Dutsin-Ma',\n",
    "    'https://en.wikipedia.org/wiki/Edo_University,_Iyamho',\n",
    "    'https://en.wikipedia.org/wiki/Umaru_Musa_Yar\\'adua_University',\n",
    "    'https://en.wikipedia.org/wiki/Nigerian_Defence_Academy',\n",
    "    'https://en.wikipedia.org/wiki/Imo_State_University',\n",
    "    'https://en.wikipedia.org/wiki/Enugu_State_University_of_Science_and_Technology',\n",
    "    '',\n",
    "    'https://en.wikipedia.org/wiki/Joseph_Ayo_Babalola_University',\n",
    "    'https://en.wikipedia.org/wiki/Federal_University_Dutse',\n",
    "    'https://en.wikipedia.org/wiki/Akwa_Ibom_State_University',\n",
    "    'https://en.wikipedia.org/wiki/Kaduna_State_University',\n",
    "    'https://en.wikipedia.org/wiki/Federal_University,_Otuoke',\n",
    "    'https://en.wikipedia.org/wiki/Lagos_Business_School',\n",
    "    'https://en.wikipedia.org/wiki/Modibbo_Adama_Federal_University_of_Technology,_Yola',\n",
    "    'https://en.wikipedia.org/wiki/Godfrey_Okoye_University',\n",
    "    'https://en.wikipedia.org/wiki/Tai_Solarin_University_of_Education',\n",
    "    'https://en.wikipedia.org/wiki/Chukwuemeka_Odumegwu_Ojukwu_University',\n",
    "    '',\n",
    "    'https://en.wikipedia.org/wiki/Igbinedion_University',\n",
    "    'https://en.wikipedia.org/wiki/Auchi_Polytechnic',\n",
    "    'https://en.wikipedia.org/wiki/Federal_University,_Lokoja',\n",
    "    'https://en.wikipedia.org/wiki/Ibrahim_Badamasi_Babangida_University',\n",
    "    'https://en.wikipedia.org/wiki/Ambrose_Alli_University',\n",
    "    'https://en.wikipedia.org/wiki/Elizade_University',\n",
    "    'https://en.wikipedia.org/wiki/Kogi_State_University',\n",
    "    'https://en.wikipedia.org/wiki/National_Open_University_of_Nigeria',\n",
    "    'https://en.wikipedia.org/wiki/Yaba_College_of_Technology',\n",
    "    'https://en.wikipedia.org/wiki/Baze_University',\n",
    "    'https://en.wikipedia.org/wiki/Nile_University_of_Nigeria',\n",
    "    'https://en.wikipedia.org/wiki/University_of_Medical_Sciences,_Ondo',\n",
    "    'https://en.wikipedia.org/wiki/Nasarawa_State_University',\n",
    "    'https://en.wikipedia.org/wiki/Federal_Polytechnic,_Ilaro',\n",
    "    'https://en.wikipedia.org/wiki/Pan-Atlantic_University',\n",
    "    'https://en.wikipedia.org/wiki/Ajayi_Crowther_University',\n",
    "    'https://en.wikipedia.org/wiki/Adeleke_University',\n",
    "    'https://en.wikipedia.org/wiki/Federal_University,_Wukari',\n",
    "    'https://en.wikipedia.org/wiki/Lead_City_University',\n",
    "    'https://en.wikipedia.org/wiki/Federal_University_of_Lafia',\n",
    "    'https://en.wikipedia.org/wiki/Benson_Idahosa_University',\n",
    "    'https://en.wikipedia.org/wiki/Al-Hikmah_University',\n",
    "    'https://en.wikipedia.org/wiki/Bauchi_State_University',\n",
    "    'https://en.wikipedia.org/wiki/Kebbi_State_University_of_Science_and_Technology',\n",
    "    'https://en.wikipedia.org/wiki/Bells_University_of_Technology',\n",
    "    'https://en.wikipedia.org/wiki/Kano_State_University_of_Technology',\n",
    "    'https://en.wikipedia.org/wiki/Bingham_University',\n",
    "    'https://en.wikipedia.org/wiki/Lagos_State_University_of_Science_and_Technology',\n",
    "    'https://en.wikipedia.org/wiki/Yobe_State_University',\n",
    "    'https://en.wikipedia.org/wiki/Taraba_State_University',\n",
    "    'https://en.wikipedia.org/wiki/Ondo_State_University_of_Science_and_Technology',\n",
    "    'https://en.wikipedia.org/wiki/Anchor_University',\n",
    "    'https://en.wikipedia.org/wiki/Federal_University,_Birnin_Kebbi'\n",
    "]\n",
    "checkpoint = ['https://en.wikipedia.org/wiki/Ebonyi_State_University',\n",
    "             'https://en.wikipedia.org/wiki/Kaduna_State_University',\n",
    "             'https://en.wikipedia.org/wiki/Bauchi_State_University',]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aeac6087",
   "metadata": {},
   "source": [
    "## Accessing the wikipedia urls and extracting their infobox data, then save it to a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e0f18396",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The code is still running, exercise patience\n",
      "The code is still running, exercise patience\n",
      "The code is still running, exercise patience\n",
      "Total time taken is 0.0hr 10.0minutes 602.9921143054962 seconds\n"
     ]
    }
   ],
   "source": [
    "\"\"\"A timed function to iterate through the above urls and extract the infobox data from each website. \n",
    "This data is saved to a text file.\n",
    "\n",
    "This code will take about 10 minutes to run.\n",
    "\"\"\"\n",
    "start = time.time()\n",
    "with open('top_100_universities_in_ng/wikipedia_info.txt', 'w') as opened_file:\n",
    "    for url in top_100_urls:\n",
    "        page = wp.page(url.split('/')[-1], silent=True)\n",
    "        pg = page.get()\n",
    "        all_info = pg.data['infobox']\n",
    "        \n",
    "#       Dumping the JSON file to the text file      \n",
    "        json.dump(all_info, opened_file)\n",
    "        opened_file.write('\\n')\n",
    "#       Checkpoint to let the user know the code is active and running\n",
    "        if url in checkpoint:\n",
    "            print(\"The code is still running, please exercise patience\")\n",
    "\n",
    "total_time = \"%s\"%(time.time() - start)\n",
    "minit, sec = divmod(float(total_time), 60)\n",
    "hr, minit = divmod(minit, 60)\n",
    "print(f'Total time taken is {hr} hr {minit} minutes {total_time} seconds')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d1c6d32",
   "metadata": {},
   "source": [
    "## Accessing the saved infobox data for extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4a97539a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "School of rank 42 has no infobox on wikipedia\n",
      "School of rank 57 has no infobox on wikipedia\n",
      "School of rank 61 has no infobox on wikipedia\n",
      "School of rank 63 has no infobox on wikipedia\n",
      "School of rank 87 has no infobox on wikipedia\n",
      "School of rank 89 has no infobox on wikipedia\n",
      "School of rank 96 has no infobox on wikipedia\n"
     ]
    }
   ],
   "source": [
    "\"\"\" The function below iterate through each infobox data found on the wikipedia.txt file.\n",
    "It extracts the required information, if the information does not exist in a particular \n",
    "infobox data, it replace it with NaN.\n",
    "\n",
    "If the page has no infobox, it prints out the school by its rank\n",
    "\"\"\"\n",
    "# Empty list to save the extracted informtion\n",
    "rank = 0\n",
    "info = []\n",
    "with open('top_100_universities_in_ng/wikipedia_info.txt', 'r') as f:\n",
    "    for line in f:\n",
    "        try:\n",
    "            rank += 1\n",
    "            each_uni = json.loads(line)\n",
    "            name = each_uni.setdefault('name', np.NaN)\n",
    "            motto = each_uni.setdefault('motto', np.NaN)\n",
    "            estab = each_uni.setdefault('established', np.NaN)\n",
    "            typ1 = each_uni.setdefault('type', np.NaN)\n",
    "            typ = each_uni.setdefault('type', np.NaN)\n",
    "            chanc = each_uni.setdefault('chancellor', np.NaN)\n",
    "            vice_chanc = each_uni.setdefault('vice_chancellor', np.NaN)\n",
    "            stu = each_uni.setdefault('students', np.NaN)\n",
    "            undrgrd = each_uni.setdefault('undergrad', np.NaN)\n",
    "            pstgrd = each_uni.setdefault('postgrad', np.NaN)\n",
    "            acad_staff = each_uni.setdefault('academic_staff', np.NaN)\n",
    "            adminstratv_staff = each_uni.setdefault('administrative_staff', np.NaN)\n",
    "            city = each_uni.setdefault('city', np.NaN)\n",
    "            state = each_uni.setdefault('state', np.NaN)\n",
    "            camp = each_uni.setdefault('campus', np.NaN)\n",
    "\n",
    "            info.append({\n",
    "                'ranking': rank,\n",
    "                'name': name,\n",
    "                'motto': motto,\n",
    "                'established': estab,\n",
    "                'type': typ,\n",
    "                'chancellor': chanc,\n",
    "                'vice_chancellor': vice_chanc,\n",
    "                'students': stu,\n",
    "                'undergraduates': undrgrd,\n",
    "                'postgraduates': pstgrd,\n",
    "                'academic_staff': acad_staff,\n",
    "                'administrative_staff': adminstratv_staff,\n",
    "                'city': city,\n",
    "                'state': state,\n",
    "                'campus': camp\n",
    "            })\n",
    "        except AttributeError as e:\n",
    "            print(f'School of rank {rank} has no infobox on wikipedia')\n",
    "\n",
    "# Creating columns name based on extracted information\n",
    "cols = ['ranking', 'name', 'motto', 'established', 'type', 'chancellor',\n",
    "       'vice_chancellor', 'students', 'undergraduates', 'postgraduates',\n",
    "       'academic_staff', 'administrative_staff', 'city', 'state', 'campus']\n",
    "\n",
    "# Converting extracted information to pandas dataframe\n",
    "df_2 = pd.DataFrame(info, columns = cols)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68a797ed",
   "metadata": {},
   "source": [
    "## Second extraction result in a data frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6d8c7cbf",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ranking</th>\n",
       "      <th>name</th>\n",
       "      <th>motto</th>\n",
       "      <th>established</th>\n",
       "      <th>type</th>\n",
       "      <th>chancellor</th>\n",
       "      <th>vice_chancellor</th>\n",
       "      <th>students</th>\n",
       "      <th>undergraduates</th>\n",
       "      <th>postgraduates</th>\n",
       "      <th>academic_staff</th>\n",
       "      <th>administrative_staff</th>\n",
       "      <th>city</th>\n",
       "      <th>state</th>\n",
       "      <th>campus</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>University of Ibadan</td>\n",
       "      <td>\"''Recte Sapere Fons''\" (To think straight is ...</td>\n",
       "      <td>{{start date and age|1948}}</td>\n",
       "      <td>[[public university|Public]]</td>\n",
       "      <td>[[Sa'adu Abubakar|Saad Abubakar]], [[Sultan of...</td>\n",
       "      <td>Professor [[Kayode Adebowale ]]</td>\n",
       "      <td>41,743</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Ibadan]]</td>\n",
       "      <td>[[Oyo State|Oyo]]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Covenant University</td>\n",
       "      <td>''Raising a New Generation of Leaders''</td>\n",
       "      <td>21 October 2002</td>\n",
       "      <td>Private</td>\n",
       "      <td>[[David Oyedepo]]</td>\n",
       "      <td>[[Abiodun H. Adebayo]]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Ota, Ogun State]]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Urban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>University of Nigeria</td>\n",
       "      <td>''To Restore the Dignity of Man''</td>\n",
       "      <td>1955</td>\n",
       "      <td>[[public university|Public]]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Charles Igwe Arizechukwu]]</td>\n",
       "      <td>36,000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Nsukka]]</td>\n",
       "      <td>[[Enugu state|Enugu]]</td>\n",
       "      <td>Rural&lt;br /&gt; {{convert|871|ha|acre}} (Nsukka ca...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>University of Lagos</td>\n",
       "      <td>In deed and in truth</td>\n",
       "      <td>1962</td>\n",
       "      <td>[[Public university|Public]] [[research univer...</td>\n",
       "      <td>Alhaji (Dr.) Abubakar IBN Umar Garbai El-Kanem...</td>\n",
       "      <td>[[Oluwatoyin Ogundipe|Prof. Oluwatoyin Ogundipe]]</td>\n",
       "      <td>55,000 (2017)</td>\n",
       "      <td>43,784 (2017)</td>\n",
       "      <td>9,070 (2017)</td>\n",
       "      <td>1,736 (2017)</td>\n",
       "      <td>552 (2017)</td>\n",
       "      <td>[[Lagos]]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Urban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Obafemi Awolowo University</td>\n",
       "      <td>For Learning and Culture</td>\n",
       "      <td>1961</td>\n",
       "      <td>[[public university|Public]]</td>\n",
       "      <td>Etsu [[Yahaya Abubakar]]</td>\n",
       "      <td>[[Adebayo Simeon Bamire]]</td>\n",
       "      <td>about 35,000</td>\n",
       "      <td>13,000</td>\n",
       "      <td>7500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Ile-Ife]]</td>\n",
       "      <td>[[Osun State|Osun]]</td>\n",
       "      <td>Urban {{convert|2020|ha|acre}}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>93</td>\n",
       "      <td>Lagos State University of Science and Technology</td>\n",
       "      <td>Mission and Professionalism</td>\n",
       "      <td>{{start date|1977}}</td>\n",
       "      <td>[[Public university|Public]]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Oluremi Nurudeen Olaleye</td>\n",
       "      <td>50,000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Above 808</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Ikorodu, Lagos State]]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Ikorodu]], [[Isolo]], [[Surulere]]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>94</td>\n",
       "      <td>Yobe State University</td>\n",
       "      <td>Knowledge Is Light</td>\n",
       "      <td>2006</td>\n",
       "      <td>Public</td>\n",
       "      <td>Ahmed Tijjani Ibn Saleh (Emir of Ngazargamu)</td>\n",
       "      <td>Professor Mala Mohammed Daura</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10,000</td>\n",
       "      <td>2400</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1300</td>\n",
       "      <td>[[Damaturu]]</td>\n",
       "      <td>[[Yobe State]]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>95</td>\n",
       "      <td>Taraba State University</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2008</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Jalingo]]</td>\n",
       "      <td>[[Taraba State]]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>97</td>\n",
       "      <td>Anchor University</td>\n",
       "      <td>Character, Competence, Courage</td>\n",
       "      <td>September 2014</td>\n",
       "      <td>Private</td>\n",
       "      <td>[[William Kumuyi]]</td>\n",
       "      <td>[[Joseph Afolayan]]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Ayobo, Lagos|Ayobo-Ipaja]]</td>\n",
       "      <td>[[Lagos State]]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>98</td>\n",
       "      <td>Federal University, Birnin Kebbi</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{{start date and age|2013}}</td>\n",
       "      <td>Public</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Professor [[Auwal H Yadudu]]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Birnin Kebbi]], [[Kebbi State]]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Urban</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>91 rows × 15 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    ranking                                              name  \\\n",
       "0         1                              University of Ibadan   \n",
       "1         2                               Covenant University   \n",
       "2         3                             University of Nigeria   \n",
       "3         4                               University of Lagos   \n",
       "4         5                        Obafemi Awolowo University   \n",
       "..      ...                                               ...   \n",
       "86       93  Lagos State University of Science and Technology   \n",
       "87       94                             Yobe State University   \n",
       "88       95                           Taraba State University   \n",
       "89       97                                 Anchor University   \n",
       "90       98                  Federal University, Birnin Kebbi   \n",
       "\n",
       "                                                motto  \\\n",
       "0   \"''Recte Sapere Fons''\" (To think straight is ...   \n",
       "1             ''Raising a New Generation of Leaders''   \n",
       "2                   ''To Restore the Dignity of Man''   \n",
       "3                                In deed and in truth   \n",
       "4                            For Learning and Culture   \n",
       "..                                                ...   \n",
       "86                        Mission and Professionalism   \n",
       "87                                 Knowledge Is Light   \n",
       "88                                                NaN   \n",
       "89                     Character, Competence, Courage   \n",
       "90                                                NaN   \n",
       "\n",
       "                    established  \\\n",
       "0   {{start date and age|1948}}   \n",
       "1               21 October 2002   \n",
       "2                          1955   \n",
       "3                          1962   \n",
       "4                          1961   \n",
       "..                          ...   \n",
       "86          {{start date|1977}}   \n",
       "87                         2006   \n",
       "88                         2008   \n",
       "89               September 2014   \n",
       "90  {{start date and age|2013}}   \n",
       "\n",
       "                                                 type  \\\n",
       "0                        [[public university|Public]]   \n",
       "1                                             Private   \n",
       "2                        [[public university|Public]]   \n",
       "3   [[Public university|Public]] [[research univer...   \n",
       "4                        [[public university|Public]]   \n",
       "..                                                ...   \n",
       "86                       [[Public university|Public]]   \n",
       "87                                             Public   \n",
       "88                                                NaN   \n",
       "89                                            Private   \n",
       "90                                             Public   \n",
       "\n",
       "                                           chancellor  \\\n",
       "0   [[Sa'adu Abubakar|Saad Abubakar]], [[Sultan of...   \n",
       "1                                   [[David Oyedepo]]   \n",
       "2                                                 NaN   \n",
       "3   Alhaji (Dr.) Abubakar IBN Umar Garbai El-Kanem...   \n",
       "4                            Etsu [[Yahaya Abubakar]]   \n",
       "..                                                ...   \n",
       "86                                                NaN   \n",
       "87       Ahmed Tijjani Ibn Saleh (Emir of Ngazargamu)   \n",
       "88                                                NaN   \n",
       "89                                 [[William Kumuyi]]   \n",
       "90                                                NaN   \n",
       "\n",
       "                                      vice_chancellor       students  \\\n",
       "0                     Professor [[Kayode Adebowale ]]         41,743   \n",
       "1                              [[Abiodun H. Adebayo]]            NaN   \n",
       "2                        [[Charles Igwe Arizechukwu]]         36,000   \n",
       "3   [[Oluwatoyin Ogundipe|Prof. Oluwatoyin Ogundipe]]  55,000 (2017)   \n",
       "4                           [[Adebayo Simeon Bamire]]   about 35,000   \n",
       "..                                                ...            ...   \n",
       "86                           Oluremi Nurudeen Olaleye         50,000   \n",
       "87                      Professor Mala Mohammed Daura            NaN   \n",
       "88                                                NaN            NaN   \n",
       "89                                [[Joseph Afolayan]]            NaN   \n",
       "90                       Professor [[Auwal H Yadudu]]            NaN   \n",
       "\n",
       "   undergraduates postgraduates academic_staff administrative_staff  \\\n",
       "0             NaN           NaN            NaN                  NaN   \n",
       "1             NaN           NaN            NaN                  NaN   \n",
       "2             NaN           NaN            NaN                  NaN   \n",
       "3   43,784 (2017)  9,070 (2017)   1,736 (2017)           552 (2017)   \n",
       "4          13,000          7500            NaN                  NaN   \n",
       "..            ...           ...            ...                  ...   \n",
       "86            NaN           NaN      Above 808                  NaN   \n",
       "87         10,000          2400            NaN                 1300   \n",
       "88            NaN           NaN            NaN                  NaN   \n",
       "89            NaN           NaN            NaN                  NaN   \n",
       "90            NaN           NaN            NaN                  NaN   \n",
       "\n",
       "                                 city                  state  \\\n",
       "0                          [[Ibadan]]      [[Oyo State|Oyo]]   \n",
       "1                 [[Ota, Ogun State]]                    NaN   \n",
       "2                          [[Nsukka]]  [[Enugu state|Enugu]]   \n",
       "3                           [[Lagos]]                    NaN   \n",
       "4                         [[Ile-Ife]]    [[Osun State|Osun]]   \n",
       "..                                ...                    ...   \n",
       "86           [[Ikorodu, Lagos State]]                    NaN   \n",
       "87                       [[Damaturu]]         [[Yobe State]]   \n",
       "88                        [[Jalingo]]       [[Taraba State]]   \n",
       "89       [[Ayobo, Lagos|Ayobo-Ipaja]]        [[Lagos State]]   \n",
       "90  [[Birnin Kebbi]], [[Kebbi State]]                    NaN   \n",
       "\n",
       "                                               campus  \n",
       "0                                                 NaN  \n",
       "1                                               Urban  \n",
       "2   Rural<br /> {{convert|871|ha|acre}} (Nsukka ca...  \n",
       "3                                               Urban  \n",
       "4                      Urban {{convert|2020|ha|acre}}  \n",
       "..                                                ...  \n",
       "86               [[Ikorodu]], [[Isolo]], [[Surulere]]  \n",
       "87                                                NaN  \n",
       "88                                                NaN  \n",
       "89                                                NaN  \n",
       "90                                              Urban  \n",
       "\n",
       "[91 rows x 15 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Prviewing extracted informatiion from wikipedia\n",
    "df_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d378576e",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ranking</th>\n",
       "      <th>world_rank</th>\n",
       "      <th>universities</th>\n",
       "      <th>website</th>\n",
       "      <th>presence_rank</th>\n",
       "      <th>impact_rank</th>\n",
       "      <th>openness_rank</th>\n",
       "      <th>excellence_rank</th>\n",
       "      <th>motto</th>\n",
       "      <th>established</th>\n",
       "      <th>...</th>\n",
       "      <th>chancellor</th>\n",
       "      <th>vice_chancellor</th>\n",
       "      <th>students</th>\n",
       "      <th>undergraduates</th>\n",
       "      <th>postgraduates</th>\n",
       "      <th>academic_staff</th>\n",
       "      <th>administrative_staff</th>\n",
       "      <th>city</th>\n",
       "      <th>state</th>\n",
       "      <th>campus</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1322</td>\n",
       "      <td>University of Ibadan</td>\n",
       "      <td>https://www.ui.edu.ng/</td>\n",
       "      <td>2113</td>\n",
       "      <td>2088</td>\n",
       "      <td>1057</td>\n",
       "      <td>1561</td>\n",
       "      <td>\"''Recte Sapere Fons''\" (To think straight is ...</td>\n",
       "      <td>{{start date and age|1948}}</td>\n",
       "      <td>...</td>\n",
       "      <td>[[Sa'adu Abubakar|Saad Abubakar]], [[Sultan of...</td>\n",
       "      <td>Professor [[Kayode Adebowale ]]</td>\n",
       "      <td>41,743</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Ibadan]]</td>\n",
       "      <td>[[Oyo State|Oyo]]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1742</td>\n",
       "      <td>Covenant University Ota</td>\n",
       "      <td>http://covenantuniversity.edu.ng/</td>\n",
       "      <td>1169</td>\n",
       "      <td>3884</td>\n",
       "      <td>1356</td>\n",
       "      <td>1797</td>\n",
       "      <td>''Raising a New Generation of Leaders''</td>\n",
       "      <td>21 October 2002</td>\n",
       "      <td>...</td>\n",
       "      <td>[[David Oyedepo]]</td>\n",
       "      <td>[[Abiodun H. Adebayo]]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Ota, Ogun State]]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Urban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1805</td>\n",
       "      <td>University of Nigeria</td>\n",
       "      <td>http://www.unn.edu.ng/</td>\n",
       "      <td>1311</td>\n",
       "      <td>3279</td>\n",
       "      <td>1038</td>\n",
       "      <td>2243</td>\n",
       "      <td>''To Restore the Dignity of Man''</td>\n",
       "      <td>1955</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Charles Igwe Arizechukwu]]</td>\n",
       "      <td>36,000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Nsukka]]</td>\n",
       "      <td>[[Enugu state|Enugu]]</td>\n",
       "      <td>Rural&lt;br /&gt; {{convert|871|ha|acre}} (Nsukka ca...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1984</td>\n",
       "      <td>University of Lagos</td>\n",
       "      <td>https://unilag.edu.ng/</td>\n",
       "      <td>161</td>\n",
       "      <td>4143</td>\n",
       "      <td>1521</td>\n",
       "      <td>2312</td>\n",
       "      <td>In deed and in truth</td>\n",
       "      <td>1962</td>\n",
       "      <td>...</td>\n",
       "      <td>Alhaji (Dr.) Abubakar IBN Umar Garbai El-Kanem...</td>\n",
       "      <td>[[Oluwatoyin Ogundipe|Prof. Oluwatoyin Ogundipe]]</td>\n",
       "      <td>55,000 (2017)</td>\n",
       "      <td>43,784 (2017)</td>\n",
       "      <td>9,070 (2017)</td>\n",
       "      <td>1,736 (2017)</td>\n",
       "      <td>552 (2017)</td>\n",
       "      <td>[[Lagos]]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Urban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>2053</td>\n",
       "      <td>Obafemi Awolowo University</td>\n",
       "      <td>http://oauife.edu.ng/</td>\n",
       "      <td>2916</td>\n",
       "      <td>4560</td>\n",
       "      <td>1616</td>\n",
       "      <td>2025</td>\n",
       "      <td>For Learning and Culture</td>\n",
       "      <td>1961</td>\n",
       "      <td>...</td>\n",
       "      <td>Etsu [[Yahaya Abubakar]]</td>\n",
       "      <td>[[Adebayo Simeon Bamire]]</td>\n",
       "      <td>about 35,000</td>\n",
       "      <td>13,000</td>\n",
       "      <td>7500</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Ile-Ife]]</td>\n",
       "      <td>[[Osun State|Osun]]</td>\n",
       "      <td>Urban {{convert|2020|ha|acre}}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>96</td>\n",
       "      <td>14054</td>\n",
       "      <td>Yobe State University (Bukar Abba Ibrahim Univ...</td>\n",
       "      <td>https://www.ysu.edu.ng/</td>\n",
       "      <td>17040</td>\n",
       "      <td>15459</td>\n",
       "      <td>6490</td>\n",
       "      <td>6084</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>97</td>\n",
       "      <td>14091</td>\n",
       "      <td>Taraba State University Jalingo</td>\n",
       "      <td>http://www.tsuniversity.edu.ng/</td>\n",
       "      <td>18418</td>\n",
       "      <td>14883</td>\n",
       "      <td>6690</td>\n",
       "      <td>6084</td>\n",
       "      <td>Character, Competence, Courage</td>\n",
       "      <td>September 2014</td>\n",
       "      <td>...</td>\n",
       "      <td>[[William Kumuyi]]</td>\n",
       "      <td>[[Joseph Afolayan]]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Ayobo, Lagos|Ayobo-Ipaja]]</td>\n",
       "      <td>[[Lagos State]]</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>98</td>\n",
       "      <td>14110</td>\n",
       "      <td>Ondo State University of Science &amp; Technology ...</td>\n",
       "      <td>http://www.osustech.edu.ng/</td>\n",
       "      <td>23528</td>\n",
       "      <td>12868</td>\n",
       "      <td>7168</td>\n",
       "      <td>6084</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{{start date and age|2013}}</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Professor [[Auwal H Yadudu]]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[[Birnin Kebbi]], [[Kebbi State]]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Urban</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>99</td>\n",
       "      <td>14347</td>\n",
       "      <td>Anchor University Lagos</td>\n",
       "      <td>https://aul.edu.ng/</td>\n",
       "      <td>9304</td>\n",
       "      <td>17841</td>\n",
       "      <td>5779</td>\n",
       "      <td>6084</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>100</td>\n",
       "      <td>14446</td>\n",
       "      <td>Federal University Birnin Kebbi</td>\n",
       "      <td>http://www.fubk.edu.ng/</td>\n",
       "      <td>26140</td>\n",
       "      <td>15000</td>\n",
       "      <td>6456</td>\n",
       "      <td>6084</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    ranking  world_rank                                       universities  \\\n",
       "0         1        1322                               University of Ibadan   \n",
       "1         2        1742                            Covenant University Ota   \n",
       "2         3        1805                              University of Nigeria   \n",
       "3         4        1984                                University of Lagos   \n",
       "4         5        2053                         Obafemi Awolowo University   \n",
       "..      ...         ...                                                ...   \n",
       "95       96       14054  Yobe State University (Bukar Abba Ibrahim Univ...   \n",
       "96       97       14091                    Taraba State University Jalingo   \n",
       "97       98       14110  Ondo State University of Science & Technology ...   \n",
       "98       99       14347                            Anchor University Lagos   \n",
       "99      100       14446                    Federal University Birnin Kebbi   \n",
       "\n",
       "                              website presence_rank impact_rank openness_rank  \\\n",
       "0              https://www.ui.edu.ng/          2113        2088          1057   \n",
       "1   http://covenantuniversity.edu.ng/          1169        3884          1356   \n",
       "2              http://www.unn.edu.ng/          1311        3279          1038   \n",
       "3              https://unilag.edu.ng/           161        4143          1521   \n",
       "4               http://oauife.edu.ng/          2916        4560          1616   \n",
       "..                                ...           ...         ...           ...   \n",
       "95            https://www.ysu.edu.ng/         17040       15459          6490   \n",
       "96    http://www.tsuniversity.edu.ng/         18418       14883          6690   \n",
       "97        http://www.osustech.edu.ng/         23528       12868          7168   \n",
       "98                https://aul.edu.ng/          9304       17841          5779   \n",
       "99            http://www.fubk.edu.ng/         26140       15000          6456   \n",
       "\n",
       "   excellence_rank                                              motto  \\\n",
       "0             1561  \"''Recte Sapere Fons''\" (To think straight is ...   \n",
       "1             1797            ''Raising a New Generation of Leaders''   \n",
       "2             2243                  ''To Restore the Dignity of Man''   \n",
       "3             2312                               In deed and in truth   \n",
       "4             2025                           For Learning and Culture   \n",
       "..             ...                                                ...   \n",
       "95            6084                                                NaN   \n",
       "96            6084                     Character, Competence, Courage   \n",
       "97            6084                                                NaN   \n",
       "98            6084                                                NaN   \n",
       "99            6084                                                NaN   \n",
       "\n",
       "                    established  ...  \\\n",
       "0   {{start date and age|1948}}  ...   \n",
       "1               21 October 2002  ...   \n",
       "2                          1955  ...   \n",
       "3                          1962  ...   \n",
       "4                          1961  ...   \n",
       "..                          ...  ...   \n",
       "95                          NaN  ...   \n",
       "96               September 2014  ...   \n",
       "97  {{start date and age|2013}}  ...   \n",
       "98                          NaN  ...   \n",
       "99                          NaN  ...   \n",
       "\n",
       "                                           chancellor  \\\n",
       "0   [[Sa'adu Abubakar|Saad Abubakar]], [[Sultan of...   \n",
       "1                                   [[David Oyedepo]]   \n",
       "2                                                 NaN   \n",
       "3   Alhaji (Dr.) Abubakar IBN Umar Garbai El-Kanem...   \n",
       "4                            Etsu [[Yahaya Abubakar]]   \n",
       "..                                                ...   \n",
       "95                                                NaN   \n",
       "96                                 [[William Kumuyi]]   \n",
       "97                                                NaN   \n",
       "98                                                NaN   \n",
       "99                                                NaN   \n",
       "\n",
       "                                      vice_chancellor       students  \\\n",
       "0                     Professor [[Kayode Adebowale ]]         41,743   \n",
       "1                              [[Abiodun H. Adebayo]]            NaN   \n",
       "2                        [[Charles Igwe Arizechukwu]]         36,000   \n",
       "3   [[Oluwatoyin Ogundipe|Prof. Oluwatoyin Ogundipe]]  55,000 (2017)   \n",
       "4                           [[Adebayo Simeon Bamire]]   about 35,000   \n",
       "..                                                ...            ...   \n",
       "95                                                NaN            NaN   \n",
       "96                                [[Joseph Afolayan]]            NaN   \n",
       "97                       Professor [[Auwal H Yadudu]]            NaN   \n",
       "98                                                NaN            NaN   \n",
       "99                                                NaN            NaN   \n",
       "\n",
       "   undergraduates postgraduates academic_staff administrative_staff  \\\n",
       "0             NaN           NaN            NaN                  NaN   \n",
       "1             NaN           NaN            NaN                  NaN   \n",
       "2             NaN           NaN            NaN                  NaN   \n",
       "3   43,784 (2017)  9,070 (2017)   1,736 (2017)           552 (2017)   \n",
       "4          13,000          7500            NaN                  NaN   \n",
       "..            ...           ...            ...                  ...   \n",
       "95            NaN           NaN            NaN                  NaN   \n",
       "96            NaN           NaN            NaN                  NaN   \n",
       "97            NaN           NaN            NaN                  NaN   \n",
       "98            NaN           NaN            NaN                  NaN   \n",
       "99            NaN           NaN            NaN                  NaN   \n",
       "\n",
       "                                 city                  state  \\\n",
       "0                          [[Ibadan]]      [[Oyo State|Oyo]]   \n",
       "1                 [[Ota, Ogun State]]                    NaN   \n",
       "2                          [[Nsukka]]  [[Enugu state|Enugu]]   \n",
       "3                           [[Lagos]]                    NaN   \n",
       "4                         [[Ile-Ife]]    [[Osun State|Osun]]   \n",
       "..                                ...                    ...   \n",
       "95                                NaN                    NaN   \n",
       "96       [[Ayobo, Lagos|Ayobo-Ipaja]]        [[Lagos State]]   \n",
       "97  [[Birnin Kebbi]], [[Kebbi State]]                    NaN   \n",
       "98                                NaN                    NaN   \n",
       "99                                NaN                    NaN   \n",
       "\n",
       "                                               campus  \n",
       "0                                                 NaN  \n",
       "1                                               Urban  \n",
       "2   Rural<br /> {{convert|871|ha|acre}} (Nsukka ca...  \n",
       "3                                               Urban  \n",
       "4                      Urban {{convert|2020|ha|acre}}  \n",
       "..                                                ...  \n",
       "95                                                NaN  \n",
       "96                                                NaN  \n",
       "97                                              Urban  \n",
       "98                                                NaN  \n",
       "99                                                NaN  \n",
       "\n",
       "[100 rows x 21 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# dropping the name column on second datafram, then merge with the first on ranking.\n",
    "sub_df_2 = df_2.drop(columns ='name')\n",
    "master = df_1.merge(sub_df_2, on = 'ranking', how = 'left')\n",
    "master"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e8d0e1de",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Saving merged dataframe to a csv file.\n",
    "master.to_csv('top_100_universities_in_nigeria.csv', index = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa596c5f",
   "metadata": {},
   "source": [
    "## Summary\n",
    "This project aims to get the top 100 universities in Nigeria and gather more information from Wikipedia. The extraction from webometrics ranking using BeautifulSoup was successful.\n",
    "\n",
    "While manually getting the URLs of each school from Wikipedia, some schools, such as Federal University Kashere Gombe State, do not have a Wikipedia page, while some school has a Wikipedia page but no infobox. Due to some schools not having the required information, the extracted JSON data has NaN as the default value.\n",
    "\n",
    "Both data from Beautilfulsoup extraction and wptools extraction were merged and saved to a file `top_100_universities_in_nigeria.csv`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}