1-data-acquisition-solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div width=50% style=\"display: block; margin: auto\">\n",
    "    <img src=\"figures/ucl-logo.svg\" width=100%>\n",
    "</div>\n",
    "\n",
    "### [UCL-ELEC0136 Data Acquisition and Processing Systems 2024]()\n",
    "University College London\n",
    "# Lab 1: Data Acquisition\n",
    "\n",
    "\n",
    "<hr width=70% style=\"float: left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Objectives\n",
    "\n",
    "The data acquisition lab aims to show how we can find and acquire various data sources (e.g., web scraping, social media, databases, sensors).   \n",
    "In this lab, you will learn:\n",
    "- How to use Python to acquire data from various sources\n",
    "- How to store data in a json format\n",
    "- How to commit and push your code to GitHub\n",
    "\n",
    "\n",
    "### Outline\n",
    "\n",
    "In this class we will cover the following topics:\n",
    "\n",
    "1. [Accepting an assignment](#1.-Accepting-an-assignment)\n",
    "2. [Cloning a repository](#2.-Cloning-a-repository)\n",
    "3. [Creating a virtual environment](#3.-Creating-a-virtual-environment)\n",
    "4. [Acquiring data using the `requests` module and the GitHub RESTful APIs](#4.-Acquiring-data-using-the-requests-module-and-the-GitHub-RESTful-APIs)\n",
    "5. [Pushing your code to GitHub](#5.-Pushing-your-code-to-GitHub)\n",
    "6. [Submitting your assignment](#6.-Submitting-your-assignment)\n",
    "\n",
    "<hr width=70% style=\"float: left\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Accepting an assignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For every week of the course, you will be given an assignment to complete.\n",
    "\n",
    "- You can find the **links** to your assignments on Moodle.\n",
    "\n",
    "- When you click on the link, you will be redirected to GitHub Classroom, where you will be asked to accept the assignment. The page will look like this:\n",
    "\n",
    "- GitHub Classroom will create a **public repository** for you, which will contain the assignment, with pattern `<assignment-name>-<your-github-username>`.\n",
    "\n",
    "\n",
    "\n",
    "<div width=50% style=\"display: block; margin-left: auto\">\n",
    "    <img src=\"figures/accepted-assignment.png\" style=\"display: block; margin: auto\" width=50%>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here, if applicable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Cloning a repository\n",
    "\n",
    "\n",
    "<div class=\"alert alert-block alert-danger\">\n",
    "<b>👩‍💻👨‍💻 Action required</b>\n",
    "\n",
    "- Open a terminal\n",
    "- Navigate to the directory where you want to store your assignment\n",
    "- Clone the repository using the command `git clone <repository-url>`\n",
    "- Navigate to the cloned repository using the command `cd <repository-name>`\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here, if applicable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Creating a virtual environment\n",
    "\n",
    "You should already have installed [**Anaconda**](https://docs.anaconda.com/free/anaconda/) or an open-source equivalent (preferred), such as [**Miniconda**](https://docs.conda.io/projects/miniconda/en/latest/miniconda-install.html).\n",
    "\n",
    "In this course we advice to mangage your python version using `conda` (e.g. Python 3.9 vs 3.10) and the packages required by Python using `pip`.\n",
    "Check our the preliminary material on the [UCL Moodle](https://moodle.ucl.ac.uk/mod/page/view.php?id=5693674) for instructions on how to use enviornment files.\n",
    "Alternatively, have a look [here](https://github.com/UCL-ELEC0136/example-requirements) to see how an environment file looks like when pip requirements are included in the file itself,\n",
    "or [here](https://github.com/UCL-ELEC0136/setup) for how an environment file where the pip requirements are stored separate into another file.\n",
    "\n",
    "\n",
    "This is a key step as it guarantees that your development environment is **reproducible** by somebody else.\n",
    "Failing do to this in the final assignment will get you a score of 0 (zero) in reproducibility.\n",
    "\n",
    "<div class=\"alert alert-block alert-danger\">\n",
    "<b>👩‍💻👨‍💻 Action required</b>\n",
    "\n",
    "- Create a `conda` environment file and name it `environment.yml`.\n",
    "- The environment (not the file) **must** have name `daps`.\n",
    "- Add `pip` dependencies **within** the environment file to specify the packages that your project needs to run. If you don't know yet which dependencies to use, yo ucan leave the file blank for now and add dependencies later.\n",
    "- Activate the environment using `conda activate daps`.\n",
    "- Install the dependencies using `pip install -r requirements.txt`.\n",
    "</div>\n",
    "\n",
    "\n",
    "<div class=\"alert alert-heading alert-danger\" style=\"background-color: white; border: 2px solid; border-radius: 5px; color: #000; border-color:#AAA; padding: 10px\">\n",
    "    <b>💎 Tip</b>\n",
    "\n",
    "You can prepend an exclamation mark `!` to any command in a Jupyter notebook to run it in the terminal. For example, `!conda activate daps` will activate the `daps` environment in the terminal.\n",
    "\n",
    "</div>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here, if applicable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Acquiring data using the `requests` module and the GitHub RESTful APIs\n",
    "\n",
    "The `requests` module is a Python module that allows you to send HTTP requests to a server and receive a response.\n",
    "We will use it to acquire data from the web.\n",
    "\n",
    "<div class=\"alert alert-block alert-danger\">\n",
    "<b>👩‍💻👨‍💻 Action required</b>\n",
    "\n",
    "- Import the `requests` module using `import requests`.\n",
    "- Use the `requests.get()` function to send a `GET` request to a server. The function takes as input the URL of the server, and returns a `Response` object.\n",
    "- Acquire 100 repositories from the `https://github.com/orgs/UCL-ELEC0136` organisation\n",
    "- Use the `Response` object to access the response of the server. For example, you can access the response status code using `Response.status_code`.\n",
    "- Use the `Response.json()` function to convert the response content to a `dict` object.\n",
    "- Use the `json` module to save the `dict` object to a file using `json.dump()`\n",
    "- Use the `json.load()` function to load the `dict` object from the file.\n",
    "- Verify that the `dict` object you loaded from the file is the same as the one you saved to the file.\n",
    "\n",
    "</div>\n",
    "\n",
    "\n",
    "<div class=\"alert alert-heading alert-danger\" style=\"background-color: white; border: 2px solid; border-radius: 5px; color: #000; border-color:#AAA; padding: 10px\">\n",
    "    <b>💎 Tip</b>\n",
    "\n",
    "Check out the GitHub REST API documentation to see how to acquire information from GitHub using the API.\n",
    "For example, you can use the following URL to retrieve information about the repository `https://api.github.com/repos/<username>/<repository-name>`, but this is not the query we want. We want **all** at least 100 repositories of an organisation.\n",
    "You can check the API's documentation here https://docs.github.com/en/rest?apiVersion=2022-11-28 or check Stackoverflow answers.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "\n",
    "# Store the URL of the GitHub API endpoint under the variable query\n",
    "query = \"https://api.github.com/orgs/UCL-ELEC0136/repos\"\n",
    "\n",
    "# send GET request to the GitHub API endpoint\n",
    "response = requests.get(query)\n",
    "\n",
    "# check if the response was successful and raise an exception if not\n",
    "response.raise_for_status()\n",
    "\n",
    "# print the response status code\n",
    "print(\"Response status code:\", response.status_code)\n",
    "print(\"Response content:\", response.content)\n",
    "\n",
    "# format the response as JSON and store it in a dictionary variable\n",
    "repositories = response.json()\n",
    "\n",
    "# print the number of repositories\n",
    "print(\"Number of repositories:\", len(repositories))\n",
    "# print the name of each repository\n",
    "print(\"Repository names:\", [repo[\"name\"] for repo in repositories])\n",
    "\n",
    "# save the response content to a file\n",
    "with open(\"repositories.json\", \"w\") as f:\n",
    "    json.dump(repositories, f)\n",
    "\n",
    "# read the response content from the file\n",
    "with open(\"repositories.json\", \"r\") as f:\n",
    "    repositories_loaded = json.load(f)\n",
    "    assert repositories == repositories_loaded\n",
    "    print(repositories_loaded)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1 Pagination\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "<b>👩‍💻👨‍💻 Optional action</b>\n",
    "\n",
    "- What if we want 200 repositories?\n",
    "\n",
    "</div>\n",
    "\n",
    "\n",
    "<div class=\"alert alert-heading alert-danger\" style=\"background-color: white; border: 2px solid; border-radius: 5px; color: #000; border-color:#AAA; padding: 10px\">\n",
    "    <b>💎 Tip</b>\n",
    "\n",
    "The method is in the title of this section\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Store the URL of the GitHub API endpoint under the variable query\n",
    "query = \"https://api.github.com/orgs/UCL-ELEC0136/repos\"\n",
    "\n",
    "\n",
    "# send GET request to the GitHub API endpoint\n",
    "repos = []\n",
    "\n",
    "# iterate over the first two pages of the response\n",
    "for page in (1, 2):\n",
    "    # send GET request to the GitHub API endpoint for the current page\n",
    "    response = requests.get(query, params={\"page\": page, \"per_page\": 100})\n",
    "    # check if the response was successful and raise an exception if not\n",
    "    response.raise_for_status()\n",
    "    # format the response as JSON and store it in a dictionary variable\n",
    "    data = response.json()\n",
    "    # add the repositories in the current page to the list of repositories\n",
    "    repos.extend(response.json())\n",
    "\n",
    "# print the number of repositories\n",
    "print(\"Number of repositories:\", len(repos))\n",
    "\n",
    "# Make sure that the number of repositories is 200\n",
    "assert len(repos) == 200"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Pushing your code to GitHub\n",
    "\n",
    "We are now ready to push our code that acquires data from GitHub to our repository (which is also GitHub, but this is just a coincidence, we could have used any other API, like Twitter's or Facebook's).\n",
    "\n",
    "<div class=\"alert alert-block alert-danger\">\n",
    "<b>👩‍💻👨‍💻 Action required</b>\n",
    "\n",
    "- Go into your terminal\n",
    "- Check the git status of your repository using the command `git status`\n",
    "- Verify that the files you want to commit are listed, and that there is no pending changes to pull from the remote\n",
    "- Add the files to the staging area using the command `git add <file-name>`\n",
    "- Commit the files using the command `git commit -m \"<commit-message>\"`\n",
    "- Use a meaningful commit message, e.g., `Acquire data from GitHub`\n",
    "- Push the files to GitHub using the command `git push`\n",
    "- Verify that the files have been pushed to GitHub by refreshing the page of your repository on GitHub. \n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the status of the repository\n",
    "!git status\n",
    "# Pull the latest changes from the remote repository\n",
    "!git pull\n",
    "# Add the new files to the staging area\n",
    "!git add repositories.json 1-data-acquisition-solutions.ipynb\n",
    "# Commit the changes with the following message\n",
    "!git commit -m \"Add a method to acquire data from GitHub API, and also the acquired data\"\n",
    "# Push the changes to the remote repository\n",
    "!git push\n",
    "# Check the status of the repository to make sure that the changes have been pushed\n",
    "!git status"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Submitting your assignment\n",
    "\n",
    "To submit this assignment and **every other future assignment**, included the **final assignment** you have to:\n",
    "- Commit and push your code to GitHub\n",
    "- Go to **your** repository of the assignment. This must be on our course organisation `UCL-ELEC0136` and usually has the pattern `https://github.com/UCL-ELEC0136/<assignment-name>-<your-github-username>`.\n",
    "- Go in the `Pull requests` tab and click on the `Feedback` pull request.\n",
    "- Click on `Files changed` and verify that the files you have changed are listed.\n",
    "- Merge the pull request by clicking on `Merge pull request` and then `Confirm merge`.\n",
    "\n",
    "We are now ready to push our code that acquires data from GitHub to our repository (which is also GitHub, but this is just a coincidence, we could have used any other API, like Twitter's or Facebook's).\n",
    "\n",
    "<div class=\"alert alert-block alert-danger\">\n",
    "<b>👩‍💻👨‍💻 Action required</b>\n",
    "\n",
    "Submit your assignment by following the steps above.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here, if applicable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}