UTD Grades is a tool to view grade distributions at UT Dallas.
This monorepo consists of 4 sub-projects split into the core application and side utilities.
Core Application:
- The
client
folder contains the project's front-end built with React and Next.js - The
functions
folder contains the project's backend code built as Node.js AWS Lambda functions with the Serverless Framework
Side Utilities:
- The
converter
folder contains a Python script that converts the Excel file of grade distributions into a JSON file fitting our own data format - The
loader
folder contains Node.js data loading script that loads the converted JSON file of grade distributions data into PostgreSQL
There is also a data
folder that contains all currently received grade data with the original Excel files and converted JSON files.
Development steps are listed within each module's README.
The two parts making up the core application are the client
and functions
modules. The deployment process for both are separate, allowing changes to be made to each service independently from the other.
Deployment steps are listed within each module's README.
To upload new grade data received from UTD, the data will first need to be converted from Excel in JSON. To do this, use the converter
Python script.
First we'll need to install all the necessary dependencies for the script.
- Change directories into the converter folder
- Run
python -m venv venv
to create a virtual environment (usepy
instead ofpython
if using Python Launcher on Windows) - Run
venv/bin/activate
to activate virtual environmentvenv\Scripts\activate
on Windows
- Finally run
pip install -r requirements.txt
to install all dependencies
Now the converter
script is ready to run.
- Load the received Excel file into the
converter/data/data.xlsx
file location. The filename must matchdata.xlsx
exactly. - Ensure the
output
folder is empty. - Within the
converter/main.py
script, edit the TERM constant to match the name of the semester of the data you are uploading. For example, the TERM constant should contain2018 Fall
for fall 2018 grade data. - Run the script with
python main.py
- Check for any errors and fix them accordingly. These errors could occur if the Excel file varies slightly from the our expected format. Check the names of the columns and check for any spelling mistakes.
- If all goes well, you should see an output file in the
output/output.json
file location. - In the
data
folder make sure to create a new folder for this semester's data and place the original Excel file as well as the converted JSON there for safe-keeping.
Now we must take our converted JSON and actually upload it to our PostgreSQL database using the loader
Node.js script. First we'll need to install all the necessary dependencies for this script.
- Change directories into the
loader
folder - Run
npm install
Now the loader
script is ready to run
-
Create a
.env
file in theloader
folder and enter the database credentials for the UTD Grades databasedbName=database name goes here dbUser=username goes here dbPass=password goes here dbHost=host goes here
-
Take the outputted JSON file from the
converter
script and place it in theloader/data
folder. -
Within the
loader/index.js
file, edit the name of the file location to match the JSON file you're loading. -
Run the script with
npm start
The script should output the records it was not able to upload. Keep track of these so you can fix whatever issue may have occured with those specific records and re-upload them later. Sometimes there are quirks with professor names that cause issues on our end. Some professors are included in the dataset with no first name, which causes a problem with our parsing. For these, just look up the professor to get their first name and edit the records accordingly. When you re-upload, don't re-upload the whole JSON file again. Instead only re-upload the specific records that didn't upload the first time by placing them into a separate errors.json
file (in the loader/data/
folder). Then, edit the loader/index.js
script to read the errors.json
file instead of the whole semester JSON file. This is just to prevent data that uploaded properly the first time from getting messed with.