Project1 utilizes the myanimelist.net dataset from Kaggle which stores it in Hive, and analyzes it using a CLI application written in Scala. The application includes a user management and access system for basic and admin users, where each user has their own list to access and implements all CRUD operations on that list.
- Java - version 1.8.0_311
- Scala - version 2.12.15
- Spark - version 3.1.2
- Spark SQL - version 3.1.2
- Hive - version 3.1.2
- HDFS - version 3.3.0
- Git + GitHub
- Login and new basic user creation
- Restricted access for basic users, full access for admins
- Execute analytical SQL queries
- Cannot add, update, or delete data from queries
- Promote basic users to admins, and delete basic users
- Can change own username and password
- Default basic account created on initialization
- Hive tables implement bucketing and partitioning
- Creates a personal list for each user
- Can add or delete data from list
- Search feature to find specfic data or a list of data from a database
- (Note, these instructions only support Windows 10 and above)
- First, download and install Git
- For Windows, navigate to https://git-scm.com/download/win and install
- Run the following Git command to create a copy of the project repository using either Command Prompt or PowerShell:
- git clone https://github.com/newyorkher/Project1
- Install Java
- Navigate to https://www.oracle.com/java/technologies/javase/javase8u211-later-archive-downloads.html and install
- Install IntelliJ
- Navigate to https://www.jetbrains.com/idea/download/download-thanks.html?platform=windows&code=IIC and install
- Download https://www.kaggle.com/datasets/CooperUnion/anime-recommendations-database
- All new users created will have basic level permissions and only admins are allowed to give admin status to other accounts.
- After logging in successfully. The user is brought to the home screen where they can choose to enter their profile or execute queries.
- From here, if there are any queries written and saved to the Hive database, the user can execute them by entering the "Execute Query" menu
- A user menu will pop up and from here the user can choose one of the following options
- The user will have access to change their username & password as well as access to their own list.
- This search feature helps users find a particular data or a list of data from a database to add to their list.
- After inputing 'Sword' as the search term. A query will be executed to search all data in the database that contains the searched input.
- To search for a more specific data. There is a unique ID designated to each data that can also be found on the 'myanimelist.net' website
- If an admin account is logged in. An admin menu will pop up for the admin user and has a few more options to choose from.
- An admin can either delete or promote a basic user to admin status (they can also demote or delete existing admins as well)
- In this case. We will be deleting the new user created from before
- After exiting the admin menu. The user menu will open and functions the same as seen before
- Return to home menu, log off and quit to exit the application.