Build an automated pipeline that takes in new data, performs appropriate transformations, and loads the data into existing tables that will update on a daily basis. This task was accomplished by creating an ETL function to collect and process data from three different sources and formats (JSON, Kaggle metadata, .csv) and adds it to a ProstgreSQL database.
- Intake 3 data files
- Write an ETL function to read data
- Extract and transform the data
- Create a final combined Movie Database