Note: This project requires GCP credentials that are not public and are no longer valid. The following instructions were for internal use only. This repo is maintained for portfolio purposes only.
A demo video showing the pipelines capabilities can be viewed here: https://www.youtube.com/watch?v=ZQttMPyt1t8.
-
Download the
keen-diode-...jsonfile from Discord and save it somewhere on your system. You will need it. -
Create virtual environment for the project
-
Activate it
-
Install all dependencies in
requirements.txt -
Navigate to the root of this project
-
From the terminal, run
$ mage start. Mage should now open on your browser. -
On the left had side, click on Pipelines:
-
Click on the pipeline
comp333_pipeline:
-
From the left hand side, click: "Edit Pipeline":
-
On the right hand side, navigate to the "Secrets" section:
-
Create a new secret using the "+ New" button. Call it
PATH. Set its value to the absolute path on your system wherekeen-diode-...jsonis stored.
Remember to hit "Enter" when done to save it.
Once the secret has been created, click the play button on a block to run it.
To run all blocks in Pipeline, go the "Triggers" pane from the left hand side.
If there does not already exist a trigger, create a new Trigger of 'Schedule' type, scheduled for now and running 'once'.
Once running, can view progress by clicking on the trigger.
To re-run the trigger (especially useful for triggers that run 'now' and 'once'), click the triggers name and click "Start trigger":
You can click on the 'Block runs' number to see a log of the status of each block in the pipeline and a tree view of the pipeline, both of which auto-update as the pipeline runs:
Note: The COMP333_Project_Data folder on GitHub will be deprecated soon. We have it on GCP.
Please try to get Mage working on your system. This not only will confirm if you can get Mage working,
but also that you can interact with GCP properly via Mage.