Skip to content

Files

Latest commit

273afb3 · Oct 27, 2023

History

History

workers

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
Jun 13, 2023
Oct 17, 2023
Oct 20, 2023
Oct 27, 2023
Sep 21, 2023
Jul 29, 2021
Apr 6, 2023
Jun 11, 2020
Sep 12, 2023
Jan 9, 2020
Sep 12, 2018
Jun 22, 2021
Jan 22, 2021
Jan 22, 2021
Jan 22, 2021
Jan 9, 2020
Oct 2, 2023
Oct 2, 2023
Jun 13, 2023
Sep 12, 2023
Jul 17, 2020

Data Refinery Workers

This is the project root for the Data Refinery Workers. This project is composed of a number of Batch jobs which can be used to download and process data from a variety of sources.

Developing

When developing a new task you will probably need to run the task repeatedly. This can be done easily by running the workers with ./run_workers and then modifying the data_refinery_workers/downloaders/management/commands/queue_task.py file to run the task you're developing. Once you've done that you can queue the task with ./run_tester.py

The worker container is run with a name of worker1 so that it's output can easily be inspected with docker logs worker1. However this means that you cannot run ./run_worker.sh twice in a row without deleting the old container. This can be done easily with

docker stop worker1 && docker container prune -f

A development workflow might look like:

./run_worker.sh
./run_tester.sh
docker logs worker1
# Review the output and make changes
docker stop worker1 && docker container prune -f