Sunset workers

[chrisnatali]

When a worker's models are out of date, it should be brought off-line in favor of new workers with up-to-date models.

Problems arise when jobs are running on workers scheduled to be sunsetted.

Does disabling access to Redis via removing ufw 'allow' rule prevent worker from taking new jobs off the queue? If so, this may be a way of taking out of date servers off line...to be 'reaped' later.

[chrisnatali]

The following redis commands may help:
CLIENT LIST: List of connected clients
CLIENT KILL: Kill specified client

I noticed that upon terminating workers, there were still 'zombie' connections from workers that needed to be killed.

[chrisnatali]

With the addition of the 'STOP_PROCESSING_QUEUE' command (part of the 0.5 release), this is partially addressed. By issuing 'STOP_PROCESSING_QUEUE' to a 'RUNNING' worker, it ensures that the worker will not pickup any new jobs while allowing it to complete the current job. Simplifying Sunsetting.

Leaving this open because this process still needs to be formalized.