Allow for a idle timeout

[Fabiosilvero]

Hello,

In jenkins GCP plugin, we can set an idle runner timeout. That permits to keep a runner idle for next jobs but deleting it afterwards, that way a runner won't exist forever.

Is it possible to implement it in Garm ? That way, we can have a min idle runners set to 0 but benefit from idle runners instead of deleting / requesting another one when min idle runners is set to 0 in pool :)

It can be really useful especially in the context of Cloud Providers when you pay per minute your VMs :/

[gabriel-samfira]

Hi @Fabiosilvero

I am not completely sure how the workflow you describe works. Perhaps a few more details or a link to some documentation would help.

GARM right now allows for pools where min-idle-runners are set to 0. In those cases, a runner is spun up on demand when a job is recorded in queued state. Once the job is done, the runner is removed and the pool drops to 0 again. That way you incur no costs except for when a job is actually running. Is that what you mean?

[Fabiosilvero]

Here's a example :

Pool set to min-idle-runners = 0.

1st job :

• A worker is spun up
• Job gets done

2nd job launched within n minutes :

• A worker is still there
• Job gets done

3rd job launched after the delay :

• A worker is spun up
• Job gets done

That can help to compromize between zero idle costs and deleting/recreating workers.

For example, in a workflow, I had 3 parallels jobs and 2 jobs, one after another : garm created 3 runners, deleted them, created 1, deleted it and created the final one. In GCP VM context that leads to several minutes between the consecutive jobs. If a idle timeout would be set to, say, 5 minutes, there would be little to no delay but the runner would be deleted afterwards.

Let me know if I wasn't clear as English isn't my native language :)

[Fabiosilvero]

Here is my usecase :

[Fabiosilvero]

In jenkins GCP plugin, it is called a retention timeout : after a job, the timeout countdown is launched : if a job requests the runner label, the deletion is aborted but if the label isn't requested the deletion is completed. You still have 0 idle runners, but you have a good reactivity between jobs if they are consecutive (since you'll be using compute anyways).

[Fabiosilvero]

It seems that the operator implements it at a global level, but I think it concern only when scaling down the pool and the age of idle runners : this usecase is different since we would like to delay the immediate deletion of runner after job completion for reusing (If I understood correctly how it works)

[gabriel-samfira]

What you seem to need is something like adaptive idle runners. Or eager spin-up of runners based on traffic. That is not implemented unfortunately, but could be implemented as an external service until we get a chance to architect things in GARM itself.

Since version 0.1.5, GARM has an events websocket endpoint that enables anyone to subscribe to database events that are happening within GARM. Using something like this, a potential application can watch for events of interest, like jobs coming in for a particular pool and it could adjust the pool min-idle-runners to a higher value. Once traffic dies down, it could adjust min-idle-runners to 0 again and allow the scale-down logic to cleanup surplus idle runners.

This is something we could add as a feature later on, but if it's a huge rush, something like I described above, can be implemented fairly easily by a 3rd party.

Would you mind opening an issue about this? Something like Add adaptive idle runners?