🐛 Bug: Investigate "No space left on device" in our pipelines

[DhanshreeA]

Describe the bug.

A number of our model pipelines, typically in the "upload model to dockerhub" stage fail because of "No space left on device".
Exhibits:

• https://github.com/ersilia-os/eos42ez/actions/runs/8191687245
• https://github.com/ersilia-os/eos18ie/actions/runs/8191708280
• https://github.com/ersilia-os/eos31ve/actions/runs/8192443412
• https://github.com/ersilia-os/eos9tyg/actions/runs/8192501865
• https://github.com/ersilia-os/eos4tcc/actions/runs/8235442007
• https://github.com/ersilia-os/eos30f3/actions/runs/8233315503
• https://github.com/ersilia-os/eos4x30/actions/runs/8232877335
• https://github.com/ersilia-os/eos7w6n/actions/runs/8250745999
• https://github.com/ersilia-os/eos4u6p/actions/runs/8251316509

Describe the steps to reproduce the behavior

Go to any one of the jobs above and re run. I tried to run these jobs when no other jobs were running on our runners and yet they failed.

Expected behavior.

These jobs should pass.

Screenshots.

No response

Operating environment

Runner OS

Additional context

Potentially useful resources:

• Github Actions randomly fail with "Error: No space left on device" actions/runner-images#1860
• https://github.com/orgs/community/discussions/26351
• GitHub Actions failes with: "no space left on device" actions/runner-images#2875
• https://github.com/orgs/community/discussions/25678

[GemmaTuron]

Hi @DhanshreeA
I am highlighting this model: https://github.com/ersilia-os/eos9taz/actions as I need it for chemsampler but it is not passing either :)

[DhanshreeA]

@GemmaTuron, I'm aware of this (ersilia-os/eos9taz#11) For now, I've pushed it manually to unblock you. However there's a related issue that I'm on top of: #1068 which I opened with specifically this model in mind.

[DhanshreeA]

Root Cause: With GitHub hosted runners, one of the guarantees is getting software updates. This could mean one or all of the following frequently: runner updates, runner provisioner updates, patches to the OS on the runner, the software bundled in the OS, etc. Every such update is quite likely to eat into the disk space. For example, here's the list of all the tools installed on the runner OS, that we do not need for our builds. So far there is no straightforward solution other than an aggressive disk clean up. This has been implemented at the level of the eos-template repository. While this works for now, it is not guaranteed that this issue will not come up again.