Bugfix Deadlock when getting the lock

[mberacochea]

This is to prevent a deadlock if the code can't get the lock file.

Example:
ERROR 2024/11/14 03:55:11 PM - lifetime has expired, breaking
ERROR 2024/11/14 03:55:11 PM - lockfile exists but isn't safe to break: /hps/nobackup/rdf/metagenomics/service-team/production/automated_jobs/ERP1432/ERP143252/download.lock

If this happens the process hangs there indefinitely. With the timeout the code will fail and exit with a Timeout error at least.

I think we need to review this lock files... we can probably remove them (or at least add unit tests to check the code is doing what we expect)

[SandyRogers]

Thanks @mberacochea
Strange that this is happening on HPS as well as NFS. We could also try using portalocker instead (in theory it works on NFS, although in practice for EMG API concurrent log handler locking, we tell it to use a non-NFS directory for the lock files). Anyway, this looks good to make things fail explicitly.

[mberacochea]

Thanks @mberacochea Strange that this is happening on HPS as well as NFS. We could also try using portalocker instead (in theory it works on NFS, although in practice for EMG API concurrent log handler locking, we tell it to use a non-NFS directory for the lock files). Anyway, this looks good to make things fail explicitly.

Yeah, it is weird. The library that we are using is supposed to handle NFS like filesystems. We will have to investigate at some point.