Add a default timepout for the lock files used when download data.

This is to prevent a deadlock if the code can't get the lock file.

Example:
ERROR 2024/11/14 03:55:11 PM - lifetime has expired, breaking
ERROR 2024/11/14 03:55:11 PM - lockfile exists but isn't safe to break: /hps/nobackup/rdf/metagenomics/service-team/production/automated_jobs/ERP1432/ERP143252/download.lock

If this happens the process hangs there indefinitely. With the timeout the code will fail and exit with a Timeout error at least

Author: mberacochea

fetchtool/abstract_fetch.py

@@ -343,8 +343,8 @@ def write_project_download_file(self, project_accession, new_rows):
             self.create_empty_file(download_file)
 
         lock_file = download_file + ".lock"
-        # Lock for 1 minute
-        with Lock(lock_file, lifetime=60):
+        # Lock for 1 minute, timeout after 10 minutes to avoid deadlocking the code if it can get the lock
+        with Lock(lock_file, lifetime=60, default_timeout=60 * 10):
             existing_rows = set(self.read_download_data(project_accession))
             existing_rows = existing_rows.union(set(new_download_rows))
             with open(download_file, "w+") as f:
@@ -412,7 +412,8 @@ def write_project_description_file(self, project_accession, new_rows):
 
         lock_file = project_file + ".lock"
         # We lock the file for 2 minutes, should be enough
-        with Lock(lock_file, lifetime=120):
+        # The timeout will prevent a deadlock, if within 10 minutes it can't get the lock it will timeout and exit
+        with Lock(lock_file, lifetime=120, default_timeout=60 * 10):
             # Fallback in case empty file exists
             try:
                 project_runs = self.read_project_description_file(project_accession)