- Fix NullPointerException when upload task logs for skipped tasks
- Gracefully handle k8s runtime errors (OOM/Deadline Exceeded) instead of throwing exception.
storeDirnow performs uploads from within the Nextflow runtime directly (as opposed to from within the process environment)
- Custom fsync binary which explicitly calls
fsyncsyscall on every file in the current directory. - Use the custom fsync binary to sync files instead of built-in sync as some distros don't support calling sync explicitly on files.
- Read exit status file before reading container exit code to ensure that output files are present in OFS mount.
- Move logs file upload to nextflow runtime because process containers may not have all
dependencies required to run
latch cp - Support downloading and uploading latch files with spaces in path
- Allow task hash to be
nullwhen processes are skipped
- Fix for downloading empty Latch paths
- Update .command polling script to use integer sleep times (decimal not always supported by host)
- Read exit code from container instead of .exitcode file
- Disable publishDir overwrite when publishing files for a cached task
- NullPointerException when relaunching cached tasks
- Race condition when concurrently creating multiple LatchFileSystem objects
- Chunk writes when downloading files to optimize write performance
- Report task hash to Latch backend
- Add retry logging for HttpRetryClient
- If latch logging directory specified, upload .command.* files to LData
- Add retires on Latch file part download
- Improved logging for dispatcher failures
- Print polling loop every 10 attempts and remove an echo
- Add explicit timeouts to all HTTP operations to prevent hanging executions
- Add explicit fsync of generated files in runtime and task steps
- Add polling for files in case the changes in the shared file system are not seen immediately after task start/finish
- Always sync written task files to disk. OFS only guarantees that files will be seen by other clients if the file is fsynced to the disk.
- Fix status for native nextflow processes
- Use v2 Nextflow scheduler
- Parallel downloads for latch paths
- Ansi color codes to warning and error messages
- Cleanup logging for DispatcherClient
- storeDir support for Latch Paths
- Add tag to task_info
- Demote K8s unschedulable warning to debug
- CopyMoveHelper.java will now attempt to copy files to Latch directly (instead of opening a stream and creating an auxiliary file to upload from)
- delete tmp file after latch file upload
- fix memory leak caused by HttpClient creation
- ability to download directories via LatchPath
- bump log column length from 100 to 150
- error handling when getting current workspace
- close LatchFileSystem thread executor after workflow onComplete
- Add support for retry from failed task
- HttpRetryClient -- retries on 429 and 500s
- Check task completion on "NodeTerminationError"
- decreasing polling interval from 15s to 5s
- Add retry on "context deadline exceeded" error for k8s executor
- Set line width to 100 so that logs don't get cut off
- Fix releasing nextflow workflows to different workspace
- Use ExecutorService instead of
ForkJoinPoolbecause ForkJoinPool doesn't cleanup threads -- causes memory to be leaked - Move memory allocation logic into thread to avoid OOM when uploading large files
- Cleanup unnecessary logs
- Use a single
ForkJoinPoolfor all LatchPaths to avoid OOM when uploading hundreds of files
- Fix race condition in LatchPath
uploadcausing file corruption
- Add versioning