v2.0.0

New Caper 2.0.0. is out.

There is no significant update for cloud-based backends (gcp and aws).
Most updates are for HPC USERS to fix various annoying Singularity/Conda and cluster (SLURM, ...) issues.

For HPC users

HPC users must initialize Caper's configuration file. This will add new important parameters and description/instruction for them. Please make a copy of your original configuration file (~/.caper/default.conf) and then initialize it with caper init.

$ cp ~/.caper/default.conf ~/.caper/default.conf.bak
$ caper init [YOUR_BACKEND]  # local, slurm, sge, pbs, lsf. See README.md for details

Follow instruction comments in the configuration file. Most parameters will be the same as those in the original configuration file.

For Conda/Singularity users on HPC

YOU DO NOT NEED TO ACTIVATE CONDA ENVIRONMENT BEFORE RUNNING PIPELINES. Just make sure that pipeline's Conda environment is correctly installed and add --conda to the command line caper run ... --conda. New Caper 2.0.0 runs each WDL task inside a Conda environment.

We strongly recommend to use Singularity for new ENCODE ATAC-seq and ChIP-seq pipelines (both >=v2.0.0). caper run ... --singularity if your cluster support Singularity.

Added resource parameters for HPC

Added a resource parameter for each HPC backend so that users can customize resource parameters for the job submission command line (squeue, qsub, ...) according to their cluster configuration. It will be appended to the job submission command line. WDL syntax is allowed in ${} notation. Please find details in the configuration file after initialization. See README for details.

• slurm backend: slurm-resource-param
• sge backend: sge-resource-param
• pbs backend: pbs-resource-param
• lsf backend: lsf-resource-param

v1.6.3

No code changes to v1.6.2 except for documentation.

However, AWS users should reinstall AWS Batch with a new CloudFormation template built for Caper.
There were several issues with the old installation method (EBS autoscaling and bucket permission issues).

Please read this instruction carefully.

v1.6.2

No changes to v1.6.1
Bumped version to avoid conflicts between wrongly-uploaded PyPI package v1.6.1.

v1.6.1

• Raises exception WomtoolValidationFailedException when Womtool validation fails (instead of returning False)
• Fixed typo in warning message related aws backend's call-caching
• Installs gsutil on aws caper server instance

v1.6.0

• AWS backend is updated and fully tested.

  • Call-caching (re-using previous outputs) works fine but Cromwell always makes cache copies on new workflow's output directory even with --aws-call-caching-dup-strat reference. See this issue for details.

• Users can use HOCON's include statements in custom backend configuration file (specified with --backend-conf).

v1.5.1

Fix --backend-conf

• --backend-conf will correctly override the auto-generated backend configuration file.
• Added a new parameter update=False to HOCONString.merge(), which will update self with merged if update==True.

v1.5.0

If you upgrade from previous version then please upgrade autouri too.

$ pip install autouri --upgrade
$ pip install caper --upgrade

Upgraded Cromwell from 52 to 59. Metadata DB is compatible between two versions.

Please check your Caper configuration file at ~/.caper/default.conf to remove cromwell and womtool from the file. Then Caper will automatically download Cromwell/Womtool 59 and use them.

# remove these two lines
cromwell=~/.caper/cromwell_jar/cromwell-52.jar
womtool=~/.caper/womtool_jar/womtool-52.jar

Fixed a bug that takes too long to recursively localize big files from S3 to GCS

• md5 checksum skipping flag was not propagated for recursive localization (in JSON, CSV, TSV) in autouri
• Fixed in autouri

If you get the following error, then check your service account's settings this document.

400 Bad Request
POST https://lifesciences.googleapis.com/v2beta/projects/XXX/locations/us-central1/operations/XXX:cancel
{
  "code" : 400,
  "errors" : [ {
    "domain" : "global",
    "message" : "Precondition check failed.",
    "reason" : "failedPrecondition"
  } ],
  "message" : "Precondition check failed.",
  "status" : "FAILED_PRECONDITION"
}

v1.4.2

Parameter name changes

• auto-update-metadata -> auto-write-metadata: in both CLI and python functions (_ instead of -).

Generation of metadata.json file on workflow's root output directory

• It was written for any change in task/subworkflow/workflows status change, which possibly stressed Cromwell server and resulted in its crash. It's written only when a main workflow is done whether successful or not.

Bug fixes

• Fix wrong graceful shutdown message even when Cromwell is done successfully.

v1.4.1

Minor release to fix several issues.

• caper cleanup does not lock a file to be deleted.
• No automatic writing of metadata.json file for subworkflows.

v1.4.0

Change of CLI parameters:

• --hide-subworkflow -> --show-subworkflow. Caper defaults to hide subworkflows.

Added features

• --disable-auto-update-metadata: Disables auto write/update of metadata.json file on workflow's output root directory.
• NBSubprocThread
  • Separated streaming (with two deamonized threads instead of one) of STDOUT and STDERR of a subprocess.
  • Better error handling. Correctly sends signal (signal.SIGTERM by default) to a subprocess thread when interrupted by stop().
• CromwellRestAPI
  • find() defaults to exclude_subworkflow.
    • Does not retrieve all workflows unless wildcard (* or ?) is in a query as workflow_id or label (as value in a key/value pair).

Bug fixes

• --java-heap-server not working
• NBSubprocThread
  • Not correctly capturing Cromwell's STDOUT/returncode while gracefully dying.
• Recursively sending too many requests to retrieve information of all workflows.
  • This recursively happened every update of workflow/subworkflow/task's status change, so lead to crash of Cromwell server running WDLs (e.g. ENCODE dnase-seq-pipeline) with nested subworkflows.