Rosetta pipeline: A showcase and comparison of pipeline frameworks

A lot of pipeline frameworks exist, each containing a different (and sometimes exotic) syntax. Because of this, it can be confusing to choose the appropriate framework for a particular use case. This is further complicated by an inconsistent terminology between frameworks. For example, rules (make and snakemake), processes (nextflow), tasks (WDL), jobs (CWL) and stages (martian) all refer to the exact same concept in different pipeline specifications.

Here we try to create:

• Example workflows for each framework with some common workflow tasks. They showcase how each framework is used, and where the similarities and differences lie. They are also useful to learn a particular framework by example. This is inspired by Rosetta code.
• A working document on the similarities and differences between pipeline frameworks. This document introduces a consistent terminology of features that a pipeline framework can have, while listing alternative terms along the way. The discussion is meant to be neutral and objective, in the sense that it simply lists possible features of pipeline frameworks, along with some advantages and disadvantages of having that particular feature.
• A comparison of the features of each framework, structured in the same way as the working document.

Structure

• tasks contains the task description and examples for each framework.
• comparison contains the working document and a comparison of frameworks.
• containers contains the dockerfiles for each framework. These are used for executing the example workflows.

Tasks & examples

Task	Frameworks
One task	make, snakemake, nextflow, luigi, airflow, toil, cromwell, drake
One task cached	make, snakemake, nextflow, luigi, cromwell
Chain	snakemake, nextflow, cromwell
Merge	snakemake, nextflow
Run in docker	nextflow, cromwell
Split merge	snakemake, nextflow
Module as is	snakemake
Alternative paths	nextflow

Running the examples

1. Install conda and docker
2. Clone this repo: git clone git@github.com:komparo/rosetta-pipeline.git && cd rosetta-pipeline
3. Install the conda environment: conda env create -f assets/env/environment.yml
4. Activate the environment: conda activate rosettapipeline
5. Run snakemake: snakemake. A first build can take a while because all docker containers have to be built. To run only one framework, use snakemake --config framework_id=nextflow.

Contributing

We welcome contributions of any kind. See assets/contributing.md.

A contribution implies that you agree with the Code of conduct.

Further reading

Opinions

• Sam Minot on how an ideal workflow manager looks like
• Experiences when creating an RNA-seq pipeline
• Reddit discussion on Toil vs Snakemake vs Nextflow

Lists and rankings

• An awesome list of frameworks
• Popularity ranking of tools

Reviews

• “A review of bioinformatic pipeline frameworks” by Jeremy Leipzig (https://github.com/leipzig)