Putting Snakemake into the HPC context

[ocaisa]

During the Sprint for this repo, it was noted that some sites explicitly recommend against Snakemake for multi-node workloads. I'm guessing that this is likely because Snakemake has no awareness of MPI runtimes so you would be required to bake the MPI runtime and it's arguments into your tasks (which is not very portable: a different site could mean a different MPI runtime).

This triggered me to go looking around to see what sites actually recommend and I found workflows recommendations from CSC.

[ocaisa]

Another argument against Snakemake was that it can potentially overwhelm the scheduler creating many small jobs. This can, however, be mitigated against through Job Grouping

[ocaisa]

There are signs that this doesn't work as one would imagine, see snakemake/snakemake#872, though this may have been resolved with snakemake/snakemake#1218 which is in release 7.11.0 (just 8 days old at the time of writing!)

[ocaisa]

I don't necessarily see those as fatal for this lesson, but they are things we will have to address in our content. And also we should make it clear that Snakemake if one of many workflow tools, some of which are heavily tailored for the HPC space.