Overriding resourceLimits

[mirpedrol]

Hello!

I am running an nf-core pipeline with --profile test,self_hosted_runner which both declare different process.resourceLimits, and I expect self_hosted_runner to be the one with priority and overriding test, but it's not the case.

In the nextflow.config we include configs in this order:

1. include nfcore_custom.config
2. nfcore_custom.config defines the profile self_hosted_runner which includes self_hosted_runner.config
3. we define the profile test which includes test.config

Is the order in which we include each config the one that has priority? or how does resourceLimits overriding work?

[bentsherman]

Looks like the same issue as #1792. Because the config parser is utterly cursed, the order in which profiles are applied is not well-defined. This applies to all config options, not just resourceLimits

It looks like @jfy133 even drew some diagrams to figure out how it works, I assume based on some empirical tests: nf-core/eager@91b33ce

I would do some tests and see if it's at least deterministic. Maybe it's resolving them in reverse order, I really can't tell. But if it's deterministic then we can just document the behavior for now.

Long term, the new config parser (#4744) should solve this problem. It will apply profiles in the order in which they are specified

[mirpedrol]

Thanks for the explanation.
In case it's useful, I have tried to change the order in which I provide the profiles and still saw the error. What fixed it was including the config with self_hosted_runner after specifying the profile test in the config. So it looks like the order in which profiles are defined is what has priority.