Include max_threads into cluster_profile.

[rajwanir]

The current method for most rules is set with attempts. For example, threads: lambda wildcards, attempt: attempt * 2. This is not ideal for rules that are known to be compute intensive and can use max threads efficiently when provided in the first attempt.

Currently, there is no way to determine what the max_threads is for each HPC system so at the rule level threads could either be assigned by attempts or hardcoded to a maximum. Ideally, there should be a way to encode the max threads in cluster profiles at cluster_profiles/<system>/config.yaml . So that at the rule level threads can be assigned to system max for select compute intensive rules to run most efficiently.

[rajwanir]

Turns out simpler than I had imagined. The known cluster_profiles now sets max-threads via config. I have currently set to what is known: 44 for CCAD2 and 56 for biowulf. The defualt slurm is set to 8 based on what some rules are already requesting in the pipeline. If a rule requests more than max-threads , it will autoscale down. If the cluster_profile do not exist for a system, a user can also set this at invocation with cgr submit --max-threads <max_threads>