Skip to content

Threads via variable. #361

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Feb 21, 2025
Merged

Threads via variable. #361

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
925d5c6
Renames `max_threads` described in #351 to `cores`.
Nov 25, 2024
acc7df7
Uses `workflow.cores` variable to define threads in select rules.
Nov 25, 2024
ed1d33c
Fixes #378. Restricts CPU overuse by adding `max-threads` as an alias…
Feb 20, 2025
070cd68
Adds cgrq as queue an additional queue that can used.
Feb 21, 2025
3a4c81d
Adds in permission checks for cgrq by checking if its a member of nci…
Feb 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions src/cgr_gwas_qc/cli/submit.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,9 @@ def main(
"If you have a very large project >50k samples, you may want to increase the number of CPUs used for this job. "
"Ignored if using `--cgems`.",
),
max_threads: int = typer.Option(
cores: int = typer.Option(
8,
help="The maximum number of threads a rule can request. If pipeline is executed in `cluster_mode`, this will scale down the threads to `max-threads`.",
help="The maximum number of threads a rule can request. If pipeline is executed in `cluster_mode`, this will scale down the threads to `cores`.",
),
):
"""Submit the CGR GwasQcPipeline to a cluster for execution.
Expand Down Expand Up @@ -147,8 +147,8 @@ def main(
if subworkflow:
payload["added_options"] += f"--subworkflow {subworkflow} " # type: ignore

if max_threads:
payload["added_options"] += f"--max-threads {max_threads} "
if cores:
payload["added_options"] += f"--cores {cores} "

# add global config options only used in cluster mode.
payload["added_options"] += "--config {} ".format( # type: ignore
Expand Down
3 changes: 2 additions & 1 deletion src/cgr_gwas_qc/cluster_profiles/biowulf/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,5 @@ printshellcmds: true
rerun-incomplete: true
restart-times: 3
use-conda: true
max-threads: 56
cores: &cores 56
max-threads: *cores
3 changes: 2 additions & 1 deletion src/cgr_gwas_qc/cluster_profiles/ccad2/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,5 @@ printshellcmds: true
rerun-incomplete: true
restart-times: 3
use-conda: true
max-threads: 44
cores: &cores 44
max-threads: *cores
10 changes: 9 additions & 1 deletion src/cgr_gwas_qc/cluster_profiles/ccad2/submit.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
#!/usr/bin/env python3
import grp
from dataclasses import dataclass, field
from datetime import timedelta
from typing import Set
Expand All @@ -15,9 +16,16 @@
)


def get_allowed_queue():
if "ncicgr_staff" in [g.gr_name for g in grp.getgrall()]:
return {"defq", "bigmemq", "cgrq"}
else:
return {"defq", "bigmemq"}


@dataclass
class Ccad2Options(ClusterOptions):
queue: Set[str] = field(default_factory=lambda: {"defq", "bigmemq"})
queue: Set[str] = field(default_factory=get_allowed_queue)
log: str = "logs/{rulename}_{job_id}.%j"

def __str__(self):
Expand Down
3 changes: 2 additions & 1 deletion src/cgr_gwas_qc/cluster_profiles/slurm_generic/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,5 @@ printshellcmds: true
rerun-incomplete: true
restart-times: 3
use-conda: true
max-threads: 8
cores: &cores 8
max-threads: *cores
Comment on lines +13 to +14
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this idea. I am still seeing the same behavior as before. That is, there is a discrepancy in the number of threads shown allocated in the job-specific log vs the whole-pipeline log. Here is what I see.

  • invocation: cgr submit --ccad2
  • commit: git rev-parse HEAD # 070cd684b8
  • sacct:
$ sacct -j 2154090 --format=JobID,JobName,Partition,AllocCPUs,State,ExitCode
JobID           JobName  Partition  AllocCPUS      State ExitCode
------------ ---------- ---------- ---------- ---------- --------
2154090      convert_b+       cgrq          8  COMPLETED      0:0
2154090.bat+      batch                     8  COMPLETED      0:0
  • job-specific log:
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 40
Rules claiming more threads will be scaled down.
Select jobs to execute...

rule convert_bcf_to_plink_bed:
    input: sample_level/samples.bcf
    output: sample_level/samples.bed, sample_level/samples.bim, sample_level/samples.fam
    log: sample_level/samples.log
    jobid: 0
    benchmark: benchmarks/convert_bcf_to_plink_bed.500.tsv
    threads: 40
    resources: mem_mb=1059, disk_mb=18132, tmpdir=/tmp, time_hr=1
  • whole-pipeline log:
[Fri Feb 21 13:15:28 2025]
rule convert_bcf_to_plink_bed:                                                                                              
    input: sample_level/samples.bcf
    output: sample_level/samples.bed, sample_level/samples.bim, sample_level/samples.fam
    log: sample_level/samples.log                                                                                           
    jobid: 2
    benchmark: benchmarks/convert_bcf_to_plink_bed.500.tsv
    threads: 8
    resources: tmpdir=/tmp, mem_mb=1059, time_hr=1
 plink2 --allow-extra-chr 0 --keep-allele-order --double-id --bcf sample_level/samples.bcf --update-sex /tmp/tmpl2_fp7jj --output-chr 26 --split-par hg38 --make-pgen --out sample_level/bcf2plink  --memory 1059 --threads 8 ;plink2 --pfile sample_level/bcf2plink --make-pgen --sort-vars --out sample_level/bcf2plink-sorted --threads 8 --memory 1059  ;plink2 --pfile sample_level/bcf2plink-sorted --make-bed --out sample_level/samples --threads 8 --memory 1059 ;rm sample_level/bcf2plink.{pgen,psam,pvar,log} sample_level/bcf2plink-sorted.{pgen,psam,pvar,log} /tmp/tmpl2_fp7jj                            
Submitted job 2 with external jobid '2154090'.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just submitted another test with invocation cgr submit --slurm to see what the logs look like when using the generic_slurm cluster profile. Should be cores/threads 8.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can confirm that when I use the invocation cgr submit --slurm both the whole-pipeline log and the job-specific log show that the number of threads allocated is 8.

2 changes: 1 addition & 1 deletion src/cgr_gwas_qc/workflow/modules/bcf.smk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ rule convert_bcf_to_plink_bed:
cfg.conda("plink2-0")
benchmark:
"benchmarks/convert_bcf_to_plink_bed." + str(len(cfg.ss)) + ".tsv"
threads: 44
threads: workflow.cores
resources:
mem_mb=ceil((0.07 * len(cfg.ss))) + 1024,
time_hr=ceil((0.11 * len(cfg.ss)) / 3600),
Expand Down
2 changes: 1 addition & 1 deletion src/cgr_gwas_qc/workflow/sub_workflows/entry_points.smk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -248,7 +248,7 @@ if cfg.config.user_files.gtc_pattern:
+ str(len(cfg.ss))
+ ".tsv"
)
threads: 44
threads: workflow.cores
resources:
time_hr=ceil((len(cfg.ss) * 0.2) / 3600 + 1),
mem_mb=len(cfg.cluster_groups) * 70,
Expand Down
Loading