Memory exceeds above 100G for WGS gVCF (pgscatalog-match)

[kpalanikannan]

Hi Team,

I am using whole genome GRCh38 gVCF as input. I configured the memory limit upto 110G and 32CPUs. The pgscatalog match method use till the 110G memory and then it get killed automatically or getting crashed with below error. Unzipped gVCF file size falls around 45G.

.command.sh: line 4: 44 Killed pgscatalog-match --dataset test01 --scorefile scorefiles.txt.gz --target GRCh38_test01_ALL.pvar.zst --only_match --outdir $PWD -v

My question: Is there any way to chunk the gVCF into small chunks and merge back the results later? Anyone can help on it?

[smlmbrt]

I would suggest splitting the data by chromosome if you haven't already (some relevant docs: https://pgsc-calc.readthedocs.io/en/latest/how-to/prepare.html). For instance by running a similar command per-chromosome:

plink2 --vcf <full_path_to_vcf.vcf.gz> \
    --allow-extra-chr \
    --chr 1 \
    -make-pgen --out <short name>_1

[kpalanikannan]

I do use gVCF generated by DRAGEN haplotypecaller. If I make the chromosome level chunks, how do I supply 24 files as input and how will it be received one by one to avoid any memory problem? Should I create seperate rows in samplesheet for each chromosome?

[smlmbrt]

You can specify a value in the chromosome column of the samplesheet (https://pgsc-calc.readthedocs.io/en/latest/how-to/samplesheet.html), the final csv will have as many rows as there are chromsomes + a header. The matching is then performed chromosome by chromosome and only the matches are aggregated to save loading the variants into memory, same with scoring.

[smlmbrt]

Note: if your gVCF doesn't have reference calls the alleles likely won't match (see #123 for discussion).

[kpalanikannan]

Thanks for details and it was useful.