feat: GWAS Catalog harmonisation prototype

[d0choa]

This PR aims to create a process to be able to harmonise GWAS Catalog studies within GCP.

(Some) Background

• Harmonised GWAS Catalog studies have been uploaded to GCP using a rsync strategy from EBI infrastructure (~25k at the moment - ~3h to synch all)
• Harmonised summary stats are stored as .tsv.gz(poorly partitioned)
• We expect the data to keep growing (incremental)
• Datasets vary in schema
• Additional row-wise QC is required to ensure several requirements (betas, positions, etc.)

This PR
Aims to produce additionally harmonise the ".tsv.gz" files by using the GWASCatalogSumstatsPreprocessStep and producing an appropriately partitioned parquet file per study.

Additional context

• Partitions are done by sorting by chromosome, position and chunking in a fixed number of 20 partitions. This ensures the partitions are equally sized within the study (no skews) and partition sizes are reasonable (~10Mb on average, to a maximum of ~100Mb). This will result in 0.5M partitions to read once the 25k summary stats are processed which I believe is still reasonable but it will require additional considerations downstream.
• Current DAG computes the list of studies to process by looking at the filesystem. The current to-do list is created based on study IDs that haven't yet been processed (.SUCCESS), so it's not bulletproof (no timestamps, no md5, etc.).
• Jobs are submitted to Dataproc as GCP python API requests as opposed to using Airflow Operators. The reason is not to overload GCP with status requests from Airflow and keep things simple. Of course, it comes with the cost of not knowing the status of every individual study in Airflow.
• The cluster geometry requires many small (n1-standard-2) workers each addressing a different summary stats file. Due to the number of workers the master node requires a decent size (ATM n1-highmem-64).
• This PR can hit Dataproc default quotas of 5,000 job submissions. It can be adjusted but I'm still keeping the default as a safety net.

After some adjustments in the geometry, the task runs at a pace of 800-1000 studies per hour using approximately 80 workers. Considering the task, I believe this is a reasonable performance balance. This will become an incremental task.

[d0choa]

Additional stats from @DSuveges:

GWAS Catalog summary stats:

• Currently 25.6k (~100 new studies from yesterday)
• 90 studies below 1MB. (I haven't considered number of rows at this point)
• 165 studies above 1GB (up to 2.4GB), The row count goes from 25M to 61M)

Size distribution of all:

Studies above 1 GB:

[codecov-commenter]

Codecov Report

Merging #270 (caa8895) into main (f2f8399) will decrease coverage by 1.21%.
The diff coverage is 40.74%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #270      +/-   ##
==========================================
- Coverage   86.44%   85.24%   -1.21%     
==========================================
  Files          86       87       +1     
  Lines        1999     2053      +54     
==========================================
+ Hits         1728     1750      +22     
- Misses        271      303      +32     

Files	Coverage Δ
src/airflow/dags/common_airflow.py	88.88% <25.00%> (-11.12%)	⬇️
src/airflow/dags/gwas_catalog_harmonisation.py	43.47% <43.47%> (ø)

[d0choa]

This DAG was used to complete the harmonisation of 25k harmonised GWAS Catalog studies.

It's currently limited to 5,000 studies at a time, which I think is a good safety net.

Similar to the other DAGs is a pragmatic solution that does the job, not a fully bulletproof implementation.

Requesting review from @ireneisdoomed because it interfaces with downstream work, but happy to listen to anybody else

[tskir]

I think this is a good interim solution which allows us to ingest and start using GWAS Catalog studies quickly.

However, I strongly believe that for production use we should not use direct orchestration and submission of jobs. I did that for my first ingestion of FinnGen, and can testify that it's not CPU efficient and seems to make the cluster less stable.

As we discovered earlier, Spark is also not great at ingesting non-Parquet files and/or partitioning them.

So I suggest that going forward, we use the non-Spark preprocessing approach developed in #295 + #296 + #297. This should be extremely fast and make downstream processing incredibly simple. Coincidentally, this will also remove the need to do rsync from the EBI side, or to do anything with the EBI infrastructure at all.

@d0choa If it's all right with you, I would keep these changes in a separate branch for now. I believe that if we merge this PR, we'll have to dismantle and rewrite this part again very soon.

However, if you'd prefer merging at least some working solution into main, I won't argue the point, and if it is the case, please let me know and I'll approve it as is.

[ireneisdoomed]

Great job, works as expected. This DAG is meant to be run independently from the other GWAS Catalog preprocess.