feat: gwas catalog top-hit + study step #808
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
documentation
|
@DSuveges, this creates the top-hits logic. It should be ready. Next Sumstats + PICS (and cleanup) |
d0choa
commented
Oct 2, 2024
| @@ -332,7 +330,6 @@ def from_source( | |||
| return ( | |||
| cls._parse_study_table(catalog_studies) | |||
| .annotate_ancestries(ancestry_file) | |||
| .annotate_sumstats_info(sumstats_lut) | |||
|
I'm having second thoughts. Because the 2 DAGs have interdependencies in the business logic, I want to see both working before we review this. I'm reverting to draft |
d0choa
changed the title
Example how to run the step:poetry run gentropy step=gwas_catalog_study_index \
step.catalog_study_files="[ '/Users/ochoa/Datasets/gwas_catalog_download_studies.tsv' ]" \
step.catalog_ancestry_files="[ '/Users/ochoa/Datasets/gwas_catalog_download_ancestries.tsv' ]" \
step.study_index_path=/Users/ochoa/Datasets/study_index_annotated \
step.gwas_catalog_study_curation_file=/Users/ochoa/Datasets/20241004_output_curation.tsv \
step.sumstats_qc_path=/Users/ochoa/Datasets/20241015_GwasCatQCLogsCopy of the outputsBreakdown of GWAS Catalog flags |
d0choa
requested review from
project-defiant
and removed request for
DSuveges
There was a problem hiding this comment.
Business logic looks good. There is minor bug might break the step when reading the curation file. (Suspect that this is copilot issue).
The logic of this step (and others) raises a single concern. The VariantIndex is calculated at the end of the genetics_etl. Yet it is required for this step (also for some other ingestions like ukb_ppp).
| gwas_catalog_study_curation = StudyIndexGWASCatalogOTCuration.from_csv( | ||
| session, gwas_catalog_study_curation_file | ||
| ) | ||
| elif gwas_catalog_study_curation_file.startswith("http"): |
There was a problem hiding this comment.
Swap the order, the http can end with .csv as well. Then it will not work.
| ) | ||
|
|
||
| if gwas_catalog_study_curation_file: | ||
| if gwas_catalog_study_curation_file.endswith(".csv"): |
There was a problem hiding this comment.
Should it not be a .tsv as in the GWASCatalogStudyIndexGenerationStep
Comment on lines
+57
to
+67
| if gwas_catalog_study_curation_file: | ||
| if gwas_catalog_study_curation_file.endswith( | ||
| ".tsv" | ||
| ) | gwas_catalog_study_curation_file.endswith(".tsv"): | ||
| gwas_catalog_study_curation = StudyIndexGWASCatalogOTCuration.from_csv( | ||
| session, gwas_catalog_study_curation_file | ||
| ) | ||
| elif gwas_catalog_study_curation_file.startswith("http"): | ||
| gwas_catalog_study_curation = StudyIndexGWASCatalogOTCuration.from_url( | ||
| session, gwas_catalog_study_curation_file | ||
| ) |
There was a problem hiding this comment.
I am not sure what the code was meant to do here, although I suspect that it could have inferr if the file is http based or not, then based on that if the file is a csv or tsv ??
|
|
||
| # Annotate with sumstats QC if provided: | ||
| if sumstats_qc_path: | ||
| schema = StructType( |
|
|
||
| !!! note This step currently only processes the GWAS Catalog curated list of top hits. | ||
| """ | ||
| class GWASCatalogTopHitIngestionStep: |
There was a problem hiding this comment.
NIT, not consistent, everywhere we say these are top hits
| """ | ||
| # Extract | ||
| gnomad_variants = VariantIndex.from_parquet(session, gnomad_variant_path) | ||
| gnomad_variants = VariantIndex.from_parquet(session, variant_annotation_path) |
There was a problem hiding this comment.
In this particullar case. Which VariantIndex dataset should be used to generate the top hits correctly?
There was a problem hiding this comment.
In theory, it should be the gnomAD variant annotation dataset in the schema of a variantIndex. We haven't run it for a while so I wonder if everything works fine there after the ETL VariantIndex work. This DAG should have no dependency on the ETL VariantIndex
|
Added changes in ancestry mapping in gwas_population_2_LD_panel_map.json since we are going to remake the study index. These changes add CSA (central south asians) from UKBB to the list of ancestries. |
d0choa
commented
Oct 18, 2024
There was a problem hiding this comment.
@addramir are you sure this won't break PICS if LDIndex does not contain csa?
Can you please double-check?
There was a problem hiding this comment.
@addramir In that case R will be 0 and the variant should be dropped before PICS.
There was a problem hiding this comment.
In this case we need to revert the change with changing mapping. I will add a quick temporary fix in susie but in the future we have to change that.
DSuveges
approved these changes
Oct 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Step to run the top-hits in isolation of everything else.
The
GWASCatalogTopHitIngestionStepincluded here was run in dataproc in 17 minutes.Subsequent PRs will handle GWAS Catalog + Summary Statistics + PICS road which might result in removing some steps.