Merge pull request #4 from opentargets/il-multiple-testing

P value multiple correction, permutation testing, and reason/phase association

.gitignore

@@ -8,4 +8,8 @@ derby.log
 .vscode/dryrun.log
 .vscode/targets.log
 .vscode/configurationCache.log
-temp
+temp
+data/baseline_predictions_aggregations
+data/predictions_aggregations_*
+data/chembl*
+reports/*

analysis/README.md

@@ -6,7 +6,7 @@ This pipeline consists of the following steps:
 - Enrichment analysis.
 To launch the script as a Pyspark job on a cluster:
 ```bash
-gcloud dataproc clusters create il-big-stop-reasons  \
+gcloud dataproc clusters create il-big-stop-reasons-2  \
         --image-version=2.1 \
         --project=open-targets-eu-dev \
         --region=europe-west1 \
@@ -24,5 +24,19 @@ gcloud dataproc jobs submit pyspark  \
         --region=europe-west1 \
         --project=open-targets-eu-dev \
         analysis/python/enrichments.py -- config.yml --stratify-therapeutic-area non_oncology
+
+# To determine a baseline between all sets of comparisons, run the analysis with a randomly generated set of associations
+gcloud dataproc jobs submit pyspark  \
+        --cluster=il-big-stop-reasons \
+        --files=config.yml \
+        --region=europe-west1 \
+        --project=open-targets-eu-dev
+
+# Adjust the p values for multiple testing using the Benjamini-Hochberg procedure by providing the path to the file with the results of the enrichment analysis
+gcloud dataproc jobs submit pyspark  \
+        --cluster=il-big-stop-reasons \
+        --region=europe-west1 \
+        --project=open-targets-eu-dev \
+        analysis/python/multiple_testing_correction.py  -- "gs://ot-team/irene/stop_reasons/predictions_aggregations_non_oncology"
 ```
 - Visualization of the results.

analysis/python/enrichments.py

@@ -2,7 +2,6 @@
 from enum import Enum
 from functools import reduce
 
-import pandas as pd
 import pyspark.sql.functions as F
 import typer
 from omegaconf import OmegaConf