Skip to content

chore: bump dataproc image version #94

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jan 28, 2025
Merged

chore: bump dataproc image version #94

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fdcc26a
chore: bump dataproc image version
SzymonSzyszkowski Jan 17, 2025
de886c2
Merge branch 'dev' into spark-image
project-defiant Jan 23, 2025
1486458
feat: install package from git
Jan 23, 2025
829218d
Merge branch 'dev' into spark-image
project-defiant Jan 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions src/ot_orchestration/dags/config/credible_set_qc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
dataproc:
python_main_module: gs://genetics_etl_python_playground/initialisation/gentropy/dev/cli.py
python_main_module: gs://genetics_etl_python_playground/initialisation/cli.py
cluster_metadata:
PACKAGE: gs://genetics_etl_python_playground/initialisation/gentropy/dev/gentropy-0.0.0-py3-none-any.whl
cluster_init_script: gs://genetics_etl_python_playground/initialisation/gentropy/dev/install_dependencies_on_cluster.sh
GENTROPY_REF: v2.0.1
cluster_init_script: gs://genetics_etl_python_playground/initialisation/install_dependencies_on_cluster.sh
cluster_name: otg-credible-set-qc
autoscaling_policy: otg-etl

Expand Down
6 changes: 3 additions & 3 deletions src/ot_orchestration/dags/config/eqtl_catalogue_ingestion.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ eqtl_catalogue_decompressed_susie_path: gs://eqtl_catalogue_data/ebi_ftp/susie_d
decompression_logs: gs://eqtl_catalogue_data/ebi_ftp/susie_decompressed_tmp/logs.log

dataproc:
python_main_module: gs://genetics_etl_python_playground/initialisation/gentropy/dev/cli.py
python_main_module: gs://genetics_etl_python_playground/initialisation/cli.py
cluster_metadata:
PACKAGE: gs://genetics_etl_python_playground/initialisation/gentropy/dev/gentropy-0.0.0-py3-none-any.whl
cluster_init_script: gs://genetics_etl_python_playground/initialisation/gentropy/dev/install_dependencies_on_cluster.sh
GENTROPY_REF: v2.0.1
cluster_init_script: gs://genetics_etl_python_playground/initialisation/install_dependencies_on_cluster.sh
cluster_name: otg-preprocess-eqtl
autoscaling_policy: eqtl-preprocess
num_workers: 4
Expand Down
6 changes: 3 additions & 3 deletions src/ot_orchestration/dags/config/finngen_ingestion.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
dataproc:
python_main_module: gs://genetics_etl_python_playground/initialisation/gentropy/dev/cli.py
python_main_module: gs://genetics_etl_python_playground/initialisation/cli.py
cluster_metadata:
PACKAGE: gs://genetics_etl_python_playground/initialisation/gentropy/dev/gentropy-0.0.0-py3-none-any.whl
cluster_init_script: gs://genetics_etl_python_playground/initialisation/gentropy/dev/install_dependencies_on_cluster.sh
GENTROPY_REF: v2.0.1
cluster_init_script: gs://genetics_etl_python_playground/initialisation/install_dependencies_on_cluster.sh
cluster_name: otg-ingestion-finngen
autoscaling_policy: finngen-preprocess
master_disk_size: 2000
Expand Down
6 changes: 3 additions & 3 deletions src/ot_orchestration/dags/config/finngen_ukb_meta_harmonisation.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
dataproc:
python_main_module: gs://genetics_etl_python_playground/initialisation/gentropy/szsz-update-package-for-dataproc-run/cli.py
python_main_module: gs://genetics_etl_python_playground/initialisation//cli.py
cluster_metadata:
PACKAGE: gs://genetics_etl_python_playground/initialisation/gentropy/szsz-update-package-for-dataproc-run/gentropy-0.0.0-py3-none-any.whl
cluster_init_script: gs://genetics_etl_python_playground/initialisation/0.0.0/install_dependencies_on_cluster.sh
GENTROPY_REF: v2.0.1
cluster_init_script: gs://genetics_etl_python_playground/install_dependencies_on_cluster.sh
cluster_name: otg-finngen-ukb-meta

nodes:
Expand Down
7 changes: 4 additions & 3 deletions src/ot_orchestration/dags/config/foldx_ingestion.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
---
dataproc:
python_main_module: gs://genetics_etl_python_playground/initialisation/gentropy/dev/cli.py
python_main_module: gs://genetics_etl_python_playground/initialisation/cli.py
cluster_metadata:
PACKAGE: gs://genetics_etl_python_playground/initialisation/gentropy/dev/gentropy-0.0.0-py3-none-any.whl
cluster_init_script: gs://genetics_etl_python_playground/initialisation/gentropy/dev/install_dependencies_on_cluster.sh
GENTROPY_REF: v2.0.1
cluster_init_script: gs://genetics_etl_python_playground/initialisation/install_dependencies_on_cluster.sh

cluster_name: foldx-ingestion
autoscaling_policy: otg-etl
allow_efm: false
Expand Down
8 changes: 4 additions & 4 deletions src/ot_orchestration/dags/config/genetics_etl.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@
l2g_gold_standard_path: gs://genetics_etl_python_playground/input/l2g/gold_standard/curation.json
release_dir: gs://ot_orchestration/releases/24.11_freeze10
dataproc:
python_main_module: gs://genetics_etl_python_playground/initialisation/gentropy/dev/cli.py
python_main_module: gs://genetics_etl_python_playground/initialisation/cli.py
cluster_metadata:
PACKAGE: gs://genetics_etl_python_playground/initialisation/gentropy/dev/gentropy-0.0.0-py3-none-any.whl
cluster_init_script: gs://genetics_etl_python_playground/initialisation/gentropy/dev/install_dependencies_on_cluster.sh
cluster_name: otg-etl
GENTROPY_REF: v2.0.1
cluster_init_script: gs://genetics_etl_python_playground/initialisation/install_dependencies_on_cluster.sh
cluster_name: test-dataproc
autoscaling_policy: otg-etl
allow_efm: false
num_workers: 2
Expand Down
7 changes: 4 additions & 3 deletions src/ot_orchestration/dags/config/gentropy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
---
python_main_module: gs://genetics_etl_python_playground/initialisation/gentropy/v{gentropy_version}/cli.py

python_main_module: gs://genetics_etl_python_playground/initialisation/cli.py

dataproc_cluster_settings:
cluster_metadata:
PACKAGE: gs://genetics_etl_python_playground/initialisation/gentropy/v{gentropy_version}/gentropy-{gentropy_version}-py3-none-any.whl
cluster_init_script: gs://genetics_etl_python_playground/initialisation/gentropy/v{gentropy_version}/install_dependencies_on_cluster.sh
GENTROPY_REF: {gentropy_version}
cluster_init_script: gs://genetics_etl_python_playground/initialisation/install_dependencies_on_cluster.sh
autoscaling_policy: otg-etl
allow_efm: false
num_workers: 2
Expand Down
6 changes: 3 additions & 3 deletions src/ot_orchestration/dags/config/gnomad_ingestion.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
dataproc:
python_main_module: gs://genetics_etl_python_playground/initialisation/gentropy/dev/cli.py
python_main_module: gs://genetics_etl_python_playground/initialisation/cli.py
cluster_metadata:
PACKAGE: gs://genetics_etl_python_playground/initialisation/gentropy/dev/gentropy-0.0.0-py3-none-any.whl
cluster_init_script: gs://genetics_etl_python_playground/initialisation/gentropy/dev/install_dependencies_on_cluster.sh
GENTROPY_REF: v2.0.1
cluster_init_script: gs://genetics_etl_python_playground/initialisation/install_dependencies_on_cluster.sh
cluster_name: gnomad-preprocess
autoscaling_policy: otg-etl
allow_efm: false
Expand Down
6 changes: 3 additions & 3 deletions src/ot_orchestration/dags/config/gwas_catalog_sumstats_pics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
dataproc:
cluster_name: otg-gwascatalog-sumstats-pics
autoscaling_policy: otg-preprocess-gwascatalog
python_main_module: gs://genetics_etl_python_playground/initialisation/gentropy/dev/cli.py
python_main_module: gs://genetics_etl_python_playground/initialisation/cli.py
cluster_metadata:
PACKAGE: gs://genetics_etl_python_playground/initialisation/gentropy/dev/gentropy-0.0.0-py3-none-any.whl
cluster_init_script: gs://genetics_etl_python_playground/initialisation/gentropy/dev/install_dependencies_on_cluster.sh
GENTROPY_REF: v2.0.1
cluster_init_script: gs://genetics_etl_python_playground/initialisation/install_dependencies_on_cluster.sh
num_workers: 5

nodes:
Expand Down
6 changes: 3 additions & 3 deletions src/ot_orchestration/dags/config/gwas_catalog_sumstats_susie_clumping.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
dataproc:
python_main_module: gs://genetics_etl_python_playground/initialisation/gentropy/dev/cli.py
python_main_module: gs://genetics_etl_python_playground/initialisation/cli.py
cluster_metadata:
PACKAGE: gs://genetics_etl_python_playground/initialisation/gentropy/dev/gentropy-0.0.0-py3-none-any.whl
cluster_init_script: gs://genetics_etl_python_playground/initialisation/gentropy/dev/install_dependencies_on_cluster.sh
GENTROPY_REF: v2.0.1
cluster_init_script: gs://genetics_etl_python_playground/initialisation/install_dependencies_on_cluster.sh
cluster_name: otg-gwas-catalog-sumstat-susie-clumping
autoscaling_policy: otg-etl

Expand Down
6 changes: 3 additions & 3 deletions src/ot_orchestration/dags/config/gwas_catalog_top_hits.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
dataproc:
cluster_name: otg-tophit-gwascatalog
autoscaling_policy: otg-gwascatalog-tophit
python_main_module: gs://genetics_etl_python_playground/initialisation/gentropy/dev/cli.py
python_main_module: gs://genetics_etl_python_playground/initialisation/cli.py
cluster_metadata:
PACKAGE: gs://genetics_etl_python_playground/initialisation/gentropy/dev/gentropy-0.0.0-py3-none-any.whl
cluster_init_script: gs://genetics_etl_python_playground/initialisation/gentropy/dev/install_dependencies_on_cluster.sh
GENTROPY_REF: v2.0.1
cluster_init_script: gs://genetics_etl_python_playground/initialisation/install_dependencies_on_cluster.sh
num_workers: 2

nodes:
Expand Down
6 changes: 3 additions & 3 deletions src/ot_orchestration/dags/config/ukb_ppp_eur_harmonisation.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
dataproc:
python_main_module: gs://genetics_etl_python_playground/initialisation/gentropy/dev/cli.py
python_main_module: gs://genetics_etl_python_playground/initialisation/cli.py
cluster_metadata:
PACKAGE: gs://genetics_etl_python_playground/initialisation/gentropy/dev/gentropy-0.0.0-py3-none-any.whl
cluster_init_script: gs://genetics_etl_python_playground/initialisation/gentropy/dev/install_dependencies_on_cluster.sh
GENTROPY_REF: v2.0.1
cluster_init_script: gs://genetics_etl_python_playground/initialisation/install_dependencies_on_cluster.sh
cluster_name: otg-ukb-ppp-eur
autoscaling_policy: otg-etl

Expand Down
2 changes: 1 addition & 1 deletion src/ot_orchestration/utils/common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
GCP_PROJECT_PLATFORM = "open-targets-eu-dev"
GCP_REGION = "europe-west1"
GCP_ZONE = "europe-west1-d"
GCP_DATAPROC_IMAGE = "2.1"
GCP_DATAPROC_IMAGE = "2.2"
GCP_AUTOSCALING_POLICY = "otg-etl"
GCP_EFM_AUTOSCALING_POLICY = "otg-efm"

Expand Down
5 changes: 3 additions & 2 deletions src/ot_orchestration/utils/dataproc.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ def create_cluster(
num_preemptible_workers=num_preemptible_workers,
num_workers=num_workers,
image_version=GCP_DATAPROC_IMAGE,
internal_ip_only=False,
enable_component_gateway=True,
metadata=cluster_metadata,
idle_delete_ttl=idle_delete_ttl,
Expand Down Expand Up @@ -229,8 +230,8 @@ def submit_pyspark_job(
"main_python_file_uri": python_main_module,
"args": args,
"properties": {
"spark.jars": "/opt/conda/miniconda3/lib/python3.10/site-packages/hail/backend/hail-all-spark.jar",
"spark.driver.extraClassPath": "/opt/conda/miniconda3/lib/python3.10/site-packages/hail/backend/hail-all-spark.jar",
"spark.jars": "/opt/conda/miniconda3/lib/python3.11/site-packages/hail/backend/hail-all-spark.jar",
"spark.driver.extraClassPath": "/opt/conda/miniconda3/lib/python3.11/site-packages/hail/backend/hail-all-spark.jar",
"spark.executor.extraClassPath": "./hail-all-spark.jar",
"spark.serializer": "org.apache.spark.serializer.KryoSerializer",
"spark.kryo.registrator": "is.hail.kryo.HailKryoRegistrator",
Expand Down
Loading