Skip to content

Commit

Permalink
fix: all files reformated by prettier
Browse files Browse the repository at this point in the history
  • Loading branch information
d0choa committed Dec 3, 2023
1 parent 550814f commit bad037f
Show file tree
Hide file tree
Showing 55 changed files with 382 additions and 372 deletions.
4 changes: 1 addition & 3 deletions .vscode/extensions.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,5 @@
"ms-python.black-formatter",
"esbenp.prettier-vscode"
],
"unwantedRecommendations": [
"ms-python.flake8"
]
"unwantedRecommendations": ["ms-python.flake8"]
}
9 changes: 2 additions & 7 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,7 @@
"program": "${file}",
"console": "integratedTerminal",
"justMyCode": true,
"args": [
"--config-dir=./configs"
]
"args": ["--config-dir=./configs"]
},
{
"name": "Python: Current File [local]",
Expand All @@ -19,10 +17,7 @@
"program": "${file}",
"console": "integratedTerminal",
"justMyCode": true,
"args": [
"--config-dir=./configs",
"environment=local"
]
"args": ["--config-dir=./configs", "environment=local"]
}
]
}
17 changes: 3 additions & 14 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,7 @@
"source.organizeImports": true
}
},
"python.terminal.launchArgs": [
"-m",
"IPython",
"--no-autoindent"
],
"python.terminal.launchArgs": ["-m", "IPython", "--no-autoindent"],
"[jsonc]": {
"editor.tabSize": 2,
"editor.insertSpaces": true,
Expand All @@ -23,16 +19,9 @@
}
},
"json.format.keepLines": true,
"isort.args": [
"--profile",
"black"
],
"isort.args": ["--profile", "black"],
"autoDocstring.docstringFormat": "google",
"python.testing.pytestArgs": [
".",
"--doctest-modules",
"--cov=src/"
],
"python.testing.pytestArgs": [".", "--doctest-modules", "--cov=src/"],
"python.testing.unittestEnabled": false,
"python.testing.pytestEnabled": true,
"mypy-type-checker.severity": {
Expand Down
366 changes: 183 additions & 183 deletions LICENSE.md

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,5 @@
[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/opentargets/genetics_etl_python/main.svg)](https://results.pre-commit.ci/badge/github/opentargets/genetics_etl_python)

# Genetics Portal Data Pipeline (experimental)

- [Documentation](https://opentargets.github.io/genetics_etl_python/)
3 changes: 1 addition & 2 deletions codecov.yml
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@

comment:
layout: "reach, diff, flags, files"
behavior: default
require_changes: false # if true: only post the comment if coverage changes
require_changes: false # if true: only post the comment if coverage changes

coverage:
status:
Expand Down
2 changes: 1 addition & 1 deletion commitlint.config.js
Original file line number Diff line number Diff line change
@@ -1 +1 @@
module.exports = {extends: ['@commitlint/config-conventional']};
module.exports = { extends: ["@commitlint/config-conventional"] };
6 changes: 3 additions & 3 deletions docs/assets/javascripts/extra.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
$('.contributors img[data-src]').each(function () {
src = $(this).attr("data-src");
$(this).attr('src', src);
$(".contributors img[data-src]").each(function () {
src = $(this).attr("data-src");
$(this).attr("src", src);
});
86 changes: 45 additions & 41 deletions docs/assets/overrides/partials/source-file.html
Original file line number Diff line number Diff line change
@@ -1,46 +1,50 @@
<hr />
<div class="md-source-file">
<small>
<ul class="metadata page-metadata" data-bi-name="page info" lang="en-us" dir="ltr">
<!-- mkdocs-git-revision-date-plugin -->
{% if page.meta.git_creation_date_localized %}
<li>
<span class="icon twemoji">
{% include ".icons/material/clock-plus-outline.svg" %}
</span>
<span class="label">
{{ page.meta.git_creation_date_localized }}
</span>
</li>
{% endif %}
<small>
<ul
class="metadata page-metadata"
data-bi-name="page info"
lang="en-us"
dir="ltr"
>
<!-- mkdocs-git-revision-date-plugin -->
{% if page.meta.git_creation_date_localized %}
<li>
<span class="icon twemoji">
{% include ".icons/material/clock-plus-outline.svg" %}
</span>
<span class="label"> {{ page.meta.git_creation_date_localized }} </span>
</li>
{% endif %} {% if page.meta.git_revision_date_localized %}
<li>
<span class="icon twemoji">
{% include ".icons/material/clock-edit-outline.svg" %}
</span>
<span class="label"> {{ page.meta.git_revision_date_localized }} </span>
</li>
{% endif %}

{% if page.meta.git_revision_date_localized %}
<li>
<span class="icon twemoji">
{% include ".icons/material/clock-edit-outline.svg" %}
</span>
<span class="label">
{{ page.meta.git_revision_date_localized }}
</span>
</li>
{% endif %}

<li>
<span class="icon twemoji">
{% include ".icons/material/github.svg" %}
</span>
<span class="label">
Contributors
</span>
</li>
<li class="contributors-holder">
<ul class="contributors" data-bi-name="contributors">
{%- for user in committers -%}
<li><a href="{{ user.url }}" title="{{ user.name }}" data-bi-name="contributorprofile"
target="_blank"><img src="{{ user.avatar }}" alt="{{ user.name }}"></a></li>
{%- endfor -%}
</ul>
</li>
<li>
<span class="icon twemoji">
{% include ".icons/material/github.svg" %}
</span>
<span class="label"> Contributors </span>
</li>
<li class="contributors-holder">
<ul class="contributors" data-bi-name="contributors">
{%- for user in committers -%}
<li>
<a
href="{{ user.url }}"
title="{{ user.name }}"
data-bi-name="contributorprofile"
target="_blank"
><img src="{{ user.avatar }}" alt="{{ user.name }}"
/></a>
</li>
{%- endfor -%}
</ul>
</small>
</li>
</ul>
</small>
</div>
90 changes: 43 additions & 47 deletions docs/assets/stylesheets/extra.css
Original file line number Diff line number Diff line change
@@ -1,79 +1,75 @@
.metadata {
list-style: none;
padding: 0;
margin: 0;
margin-bottom: 15px;
color: #999;
list-style: none;
padding: 0;
margin: 0;
margin-bottom: 15px;
color: #999;
}

.metadata.page-metadata .contributors-text {
margin-right: 5px;
display: inline-block;
margin-right: 5px;
display: inline-block;
}

.metadata.page-metadata {
display: flex !important;
flex-direction: row;
align-items: center;
font-size: 1.2em;
display: flex !important;
flex-direction: row;
align-items: center;
font-size: 1.2em;
}


.metadata.page-metadata li {
list-style: none;
display: flex;
flex-direction: row;
justify-content: center;
align-items: center;
margin-bottom: 0 !important;
list-style: none;
display: flex;
flex-direction: row;
justify-content: center;
align-items: center;
margin-bottom: 0 !important;
}


.metadata.page-metadata li .icon {
width: 1.1rem;
height: 1.1rem;
margin-right: 2px;
width: 1.1rem;
height: 1.1rem;
margin-right: 2px;
}


.metadata.page-metadata li .icon svg {
fill: #999;
fill: #999;
}


.page-metadata .contributors-holder {
margin: 0 !important;
margin-left: 1em;
margin: 0 !important;
margin-left: 1em;
}

.page-metadata .contributors {
margin-right: 5px;
display: inline-block;
list-style: none;
display: flex;
flex-direction: row;
margin-right: 5px;
display: inline-block;
list-style: none;
display: flex;
flex-direction: row;
}

.page-metadata .contributors li {
display: inline-flex;
margin: 0 !important;
padding: 0 !important;
display: inline-flex;
margin: 0 !important;
padding: 0 !important;
}

.page-metadata .contributors li img {
border-radius: 50%;
filter: grayscale(100%);
-webkit-filter: grayscale(100%);
opacity: 0.8;
vertical-align: middle;
width: 1.6rem;
transition: all 0.2s ease-in-out;
border-radius: 50%;
filter: grayscale(100%);
-webkit-filter: grayscale(100%);
opacity: 0.8;
vertical-align: middle;
width: 1.6rem;
transition: all 0.2s ease-in-out;
}

.page-metadata .contributors li img:hover {
opacity: 1;
filter: grayscale(0%);
-webkit-filter: grayscale(0%);
vertical-align: middle;
transform: scale(1.2);
opacity: 1;
filter: grayscale(0%);
-webkit-filter: grayscale(0%);
vertical-align: middle;
transform: scale(1.2);
}
28 changes: 12 additions & 16 deletions docs/development/airflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,12 @@ This section describes how to set up a local Airflow server which will orchestra
- [Google Cloud SDK](https://cloud.google.com/sdk/docs/install)

!!!warning macOS Docker memory allocation
On macOS, the default amount of memory available for Docker might not be enough to get Airflow up and running. Allocate at least 4GB of memory for the Docker Engine (ideally 8GB). [More info](https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#)

On macOS, the default amount of memory available for Docker might not be enough to get Airflow up and running. Allocate at least 4GB of memory for the Docker Engine (ideally 8GB). [More info](https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#)

## Configure Airflow access to Google Cloud Platform

!!!warning Specifying Google Cloud parameters
Run the next two command with the appropriate Google Cloud project ID and service account name to ensure the correct Google default application credentials are set up.
Run the next two command with the appropriate Google Cloud project ID and service account name to ensure the correct Google default application credentials are set up.

Authenticate to Google Cloud:

Expand All @@ -28,7 +27,6 @@ Create the service account key file that will be used by Airflow to access Googl
gcloud iam service-accounts keys create ~/.config/gcloud/service_account_credentials.json --iam-account=<PROJECT>@appspot.gserviceaccount.com
```


## Set up Airflow

Change the working directory so that all subsequent commands will work:
Expand All @@ -40,7 +38,7 @@ cd src/airflow
### Build Docker image

!!!note Custom Docker image for Airflow
The custom Dockerfile built by the command below extends the official [Airflow Docker Compose YAML](https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml). We add support for Google Cloud SDK, Google Dataproc operators, and access to GCP credentials.
The custom Dockerfile built by the command below extends the official [Airflow Docker Compose YAML](https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml). We add support for Google Cloud SDK, Google Dataproc operators, and access to GCP credentials.

```bash
docker build . --tag extending_airflow:latest
Expand All @@ -49,7 +47,7 @@ docker build . --tag extending_airflow:latest
### Set Airflow user ID

!!!note Setting Airflow user ID
These commands allow Airflow running inside Docker to access the credentials file which was generated earlier.
These commands allow Airflow running inside Docker to access the credentials file which was generated earlier.

```bash
# If any user ID is already specified in .env, remove it.
Expand Down Expand Up @@ -78,7 +76,6 @@ Airflow UI will now be available at `http://localhost:8080/`. Default username a

For additional information on how to use Airflow visit the [official documentation](https://airflow.apache.org/docs/apache-airflow/stable/index.html).


### Cleaning up

At any time, you can check the status of your containers with:
Expand All @@ -105,18 +102,17 @@ More information on running Airflow with Docker Compose can be found in the [off

1. **Increase Airflow concurrency**. Modify the `docker-compose.yaml` and add the following to the x-airflow-common → environment section:

```yaml
AIRFLOW__CORE__PARALLELISM: 32
AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG: 32
AIRFLOW__SCHEDULER__MAX_TIS_PER_QUERY: 16
AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG: 1
# Also add the following line if you are using CeleryExecutor (by default, LocalExecutor is used).
AIRFLOW__CELERY__WORKER_CONCURRENCY: 32
```
```yaml
AIRFLOW__CORE__PARALLELISM: 32
AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG: 32
AIRFLOW__SCHEDULER__MAX_TIS_PER_QUERY: 16
AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG: 1
# Also add the following line if you are using CeleryExecutor (by default, LocalExecutor is used).
AIRFLOW__CELERY__WORKER_CONCURRENCY: 32
```
1. **Additional pip packages**. They can be added to the `requirements.txt` file.


## Troubleshooting

Note that when you a a new workflow under `dags/`, Airflow will not pick that up immediately. By default the filesystem is only scanned for new DAGs every 300s. However, once the DAG is added, updates are applied nearly instantaneously.
Expand Down
Loading

0 comments on commit bad037f

Please sign in to comment.