Skip to content

Commit

Permalink
Merge branch 'dev-bfabric-2' of github.com:fgcz/bfabricPy into dev-bf…
Browse files Browse the repository at this point in the history 
…abric-2
  • Loading branch information
Aleksejs Fomins committed May 17, 2024
2 parents 88ab710 + 7ef5729 commit b9e47f0
Show file tree
Hide file tree
Showing 60 changed files with 1,269 additions and 1,048 deletions.
132 changes: 78 additions & 54 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,55 +1,70 @@
![functionTest](https://github.com/fgcz/bfabricPy/workflows/functionalTest/badge.svg)
![unitTests](https://github.com/fgcz/bfabricPy/workflows/unit%20tests/badge.svg)
[![EDBT'10](https://img.shields.io/badge/EDBT-10.1145%2F1739041.1739135-brightgreen)](https://doi.org/10.1145/1739041.1739135)
[![JIB](https://img.shields.io/badge/JIB-10.1515%2Fjib.2022.0031-brightgreen)](https://doi.org/10.1515/jib-2022-0031)
![Downloads](https://img.shields.io/github/downloads/fgcz/bfabricPy/total)



# bfabricPy

This package connects the [bfabric](https://fgcz-bfabric.uzh.ch/bfabric/) system to the [python](https://www.python.org/) and [R](https://cran.r-project.org/) world while providing a JSON and REST interface using [Flask](https://www.fullstackpython.com).
The [bfabricShiny](https://github.com/cpanse/bfabricShiny) R package is an extension and provides code snippets and sample implementation for a seamless R shiny bfabric integration.
For more advanced users the *bfabricPy* package also provides a powerful query interface on the command-line though using the provided scripts.

## Install
The package can be installed like any other Python package, so if you are familiar you might not need to read this section.
Currently, it's only available from GitHub.

![bfabricPy-read](https://user-images.githubusercontent.com/4901987/65025926-db77c900-d937-11e9-8c92-f2412d6793ee.gif)
[see also #14](https://github.com/fgcz/bfabricPy/issues/14)
The best way to install the package depends on your use case, i.e. whether you want to:

1. Use the command line scripts
2. Use the Python API
3. Develop on the package

## Install
There are many ways to install Python packages.
Generally it's recommended to use some type of virtual environment manager, like [conda](https://docs.conda.io/en/latest/), [uv](https://github.com/astral-sh/uv), or Python's [venv](https://docs.python.org/3/library/venv.html). Then the following commands work.
If you don't, you might need to specify `--user` to the pip commands, so they get installed into the user's Python package directory.
The command line scripts are currently included in all cases.

To use bfabricPy a normal installation is good enough:
```{bash}
pip install git+https://github.com/fgcz/bfabricPy.git
### Command line scripts
To use the command line scripts, it's recommended to install `bfabricPy` with [pipx](https://pipx.pypa.io/).
If you don't have `pipx` installed, refer to the [pipx documentation](https://pipx.pypa.io/stable/installation/) for instructions.

You can execute a command using a specific version of `bfabricPy` with the `pipx run` command.
This command handles the dependencies of multiple concurrent installations:

```bash
pipx run --spec "git+https://github.com/fgcz/bfabricPy.git@0.13.8" bfabric_read.py --help
```

To install a specific version of bfabricPy on your system and make the command available without `pipx run` prefix, use the following command:
```bash
pipx install "git+https://github.com/fgcz/bfabricPy.git@0.13.8"
bfabric_read.py --help
```

As a user: (i.e. a regular install, files will be used from your current directory instead of properly installing a copy of it)
### Python API
If you're interested in using the Python API of `bfabricPy`, you have two options:

```{bash}
# variant 1) clone to a folder
git clone https://github.com/fgcz/bfabricPy.git && cd bfabricPy
pip install .
#### 1. Configure it in your `pyproject.toml` file.
```toml
[project]
dependencies = [
"bfabricPy @ git+https://github.com/fgcz/bfabricPy.git@main"
]
```

# variant 2) direct install from GitHub
#### 2. Install the `bfabricPy` package directly using pip.
```bash
pip install git+https://github.com/fgcz/bfabricPy.git
```
````

### Development
As a bfabricPy developer: (i.e. an editable install)

```{bash}
pip install -e ".[dev]"
```

## Configuration [outdated]

```{bash}
cat ~/.bfabricpy.yml
```
## Configuration
Create a file as follows: (note: the password is not your login password, but the web service password)

```{yaml}
# ~/.bfabricpy.yml
GENERAL:
default_config: PRODUCTION
Expand All @@ -59,6 +74,15 @@ PRODUCTION:
base_url: https://fgcz-bfabric.uzh.ch/bfabric
```

You can also include an additional config for the TEST instance

```{yaml}
TEST:
login: yourBfabricLogin
password: yourBfabricWebPassword
base_url: https://fgcz-bfabric-test.uzh.ch/bfabric
```

## CheatSheet

### Read
Expand All @@ -83,41 +107,50 @@ bfabric_read.py workunit status failed
bfabric_read.py resource filechecksum d41d8cd98f00b204e9800998ecf8427e
```

call the `python3` interpreter and enter
Using the Python API:

```{py}
import bfabric
from bfabric import Bfabric
B = bfabric.Bfabric()
client = Bfabric.from.config()
user = B.read_object(endpoint = 'user', obj={'login': 'cpanse'})
resource = B.read_object(endpoint = 'resource', obj={'id': 550327 })
user = B.read(endpoint = 'user', obj={'login': 'cpanse'})
resource = B.read(endpoint = 'resource', obj={'id': 550327 })
```

### save
```{bash}
bfabric_save_workunit_attribute.py 199387 status available
```
rv = B.save_object('workunit', {'id': 254063, 'status': 'available'})
B.print_json(rv)
# print(rv)
```

### Command line code snippets

remove pending workunits from the past
```{bash}
bfabric_read.py workunit status pending \
| awk '$2~/cpanse/ && $3~/2015/{print $1}'
| fgcz_bfabric_delete_workunits.py
```{python}
import json
rv = client.save('workunit', {'id': 254063, 'status': 'available'})
print(json.dumps(rv.to_list_dict(), indent=2))
```

find empty resource files in bfabric
### Command line code snippet
Find empty resource files in bfabric
```{bash}
bfabric_read.py resource filechecksum `md5sum < /dev/null | cut -c-32` \
| cat -n \
| tail
```

## Examples
## Testing
Please be advised that integration tests will write to the `TEST` instance configured in your `~/.bfabricpy.yml` config file.

Run unit tests:
```{bash}
python3 -m unittest discover -s "bfabric/tests/unit"
```

Run integration tests (see note above):
```{bash}
python3 -m unittest discover -s "bfabric/tests/integration"
```

## Examples [outdated]

### bash script generated by the yaml wrapper creator / submitter

Expand Down Expand Up @@ -276,7 +309,7 @@ bfabric_read.py importresource \
done
```
## Send an E-mail
## Send an E-mail [outdated]
```
# by CT,CP
Expand All @@ -290,11 +323,7 @@ rv = B.save_object(endpoint = 'mail',
# shown as mail for user id 482
```
## Testing
```{sh}
cd bfabric/tests/ && python3 -m unittest discover; echo $?; cd -
```
## See also
Expand All @@ -305,16 +334,13 @@ cd bfabric/tests/ && python3 -m unittest discover; echo $?; cd -
## FAQ

### How to resolve `<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed`?

on macOSX
```
cd /Applications/Python 3.12 && ./Install\ Certificates.command
```
### How is the version numbering working?

X.Y.Z
X is not used
Expand All @@ -323,8 +349,6 @@ Y should be the bfabric release
Z increment for significant changes
Also, please note that the branch ID should correspond with the bfabric stable release number.

### Howto cite?
Panse, Christian, Trachsel, Christian and Türker, Can. "Bridging data management platforms and visualization tools to enable ad-hoc and smart analytics in life sciences" Journal of Integrative Bioinformatics, 2022, pp. 20220031. [doi: 10.1515/jib-2022-0031](https://doi.org/10.1515/jib-2022-0031).
20 changes: 11 additions & 9 deletions bfabric/__init__.py
View file
Original file line number Diff line number Diff line change
@@ -1,11 +1,19 @@
import importlib.metadata

__version__ = importlib.metadata.version("bfabric")

from bfabric.bfabric import Bfabric, BfabricAPIEngineType
from bfabric.bfabric_config import BfabricAuth, BfabricConfig


__all__ = [
"Bfabric",
"BfabricAPIEngineType",
"BfabricAuth",
"BfabricConfig",
]


__version__ = importlib.metadata.version("bfabric")


endpoints = sorted(
[
"annotation",
Expand Down Expand Up @@ -41,9 +49,3 @@
project = 403
container = project
application = 217


from bfabric.bfabric_legacy import BfabricLegacy
from bfabric.wrapper_creator.bfabric_wrapper_creator import BfabricWrapperCreator
from bfabric.wrapper_creator.bfabric_submitter import BfabricSubmitter
from bfabric.wrapper_creator.bfabric_feeder import BfabricFeeder
16 changes: 6 additions & 10 deletions bfabric/bfabric.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@
import base64
import importlib.metadata
import logging
import os
from contextlib import contextmanager
from datetime import datetime
from enum import Enum
from pathlib import Path
from pprint import pprint
from typing import Literal, ContextManager, Any

Expand Down Expand Up @@ -293,20 +293,17 @@ def get_system_auth(
have_config_path = config_path is not None
if not have_config_path:
# Get default path config file path
config_path = os.path.normpath(os.path.expanduser("~/.bfabricpy.yml"))
config_path = Path("~/.bfabricpy.yml").expanduser()

# Use the provided config data from arguments instead of the file
if not os.path.isfile(config_path):
if not config_path.is_file():
if have_config_path:
# NOTE: If user explicitly specifies a path to a wrong config file, this has to be an exception
raise OSError(f"Explicitly specified config file does not exist: {config_path}")
# TODO: Convert to log
print(f"Warning: could not find the config file in the default location: {config_path}")
config = BfabricConfig(base_url=base_url)
if login is None and password is None:
auth = None
else:
auth = BfabricAuth(login=login, password=password)
auth = None if login is None and password is None else BfabricAuth(login=login, password=password)

# Load config from file, override some of the fields with the provided ones
else:
Expand All @@ -321,9 +318,8 @@ def get_system_auth(

if not config.base_url:
raise ValueError("base_url missing")
if not optional_auth:
if not auth or not auth.login or not auth.password:
raise ValueError("Authentication not initialized but required")
if not optional_auth and (not auth or not auth.login or not auth.password):
raise ValueError("Authentication not initialized but required")

if verbose:
pprint(config)
Expand Down
Loading

0 comments on commit b9e47f0

Please sign in to comment.