Refactor gnps classes #169

CunliangGeng · 2023-08-25T08:42:10Z

This PR intends to check GNPS data, refactor classes and functions on downloading, extracting and loading GNPS data, add detailed docstring to explain how GNPS data is processed, and update unit tests for all types of GNPS data.

👉 It's easier to review the code commit by commit or class by class.
Though this PR has so many commits, actually the changes on GNPS classes followed the similar pattern.

Major changes

Removed dependency requests and only use httpx to handle http requests
Added dependency pytemoics to parse MGF file
Added the missing GNPS workflow (SNETS-V2) to enum GNPSFormat
Refactored all classes and functions in the module gnps, including GNPSDownloader, GNPSExtractor, GNPSAnnotationLoader, GNPSFileMappingLoader, GNPSMolecularFamilyLoader, GNPSSpectrumLoader, and functions from gnps_format.py
- Updated relevant download/extract/load logics to consider all GNPS workflows in GNPSFormat
- Added data validation before loading
- Added detailed docstring for each GNPS data type
- Fixed some bugs or invalid URLs
Refactored all unit tests relevant to the gnps module
- Replaced minimal working test data with complete GNPS data (zip file)
- Restructured and refactored module-level fixtures to make all tests consistent with each other and easier to understand
- Refactored all tests to test all types of GNPS workflows
Added, removed or updated some util functions in utils.py

Expected failed tests

all tests in pairedomics folder
test_loader.py

Add the detection of GNPS workflow "METABOLOMICS-SNETS-V2"

- rename enum names - change enum values to GNPS workflow names - add docstrings

Add the scenario of GNPSFormat.Unknown

- replace workflow name with enum value - update docstrings

… functions

- move existing GNPS files - add new file `ProteoSAFe-METABOLOMICS-SNETS-V2-189e8bf1-download_clustered_spectra.zip` - add new file `ProteoSAFe-Unknown.zip`

Move some fixtures from top level to metabolomics level

- add tests for workflow SNETSV2 and Unknown - use new fixtures `gnps_zip_files` and `gnps_file_mappings_files`

- use httpx to replace urllib - remove private functions not needed any more

Add GNPSFormat checking

Checking GNPSFormat has been moved to `__init__`

…loader`

- add detection of gnps website availability - add tests for all supported GNPS workflows

- add detection of GNPS workflow in initiation - change method `extract` to private `_extract` - rewrite the private methods based on GNPS workflow types - add detailed docstring to explain which file is extracted and renamed update extractor

- update URL for GNPS USI - add `_validate` to validate annotation file (.tsv) - add `_load` method to modularise loading code - change method `get_annotations` to property `annotations` - update docstring

- add `_validate` to validate file mappings file - add `_load*` methods to modularise loading code - change method `mappings` to property - add detailed docstring

- add `_validate` method - change method `families` to property - refactor `_load` method to make loading code more modular - add detailed docstring

- replace mgf parser with community pakcage `pyteomics` - add `_validate` method - add `_load` method - add detailed docstring

…t_antismash_data` to make sure deletion of exttract_path can always happen.

gcroci2

Minor comments :)

src/nplinker/metabolomics/gnps/gnps_format.py

src/nplinker/utils.py

src/nplinker/metabolomics/gnps/gnps_annotation_loader.py

CunliangGeng added 30 commits August 25, 2023 09:58

add deprecated to load_gnps file

503db0d

update the logic of function gnps_format_from_archive

10316da

update function gnps_format_from_task_id

e308213

Add the detection of GNPS workflow "METABOLOMICS-SNETS-V2"

update function gnps_format_from_file_mapping

761c544

update GNPSFormat class

4dd7584

- rename enum names - change enum values to GNPS workflow names - add docstrings

format utils.py and remove os.fspath

5d00361

update GNPSDownloader.get_url method

dd51fae

Add the scenario of GNPSFormat.Unknown

format gnps_extractor.py file

5a26b28

update function gnps_format_from_task_id

14e2a72

- replace workflow name with enum value - update docstrings

update function gnps_format_from_archive

6bc328d

- replace workflow name with enum value - update docstrings

move function gnps_format_from_file_mapping after other gnps format…

c6e2695

… functions

move all gnps-related test data to gnps data folder

9557d19

- move existing GNPS files - add new file `ProteoSAFe-METABOLOMICS-SNETS-V2-189e8bf1-download_clustered_spectra.zip` - add new file `ProteoSAFe-Unknown.zip`

add GNPS_DATA_DIR variable for tests

1c92cce

create conftest.py for metabolomics tests

894cfef

Move some fixtures from top level to metabolomics level

remove dependency Requests and use httpx for hanlding http requests

d280307

Update test_gnps_format.py

f20b925

- add tests for workflow SNETSV2 and Unknown - use new fixtures `gnps_zip_files` and `gnps_file_mappings_files`

update function download_url with package httpx

639f971

- use httpx to replace urllib - remove private functions not needed any more

Update GNPSDownloader.__init__

4d7d9fd

Add GNPSFormat checking

update GNPSDownloader. get_task_id method

1b5f50e

Checking GNPSFormat has been moved to `__init__`

rename method get_download_path to get_download_file in `GNPSDown…

e327963

…loader`

update GNPSDownloader.download using utils function

4245196

add property GNPSDownloader.gnps_format

0c42010

Update test_gnps_downloader.py

bf5e870

- add detection of gnps website availability - add tests for all supported GNPS workflows

remove utils function extract_file_matching_pattern

5dc23ed

update utils extract_archive to allow select members

eab7ce1

fix path bug

512e136

change extension from .pairsinfo to .tsv

e4f8110

refactor class GNPSExtractor

5236a88

- add detection of GNPS workflow in initiation - change method `extract` to private `_extract` - rewrite the private methods based on GNPS workflow types - add detailed docstring to explain which file is extracted and renamed update extractor

Update test_gnps_extractor.py

7501a3a

Update annotations.py

05e1f7b

CunliangGeng added 17 commits August 25, 2023 09:58

add function is_file_format to utils

e37c720

update GNPSAnnotationLoader class

c4a6e5d

- update URL for GNPS USI - add `_validate` to validate annotation file (.tsv) - add `_load` method to modularise loading code - change method `get_annotations` to property `annotations` - update docstring

Update test_gnps_annotation_loader.py

542de65

refactor GNPSFileMappingLoader class

dbf9ba1

- add `_validate` to validate file mappings file - add `_load*` methods to modularise loading code - change method `mappings` to property - add detailed docstring

Update test_gnps_file_mapping_loader.py

2b8bffa

update GNPSMolecularFamilyLoader class

1f7a182

- add `_validate` method - change method `families` to property - refactor `_load` method to make loading code more modular - add detailed docstring

Update test_gnps_molecular_family_loader.py

8e7ce86

add TODO comment

4bac97a

add dependency pyteomics

7972a4b

refactor GNPSSpectrumLoader class

9469251

- replace mgf parser with community pakcage `pyteomics` - add `_validate` method - add `_load` method - add detailed docstring

replace minimal working GNPS zip files with complete zip files

777b006

Update test_gnps_spectrum_loader.py

778cf4b

remove unused test data file nodes_fbmn_mwe.csv

6e95d03

fix root path for gnps test data

ffdabe5

remove test data and tests for load_annotations function

f410a6f

add success condition for download_url function

d96c159

replace HTTPError with more robust Exception for `download_and_extrac…

d08851d

…t_antismash_data` to make sure deletion of exttract_path can always happen.

CunliangGeng requested a review from gcroci2 August 25, 2023 08:42

CunliangGeng self-assigned this Aug 25, 2023

gcroci2 approved these changes Aug 29, 2023

View reviewed changes

CunliangGeng added 2 commits August 29, 2023 14:01

change double backticks to single one

ac60f24

update the logics of validating unique ids

69b73ae

CunliangGeng requested a review from gcroci2 August 29, 2023 12:16

gcroci2 approved these changes Aug 29, 2023

View reviewed changes

CunliangGeng merged commit 4d230e9 into dev Aug 29, 2023
2 of 4 checks passed

CunliangGeng deleted the refactor_gnps_classes branch August 29, 2023 13:58

CunliangGeng mentioned this pull request Dec 22, 2023

Improve GNPS format detection #89

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor gnps classes #169

Refactor gnps classes #169

CunliangGeng commented Aug 25, 2023 •

edited

Loading

gcroci2 left a comment

Refactor gnps classes #169

Refactor gnps classes #169

Conversation

CunliangGeng commented Aug 25, 2023 • edited Loading

Major changes

Expected failed tests

gcroci2 left a comment

Choose a reason for hiding this comment

CunliangGeng commented Aug 25, 2023 •

edited

Loading