Feature/metagenomics exchange #325

KateSakharova · 2023-09-12T15:36:27Z

Scripts to populate and track Metagenomics Exchange https://www.ebi.ac.uk/ena/registry/metagenome/api/
ME is based on EBI Search model.
System for ME works a bit differently. For update process we do not need to remove and then add a record to "XML" again. That is why I do not use to_delete() function.
I've created get_suppressed() to filter records that need to be removed from ME API.

…OST request

- Refactored the cli wrapper - Refactored the settings to be in line with the rest - Modified the unit tests

… api

The test tests/me/test_populate_metagenomics_exchange.py::TestMeAPI::test_removals_dry_mode it's working now.

…nomics/emgapi into feature/metagenomics_exchange

mberacochea

Great stuff Kate, I left some comments :)

mberacochea · 2024-01-31T20:42:24Z

config/local-tests.yml

+  # metagenomics exchange
+  me_api: 'https://wwwdev.ebi.ac.uk/ena/registry/metagenome/api'
+  me_api_token: 'mgx 871cd915-2826-46bb-94ed-8e6a3d9b6014'


We have to remove this token and ask Suran to grant us a new one, this has write access and this repo is public (even if it's dev env). This one should be in the secrets config file

mberacochea · 2024-01-31T20:49:27Z

emgapi/management/commands/populate_metagenomics_exchange.py

+                            logging.info(f"Analysis {ajob} updated successfully")
+                            # Just to be safe, update the MGX accession
+                            ajob.mgx_accession = registry_id
+                            ajob.last_mgx_indexed = timezone.now()


This one should also use the "nowish" timestamp: "timezone.now() + timedelta(minutes=1)"

emgapi/management/commands/populate_metagenomics_exchange.py

mberacochea · 2024-01-31T20:52:52Z

emgapi/metagenomics_exchange.py

+        return response
+
+    def add_analysis(self, mgya: str, run_accession: str, public: bool):
+        data = {


This "data" object is repeated in the command:

def generate_metadata(self, mgya, run_accession, status): return { "confidence": "full", "endPoint": f"https://www.ebi.ac.uk/metagenomics/analyses/{mgya}", "method": ["other_metadata"], "sourceID": mgya, "sequenceID": run_accession, "status": status, "brokerID": settings.METAGENOMICS_EXCHANGE_MGNIFY_BROKER, }

I would keep one copy of this, probably here

mberacochea · 2024-01-31T20:59:48Z

emgapi/models.py

@@ -321,20 +331,65 @@ def to_add(self):

        return self.filter(never_indexed | updated_after_indexing, not_suppressed, not_private)

+    def get_suppressed(self):


Instead of adding this method, it's better to override "to_delete" with your implementation. This way we stick to the "interface" that IndexableModelQueryset provides

class MetagenomicsExchangeQueryset(IndexableModelQueryset): index_field = "last_mgx_indexed" def to_delete(self): try: self.model._meta.get_field("suppressed_at") except FieldDoesNotExist: return Q() else: return self.filter( Q(suppressed_at__gte=F(self.index_field)) )

mberacochea · 2024-01-31T21:00:36Z

emgapi/models.py

@@ -1634,6 +1687,70 @@ def __str__(self):
        return 'Assembly:{} - Sample:{}'.format(self.assembly, self.sample)


+class MetagenomicsExchangeQueryset(models.QuerySet):


This class should be removed

SandyRogers

Thank you @KateSakharova , great stuff! Couple of small suggestions and queries inline.

SandyRogers · 2024-02-02T14:33:00Z

emgapi/models.py

+class MetagenomicsExchangeModel(models.Model):
+    """Model to track Metagenomics Exchange population
+    https://www.ebi.ac.uk/ena/registry/metagenome/api/
+    """
+    last_updated_me = models.DateTimeField(
+        db_column='LAST_UPDATED_ME',
+        auto_now=True
+    )
+    last_populated_me = models.DateTimeField(
+        db_column='LAST_POPULATED_ME',
+        null=True,
+        blank=True,
+        help_text="Date at which this model was last appeared in Metagenomics Exchange"
+    )
+
+    objects_for_population = MetagenomicsExchangeQueryset.as_manager()
+
+    class Meta:
+        abstract = True


What is this abstract model for? Isn't last_populated_me now done by last_mgx_indexed in the MetagenomicsExchangeIndexedModel? Maybe I misunderstood.

good catch 🤦, it was left after ebi_search merge by mistake, removed

SandyRogers · 2024-02-02T14:37:07Z

emgcli/settings.py

+try:
+    METAGENOMICS_EXCHANGE_API = EMG_CONF['emg']['me_api']
+    if EMG_CONF['emg'].get('me_api_token'):
+        METAGENOMICS_EXCHANGE_API_TOKEN = EMG_CONF['emg']['me_api_token']
+    else:
+        METAGENOMICS_EXCHANGE_API_TOKEN = os.getenv('METAGENOMICS_EXCHANGE_API_TOKEN')
+except KeyError:
+    warnings.warn("The metagenomics exchange API and Token are not configured properly")


Suggested change

try:

METAGENOMICS_EXCHANGE_API = EMG_CONF['emg']['me_api']

if EMG_CONF['emg'].get('me_api_token'):

METAGENOMICS_EXCHANGE_API_TOKEN = EMG_CONF['emg']['me_api_token']

else:

METAGENOMICS_EXCHANGE_API_TOKEN = os.getenv('METAGENOMICS_EXCHANGE_API_TOKEN')

except KeyError:

warnings.warn("The metagenomics exchange API and Token are not configured properly")

try:

METAGENOMICS_EXCHANGE_API = EMG_CONF['emg']['me_api']

METAGENOMICS_EXCHANGE_API_TOKEN = os.getenv('METAGENOMICS_EXCHANGE_API_TOKEN')

if not METAGENOMICS_EXCHANGE_API_TOKEN:

METAGENOMICS_EXCHANGE_API_TOKEN = EMG_CONF['emg']['me_api_token']

except KeyError:

warnings.warn("The metagenomics exchange API and Token are not configured properly")

Just to make sure that a missing token (not in env or in config) triggers the warning.

emgapi/migrations/0017_auto_20240129_1401.py

SandyRogers · 2024-02-02T14:43:34Z

emgapi/metagenomics_exchange.py

+    """Metagenomics Exchange API Client"""
+
+    def __init__(self, base_url=None):
+        self.base_url = base_url if base_url else settings.METAGENOMICS_EXCHANGE_API


Suggested change

self.base_url = base_url if base_url else settings.METAGENOMICS_EXCHANGE_API

self.base_url = base_url or settings.METAGENOMICS_EXCHANGE_API

SandyRogers · 2024-02-02T14:45:01Z

emgapi/management/commands/populate_metagenomics_exchange.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Copyright 2017-2022 EMBL - European Bioinformatics Institute


Suggested change

# Copyright 2017-2022 EMBL - European Bioinformatics Institute

# Copyright 2017-2024 EMBL - European Bioinformatics Institute

SandyRogers · 2024-02-02T14:45:19Z

emgapi/metagenomics_exchange.py

+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Copyright 2018-2023 EMBL - European Bioinformatics Institute


Suggested change

# Copyright 2018-2023 EMBL - European Bioinformatics Institute

# Copyright 2018-2024 EMBL - European Bioinformatics Institute

SandyRogers · 2024-02-02T14:46:41Z

emgapi/management/commands/populate_metagenomics_exchange.py

+    def process_to_index_and_update_records(self, analyses_to_index_and_update):
+        logging.info(f"Indexing {len(analyses_to_index_and_update)} new analyses")
+
+        for page in Paginator(analyses_to_index_and_update, 100):


This 100 is probably worth making a parameter – we had some trouble with EBI dump where reasonable sounding page sizes still caused query problem on the DB. If parametrized we can at least easily try different values.

SandyRogers · 2024-02-02T14:49:33Z

emgapi/management/commands/populate_metagenomics_exchange.py

+                            logging.info(f"Analysis {ajob} updated successfully")
+                            # Just to be safe, update the MGX accession
+                            ajob.mgx_accession = registry_id
+                            ajob.last_mgx_indexed = timezone.now() + timedelta(minutes=1)


One thing to consider here, is that if we offset individual jobs indexation by only 1 minute individually, by the time this loop exists and the bulk_update on L154 happens – that +minute may be in the past. This means that the auto last_update will now be after the last indexation (if Django decided to update the last_update for bulk updates... which I think it currently doesn't, but might in future). Perhaps worth a quick test / check the Django docs to be sure this won't be a problem.

…ndexed This is needed because RenameField doesn't allow you to change the db_column name. With this change it seems to work, and the data should be kept: ```sql -- -- Alter field last_indexed on analysisjob -- ALTER TABLE "ANALYSIS_JOB" RENAME COLUMN "LAST_INDEXED" TO "LAST_EBI_SEARCH_INDEXED"; -- -- Alter field last_indexed on study -- ALTER TABLE "STUDY" RENAME COLUMN "LAST_INDEXED" TO "LAST_EBI_SEARCH_INDEXED"; -- -- Rename field last_indexed on analysisjob to last_ebi_search_indexed -- -- -- Rename field last_indexed on study to last_ebi_search_indexed -- ```

SandyRogers

Thank you @KateSakharova !

mberacochea

Thanks Kate!

KateSakharova requested review from SandyRogers and removed request for SandyRogers October 23, 2023 10:36

KateSakharova marked this pull request as draft October 23, 2023 12:54

mberacochea force-pushed the feature/metagenomics_exchange branch from 554e0b0 to 355bee0 Compare January 17, 2024 16:58

mberacochea and others added 23 commits January 29, 2024 11:09

adds incremental ebi search dumping for analysisjobs

0a8a761

integrates indexable sample data into analyses ebi search dump

4d51fe3

adds ebi search dump for studies/projects (plus some tweaks to analyses)

240d8f4

resolve migration merge conflicts

ed97ccb

adds support for chunking of analysis-runs ebi search dump

ab829f4

bugfixes on analysis-run dumper

fcb9300

Add metagenomics exchange support

6438c3b

Added 2 tests for ME, population script WIP

64f4900

Move metagenomic exchange commands to separate Class. Tests fail in P…

7484116

…OST request

My revised version:

fee19bd

- Refactored the cli wrapper - Refactored the settings to be in line with the rest - Modified the unit tests

Modified ME dj function to add new ME records, fixed tests for ME and…

e7d9b17

… api

Move test for mock API to test_mgx_api.py

2e0b019

Add settings

c92c381

change models and add patch and delete to ME

bdac39c

Add delete and patch commands with tests to ME API

65e64e0

add population scripts and tests for dry-run

3c664e6

add population scripts and tests for dry-run

f082415

change api adress

bb1ce81

change api adress

0f309c4

WIP

83b6b3a

Minor refactor of the MGX indexation process WIP

97e226e

This is still a WIP - MGX

e229f5c

Fixed the migrations for EBI Search and MGX.

a6e9eda

The test tests/me/test_populate_metagenomics_exchange.py::TestMeAPI::test_removals_dry_mode it's working now.

mberacochea force-pushed the feature/metagenomics_exchange branch from c0a15d7 to a6e9eda Compare January 29, 2024 13:06

KateSakharova and others added 2 commits January 29, 2024 13:19

Fix some ME tests

2f83224

WIP - migration tidy up

5961f7d

mberacochea and others added 6 commits January 29, 2024 14:07

Merge branch 'feature/metagenomics_exchange' of github.com:EBI-Metage…

02d3943

…nomics/emgapi into feature/metagenomics_exchange

Adjust field names to new convention (last_ebi_search_indexed)

60d2a37

WIP testing

ad539e7

Tests for populate ME

986f222

rm dev from tests and leave only DEV API

916ce45

fix pipeline mock

948caa8

KateSakharova requested review from SandyRogers and mberacochea January 31, 2024 16:29

KateSakharova marked this pull request as ready for review January 31, 2024 16:29

mberacochea requested changes Feb 1, 2024

View reviewed changes

KateSakharova added 2 commits February 1, 2024 11:52

fixes

6a55c40

add mock to patch test

18d4641

SandyRogers requested changes Feb 2, 2024

View reviewed changes

KateSakharova and others added 3 commits February 6, 2024 16:08

some fixes after review

bfb1b54

Rename EBI Seach last_indexed fields

7dbbb14

KateSakharova requested review from SandyRogers and mberacochea February 8, 2024 09:47

SandyRogers approved these changes Feb 8, 2024

View reviewed changes

mberacochea approved these changes Feb 8, 2024

View reviewed changes

mberacochea merged commit 633b3f7 into develop Feb 8, 2024
4 checks passed

mberacochea deleted the feature/metagenomics_exchange branch February 29, 2024 12:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/metagenomics exchange #325

Feature/metagenomics exchange #325

KateSakharova commented Sep 12, 2023 •

edited

Loading

mberacochea left a comment

mberacochea Jan 31, 2024

mberacochea Jan 31, 2024

mberacochea Jan 31, 2024

mberacochea Feb 1, 2024

mberacochea Jan 31, 2024

mberacochea Jan 31, 2024

SandyRogers left a comment

SandyRogers Feb 2, 2024

KateSakharova Feb 6, 2024 •

edited

Loading

SandyRogers Feb 2, 2024

SandyRogers Feb 2, 2024

SandyRogers Feb 2, 2024

SandyRogers Feb 2, 2024

SandyRogers Feb 2, 2024

SandyRogers Feb 2, 2024

SandyRogers left a comment

mberacochea left a comment

		@@ -321,20 +331,65 @@ def to_add(self):

		return self.filter(never_indexed \| updated_after_indexing, not_suppressed, not_private)

		def get_suppressed(self):

		@@ -1634,6 +1687,70 @@ def __str__(self):
		return 'Assembly:{} - Sample:{}'.format(self.assembly, self.sample)


		class MetagenomicsExchangeQueryset(models.QuerySet):

	self.base_url = base_url if base_url else settings.METAGENOMICS_EXCHANGE_API
	self.base_url = base_url or settings.METAGENOMICS_EXCHANGE_API

	# Copyright 2017-2022 EMBL - European Bioinformatics Institute
	# Copyright 2017-2024 EMBL - European Bioinformatics Institute

	# Copyright 2018-2023 EMBL - European Bioinformatics Institute
	# Copyright 2018-2024 EMBL - European Bioinformatics Institute

Feature/metagenomics exchange #325

Feature/metagenomics exchange #325

Conversation

KateSakharova commented Sep 12, 2023 • edited Loading

mberacochea left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SandyRogers left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KateSakharova Feb 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SandyRogers left a comment

Choose a reason for hiding this comment

mberacochea left a comment

Choose a reason for hiding this comment

KateSakharova commented Sep 12, 2023 •

edited

Loading

KateSakharova Feb 6, 2024 •

edited

Loading