Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issue 12040 #12155

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

Fix issue 12040 #12155

wants to merge 8 commits into from

Conversation

vkuznet
Copy link
Contributor

@vkuznet vkuznet commented Oct 22, 2024

Fixes #12040

Status

In development

Description

Introduce new logic to update sites and associated rules:

  • Add new POST API for transferor into MSManager, POST JSON payload to /ms-transferor/data/transferor
  • Add new updateSites API to handler POST request and return status to upstream caller (ReqMgr2)
  • Use local file system as persistent storage to store JSON payloads
  • define JSON payload as list with the following data-structure:
[{"workflow": <name>, "SiteWhiteList": [T1,...], "SiteBlackList": [T2, ...]}, {...}]
  • define response from MSTransferor to upstream caller as following:
[{"workflow": <name>, "error": <error>}, {...}]
  • implement saveData and readData to perform IO operations for provided JSON payload and handle its persistent storage. So far these APIs rely on usage of local file system where it store JSON as file whose name is workflow name. If we will decide to use other storage, e.g. database only these two APIs will need a change to perform IO operations
  • provide business logic of _updateSites API which will be executed by execute API of MSTransferor daemon.

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

<If it's a follow up work; or porting a fix from a different branch, please mention them here.>

External dependencies / deployment changes

<Does it require deployment changes? Does it rely on third-party libraries?>

@vkuznet vkuznet requested a review from amaltaro October 22, 2024 12:40
@vkuznet
Copy link
Contributor Author

vkuznet commented Oct 22, 2024

@amaltaro , this is initial logic based on provided requirements. I would appreciate if you will reviewed and let me know if it has expected behavior. In particular, I need to know decision about persistent storage and overview of acknowledged responses to upstream caller. Once we settle on this the rest would be implementation of site update/rules only.

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 293 new failures
    • 4 tests deleted
    • 13 changes in unstable tests
  • Python3 Pylint check: failed
    • 2 warnings and errors that must be fixed
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 20 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15353/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 4 new failures
    • 1 tests no longer failing
    • 2 changes in unstable tests
  • Python3 Pylint check: failed
    • 14 warnings and errors that must be fixed
    • 13 warnings
    • 65 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 22 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15354/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 5 new failures
    • 1 tests no longer failing
    • 2 changes in unstable tests
  • Python3 Pylint check: failed
    • 17 warnings and errors that must be fixed
    • 13 warnings
    • 65 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 22 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15355/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 2 new failures
    • 1 tests no longer failing
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 3 warnings and errors that must be fixed
    • 13 warnings
    • 69 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 21 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15357/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 2 new failures
    • 1 tests no longer failing
  • Python3 Pylint check: failed
    • 1 warnings and errors that must be fixed
    • 13 warnings
    • 69 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 21 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15361/artifact/artifacts/PullRequestReport.html

@khurtado
Copy link
Contributor

test this please

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 4 new failures
  • Python3 Pylint check: failed
    • 1 warnings and errors that must be fixed
    • 13 warnings
    • 69 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 21 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15366/artifact/artifacts/PullRequestReport.html

@khurtado
Copy link
Contributor

test this please

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 2 new failures
    • 2 changes in unstable tests
  • Python3 Pylint check: failed
    • 1 warnings and errors that must be fixed
    • 13 warnings
    • 69 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 21 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15367/artifact/artifacts/PullRequestReport.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Remake input data placement upon site list changes
3 participants