Fix issue 12040

[vkuznet]

Fixes #12040

Status

In development

Description

Introduce new logic to update sites and associated rules:

• Add new POST API for transferor into MSManager, POST JSON payload to /ms-transferor/data/transferor
• Add new updateSites API to handler POST request and return status to upstream caller (ReqMgr2)
• Use local file system as persistent storage to store JSON payloads
• define JSON payload as list with the following data-structure:

[{"workflow": <name>, "SiteWhiteList": [T1,...], "SiteBlackList": [T2, ...]}, {...}]

• define response from MSTransferor to upstream caller as following:

[{"workflow": <name>, "error": <error>}, {...}]

• implement saveData and readData to perform IO operations for provided JSON payload and handle its persistent storage. So far these APIs rely on usage of local file system where it store JSON as file whose name is workflow name. If we will decide to use other storage, e.g. database only these two APIs will need a change to perform IO operations
• provide business logic of _updateSites API which will be executed by execute API of MSTransferor daemon.

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

<If it's a follow up work; or porting a fix from a different branch, please mention them here.>

External dependencies / deployment changes

<Does it require deployment changes? Does it rely on third-party libraries?>

[vkuznet]

@amaltaro , this is initial logic based on provided requirements. I would appreciate if you will reviewed and let me know if it has expected behavior. In particular, I need to know decision about persistent storage and overview of acknowledged responses to upstream caller. Once we settle on this the rest would be implementation of site update/rules only.

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: failed
  • 293 new failures
  • 4 tests deleted
  • 13 changes in unstable tests
• Python3 Pylint check: failed
  • 2 warnings and errors that must be fixed
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 20 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15353/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: failed
  • 4 new failures
  • 1 tests no longer failing
  • 2 changes in unstable tests
• Python3 Pylint check: failed
  • 14 warnings and errors that must be fixed
  • 13 warnings
  • 65 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 22 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15354/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: failed
  • 5 new failures
  • 1 tests no longer failing
  • 2 changes in unstable tests
• Python3 Pylint check: failed
  • 17 warnings and errors that must be fixed
  • 13 warnings
  • 65 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 22 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15355/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: failed
  • 2 new failures
  • 1 tests no longer failing
  • 1 changes in unstable tests
• Python3 Pylint check: failed
  • 3 warnings and errors that must be fixed
  • 13 warnings
  • 69 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 21 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15357/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: failed
  • 2 new failures
  • 1 tests no longer failing
• Python3 Pylint check: failed
  • 1 warnings and errors that must be fixed
  • 13 warnings
  • 69 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 21 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15361/artifact/artifacts/PullRequestReport.html

[khurtado]

test this please

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: failed
  • 4 new failures
• Python3 Pylint check: failed
  • 1 warnings and errors that must be fixed
  • 13 warnings
  • 69 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 21 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15366/artifact/artifacts/PullRequestReport.html

[khurtado]

test this please

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: failed
  • 2 new failures
  • 2 changes in unstable tests
• Python3 Pylint check: failed
  • 1 warnings and errors that must be fixed
  • 13 warnings
  • 69 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 21 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15367/artifact/artifacts/PullRequestReport.html