Is it possible to make Transformation System aware of Tag?

[hmiyake]

Here, SE1-EL7 and SE2-EL7 are EL7 sites, and SE3-EL9 and SE4-EL9 are EL9 sites.

Assume we have two replicas for each input file and they always cover EL7 and EL9 sites.

files = {'LFN1': ['SE1-EL7', 'SE3-EL9'],
'LFN2': ['SE1-EL7', 'SE3-EL9'],
'LFN3': ['SE1-EL7', 'SE3-EL9'],
'LFN4': ['SE2-EL7', 'SE3-EL9'],
'LFN5': ['SE2-EL7', 'SE3-EL9'],
'LFN6': ['SE1-EL7', 'SE4-EL9'],
'LFN7': ['SE1-EL7', 'SE4-EL9'],
'LFN8': ['SE2-EL7', 'SE4-EL9'],
'LFN9': ['SE2-EL7', 'SE4-EL9'],
}

groupByReplicas() returns following tasks under GroupSize=3

SE3-EL9,SE1-EL7 ['LFN1', 'LFN2', 'LFN3']
SE2-EL7 ['LFN4', 'LFN5', 'LFN8']
SE4-EL9 ['LFN6', 'LFN7', 'LFN9']

Important thing is this workflow is independent from user's JDL.

In our use case, user (production requester) could prefer EL7 or EL9, then apply Tags="EL7" or "EL9" in their JDL.

Matcher can assign 1st job to both EL7 and EL9 sites even if Tags are specified (in my understanding), while the second job gets stuck if job requires EL9.
Same thing happens for the third job.

So ideally

Under Tags=['EL7']
SE1-EL7 ['LFN1', 'LFN2', 'LFN3']
SE2-EL7 ['LFN4', 'LFN5', 'LFN8']

Or, under Tags=['EL9']
SE3-EL9 ['LFN1', 'LFN2', 'LFN3']
SE4-EL9 ['LFN6', 'LFN7', 'LFN8']

(the other files could be processed by Flush)

Do you think if it is feesible? (including development for TS)
Or do we have other way to avoid such dead lock?

[fstagni]

Regarding your question: none of the existing Transformation Plugins (https://dirac.readthedocs.io/en/latest/DeveloperGuide/Systems/Transformation/transformationplugin.html) that are in Core DIRAC consider the tags. If you want to do something there you'd need to develop a new plugin.

Now, TBH I am not sure that you are thinking to do with these tags is the best way to proceed. First of all, you can assign a SE to more than 1 site. But, mostly, if I was you I would make use of the platform concept:

• In DIRAC there is the concept of platform, which is what you obtain BTW with

$ dirac-platform
Linux_x86_64_glibc-2.34

This platform is discovered by the pilots, and is used for matching jobs : https://dirac.readthedocs.io/en/latest/AdministratorGuide/Systems/WorkloadManagement/JobsMatching.html#matching-wns-capabilities-to-jobs-requirements (each job has a platform)
You can define lists of compatible platforms in the CS. If you do not like what base DIRAC gives you, you can build something like https://gitlab.cern.ch/lhcb-core/LbPlatformUtils (this is a "core SW" issue).

• There is no way that you are guaranteed that a Site that declares that all its WNs are EL7 are effectively all EL7 (some can just have another platform). Site admins can also embark in migrations at any point in time
• You can define another script for determining the platform from the Pilot: https://github.com/search?q=repo%3ADIRACGrid%2FPilot%20dirac-platform&type=code

[hmiyake]

Thank you for the information.

I see current situation and necessity of development to utilize Tag in TS.

Regarding as Tag vs platform, yes platform is more flexible when a site serves mixed architectures and I prefer the way in normal usage.

Our problem is, when we use apptainer, we want to run EL7 jobs at specific EL9 site allowing to run EL7 image. For such complicated condition, Tag is easier to control jobs.

However, we may consider to extend ConfigureArchitecture following your advice. I'll give the information to my colleagues and discuss feasibility.