Validate the DIRAC Job format (JDL) using Pydantic

[simon-mazenoux]

Currently, the list of fields in the JDL format when submitting a job is not completely validated. I identified a list of fields currently used by the JDL (in alphabetical order):
arguments: str
bannedSites: set[str]
cpuTime: int
executable: str
executionEnvironment: set[str]
gridCE: set[str]
inputSandbox: set[str]
inputData: set[str]
inputDataPolicy: str
jobGroup: str
jobType: str
jobName: str
logLevel: str
maxNumberOfProcessors: int
minNumberOfProcessors: int
outputData: set[str]
outputSandbox: set[str]
outputSE: str
platform: set[str]
priority: int
sites: set[str]
stderr: str
stdout: str
tags: set[str]

To those fields are added in the server side:
diracSetup: str
owner: str
ownerDN: str
ownerGroup: str
vo: str

There is also the fields for the parametric jobs:
parameters: set(str)
parameterStart: float
parameterStep: float
parameterFactor: float

Pydantic would enforces type hints at runtime, and provides user-friendly errors when data is invalid (instead of putting the job in the FAILED state). The issue we may face while implementing Pydantic is that VO may use their own fields in the JDL. We would then like to know if some VOs use extra/custom fields, and have some feedback about this idea.

Edit: feedback from @sfayer (thanks for the very useful feedback)

[sfayer]

Hi,

We (GridPP) don't use any extra/custom fields in the JDL but there are a few standard fields that we use that seem to be missing from the list? Specifically GridCE, Platform and the Parameter* options for parametric submission:
https://github.com/DIRACGrid/DIRAC/blob/integration/docs/source/UserGuide/GettingStarted/UserJobs/JDLReference/index.rst
(This also says the field for the site list is just "site" rather than "sites", not sure which one is correct?).

Regards,
Simon

[simon-mazenoux]

Yes exactly. Which kind of parameters are you adding, and if so where are you using them ?

[hmiyake]

Our variables are string and integer.
We retrieve these variables on grid to give some parameters to our executable or register metadata.

[simon-mazenoux]

Are you retrieving them from the jobDescription.xml file or are you querying the JDL from jobDB?

[hmiyake]

Actually several ways. On grid, we take it from jobDescription.xml or JDL file put by pilot.
Our command line tool queries it from JobDB, but not at job's run time.

[aldbr]

Our variables are string and integer. We retrieve these variables on grid to give some parameters to our executable or register metadata.

@hmiyake is GBasf2Rel a concrete example? Could you give some other examples?

Knowing if (i) our JDL base model is incomplete (in which case we may need to enrich it with some of your attributes); (ii) your JDL attributes could be used as part of the tags JDL attribute; (iii) your and our JDL models are "incompatible" (e.g. you really need some attributes that cannot be included in tags and that cannot be added to the JDL base model); would help us a lot in defining a job validation tool.

Thanks

[hmiyake]

Our parameters are quite VO specific and would not be adequate to add to base model.
Other our parameters are Basf2Rel (str), ProductionID (int), Campaign (str), BeamEnergy (str). Last three are only added to our production jobs. Recently we discussed possibility to add some more variables (str), but it depends on our situation.

Does tags mean what is set by job.setTag()? In my understanding Tags is used to control job execution site, then I hesitate to add unrelated parameters to there...

Of course, if our usage is a kind of 'abuse', we need to change our strategy. I want to discuss the issue in our group.
However, I hope DIRAC provides 'soft landing model' allowing such extra parameters in some grace period, or allows flexible validation (as mentioned above, we have different parameters between user and production jobs).

[simon-mazenoux]

Thanks for your more detailed answer. After some reflection, we managed to include attributes added with the _addJDLParameter method in the pydantic model. Therefore, you don't have to worry about a change here.

[hmiyake]

Excellent! Thank you so much for your smart work!

[simon-mazenoux]

I have another question related to the topic for you. Do you use the parameters added with the _addJDLParameter from the JDL or from the workflow file (meaning JobDescription.xml) ?

[hmiyake]

It depends on the use case. We take the parameter from both the workflow file and the JDL file, on grid.
Our command line tool retrieves JDL from JobDB.

[simon-mazenoux]

Let me be more specific : are you using the JDL to make matching decisions on the server side (meaning to determine the JobRequirements section of the JDL, i.e. Platforms, Sites, BannedSites, GridCEs or Tags) ?

[hmiyake]

Sometimes we update existing JDL parameters by _addJDLParameter...but our "customized" variables are not used for the matching.