MaxJobsInFillMode replacements

[IgorPelevanyuk]

Hello!

Since version v7r1 MaxJobsInFillMode was removed as an option from SiteDirector. How can we confine amount of jobs running by one pilot?

Kind regards,
Igor Pelevanyuk

[IgorPelevanyuk]

I have also found some instruction to achieve that in #5912.

You can set ExtraPilotOptions as -o FillingModeFlag=False (not entirely sure it still works) or --MaxCycles=1 in the SiteDirector CS section.

I put that in config in configuration of SiteDirector, but I worry that it is not working. Since ExtraPilotOptions already defined in /Resources/Computing/CEDefaults

[fstagni]

You can set the maximum amount of cycles a JobAgent would do, for example setting ExtraPilotOptions = -M 3 in a CE CS definition (works also in the CEDefaults).
What do you have already as ExtraPilotOptions in CEDefaults?

Beware that this option can conflict with using Pool as inner CE.

[IgorPelevanyuk]

Right in /Resources/Computing/CEDefaults I have:
ExtraPilotOptions=--userEnvVariables DISABLE_WATCHDOG_CPU_WALLCLOCK_CHECK:::True, --MaxCycles=1

Am I understand correctly that there is no need in setting ExtraPilotOptions in SiteDirector configuration?
So, I am going to replace /Resources/Computing/CEDefaults/ExtraPilotOptions
--userEnvVariables DISABLE_WATCHDOG_CPU_WALLCLOCK_CHECK:::True, --MaxCycles=1
with
--userEnvVariables DISABLE_WATCHDOG_CPU_WALLCLOCK_CHECK:::True, -M 1
Is that correct?

[IgorPelevanyuk]

BTW, ExtraPilotOptions=--userEnvVariables DISABLE_WATCHDOG_CPU_WALLCLOCK_CHECK:::True, --MaxCycles=1 is not working since pilot-commands.py split parameters only by :::

Traceback (most recent call last):
  File "dirac-pilot.py", line 71, in <module>
    command.execute()
  File "/tmp/dirac/DIRAC_Q9KVO2pilot/pilotCommands.py", line 371, in execute
    self._installDIRACpy3()
  File "/tmp/dirac/DIRAC_Q9KVO2pilot/pilotCommands.py", line 302, in _installDIRACpy3
    [value.split(":::")[1] for value in self.pp.userEnvVariables.replace(" ", "").split(",")],
IndexError: list index out of range

[fstagni]

--userEnvVariables DISABLE_WATCHDOG_CPU_WALLCLOCK_CHECK:::True, --MaxCycles=1 is equivalent to --userEnvVariables DISABLE_WATCHDOG_CPU_WALLCLOCK_CHECK:::True, -M 1

and means that ALL your pilots will run at most 1 job.

You can do with --userEnvVariables DISABLE_WATCHDOG_CPU_WALLCLOCK_CHECK:::True,-M:::1

But why do you need DISABLE_WATCHDOG_CPU_WALLCLOCK_CHECK:::True ?

[IgorPelevanyuk]

Thank you for reply, Federico. Let me test that approach, so I can mark reply as the answer.

Considering DISABLE_WATCHDOG_CPU_WALLCLOCK_CHECK:::True. As far as I can remember(and I really forgot about this option), this was done to disable job termination triggered by watchdog when cpu usage is very low. That usually happens during long data transfers on some remote clouds.

[IgorPelevanyuk]

Hello, so far it looks like the option I used is not working.
/Resources/Computing/ExtraPilotOptions=--MaxCycles=1, --userEnvVariables DISABLE_WATCHDOG_CPU_WALLCLOCK_CHECK:::True

I have checked logs during pilot installation and there is written:
2022-11-23 00:00:33 UTC INFO Executing: python dirac-pilot.py -S JINR-Production --pythonVersion=3 -N hydra.jinr.ru -Q mpd_ram_high -n DIRAC.GOVORUN.ru -o '/LocalSite/SharedArea=/zfs/hybrilit.jinr.ru/user/d/dirac/diracce2' --MaxCycles=1, --userEnvVariables DISABLE_WATCHDOG_CPU_WALLCLOCK_CHECK:::True
and
2022-11-23 00:00:33 UTC DEBUG [Pilot] PARAMETER [('-S', 'JINR-Production'), ('--pythonVersion', '3'), ('-N', 'hydra.jinr.ru'), ('-Q', 'mpd_ram_high'), ('-n', 'DIRAC.GOVORUN.ru'), ('-o', '/LocalSite/SharedArea=/zfs/hybrilit.jinr.ru/user/d/dirac/diracce2'), ('--MaxCycles', '1,'), ('--userEnvVariables', 'DISABLE_WATCHDOG_CPU_WALLCLOCK_CHECK:::True')]

But when I look in the directory of pilot I see that many jobs have been executed:

[dirac@n02p042 DIRAC_QXsQ8Dpilot]$ ll
total 644
-rw-r--r--  1 dirac hybrilit   1337 ноя 23 03:01 2149772.jdl
-rw-r--r--  1 dirac hybrilit   1337 ноя 24 18:23 2150720.jdl
-rw-r--r--  1 dirac hybrilit   1337 ноя 23 16:19 2155936.jdl
drwxr-xr-x  2 dirac hybrilit   4096 ноя 25 07:12 2161702
-rw-r--r--  1 dirac hybrilit   1337 ноя 25 07:12 2161702.jdl
-rw-r--r--  1 dirac hybrilit   1337 ноя 24 05:09 2166294.jdl
-rw-r--r--  1 dirac hybrilit   1448 ноя 23 03:00 checksums.sha512

And in later log of Pilot there are the following log:

2022-11-23 00:01:26 UTC INFO [LaunchAgent] Starting JobAgent
2022-11-23 00:01:26 UTC INFO [LaunchAgent] Executing command dirac-agent WorkloadManagement/JobAgent -o MaxCycles=10 -o PollingTime=20 -o StopOnApplicationFailure=True -o StopAfterFailedMatches=10 -o LogLevel=INFO -s /Resources/Computing/CEDefaults -o WorkingDirectory=/tmp/dirac/DIRAC_QXsQ8Dpilot -o /LocalSite/CPUTime=6556343 -o /LocalSite/InstancePath=/tmp/dirac/DIRAC_QXsQ8Dpilot  -o /AgentJobRequirements/ExtraOptions=pilot.cfg --cfg pilot.cfg

It looks like JobAgent takes default value.
Maybe I have wrote the configuration incorrectly? Probably the "," afrer 1 is the reason. But I do not know how to write that correctly.

[fstagni]

Mmm indeed the 1, must be the issue: https://github.com/DIRACGrid/Pilot/blob/master/Pilot/pilotTools.py#L706-L710

I will patch this.

[fstagni]

Should be fixed within #6602 for v7r3

[IgorPelevanyuk]

Thanks!