Error submitting pilots to AREX CE

[arrabito]

Dear all,

I've just updated to DIRAC 8.0.5 and I'm trying to use AREX CE, but I get some errors:

2022-12-07 08:02:19 UTC WorkloadManagement/SiteDirectorARCCSCS/WorkloadManagement/SiteDirectorARCCSCS ERROR: Agent exception while calling method <bound method SiteDirector.execute of <DIRAC.WorkloadMana\
gementSystem.Agent.SiteDirector.SiteDirector object at 0x7fef283a30a0>>
Traceback (most recent call last):
  File "/opt/dirac/versions/v2.2.0-1670311786/Linux-x86_64/lib/python3.9/site-packages/DIRAC/Core/Base/AgentModule.py", line 311, in am_secureCall
    result = functor(*args)
  File "/opt/dirac/versions/v2.2.0-1670311786/Linux-x86_64/lib/python3.9/site-packages/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py", line 320, in execute
    result = self.submitPilots()
  File "/opt/dirac/versions/v2.2.0-1670311786/Linux-x86_64/lib/python3.9/site-packages/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py", line 439, in submitPilots
    res = self._submitPilotsToQueue(pilotsToSubmit, ce, queueName)
  File "/opt/dirac/versions/v2.2.0-1670311786/Linux-x86_64/lib/python3.9/site-packages/DIRAC/WorkloadManagementSystem/Agent/SiteDirector.py", line 706, in _submitPilotsToQueue
    submitResult = ce.submitJob(executable, "", pilotsToSubmit)
  File "/opt/dirac/versions/v2.2.0-1670311786/Linux-x86_64/lib/python3.9/site-packages/DIRAC/Resources/Computing/AREXComputingElement.py", line 248, in submitJob
    self.arcQueue = self.queue.split("-", 2)[2]
IndexError: list index out of range

It seems that it assumes the queue name being:

nordugrid-<batchsystem>-<queue>

while for the queue name we simply have "normal":

So I simply fixed (I hope it's fine) with:

self.arcQueue = self.queue

Then, I have another issue with AREX, I get:

2022-12-07 08:13:09 UTC WorkloadManagement/SiteDirectorARCCSCS/WorkloadManagement/SiteDirectorARCCSCS INFO: arc-noir.cta.cscs.ch_normal: Slots=297, TQ jobs(pilotsWeMayWantToSubmit)=42, Pilots: waiting 3,\
 to submit=39
2022-12-07 08:13:10 UTC WorkloadManagement/SiteDirectorARCCSCS/WorkloadManagement/SiteDirectorARCCSCS INFO: Going to submit pilots (a maximum of 39 pilots to arc-noir.cta.cscs.ch_normal queue)
2022-12-07 08:13:11 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch WARN: Could not get a delegation For CE arc-noir.cta.cscs.ch
2022-12-07 08:13:11 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch WARN: Continue without a delegation
2022-12-07 08:13:11 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch INFO: Executable correctly uploaded
2022-12-07 08:13:11 UTC WorkloadManagement/SiteDirectorARCCSCS/arc-noir.cta.cscs.ch WARN: Failed to submit job to CE arc-noir.cta.cscs.ch with error - 501 - and messages : Not Implemented
2022-12-07 08:13:11 UTC WorkloadManagement/SiteDirectorARCCSCS/WorkloadManagement/SiteDirectorARCCSCS INFO: Submitted 1 pilots to [normal@arc-noir.cta.cscs.ch](mailto:normal@arc-noir.cta.cscs.ch)

and pilots get 'scheduled' forever.
Since it's the first time I use AREX is maybe a problem on the site side, but do you have any idea already?

Then, I also reverted back to the use of ARC6 (that I was successfully using some times ago with version 7.3) but in this case I get:

2022-12-07 15:13:29 UTC WorkloadManagement/SiteDirectorARCCSCS ALWAYS: Site: ARC.CSCS.ch, CE: arc-noir.cta.cscs.ch, Queue: arc-noir.cta.cscs.ch_normal
2022-12-07 15:13:29 UTC WorkloadManagement/SiteDirectorARCCSCS INFO: Total jobs : number of task queues : number of waiting pilots 10 : 1 : 3
2022-12-07 15:13:30 UTC WorkloadManagement/SiteDirectorARCCSCS INFO: PilotAgentsDB report (arc-noir.cta.cscs.ch_normal): Wait=42, Run=0, Max=2500
2022-12-07 15:13:30 UTC WorkloadManagement/SiteDirectorARCCSCS INFO: arc-noir.cta.cscs.ch_normal: Slots=258, TQ jobs(pilotsWeMayWantToSubmit)=10, Pilots: waiting 3, to submit=7
2022-12-07 15:13:30 UTC WorkloadManagement/SiteDirectorARCCSCS INFO: Going to submit pilots (a maximum of 7 pilots to arc-noir.cta.cscs.ch_normal queue)
VERBOSE: LDAPQuery: Initializing connection to arc-noir.cta.cscs.ch:2135
VERBOSE: Failed to bind to ldap server (arc-noir.cta.cscs.ch)
2022-12-07 15:13:31 UTC WorkloadManagement/SiteDirectorARCCSCS ERROR: Failed submission to queue Queue arc-noir.cta.cscs.ch_normal:
, No pilot references obtained from the ARC job submission
2022-12-07 15:13:31 UTC WorkloadManagement/SiteDirectorARCCSCS INFO: Failed pilot submission Queue: arc-noir.cta.cscs.ch_normal
2022-12-07 15:13:31 UTC WorkloadManagement/SiteDirectorARCCSCS INFO: Total number of pilots submitted in this cycle 0

Again, it might be an issue on the site side, but do you have any idea of the possible problem?

Thank you,
Luisa

[aldbr]

Hi,

while for the queue name we simply have "normal"

Indeed, the queue name is normal but not sure why, it is reported in DIRAC (by BDII I guess) as nordugrid-<batch-system>-normal.
I suggest you to rename the queue in the CS as nordugrid-slurm-normal instead of changing the code.

Since it's the first time I use AREX is maybe a problem on the site side, but do you have any idea already?

Interesting, I never saw this message at this level.
I will try it and let you know.
The AREX CE is still experimental. It works with a few ARC CEs but there are still a few details to fix.

Again, it might be an issue on the site side, but do you have any idea of the possible problem?

The issue with this particular CE and ARC6 is known.
The CE doesn't support ldapglue2 anymore, so we have to use a different endpoint to get details about the CE.
Add ComputingInfoEndpoint = org.nordugrid.rest to the CE parameters in the CS and it should be fine.

[arrabito]

Thank you for the clarifications.
Unfortunately adding:

ComputingInfoEndpoint = org.nordugrid.rest
to the CE parameters in the CS did not help.

Any other suggestion?

Thank you,
Luisa

[aldbr]

Ok, and what about this?

ComputingInfoEndpoint = org.ogf.glue.emies.resourceinfo

[arrabito]

It seems to work fine thank you !

[aldbr]

Also related to AREX, there were a lot of changes between v8.0.5 and v8.0.6 and we fixed a major issue.
If you would want to give a try to the AREX CE, I would suggest you to use the one from v8.0.6: https://github.com/DIRACGrid/DIRAC/blob/v8.0.6/src/DIRAC/Resources/Computing/AREXComputingElement.py

But it would not be enough because I introduced a few bugs while fixing it: https://github.com/DIRACGrid/DIRAC/pull/6615/files

To sum up, I suggest you to wait for v8.0.7 to try the AREX CE.