Proxies not getting renewed

[FA-VS]

Hi all,

I am using the CVMFS EGI DIRAC, which currently runs dirac version 8.0.28. My jobs that run longer than ~33h lose connection to the Grid due to not being able to renew their proxy (the job/pilot is still running on the relevant cluster, it just can't communicate with Dirac or access storage past that point). I had a similar issue in the past, which turned out to be due to the VOMS extension being shorter-lived than the proxy, but that was fixed already. The only potential hint is that the starting proxy lifetime of the job (once the pilot receives it) is just 33h, despite me launching the job with a 7 day long proxy.

The relevant section of the pilot stdout seems to be the following:

=====================================================================================
Last 8 lines of application output from Watchdog on 2024-02-04 17:33:09.008910 [UTC]:
Last reported CPU consumed for job is 28:43:18 (h:m:s), Batch Queue Time Left 7011577.9 (s @ HS06)
=====================================================================================

WARNING: Parsing of '.cfg' files as command line arguments is changing!
          Set the environment variable 'export DIRAC_NO_CFG=1' to pass the file as a positional
          argument (this will become the default).
          To modify the local configuration use '--cfg <configfile>' instead.
2024-02-03 12:03:01 UTC dirac-jobexec INFO: JobID: 151770664
2024-02-03 12:03:01 UTC dirac-jobexec INFO: DIRAC JobID 151770664 is running at site EGI.SARA.nl
Executing StepInstance RunScriptStep1 of type ScriptStep1 ['ScriptStep1']
2024-02-03 12:03:02 UTC dirac-jobexec/Script INFO: Command is: ./Run-Snakemake-GridWN.sh -D 00000132 -R 00014417 -C v9.0_ORCA.yaml -v v9.0 -c 1
2024-02-04 18:02:49 UTC WorkloadManagement/JobAgent/InProcess NOTICE: Pilot Proxy: /tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/proxy
2024-02-04 18:02:49 UTC WorkloadManagement/JobAgent/InProcess INFO: Using Pilot credentials to get a new payload Proxy
2024-02-04 18:02:49 UTC WorkloadManagement/JobAgent ERROR: proxyString parameter is not a valid type <class 'str'>
2024-02-04 18:02:49 UTC WorkloadManagement/JobAgent WARN: Could not get stored proxy strength /DC=org/DC=terena/DC=tcs/C=NL/O=Nikhef/CN=Francisco Vazquez de Sola Fernandez pvazquez@nikhef.nl, km3net_user, /km3net.org: {'OK': False, 'Errno': 0, 'Message': 'proxyString parameter is not a valid type', 'CallStack': ['  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/threading.py", line 1002, in _bootstrap\n    self._bootstrap_inner()\n', '  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/threading.py", line 1045, in _bootstrap_inner\n    self.run()\n', '  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/threading.py", line 982, in run\n    self._target(*self._args, **self._kwargs)\n', '  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/site-packages/DIRAC/Core/Utilities/ThreadScheduler.py", line 111, in __executorThread\n    timeToNext = self.executeNextTask()\n', '  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/site-packages/DIRAC/Core/Utilities/ThreadScheduler.py", line 127, in executeNextTask\n    self.__executeTask(taskId)\n', '  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/site-packages/DIRAC/Core/Utilities/ThreadScheduler.py", line 180, in __executeTask\n    task["func"](*task["args"])\n', '  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/site-packages/DIRAC/Resources/Computing/ComputingElement.py", line 410, in _monitorProxy\n    return gProxyManager.renewProxy(\n', '  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/site-packages/DIRAC/FrameworkSystem/Client/ProxyManagerClient.py", line 632, in renewProxy\n    retVal = self.downloadVOMSProxy(\n', '  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/site-packages/DIRAC/Core/Utilities/ThreadSafe.py", line 19, in lockedFunc\n    return funcToCall(*args, **kwargs)\n', '  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/site-packages/DIRAC/FrameworkSystem/Client/ProxyManagerClient.py", line 342, in downloadVOMSProxy\n    res = rpcClient.getStoredProxyStrength(userDN, userGroup, requiredVOMSAttribute)\n', '  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/site-packages/DIRAC/Core/Base/Client.py", line 93, in executeRPC\n    rpcClient = self._getRPC(rpc=rpc, url=url, timeout=timeout)\n', '  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/site-packages/DIRAC/Core/Base/Client.py", line 112, in _getRPC\n    rpc = RPCClientSelector(url, httpsClient=self.httpsClient, **self.__kwargs)\n', '  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/site-packages/DIRAC/Core/Tornado/Client/ClientSelector.py", line 76, in ClientSelector\n    rpc = disetClient(*args, **kwargs)\n', '  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/site-packages/DIRAC/Core/DISET/RPCClient.py", line 72, in __init__\n    self.__innerRPCClient = InnerRPCClient(*args, **kwargs)\n', '  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/site-packages/DIRAC/Core/DISET/private/BaseClient.py", line 95, in __init__\n    result = initFunc()\n', '  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/site-packages/DIRAC/Core/DISET/private/BaseClient.py", line 608, in __checkTransportSanity\n    retVal = gProtocolDict[self.__URLTuple[0]]["sanity"](self.__URLTuple[1:3], self.kwargs)\n', '  File "/tmp/7CcLDmZCJq4nKmfV3nlrJULmABFKDmABFKDmU8HKDmABFKDmA27Qun/DIRAC_1X_GWnpilot/diracos/lib/python3.11/site-packages/DIRAC/Core/DISET/private/Transports/SSLTransport.py", line 53, in checkSanity\n    return S_ERROR("proxyString parameter is not a valid type")\n'], 'rpcStub': [['Framework/ProxyManager', {'timeout': 600, 'skipCACheck': True, 'proxyString': '***REDACTED***\n-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCQYCLpAo/ks8EC\nLtI1Vy9kAdDIznsbRpyjT55MnOEOL8VfSndepr3EAkwpJlI8r6sV+ws61TlYnMmt\nLNpsGvHinTN0/QWcxnkhAxIUODvwBtZux03MGsdPMnI0avZCZM3XC64S6Y20Zfrm\n4Boj0tp+vH0Bsbw2gDMav1+1LNchw8xtRO02fYszPv1tPkdrbfJHK0H2xgiTG0tL\n3BQgrvD8zQkhk6KwnS6dI/YzYmMZV5FH1Y/GtGE4ajKK0jYOIYPkY+b4P1eTwQbD\nRwB8Zz/ZtdKH0Hr8xwQBhMUaLGea6ggQbemEEsHu37TOC80xsA3Evoo1ckkSQjqX\nr0VvbQibAgMBAAECggEAL7XtY3YqyifS9R827VSp/xEQh4OgsCNWGSvzAptnTjJg\nNbzclMViebdOptyYmaBt19412FPlVxC9HwLvDYGcBMjxW8xPwf7ewZeLrm3k9M21\nqDcp0PTaf6QKlKLyTX+VfGbKOErBPufikdBmIweHmBE++EPdkF8uRD0Q6kTutSPJ\nqOjkdZm/CkQExOvsOEjIPnLmvYVuyx8Ffwyq8wI68HsV+EOW2//dykqutbf/nh8r\nQatRFNIMd7uf0Es7qIOZmIdJoaFUPXrVUkiWGjVqemUFWgTXlX5H3zwII30m1snB\ndP8GvEXAIgaI+sCv+FAe/+CALXNn2pfMgJC3tJaT2QKBgQDIosUe1ZrGI3/NHBLk\nlfTpgkQeq2Be/RsgSoKarfBjFoQL+bcKNrzZwvVrUqHKw6z9CSC/o8fQzJEyI6Qv\nlPtdfm2CY54m9ku+bNnhrJT1Ir0N4yBqx1TvUC9avM8tjAIAyS75kMpxmnl3REVj\nRHQhALG/dr7eD6uVuWc3oq2XyQKBgQC4Nw0XQ1adIdHLKjIFg/EQY/xe9vlQJXeE\nXFK7G76Ddy9c5Y33BrszQD7In3XAFEpQt40X9Jk0ELYeeyHNtvj1mKfTqeOasYc9\ndiKx1Iyk2PAJLu/KrWOAgZqkoIw9Di8nmqUQDQuQZJXdWjCrpeWgXuWi5KAlrXrY\n5bbBKKFXQwKBgQCro1ugyjly4bGWRTsAZCy1iqQ93HxAJqULFdLvAnhQPFXQbNsi\nmLssS3VqT7FrZcRaX3GtoC3zON3Jl+YAcqIrVFnYnuIQEMmxaStnvR1dLE+xTqlP\nz6Cgbedk53sRSuH3zCQsiao2T2E0H6qAhnIWPaLR2YLlM2ClYIulc7yvyQKBgEK5\nXfqBH4xGxbCrFfc9dUGUlcLn/iHzkDjXH/g58mW38+XYF/XT8FZ9/L+6UnJCZzCZ\na3ez1xIuw2iC/RmAHukGZDOco55Sk0PF2IdFD7ReB4OsBpsI9rODPjpHFMTAsH/N\n2d/YecnRRR8qHdrueh/+f3rjCMbc8VRC0FFioB31AoGAWHocgkpu3LrwmDz/pbrL\nEhRCCV7XL2cUce7O9Nslcm7n0/QgUCf33JQaGxOC57r3Ws8ixcCvb7m7N/208TCx\nz+cvWYmUh1tm69NU6q/BSc/eZxnGidXACXdHB+U97QHpmLgYPpXLMleIlOr6m/Xe\npC+V+PFL/1/nKskS8tbD02U=\n-----END PRIVATE KEY-----\n***REDACTED***\n***REDACTED***\n***REDACTED***\n***REDACTED***\n', 'keepAliveLapse': 150}], 'getStoredProxyStrength', ['/DC=org/DC=terena/DC=tcs/C=NL/O=Nikhef/CN=Francisco Vazquez de Sola Fernandez pvazquez@nikhef.nl', 'km3net_user', '/km3net.org']]}
2024-02-04 18:02:49 UTC WorkloadManagement/JobAgent ERROR: proxyString parameter is not a valid type <class 'str'>

I would appreciate any help figuring out what is going wrong. Submitting jobs through a different server (SURF-SARA's, which runs dirac version 7.3.17) did not have this issue, so I do not know if the problem is the dirac version, the server configuration, or something else.

[fstagni]

This might be for @atsareg : did you install the ProxyRenewalAgent https://github.com/DIRACGrid/DIRAC/wiki/DIRAC-8.0#new-agent server side?

[atsareg]

This has nothing to do with the ProxyRenewalAgent. The problem is supposed to be fixed in #7478