Job owner problems in transformations after migrating to v8.0.39

[zhangxiaomei]

I didn't set shifterProxy for WorkflowTaskAgent-JUNO of TS [1]. In 8.0.24, the jobs created by WorkflowTaskAgent are using owner proxy of the transformation. But after migrating to v8.0.39, all the jobs created from WorkflowTaskAgent are using a single shifter proxy with wrong VO group [1]. I am confused by that Any suggestions to debug that?
[1]

[2] submit a transformation by a "cjiang" juno user, but jobs submitted with "zhangxm" bes user
4000000065fbd0361fa52494.s:2024-03-21 06:07:56 UTC Transformation/WorkflowTaskAgent-JUNO/Transformation/WorkflowTaskAgent-JUNO [139749702289216] INFO: ShifterProxy: Tasks will be submitted with the credentials zhangxm:bes_user

[fstagni]

Can it be a consequence of #7113 ? @sfayer @marianne013

[andresailer]

My guess is that:

The issue is that there is a /tmp/x509_u$(id -u dirac) file on the server, and the WTA picks that up via

DIRAC/src/DIRAC/TransformationSystem/Agent/TaskManagerAgentBase.py

Lines 106 to 112 in 26f60b6

	resProxy = getProxyInfo(proxy=False, disableVOMS=False)
	if resProxy["OK"]: # there is a shifterProxy
	proxyInfo = resProxy["Value"]
	owner = proxyInfo["username"]
	ownerGroup = proxyInfo["group"]
	ownerDN = proxyInfo["identity"]
	self.log.info(f"ShifterProxy: Tasks will be submitted with the credentials {owner}:{ownerGroup}")

usually the shifter proxy will be created in some randomized x509 file and cross-talk is unlikely.
But then there is the fall back to

DIRAC/src/DIRAC/Core/Security/Locations.py

Lines 19 to 22 in 26f60b6

	proxyName = "x509up_u%d" % os.getuid()
	if os.path.isfile(f"/tmp/{proxyName}"):
	return f"/tmp/{proxyName}"

Probably we should make that fall back optional. I think this happened before already.

So for now delete the /tmp/x509_u$(id -u dirac) file from the sever.

[zhangxiaomei]

Yes, exactly. There is accidently /tmp/x509_u$(id -u dirac) in the server. I just realized that.

[marianne013]

If there's really a /tmp/x509_ *on the server, then usually all bets are off. I've left one accidentally more than once to all kinds of odd effects :-(

[zhangxiaomei]

At the beginning I didn't realize that "ProductionManagement" property need to be added to the host in 8.0.39 and got a permission error. I put a proxy there for testing and forgot to delete it.
Also I take it for granted that only valid proxy in the server would cause problems and didn't know even the proxy file would cause problem too :(