no longer possible to purge jobs

[eric1708]

Hi everyone. I currently have a problem that I remained unable to solve. I guess this problem came with the last version of DIRAC (i.e., version 8.0.49), that I've installed last week.

If I want to delete jobs, I use dirac-wms-job-delete. For years, doing that returned the list of the jobs deleted. This is not the case anymore. Even more, I now collect back the following message only, and this for a almost a week:

Some jobs failed deletion

The (other) point is that I still have a lot of jobs listed for days that I cannot get rid of:

dstat | wc
2736 30699 361018

They are all "killed" or "deleted", and this is not making my life really easy, since I have to look for those that are running or not only.

Is there something I am missing? How can these problems be solved?

Any help on that would be more than welcomed.

Thanks in advance for any help & cheers,

Eric.

[fstagni]

Hi, a few basic questions:

• did you update both client and server?
• from which version did you update? (what was the version installed prior to v8.0.49?)
• the non-purged jobs are from different users?
• if you can, it would be useful to get the log of "JobCleaningAgent". Can you get that?

[eric1708]

Thanks Federico for your time on this.

1. I had to "fight" to update the current version. So, I actually deleted the previous one, and I re-installed the current version from scratch.

2. As far as I understand (not sure) I only installed the client.

3. All these remaining jobs are mine. I am the only user of Dirac on the machine.

4. I do not know how to get the log of JobCleaningAgent. I'll be happy to do that if you explain..

Can this problem be solved?

Thanks again for your time!

Eric.

[fstagni]

I see, more questions then:

• to which server are you connecting to?
• what is the version of the server (I guess you are not its admin...)
• did you recently change the certificate?

The jobCleaningAgent log are (at a minimum) at this location on the server: /opt/dirac/runit/WorkloadManagement/JobCleaningAgent/log/current

[eric1708]

Thanks again for your time!

I'll try to answer (but I am not sure I can..).

1. I guess the server is https://dirac.egi.eu/.
2. Do not know the version of the sever.
3. I changed the certificate indeed rather recently, but the problem did not come right after that.
4. There is nothing like /opt/dirac/runit/WorkloadManagement/JobCleaningAgent/log/ on my machine. I also search the entire machine for the string "JobCleaningAgent". Nothing.. And I do not think I can have access to the server.

Thanks again for your time!

Eric.

[marianne013]

Hi Eric,

To see which DIRAC server you are submitting to, you can check diracos/etc/dirac.cfg in your client.

Daniela

[eric1708]

Thanks Daniela,

The information in this file is:

dips://dirac-conf.in2p3.fr:9135/Configuration/Server

And sorry for not being an expert in this. I am just a simple & standard user..

Eric.

[fstagni]

So, you can't solve this by yourself, an admin of the server should look into this. CC @atsareg @hamar

Last question: when you changed the certificate, was your DN changed too?

[eric1708]

Thanks!

I now Andrei. I'll ask him then.

No, my DN did not change.

Thanks again for your help!

Eric.

[atsareg]

Actually, the JobCleaningAgent got stuck for a while. This is fixed now. Sorry for the inconvenience

[eric1708]

Thanks Andrei. Things are now indeed working, and the jobs are now purged.

However the purging process is really slow. I estimated that about 700 jobs are purged per hour, which is at least 4 or 5 times slower than what is was on previous version of dirac. If I launch, e.g., 5000 jobs, I will have to wait 7-8 h before all of them to disappear. Can this be improved?

In all cases, thanks for your help & cheers,

Eric Wajnberg