CS saturated and not able to commit any change

[arrabito]

Dear all,

We are having troubles with the Configuration Service which seems to be saturated. I've observed that many clients (jobs) get timeout errors when connecting to the master instance dips://dcta-servers02.pic.es:9135/Configuration/Server (while I don't observe errors for connecting to the other slave instances). We have several slave CS services, but I suspect that pilots are currently pointing to a single CS instance (the master) and it's maybe the cause of CS getting saturated.
I suspect this because the default client/pilot configuration comes with only the master instance configured:

DIRAC
{
  Configuration
  {
    Servers = dips://dcta-servers02.pic.es:9135/Configuration/Server

I was thinking about adding ExtraPilotOptions to add the other CS endpoints, something like:

ExtraPilotOptions= -o /DIRAC/Configuration/Servers=dips://dcta-servers.pic.es:9135/Configuration/Server,dips://dcta-agents.pic.es:9135/Configuration/Server,dips://ccdcta-server04.in2p3.fr:9135/Configuration/Server

(Even if I'm not sure this is the right way to do.)

Then, we also have only 1 CS endpoint (not the master) configured under:

DIRAC
{
  Configuration
  {
    Servers = dips://ccdcta-web01.in2p3.fr:9135/Configuration/Server

and I would like to add all other endpoints.

However, since I'm not able to connect to the CS (neither from the Web portal, neither from dirac-configuration-cli) I'm not able to change any configuration parameter.

I've tried to change the section directly on the dirac.cfg on the server where the CS master instance is running:

DIRAC
{
  Setup = CTA
  Hostname = dcta-servers02.pic.es
  Configuration
  {
    Servers = dips://dcta-agents02.pic.es:9135/Configuration/Server
    Servers += dips://ccdcta-server04.in2p3.fr:9135/Configuration/Server
    Servers += dips://ccdcta-server05.in2p3.fr:9135/Configuration/Server
    Servers += dips://ccdcta-web01.in2p3.fr:9135/Configuration/Server
    MasterServer = dips://dcta-servers02.pic.es:9135/Configuration/Server   

but it doesn't seem to have any positive effect.

Can you suggest some other means to fix this problem and to commit changes to the CS in this particular situation?

Thank you.

[arrabito]

Just to let you know that when the load went down, I've been able to update CS with:

DIRAC
{
  Configuration
  {
    Servers = dips://dcta-agents02.pic.es:9135/Configuration/Server
    Servers += dips://ccdcta-web01.in2p3.fr:9135/Configuration/Server
    Servers += dips://cta-dirac.zeuthen.desy.de:9135/Configuration/Server
    Servers += dips://ccdcta-server04.in2p3.fr:9135/Configuration/Server
    Servers += dips://ccdcta-server05.in2p3.fr:9135/Configuration/Server
    MasterServer = dips://dcta-servers02.pic.es:9135/Configuration/Server

and as far as I understand, this is the list of CS endpoints also used for pilots. Is that correct?

In the pilot log I find:

2022-02-23 08:13:01 UTC DEBUG [PilotParams] CS list: dips://dcta-servers02.pic.es:9135/Configuration/Server,dips://dcta-agents02.pic.es:9135/Configuration/Server,dips://ccdcta-server04.in2p3.fr:9135/Configuration/Server,dips://ccdcta-server05.in2p3.fr:9135/Configuration/Server,dips://ccdcta-web01.in2p3.fr:9135/Configuration/Server,dips://cta-dirac.zeuthen.desy.de:9135/Configuration/Server

which seems to confirm that all the list is taken into account.

So in principle I don't need any ExtraPilotOptions.

I will monitor anyway if the timeouts to the Master endpoints will decraese:

2022-02-23 08:08:48 UTC Transformation/RequestTaskAgent WARN: Non-responding URL temporarily banned dips://dcta-servers02.pic.es:9135/Configuration/Server

Thanks.

[fstagni]

Hi Luisa,

1. first, operationally "how to commit changes to the CS when the service is overloaded". You can't go through the web interface in this case. You can change directly the .cfg file, though. The CS (the "service") would read and update the a specific cfg, file, normally it's called "Production.cfg" or something like that (do not change the etc/dirac.cfg which is for internal use -- and I suspect its content on this specific option is never used)
2. the ExtraPilotOptions are not for adding options to the pilot configuration (what you were thinking to do would actually make the pilots to crash)
3. do you have a pilot extension? Do you use a (VO)DIRAC already deployed on CVMFS? In not, the "first" CS (list) used might come down to what is described in Specify a list of CS url in the extension_metadata #5704

[arrabito]

Hi Federico,
My comments below.

1. OK, I was not sure that manually editing the "Production.cfg" was fair. I guess it should be done only in 'extreme' cases. Anyway, it's good to know that etc/dirac.cfg is probably not used for this specific option.
2. ok. I was just trying to find a workaround
3. We don't have a pilot extension. And we don't use CTADIRAC deployed on CVMFS. So, I was aware indeed that the first list comes as described in Specify a list of CS url in the extension_metadata #5704 . The problem is exactly the one described in that issue: we haven't found yet a way to pass a list. Do you have any suggestion? How do you do in LHCb? Should we add a pilot extension for this purpose?

So reading your questions, I understand that we still have a unique endpoint as first list used by the pilot and that we will probably hit the saturation problem again. Is that correct?

Thank you.

[fstagni]

1. Yes, in a way it should be done only in extreme cases (you need to be sure no-one else is changing it!)
2. In LHCb we deploy on CVMFS, and we just "source" a bashrc, and the .cfg file in use is also on CVMFS. I don't think anyway that you should have a pilot extension for that purpose, solving Specify a list of CS url in the extension_metadata #5704 should be enough (I didn't look into it).

The risk of saturation in the current situation is still real.

[arrabito]

See my answer here: #5704
It seems that it was already fixed.
Thank you.

[arrabito]

1. OK.
2. OK thank you for the explanations. Is there any doc or some instructions to deploy an extension on CVMFS? In any case, we are also interested in solving About Specify a list of CS url in the extension_metadata #5704.

Thanks.

[arrabito]

I think sorting out for #5704 would be best. Could you give it a try?

OK. Sorry, what should I try?

I am not fully aware of docs for deploying on CVMFS, and LHCb's solution might be a bit too specific. Maybe @atsareg have a better answer here.

OK

[fstagni]

I think sorting out for #5704 would be best. Could you give it a try?

OK. Sorry, what should I try?

Try to code a solution, in case it's not already coded (I didn't check what's possible)

[arrabito]

OK fine, I admit that I haven't looked into it because I thought that somebody had the answer already, but I will do. Any suggestion on which part of code I could start to look at?

[fstagni]

start from DIRAC/Core/scripts/dirac_configure.py

[arrabito]

OK thank you.