Configure Keycloak worker thread pool to be aligned with JGroup thread pool size

[ahus1]

This prepares the setup for the next upcoming load test where we align the Keycloak workers.

@kami619 - please review the docs and test in a single cluster with 100_000 sessions, 200 ups sustained load 100 pod kills (like the previous run where we saw the spikes).

Thanks!

[pruivo]

That is not entirely true since you can still configure the thread pool in the extended stack. As an example

    <jgroups>
        <stack name="kubernetes-with-fdsock" extends="kubernetes">
            <TCP thread_pool.max_threads="${env.JGROUPS_MAX_THREADS:200}" />
            <!-- When using an embedded stack, replacement is done by Infinispan which requires environment variables to be
            prefixed with ".env", while JGroups falls back to environment variables without the prefix.
            The Keycloak Operator passes this information in an enviornment variable `jgroups.dns.query`.
            See https://github.com/keycloak/keycloak/issues/21830 for a discussion.
            -->
            <dns.DNS_PING dns_query="${env.jgroups.dns.query}" />
            <!-- Workaround for problems with FD_SOCK2, which might fail when the other nodes are not ready yet.
            See https://github.com/keycloak/keycloak/issues/21092 -->
            <FD_SOCK stack.combine="REPLACE" stack.position="FD_SOCK2"/>
        </stack>
    </jgroups>

ps. It looks for JGROUPS_MAX_THREADS and if not set, it uses 200 threads

[ahus1]

@pruivo - I tried this. I debugged that the value was correctly picked up in StringPropertyReplacer, but it never made it to the TCP instance, where the thread pool was still set up to be 100 threads only.

So I guess I'll wait until ISPN-14780 fixed in an upcoming release.

[kami619]

@ahus1 do you think we need a run based off of the latest changes, if not I think the changes look good from the results we had from yday.

[ahus1]

Do you think we need a run based off of the latest changes, if not I think the changes look good from the results we had from yday.

@kami619 - I think we're good to merge this one. We should just let everyone know that we now restrict the number of Keycloak worker threads. We might lift that later with the next Infinispan upgrade to 200.