Lettuce not sending keepAlives

[sklakegg]

Hello 👋

Followed: https://redis.io/docs/latest/develop/clients/lettuce/produsage/ and https://github.com/redis/lettuce/wiki/Spring-Support/450db075877f2e3baf522380881272454c29ba61

But I dont see any keep alives (ping/pongs) beeing sent over the connection with: logging.level.io.lettuce.core.protocol=debug

I can manually call the following snippet to see the ping/pong: connection.sync().ping();

I set up the redis client with keep alives as follow:

@Configuration
public class RedisConfig {
    private static final Logger logger = Logger.getLogger(RedisConfig.class.getName());

    @Bean(destroyMethod = "shutdown")
    ClientResources clientResources() {
        return DefaultClientResources.create();
    }

    @Bean(destroyMethod = "shutdown")
    public RedisClient redisClient(ClientResources clientResource) {
        RedisURI redisURI = RedisURI.Builder
            .redis("localhost")
            .withTimeout(Duration.ofSeconds(30))
            .build();

        RedisClient client = RedisClient.create(clientResource, redisURI);

        SocketOptions.TcpUserTimeoutOptions tcpUserTimeout = SocketOptions.TcpUserTimeoutOptions.builder()
            .tcpUserTimeout(Duration.ofSeconds(20))
            .enable().build();

        SocketOptions.KeepAliveOptions keepAliveOptions = SocketOptions.KeepAliveOptions.builder()
            .interval(Duration.ofSeconds(5))
            .idle(Duration.ofSeconds(5))
            .count(3).enable().build();

        SocketOptions socketOptions = SocketOptions.builder()
            .tcpUserTimeout(tcpUserTimeout)
            .keepAlive(keepAliveOptions)
            .build();

        client.setOptions(ClientOptions.builder()
            .socketOptions(socketOptions)
            .build());

        return client;

    }

    @Bean(destroyMethod = "close")
    StatefulRedisConnection<String, String> connection(RedisClient redisClient) {
        return redisClient.connect();
    }
}

To verify settings and to see the ping/pong I do:

    @GetMapping("/test-redis2")
    public void testRedisConnection2() {
        logger.info(connection.getOptions().getSocketOptions().getKeepAlive().getInterval().toString());
        logger.info(connection.getOptions().getSocketOptions().getKeepAlive().getIdle().toString());
        logger.info(String.valueOf(connection.getOptions().getSocketOptions().getKeepAlive().getCount()));

        connection.sync().ping();
    }

Prints:

2025-02-11T14:10:57.077+01:00  INFO 5551 --- [demo] [nio-8080-exec-3] com.example.demo.RedisTestController     : PT5S
2025-02-11T14:10:57.077+01:00  INFO 5551 --- [demo] [nio-8080-exec-3] com.example.demo.RedisTestController     : PT5S
2025-02-11T14:10:57.077+01:00  INFO 5551 --- [demo] [nio-8080-exec-3] com.example.demo.RedisTestController     : 3
2025-02-11T14:10:57.077+01:00 DEBUG 5551 --- [demo] [nio-8080-exec-3] i.lettuce.core.protocol.DefaultEndpoint  : [channel=0x8920dabb, /127.0.0.1:52488 -> localhost/127.0.0.1:6379, epid=0x1] write() writeAndFlush command AsyncCommand [type=PING, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
2025-02-11T14:10:57.077+01:00 DEBUG 5551 --- [demo] [nio-8080-exec-3] i.lettuce.core.protocol.DefaultEndpoint  : [channel=0x8920dabb, /127.0.0.1:52488 -> localhost/127.0.0.1:6379, epid=0x1] write() done
2025-02-11T14:10:57.077+01:00 DEBUG 5551 --- [demo] [ioEventLoop-4-1] i.lettuce.core.protocol.CommandHandler   : [channel=0x8920dabb, /127.0.0.1:52488 -> localhost/127.0.0.1:6379, epid=0x1, chid=0x1] write(ctx, AsyncCommand [type=PING, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command], promise)
2025-02-11T14:10:57.078+01:00 DEBUG 5551 --- [demo] [ioEventLoop-4-1] i.lettuce.core.protocol.CommandEncoder   : [channel=0x8920dabb, /127.0.0.1:52488 -> localhost/127.0.0.1:6379] writing command AsyncCommand [type=PING, output=StatusOutput [output=null, error='null'], commandType=io.lettuce.core.protocol.Command]
2025-02-11T14:10:57.078+01:00 DEBUG 5551 --- [demo] [ioEventLoop-4-1] i.lettuce.core.protocol.CommandHandler   : [channel=0x8920dabb, /127.0.0.1:52488 -> localhost/127.0.0.1:6379, epid=0x1, chid=0x1] Received: 7 bytes, 1 commands in the stack
2025-02-11T14:10:57.078+01:00 DEBUG 5551 --- [demo] [ioEventLoop-4-1] i.lettuce.core.protocol.CommandHandler   : [channel=0x8920dabb, /127.0.0.1:52488 -> localhost/127.0.0.1:6379, epid=0x1, chid=0x1] Stack contains: 1 commands
2025-02-11T14:10:57.078+01:00 DEBUG 5551 --- [demo] [ioEventLoop-4-1] i.l.core.protocol.RedisStateMachine      : Decode done, empty stack: true
2025-02-11T14:10:57.078+01:00 DEBUG 5551 --- [demo] [ioEventLoop-4-1] i.lettuce.core.protocol.CommandHandler   : [channel=0x8920dabb, /127.0.0.1:52488 -> localhost/127.0.0.1:6379, epid=0x1, chid=0x1] Completing command AsyncCommand [type=PING, output=StatusOutput [output=PONG, error='null'], commandType=io.lettuce.core.protocol.Command]

[tishun]

Hey @sklakegg ,

The keep-alice in your first snippet is a TCP socket keep alive and not a RESP-level keep alive (sending some command - like PING - and expecting a result to verify the health of the connection).

Wouldn't this be enough for your use case?

[sklakegg]

Thanks for the reply @tishun 🙏

I see, I also came across #2256 which probably alludes to a similar scenario. We have nginx between our client and the azure redis server. But reading the Azure Redis documentation:

https://learn.microsoft.com/en-us/azure/azure-cache-for-redis/cache-best-practices-connection#idle-timeout
Azure Cache for Redis has a timeout of 10 minutes before cleaning leaky connections, which is what we are experiencing. They are suggesting that:

If your application is using a client library that doesn't have native support for keepalive functionality, you can implement it in your application by periodically sending a PING command.

My understanding is that we would need to periodically send a PING to keep the connection to Azure Cache for Redis open? I guess in addition to any TCP keepalives.

[tishun]

In this case my recommended approach would be to implement a NettyCustomizer (see #499) that add a custom IdleStateHandler that sends PING commands when the channel is inactive.

[sklakegg]

Hey, thanks again for the reply. Yes, some more docs is always appreciated 👍 I guess there is quite a few who is also using Azure Cache for Redis.

I am trying to wrap my head around how implementing ping support would look when using the LettucePoolingClientConfiguration
with e.g., setMinIdle of 1 and setMaxTotal 4. As far as I can tell, you would want the ping handler to only be active on only one of the connections, in order for the remaining ones to be scaled up/down?

[tishun]

Hey, thanks again for the reply. Yes, some more docs is always appreciated 👍 I guess there is quite a few who is also using Azure Cache for Redis.

I am trying to wrap my head around how implementing ping support would look when using the LettucePoolingClientConfiguration with e.g., setMinIdle of 1 and setMaxTotal 4. As far as I can tell, you would want the ping handler to only be active on only one of the connections, in order for the remaining ones to be scaled up/down?

Not sure you are on the right path there. These settings strictly control the size of the pool and not some heartbeat (which is what you need if I understand correctly). Btw why do you need a pool in the first place?

[tishun]

In this case my recommended approach would be to implement a NettyCustomizer (see #499) that add a custom IdleStateHandler that sends PING commands when the channel is inactive.

My idea was a bit simpler and should work with any type of connection made with a client that has the appropriate ClientResources configured.

You need 3 parts.

First you configure the client to use a Netty customizer of your own:

        StatefulRedisConnection<String, String> connection;
        ClientResources resources = ClientResources.builder().nettyCustomizer(new HeartBeatCustomizer()).build();

        try (RedisClient redisClient = RedisClient.create(resources, RedisURI.create("com.acme", 6474))) {
            asyncCommands = redisClient.connect().async();

Then you need to make a customizer that patches your pipeline to monitor for inactivity and call your custom handler:

class HeartBeatCustomizer implements NettyCustomizer {

    public static final int TIMEOUT_SECONDS = 5; // 5 seconds for testing purposes, use something else in production, e.g. if
                                                 // this is Azure Cache for Redis you need to send hearth beat every 10 minutes

    @Override
    public void afterChannelInitialized(Channel channel) {
        // Add IdleStateHandler to make sure an event is fired when the connection is idle
        channel.pipeline().addLast(new IdleStateHandler(0, 0, TIMEOUT_SECONDS));
        // Add another handler to process this event and send a PING to the server
        channel.pipeline().addFirst(new HeartBeatHandler());
    }
}

Last you need to send a PING whenever the connection is idle:

class HeartBeatHandler extends ChannelDuplexHandler {
    private static final Logger logger = LoggerFactory.getLogger(HeartBeatHandler.class);

    @Override
    public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception {
        if (evt instanceof IdleStateEvent event) {
            if (event.state() == IdleState.ALL_IDLE && MyApp.asyncCommands != null) {
                RedisFuture<String> heartBeat = MyApp.asyncCommands.ping();
                heartBeat.get();
            }
        }
    }
}

Pooled or not each connection created with the client would watch for idle time and send a ping.

[tishun]

Oh ... an even simpler example is given in #2078

[sklakegg]

Thanks a lot @tishun

I did some QA on our current setup and I found that pooled connection was some legacy config that was passed down from our previous Redis client. I removed it and implemented the ping functionality in #2078
Problem solved ✅ 🙏

[tishun]

Awesome, happy to hear that works and thanks for letting me know!