NATS PING/PONG does not reconnects correctly when client can't talk with the server

[liviozanol]

Observed behavior

It seems that even with the ping_interval and max_outstanding_pings configured properly if the nats client lose connection to the server (IE the server is network unreachable), the nats client will not teardown the connection on the expected time.

Below a pcap showing the behavior with default timers. 192.168.0.1:1234 is a NATS Jetstream server

#OK - ping/pong working properly
15:16:45.510365 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 964888961:964888989, ack 3349841365, win 501, options [nop,nop,TS val 3739277461 ecr 3920449588], length 28
15:16:45.510693 IP 192.168.0.1.1234 > 192.168.0.2.53674: Flags [P.], seq 1:29, ack 28, win 501, options [nop,nop,TS val 3920569588 ecr 3739277461], length 28
15:16:45.510710 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [.], ack 29, win 501, options [nop,nop,TS val 3739277461 ecr 3920569588], length0

# Traffic was blocked. Client sends PING, does not receive response but doesn't tear down the connection.
15:18:45.511920 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3739397462 ecr 3920569588], length 28
15:18:45.716279 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3739397667 ecr 3920569588], length 28
15:18:45.924277 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3739397875 ecr 3920569588], length 28
15:18:46.360284 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3739398311 ecr 3920569588], length 28
15:18:47.192309 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3739399143 ecr 3920569588], length 28
15:18:48.856277 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3739400807 ecr 3920569588], length 28
15:18:52.184299 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3739404135 ecr 3920569588], length 28
15:18:58.840290 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3739410791 ecr 3920569588], length 28
15:19:12.152328 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3739424103 ecr 3920569588], length 28
15:19:39.800289 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3739451751 ecr 3920569588], length 28
15:20:33.048292 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3739504999 ecr 3920569588], length 28
15:22:19.544297 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3739611495 ecr 3920569588], length 28
15:24:20.376285 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3739732327 ecr 3920569588], length 28
15:26:21.208235 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3739853159 ecr 3920569588], length 28
15:28:22.040312 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3739973991 ecr 3920569588], length 28
15:30:22.872301 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3740094823 ecr 3920569588], length 28
15:32:23.704231 IP 192.168.0.2.53674 > 192.168.0.1.1234: Flags [P.], seq 28:56, ack 29, win 501, options [nop,nop,TS val 3740215655 ecr 3920569588], length 28


# Reconnection finally attempted
15:34:24.538864 IP 192.168.0.2.52702 > 192.168.0.1.1234: Flags [S], seq 870453250, win 65495, options [mss 65495,sackOK,TS val 3740336489 ecr 0,nop,wscale 7], length 0
15:34:25.560336 IP 192.168.0.2.52702 > 192.168.0.1.1234: Flags [S], seq 870453250, win 65495, options [mss 65495,sackOK,TS val 3740337511 ecr 0,nop,wscale 7], length 0

Expected behavior

Connection should be teardown as soon as the max_outstanding_pings threshold is reached

Server and client version

nats-py==2.7.2

Host environment

No response

Steps to reproduce

This is easy to reproduce. Just spin a server, connect a client to it and make the client not able to reach the server anymore (add route to null/loopback, or create an iptables rule for example)