"Idle" SSH connections are forcefully disconnected

[otavio]

Discussed in #5945

^{Originally posted by zoopp March 5, 2026}
I'm running into an interesting issue with my setup. I host ShellHub CE on GCP behind an External Proxy Network Load Balancer, essentially with an architecture like this:

The problem I keep encountering is that idle SSH connections get forcibly disconnected after a couple of minutes of inactivity:

This happens both when connecting through ShellHub and when using native ssh directly. Interestingly, the disconnect does not occur if there is periodic activity on the connection. For example, running something like watch <cmd> (which produces output every couple of seconds) keeps the connection alive indefinitely.

However, I expected that SSH keep alive settings might mitigate this, at least for native SSH connections. I've experimented with options like:

• TCPKeepAlive
• ServerAliveInterval
• ServerAliveCountMax

... but the connection still gets dropped after a few minutes of inactivity.

Does anyone have suggestions on what could be done to prevent this, at least for native SSH connections, without changing the overall GCP setup?

[otavio]

@zoopp, Thanks for the detailed report! We've been investigating this on our side and have a strong hypothesis about what's happening, but we need a few pieces of information from your setup to confirm before we can move forward with a fix.

Our current hypothesis: The connection that's actually dropping isn't the SSH client → ShellHub server link, but rather the agent's reverse tunnel (the WebSocket connection between your device agent and the SSH service, which goes through Nginx). When that tunnel dies, the SSH session ends — regardless of whether the client-side keepalive is working. This would explain why ServerAliveInterval doesn't help: it keeps the wrong connection alive.

Could you help us narrow this down?

1. Agent version: What version of the ShellHub agent are you running on your devices? (This determines which transport protocol is used — newer agents use yamux/V2, older ones use revdial/V1, and they have different keepalive behaviors.)

2. Exact disconnect timing: Is the disconnect consistently around 60 seconds, 120 seconds, or some other interval? If you can time it precisely (e.g., connect, do nothing, measure until disconnect), that would be very helpful.

3. SSH service logs: Could you check the SSH service container logs around the time of a disconnect? We're looking for messages like:

   • "close connection due pong timeout" (WebSocket keepalive failure)
   • "failed to ping yamux session" (yamux keepalive failure)
   • "failed to send keep-alive message to device" (revdial keepalive failure)

   You can capture these with:

   docker compose logs ssh --since 5m
   

4. Nginx error logs: Similarly, could you check if Nginx reports any upstream timed out errors on the WebSocket proxy locations?

   docker compose logs gateway --since 5m
   

5. GCP Load Balancer ports: Does your LB proxy both port 2222 (SSH) and port 80/443 (HTTP) to the ShellHub backend? Or only one of them?

This will help us confirm the root cause and target the right fix.

[zoopp]

Thank you for the follow-up, @otavio. Please give me a bit of time to collect and organize the information you requested, and I'll respond here once it's ready.

[ltan10]

Is it something as trivial as the shellhub-agent keepalive being set longer than a certain time?

When I set the agent KEEPALIVE_INTERVAL to 5 mins and seeing this issue. I tried to see how long an idle device connection would last.
I am observing the same behavior where the connection was ended prematurely.

When i changed the KEEPALIVE_INTERVAL back to 30s. The idle connection has been solid for a good 10 mins without any drops.

The watch command is i suppose keeping the connection alive by transmitting a packet to update the output.

Shellhub Server:

• Version: v0.21.5
• Configuration: Sitting behind reverse proxy, no load balancing, proxy timeout set at 60s

Shellhub-Agent: Self compiled Latest Go Version

[zoopp]

@otavio

1. Agent version: v0.21.4.

2. Disconnect timing: I used time ssh ... to time the sessions. So far:

   • 1 session timed out after 2m3s.
   • 2 consecutive sessions timed out after 1m33s.
   • 1 session timed out after 5m38s.
   • 1 session timed out after 2m31s.
   • 2 session timed out after 1m4s.

   As you can see, it's a bit all over the place.

3. SSH service logs (filtered by uid):

   5m38s session:

   ssh-1  | {"level":"info","msg":"new connection established","sshid": "-redacted-","time":"2026-03-05T12:25:34Z","uid":"721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04"}
   ssh-1  | {"attempt":1,"id":"CLZLDMUlDINxoJRwsPIdKOwzrGCeEGUz","level":"info","method":"PATCH","msg":"internal client response received","status":200,"time":"2026-03-05T12:25:42Z","url":"http://api:8080/internal/sessions/721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04"}
   ssh-1  | {"key": "-redacted-","level":"info","msg":"succeeded to use public key authentication.","sshid": "-redacted-","time":"2026-03-05T12:25:42Z","uid":"721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04"}
   ssh-1  | {"device":"03045b535ee22f67f500d71f59e53221d1de36090abb4e4ef9f1f155418e382f","ip": "-redacted-","level":"info","msg":"session channel started","sshid": "-redacted-","time":"2026-03-05T12:25:42Z","uid":"721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04","username": "-redacted-"}
   ssh-1  | {"attempt":1,"id":"kOOoJlGwQlFfdJdXXjqSoTKRrjObNzEB","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:26:12Z","url":"http://api:8080/internal/sessions/721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04/keepalive"}
   ssh-1  | {"attempt":1,"id":"lcBCXTYjtyYcPSZrSgWChJqGOwihqctU","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:26:42Z","url":"http://api:8080/internal/sessions/721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04/keepalive"}
   ssh-1  | {"attempt":1,"id":"hbyjTllZsqUsdGfveOcoJhQCtVupjcpP","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:27:12Z","url":"http://api:8080/internal/sessions/721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04/keepalive"}
   ssh-1  | {"attempt":1,"id":"IfdsZSlUKbczvZiSEgPQbzwcgYtiDmwS","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:27:42Z","url":"http://api:8080/internal/sessions/721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04/keepalive"}
   ssh-1  | {"attempt":1,"id":"VXrcdOsmGXiPaIMJFKHJddxoiXcemYPJ","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:28:12Z","url":"http://api:8080/internal/sessions/721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04/keepalive"}
   ssh-1  | {"attempt":1,"id":"HFADKuRCEYElJIzFgyjRWnTCsUWENwTF","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:28:42Z","url":"http://api:8080/internal/sessions/721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04/keepalive"}
   ssh-1  | {"attempt":1,"id":"kckjuREbdLmrkDdWRppmqfYrnoxrpIvd","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:29:12Z","url":"http://api:8080/internal/sessions/721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04/keepalive"}
   ssh-1  | {"attempt":1,"id":"QZJaSTiEYALUwroTjAzRUZXQqYHUdMvk","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:29:42Z","url":"http://api:8080/internal/sessions/721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04/keepalive"}
   ssh-1  | {"attempt":1,"id":"jboKIxTyeQOawRdDhlWGVyBDOpURHBmt","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:30:12Z","url":"http://api:8080/internal/sessions/721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04/keepalive"}
   ssh-1  | {"attempt":1,"id":"UuGtKwAFEoNxwKFrHdfrOleyUtIhiwsz","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:30:42Z","url":"http://api:8080/internal/sessions/721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04/keepalive"}
   ssh-1  | {"device":"03045b535ee22f67f500d71f59e53221d1de36090abb4e4ef9f1f155418e382f","ip": "-redacted-","level":"info","msg":"session channel done","sshid": "-redacted-","time":"2026-03-05T12:31:12Z","uid":"721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04","username": "-redacted-"}
   ssh-1  | {"attempt":1,"id":"UiDiwVMQDioIaFCAVCUsIKvYlnXBmekQ","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:31:12Z","url":"http://api:8080/internal/sessions/721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04/finish"}
   ssh-1  | {"device":"03045b535ee22f67f500d71f59e53221d1de36090abb4e4ef9f1f155418e382f","ip": "-redacted-","level":"info","msg":"session finished","time":"2026-03-05T12:31:12Z","uid":"721c17ea9cac13a62016ec68b2440181c82e217d967ea756c017e5b007729d04","username": "-redacted-"}
   

   2m31s session:

   ssh-1  | {"level":"info","msg":"new connection established","sshid": "-redacted-","time":"2026-03-05T12:33:53Z","uid":"224f8065590428819ac2030e460fa5b0f4a20ace67b16ef64c92f8f08be3f10a"}
   ssh-1  | {"attempt":1,"id":"aiyetEVAhyEHQoLrHYGWZqgHbotGnbYA","level":"info","method":"PATCH","msg":"internal client response received","status":200,"time":"2026-03-05T12:33:55Z","url":"http://api:8080/internal/sessions/224f8065590428819ac2030e460fa5b0f4a20ace67b16ef64c92f8f08be3f10a"}
   ssh-1  | {"key": "-redacted-","level":"info","msg":"succeeded to use public key authentication.","sshid": "-redacted-","time":"2026-03-05T12:33:55Z","uid":"224f8065590428819ac2030e460fa5b0f4a20ace67b16ef64c92f8f08be3f10a"}
   ssh-1  | {"device":"03045b535ee22f67f500d71f59e53221d1de36090abb4e4ef9f1f155418e382f","ip": "-redacted-","level":"info","msg":"session channel started","sshid": "-redacted-","time":"2026-03-05T12:33:55Z","uid":"224f8065590428819ac2030e460fa5b0f4a20ace67b16ef64c92f8f08be3f10a","username": "-redacted-"}
   ssh-1  | {"attempt":1,"id":"zAYJxPELSDZbIjSOtDgkbBLnQUNIRubn","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:34:25Z","url":"http://api:8080/internal/sessions/224f8065590428819ac2030e460fa5b0f4a20ace67b16ef64c92f8f08be3f10a/keepalive"}
   ssh-1  | {"attempt":1,"id":"iAFMTVDQZHSdQWUNrEIqqBetcAqbsBiJ","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:34:55Z","url":"http://api:8080/internal/sessions/224f8065590428819ac2030e460fa5b0f4a20ace67b16ef64c92f8f08be3f10a/keepalive"}
   ssh-1  | {"attempt":1,"id":"zbZUeHwVPJggNBIjIIubxFJjDaDucInV","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:35:25Z","url":"http://api:8080/internal/sessions/224f8065590428819ac2030e460fa5b0f4a20ace67b16ef64c92f8f08be3f10a/keepalive"}
   ssh-1  | {"attempt":1,"id":"BcGQMRilhuFGTfJjGXuWhmMDqwHwoGEz","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:35:55Z","url":"http://api:8080/internal/sessions/224f8065590428819ac2030e460fa5b0f4a20ace67b16ef64c92f8f08be3f10a/keepalive"}
   ssh-1  | {"device":"03045b535ee22f67f500d71f59e53221d1de36090abb4e4ef9f1f155418e382f","ip": "-redacted-","level":"info","msg":"session channel done","sshid": "-redacted-","time":"2026-03-05T12:36:25Z","uid":"224f8065590428819ac2030e460fa5b0f4a20ace67b16ef64c92f8f08be3f10a","username": "-redacted-"}
   ssh-1  | {"attempt":1,"id":"DQyoopQTPeEfrUgrxNigrGOCXQnYlUUO","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:36:25Z","url":"http://api:8080/internal/sessions/224f8065590428819ac2030e460fa5b0f4a20ace67b16ef64c92f8f08be3f10a/finish"}
   ssh-1  | {"device":"03045b535ee22f67f500d71f59e53221d1de36090abb4e4ef9f1f155418e382f","ip": "-redacted-","level":"info","msg":"session finished","time":"2026-03-05T12:36:25Z","uid":"224f8065590428819ac2030e460fa5b0f4a20ace67b16ef64c92f8f08be3f10a","username": "-redacted-"}
   

   1m4s session:

   ssh-1  | {"level":"info","msg":"new connection established","sshid": "-redacted-","time":"2026-03-05T12:39:13Z","uid":"8753e313dca1d382048d900df23371686802befab162506df29107058cc7dafc"}
   ssh-1  | {"attempt":1,"id":"BIIPPwOUiupvqRAmsYmVjCBAaQcchDGb","level":"info","method":"PATCH","msg":"internal client response received","status":200,"time":"2026-03-05T12:39:17Z","url":"http://api:8080/internal/sessions/8753e313dca1d382048d900df23371686802befab162506df29107058cc7dafc"}
   ssh-1  | {"key": "-redacted-","level":"info","msg":"succeeded to use public key authentication.","sshid": "-redacted-","time":"2026-03-05T12:39:17Z","uid":"8753e313dca1d382048d900df23371686802befab162506df29107058cc7dafc"}
   ssh-1  | {"device":"03045b535ee22f67f500d71f59e53221d1de36090abb4e4ef9f1f155418e382f","ip": "-redacted-","level":"info","msg":"session channel started","sshid": "-redacted-","time":"2026-03-05T12:39:17Z","uid":"8753e313dca1d382048d900df23371686802befab162506df29107058cc7dafc","username": "-redacted-"}
   ssh-1  | {"attempt":1,"id":"fmJLdmExtPTWEUQDbPsqwlEOckhvRqSz","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:39:47Z","url":"http://api:8080/internal/sessions/8753e313dca1d382048d900df23371686802befab162506df29107058cc7dafc/keepalive"}
   ssh-1  | {"device":"03045b535ee22f67f500d71f59e53221d1de36090abb4e4ef9f1f155418e382f","ip": "-redacted-","level":"info","msg":"session channel done","sshid": "-redacted-","time":"2026-03-05T12:40:17Z","uid":"8753e313dca1d382048d900df23371686802befab162506df29107058cc7dafc","username": "-redacted-"}
   ssh-1  | {"attempt":1,"id":"yZONdcfvbbzymUKeIWvYAbeSgyaEVzWx","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T12:40:17Z","url":"http://api:8080/internal/sessions/8753e313dca1d382048d900df23371686802befab162506df29107058cc7dafc/finish"}
   ssh-1  | {"device":"03045b535ee22f67f500d71f59e53221d1de36090abb4e4ef9f1f155418e382f","ip": "-redacted-","level":"info","msg":"session finished","time":"2026-03-05T12:40:17Z","uid":"8753e313dca1d382048d900df23371686802befab162506df29107058cc7dafc","username": "-redacted-"}
   

   I've also grep-ed the logs, looking for suspicious failed/closed/error messages but found none.

4. Nginx error logs:

   I'm not seeing any error logs when the disconnect happens. I'm attaching the logs around a disconnect:

   gateway-1  | { "timestamp": "2026-03-05T13:28:27+00:00", "remote_addr": "-redacted-", "body_bytes_sent": 4964, "request_time": 214.491, "response_status": 101, "request": "GET /ssh/revdial?revdial.dialer=23047bb055230534567bb1fb7a259413&uuid=79f8f7be-7523-40c3-a7f5-32f07cf40f39 HTTP/1.1", "request_method": "GET", "host": "-redacted-","upstream_addr": "-redacted-","http_x_forwarded_for": "","http_referrer": "", "http_user_agent": "Go-http-client/1.1", "http_version": "HTTP/1.1", "request_id": "474f8b294ba1e45db4e109c372313edc", "nginx_access": true }
   ssh-1      | {"attempt":1,"id":"ZpipkcayndFmBsNzlOhOIQrypWOYDMjl","level":"info","method":"POST","msg":"internal client request send","time":"2026-03-05T13:28:27Z","url":"http://api:8080/internal/sessions/{uid}/keepalive"}
   api-1      | time="2026-03-05T13:28:27Z" level=info bytes_in=0 bytes_out=0 host="api:8080" id=ZpipkcayndFmBsNzlOhOIQrypWOYDMjl latency=4576 latency_human=4.576684ms method=POST remote_ip=-redacted- status=200 uri=/internal/sessions/8e17058b5c2294ed2f80969637913850e2b21c5f3bf7dd7a0bea0c49dd6a3588/keepalive user_agent="go-resty/2.10.0 (https://github.com/go-resty/resty)"
   ssh-1      | {"attempt":1,"id":"ZpipkcayndFmBsNzlOhOIQrypWOYDMjl","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T13:28:27Z","url":"http://api:8080/internal/sessions/8e17058b5c2294ed2f80969637913850e2b21c5f3bf7dd7a0bea0c49dd6a3588/keepalive"}
   ssh-1      | {"device":"03045b535ee22f67f500d71f59e53221d1de36090abb4e4ef9f1f155418e382f","ip": "-redacted-","level":"info","msg":"session channel done","sshid": "-redacted-","time":"2026-03-05T13:28:27Z","uid":"958465c8e5d8350733aa509c4521ec9ff73bfd69dcd6f79a3c7c42014f694e64","username": "-redacted-"}
   ssh-1      | {"attempt":1,"id":"CSwdNuHwKccymHFqRGHVyTigDuSxPnWs","level":"info","method":"POST","msg":"internal client request send","time":"2026-03-05T13:28:27Z","url":"http://api:8080/internal/sessions/{uid}/finish"}
   api-1      | time="2026-03-05T13:28:27Z" level=info bytes_in=0 bytes_out=0 host="api:8080" id=CSwdNuHwKccymHFqRGHVyTigDuSxPnWs latency=4416 latency_human=4.416508ms method=POST remote_ip=-redacted- status=200 uri=/internal/sessions/958465c8e5d8350733aa509c4521ec9ff73bfd69dcd6f79a3c7c42014f694e64/finish user_agent="go-resty/2.10.0 (https://github.com/go-resty/resty)"
   api-1      | time="2026-03-05T13:28:27Z" level=info bytes_in=0 bytes_out=0 host="api:8080" id= latency=214795539 latency_human=3m34.795539278s method=GET remote_ip=-redacted- status=200 uri=/internal/sessions/958465c8e5d8350733aa509c4521ec9ff73bfd69dcd6f79a3c7c42014f694e64/events user_agent=Go-http-client/1.1
   ssh-1      | {"attempt":1,"id":"CSwdNuHwKccymHFqRGHVyTigDuSxPnWs","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-05T13:28:27Z","url":"http://api:8080/internal/sessions/958465c8e5d8350733aa509c4521ec9ff73bfd69dcd6f79a3c7c42014f694e64/finish"}
   ssh-1      | {"device":"03045b535ee22f67f500d71f59e53221d1de36090abb4e4ef9f1f155418e382f","ip": "-redacted-","level":"info","msg":"session finished","time":"2026-03-05T13:28:27Z","uid":"958465c8e5d8350733aa509c4521ec9ff73bfd69dcd6f79a3c7c42014f694e64","username": "-redacted-"}
   api-1      | time="2026-03-05T13:28:28Z" level=info msg="executing heartbeat task" task="api:heartbeat"
   api-1      | time="2026-03-05T13:28:28Z" level=info msg="finishing heartbeat task" modified_count=9 task="api:heartbeat"
   

5. LB proxies all ports (I use 22222 for SSH and 80/443 for HTTP/HTTPS).

[zoopp]

I've also tested @ltan10 suggestion on changing the keepalive interval but the connection still drops.

[otavio]

We've identified the root cause and opened a fix in #5947.

Root cause: In V1 (revdial) transport, the main control WebSocket connection (/ssh/connection) has full bidirectional keepalive (ping/pong + JSON keep-alive), but per-session WebSocket connections (/ssh/revdial) created for each SSH session have none. During idle sessions, the only traffic on the per-session WebSocket is the agent's SSH keepalive flowing agent→server every 30s — the server→agent direction is completely silent. Intermediaries like load balancers detect this half-idle connection and close it.

Fix: We now enable WebSocket ping/pong on the server side of per-session connections, creating bidirectional traffic that keeps the connection alive through all intermediaries.

@zoopp — could you test this fix? You can apply it by adding one line to pkg/revdial/revdial.go in the ConnHandler() function, after c.UUID = uuid:

c.Ping()

Or you can pull the branch directly: fix/revdial-session-keepalive

[gustavosbarreto]

I investigated the GCP LB behavior to understand what's going on.

From what I found, it appears to track TCP sequence numbers per direction independently. TCP ACKs alone don't increment sequence numbers, so they may not count as activity. If that's the case, the LB would need actual application data in both directions to consider the connection alive (ref).

Others have reported similar behavior: Cockpit, GCE discussion.

[otavio]

Update: per @gustavosbarreto's review on #5947, we've moved the WebSocket ping/pong from the server side to the agent side in grabConn(). This distributes the keepalive goroutine cost across agents rather than concentrating it on the server.

This fix addresses the immediate problem, but we'll need to think more deeply about the long-term implementation. Since we've never had the agent integration in other languages, we need to find a way to enable this bidirectional keepalive that works across all potential client implementations. Additionally, some customers rely on a large number of devices, and the extra per-session WebSocket ping/pong traffic could impact their performance at scale.

@zoopp — the testing instructions remain the same, just make sure to pull the latest from fix/revdial-session-keepalive.

[zoopp]

Thanks again for the follow-up.

Right now I'm away from home, so the earliest I'll be able to pull the latest changes and test them is sometime tomorrow morning. I'll report back once I've had a chance to verify it with the updated branch.

[ltan10]

Recompiled the agent with fix/revdial-session-keepalive.

@zoopp changing the keepalive to 30s would be a temporary solution anyhow. As the 'fix/revdial-session-keepalive` is just replacing the keepalive packet with a ping packet...

Shellhub Server:

• Version: v0.21.5
• Configuration: Sitting behind reverse proxy, no load balancing, websocket proxy timeout set at 60s

Shellhub Agent:

• Recompiled latest agent (GO-framework) with fix/revdial-session-keepalive
• KEEPALIVE_INTERVAL=300

Result:
Number of Tries: 2
Dropped at approx 20min and 27min

logs from ssh container for connection dropped at 27min.

{"attempt":1,"id":"lUahjSsLCNnMpwMUKFoqSALbEhLNAcpZ","level":"info","method":"POST","msg":"internal client request send","time":"2026-03-06T01:13:08Z","url":"http://api:8080/internal/sessions/{uid}/keepalive"}
{"attempt":1,"id":"lUahjSsLCNnMpwMUKFoqSALbEhLNAcpZ","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-06T01:13:08Z","url":"http://api:8080/internal/sessions/2ce4f94670204c20e0ea58525f95db3a0d3a70436516ace553edd114fc3359de/keepalive"}
{"attempt":1,"id":"vXdHoJhfXBolBEeuXwlyonJOrUWFXuuR","level":"info","method":"POST","msg":"internal client request send","time":"2026-03-06T01:15:14Z","url":"http://api:8080/internal/devices/{uid}/offline"}
{"attempt":1,"id":"vXdHoJhfXBolBEeuXwlyonJOrUWFXuuR","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-06T01:15:14Z","url":"http://api:8080/internal/devices/b03dabd9736bc11fa79561adb30cf0f07c9bf40b567f2ee61deb8a1631876283/offline"}
{"device":"b03dabd9736bc11fa79561adb30cf0f07c9bf40b567f2ee61deb8a1631876283","ip":"#######","level":"info","msg":"session channel done","sshid":"#######","time":"2026-03-06T01:15:30Z","uid":"2ce4f94670204c20e0ea58525f95db3a0d3a70436516ace553edd114fc3359de","username":"###"}
{"attempt":1,"id":"tdFyKNNEjvFpxeDZPCnGwFkXecTXbsIt","level":"info","method":"POST","msg":"internal client request send","time":"2026-03-06T01:15:30Z","url":"http://api:8080/internal/sessions/{uid}/finish"}
{"attempt":1,"id":"tdFyKNNEjvFpxeDZPCnGwFkXecTXbsIt","level":"info","method":"POST","msg":"internal client response received","status":200,"time":"2026-03-06T01:15:30Z","url":"http://api:8080/internal/sessions/2ce4f94670204c20e0ea58525f95db3a0d3a70436516ace553edd114fc3359de/finish"}
{"device":"b03dabd9736bc11fa79561adb30cf0f07c9bf40b567f2ee61deb8a1631876283","ip":"#######","level":"info","msg":"session finished","time":"2026-03-06T01:15:30Z","uid":"2ce4f94670204c20e0ea58525f95db3a0d3a70436516ace553edd114fc3359de","username":"###"}

Notes:
Question: Is having a timeout on idle connections a problem or just inconvenient? I would think having idle connections terminated is fine. But the timeout needs to be something reasonable.

[zoopp]

@otavio I've tested the fix and it resolves the issue. I kept four sessions idle (1 web, 3 native) for over 30 minutes and none of them were disconnected.

@ltan10 I don't have a strong opinion on this. Timing out long-running idle connections to free up resources makes sense, but as you mentioned, the timeout should be reasonable. It might also make sense if this behavior were configurable.

[otavio]

Cross-Reference Analysis

After investigating the logs from both @zoopp and @ltan10, I believe they're experiencing two different issues:

	@zoopp	@ltan10
What drops	Per-session revdial WebSocket	Main control connection
Fix addresses it?	Yes	No
Intermediary	GCP Load Balancer	External reverse proxy (60s timeout)
KEEPALIVE_INTERVAL	Default (30s)	300s (5 min)
Log evidence	Clean session done	Device goes offline before session ends
Disconnect timing	1–5 min (proxy idle timeout)	20–27 min (non-deterministic)

@zoopp — Resolved

The per-session WebSocket keepalive fix (fe11347) works correctly. The GCP Load Balancer was killing idle per-session revdial connections, and the new WebSocket ping/pong keeps them alive. This is the issue the fix targets.

@ltan10 — Different issue: main connection dying

The critical log sequence tells the story:

01:13:08  Last session keepalive (success)
01:15:14  /internal/devices/{uid}/offline    ← MAIN CONNECTION DIED
01:15:30  session channel done + finish

The /internal/devices/{uid}/offline API call proves the main control connection (/ssh/connection) died — not the per-session WebSocket. When the main connection drops, the device goes offline and all sessions are torn down as a consequence.

Why the main connection shouldn't be dying

The main connection has four independent keepalive traffic sources, all at hardcoded 30s intervals (completely independent of KEEPALIVE_INTERVAL):

1. Server → Agent: WebSocket PING every 30s
2. Agent → Server: WebSocket PONG auto-response
3. Server → Agent: JSON keep-alive every 30s
4. Agent → Server: JSON keep-alive every 30s

With this much bidirectional traffic, a 60s proxy timeout should never fire. The 20–27 minute non-deterministic failure suggests the external reverse proxy may have behavior beyond simple idle timeouts (e.g., max connection duration limits, connection pool cycling, or proxy-specific quirks).

About KEEPALIVE_INTERVAL

To clarify: KEEPALIVE_INTERVAL controls SSH-level keepalive within active sessions only. It does not affect the main control connection or any WebSocket-level keepalive, which are all hardcoded at 30s. Setting it to 300s means per-session connections see SSH keepalive traffic only every 5 minutes, but the new per-session WebSocket ping (from this fix) fills that gap with 30s pings. So the fix helps @ltan10's per-session connections too — but the main connection drop is a separate problem.

Regarding idle timeout concerns

To answer the broader question about whether idle timeouts are a problem vs. an inconvenience: for the main connection, any drop means the device goes fully offline, which disrupts all active sessions and requires reconnection — so it's definitely a real problem, not just an inconvenience. For per-session connections, idle timeouts during an SSH session would kill that session mid-use, which is also a real issue (and what this fix addresses).

Additional hardening in this branch

I've also added proxy_read_timeout 0; to ShellHub's own gateway Nginx config for all WebSocket locations (/ssh/connection, /ssh/revdial, /agent/connection, /ws). This disables Nginx's read timeout for these long-lived connections, since application-level keepalive already handles liveness detection. Previously, the default 60s timeout only worked because keepalive is 30s — that coupling was fragile.

@ltan10 — Could you share:

1. What reverse proxy software are you using? (nginx, HAProxy, Caddy, Traefik, etc.)
2. Your full WebSocket-related proxy configuration
3. Whether your proxy has any max connection duration or connection lifetime settings (separate from idle timeout)
4. Whether the issue reproduces without the external reverse proxy (connecting directly to ShellHub's gateway)

This will help narrow down what's causing the 20–27 minute disconnect pattern.