Container stops with "API response: fatal error" but healthcheck says healthy

[grasshide]

The container stops working after some days with this message:

Fri Jan 10 17:07:22 UTC 2025: This script should remain running to keep the forwarded port alive
Fri Jan 10 17:07:22 UTC 2025: Press Ctrl+C to exit
Fri Jan 10 17:07:22 UTC 2025: Running /scripts/pf_success.sh
Fri Jan 10 17:07:22 UTC 2025: Allowing incoming traffic on port 25420
Wed Jan 15 02:48:51 UTC 2025: bindPort error
Wed Jan 15 02:48:51 UTC 2025: Curl error/s:
curl: (28) Connection timed out after 30002 milliseconds
curl: (28) Connection timed out after 30001 milliseconds
curl: (28) Connection timed out after 30002 milliseconds
curl: (28) Connection timed out after 30003 milliseconds
curl: (28) Connection timed out after 30002 milliseconds
curl: (28) Connection timed out after 30002 milliseconds
curl: (28) Connection timed out after 30002 milliseconds
curl: (28) Connection timed out after 30002 milliseconds
curl: (28) Connection timed out after 30002 milliseconds
curl: (28) Connection timed out after 30002 milliseconds
curl: (28) Connection timed out after 30002 milliseconds
Wed Jan 15 02:48:51 UTC 2025: API response:
Wed Jan 15 02:48:51 UTC 2025: Fatal error

But the healthcheck for some reason says the container is healthy.

My healthcheck from the docker-compose file:

    healthcheck:
      test: ping -c 1 www.google.com || exit 1
      interval: 30s
      timeout: 10s
      retries: 3

[thrnz]

Thats an error from the port forwarding script. To keep the forwarded port active, an api request gets made every 15 mins and it looks like its timing out for some reason. If the healthcheck is still passing then the vpn connection itself would still be active, but port forwarding would no longer work.

[grasshide]

Hmm, is there a way to react to this? For my use case the container is broken in this state.

[thrnz]

You might be able to catch it via the healthcheck if you test to see if pf.sh is still running or not, and using something that then restarts unhealthy containers. Maybe something like ping -c 1 www.google.com && pidof pf.sh might work.

Another way would be to set PORT_FATAL=1 along with EXIT_ON_FATAL=1 which would cause the container itself to shutdown when something goes wrong with the port forwarding script, although the container would still then need restarting again.

The port forwarding script retires several times before dying, so simply restarting the script probably won't fix any issues. I think I've only seen it die when the vpn endpoint itself has stopped responding.

Issue #69 might have a bit more info.

[grasshide]

Ok, so I changed my healthcheck to:

    healthcheck:
      test: ping -c 1 www.google.com && pidof pf.sh || exit 1
      interval: 30s
      timeout: 10s
      retries: 3

And added a "docker autoheal" container (https://github.com/willfarrell/docker-autoheal).
It always takes some days till the port error occurs. I'll update here if I see that it is working.

[thrnz]

Bear in mind that other containers sharing the network might also need to be restarted at the same time. If the vpn container is stopped, then I think Docker brings down the shared network stack with it, leaving the other containers networkless. When it gets re-created the other containers might remain detatched from any networking and need to be restarted to regain network access.