Flow runs created with Prefect client <=3.6.13 crash on server 3.6.21

[tonal]

Bug summary

Flow runs created with Prefect client <=3.6.13 crash on server 3.6.21 due to WebSocket connection error (HTTP 403)

Flows are registered, but they don't start - they crash

Version info

$ prefect version
Version:              3.6.13
API version:          0.8.4
Python version:       3.14.2
Git commit:           9eb03a74
Built:                Fri, Jan 23, 2026 03:52 AM
OS/Arch:              linux/x86_64
Profile:              ephemeral
Server type:          server
Pydantic version:     2.12.5
Server:
  Database:           sqlite
  SQLite version:     3.50.4
Integrations:
  prefect-aws:        0.7.4
  prefect-redis:      0.2.8
$ python3 prefect2_register_flows.py
2026-03-12 09:39:36.132646 1 deploy tech-cable
2026-03-12 09:39:36.149011 2 deploy tech-prop
Successfully created/updated all deployments!
               Deployments                
┏━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
┃ Name               ┃ Status  ┃ Details ┃
┡━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
│ tech-cable/wp-tech │ applied │         │
├────────────────────┼─────────┼─────────┤
│ tech-prop/wp-tech  │ applied │         │
└────────────────────┴─────────┴─────────┘

Additional context

Log from server ui:

timestamp,level,flow_run_id,task_run_id,message
2026-03-12 06:20:34.097833+00:00,20,666fa99e-b100-4ea3-9d46-d96d00b77808,,Worker 'bora' submitting flow run '666fa99e-b100-4ea3-9d46-d96d00b77808'
2026-03-12 06:21:45.835486+00:00,20,666fa99e-b100-4ea3-9d46-d96d00b77808,,Completed submission of flow run '666fa99e-b100-4ea3-9d46-d96d00b77808'
2026-03-12 06:21:49.882006+00:00,20,666fa99e-b100-4ea3-9d46-d96d00b77808,,Reported flow run '666fa99e-b100-4ea3-9d46-d96d00b77808' as crashed: Flow run infrastructure exited with non-zero status code 1. Process exited with a general error. Check the flow run logs for an unhandled exception or assertion error.

Log from worker docker:

prefect-worker  | 06:20:34.097 | INFO    | prefect.flow_runs.worker - Worker 'bora' submitting flow run '666fa99e-b100-4ea3-9d46-d96d00b77808'
prefect-worker  | 06:20:34.752 | INFO    | prefect.workers.docker.bora - Logging into Docker registry...
prefect-worker  | 06:20:35.442 | INFO    | prefect.workers.docker.bora - Pulling image 'gitlab.ex-ample.com:6789/technova/tech_prop_accepted:2026-03-12T06-18'...
prefect-worker  | 06:21:44.661 | INFO    | prefect.workers.docker.bora - Creating Docker container 'dazzling-woodpecker'...
prefect-worker  | 06:21:44.662 | INFO    | prefect.workers.docker.bora - Creating Docker container 'dazzling-woodpecker'...
prefect-worker  | 06:21:45.555 | INFO    | prefect.workers.docker.bora - Docker container 'dazzling-woodpecker' has status 'created'
prefect-worker  | 06:21:45.556 | INFO    | prefect.workers.docker.bora - Docker container 'dazzling-woodpecker' created successfully.
prefect-worker  | 06:21:45.777 | INFO    | prefect.workers.docker.bora - Docker container 'dazzling-woodpecker' has status 'running'
prefect-worker  | 06:21:45.835 | INFO    | prefect.flow_runs.worker - Completed submission of flow run '666fa99e-b100-4ea3-9d46-d96d00b77808'
prefect-worker  | 08:00:06.021 | DEBUG   | prefect.FlowRunCancellingObserver - Removing in-flight flow run ID: 019cd0f7-a2f9-7812-a13c-83798c578b12
prefect-worker  | 08:00:06.051 | DEBUG   | prefect.runner - Stopping runner...
prefect-worker  | 08:00:06.051 | DEBUG   | prefect.FlowRunCancellingObserver - Shutting down FlowRunCancellingObserver
prefect-worker  | 08:00:06.103 | DEBUG   | prefect.events.clients - Connection closed with "OK" status
prefect-worker  | 08:25:54.210 | DEBUG   | prefect.profiles - Using profile 'ephemeral'
prefect-worker  | 08:25:54.715 | DEBUG   | prefect.runner - Starting runner...
prefect-worker  | 08:25:54.736 | DEBUG   | prefect.client - Connecting to API at https://prefect.ex-ample.com/api/
prefect-worker  | 08:25:54.737 | DEBUG   | prefect.events.clients - Reconnecting websocket connection.
prefect-worker  | 08:25:54.737 | DEBUG   | prefect.events.clients - Opening websocket connection.
prefect-worker  | 08:25:54.922 | DEBUG   | prefect.events.clients - Unable to connect to 'wss://prefect.ex-ample.com/api/events/in', will retry. Reason: server rejected WebSocket connection: HTTP 403
prefect-worker  | 08:25:54.923 | WARNING | prefect.events.clients - Unable to connect to 'wss://prefect.ex-ample.com/api/events/in'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: server rejected WebSocket connection: HTTP 403. Set PREFECT_DEBUG_MODE=1 to see the full error.
prefect-worker  | Traceback (most recent call last):
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/prefect/cli/_utilities.py", line 44, in wrapper
prefect-worker  |     return fn(*args, **kwargs)
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/prefect/cli/_types.py", line 159, in sync_fn
prefect-worker  |     return asyncio.run(async_fn(*args, **kwargs))
prefect-worker  |            ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-worker  |   File "/usr/local/lib/python3.14/asyncio/runners.py", line 204, in run
prefect-worker  |     return runner.run(main)
prefect-worker  |            ~~~~~~~~~~^^^^^^
prefect-worker  |   File "/usr/local/lib/python3.14/asyncio/runners.py", line 127, in run
prefect-worker  |     return self._loop.run_until_complete(task)
prefect-worker  |            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
prefect-worker  |   File "/usr/local/lib/python3.14/asyncio/base_events.py", line 719, in run_until_complete
prefect-worker  |     return future.result()
prefect-worker  |            ~~~~~~~~~~~~~^^
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/prefect/cli/flow_run.py", line 676, in execute
prefect-worker  |     await runner.execute_flow_run(id)
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/prefect/runner/runner.py", line 598, in execute_flow_run
prefect-worker  |     async with context:
prefect-worker  |                ^^^^^^^
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/prefect/runner/runner.py", line 1553, in __aenter__
prefect-worker  |     await self._exit_stack.enter_async_context(self._events_client)
prefect-worker  |   File "/usr/local/lib/python3.14/contextlib.py", line 668, in enter_async_context
prefect-worker  |     result = await _enter(cm)
prefect-worker  |              ^^^^^^^^^^^^^^^^
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/prefect/events/clients.py", line 297, in __aenter__
prefect-worker  |     await self._reconnect()
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/prefect/events/clients.py", line 360, in _reconnect
prefect-worker  |     self._websocket = await self._connect.__aenter__()
prefect-worker  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/websockets/asyncio/client.py", line 590, in __aenter__
prefect-worker  |     return await self
prefect-worker  |            ^^^^^^^^^^
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/websockets/asyncio/client.py", line 546, in __await_impl__
prefect-worker  |     await self.connection.handshake(
prefect-worker  |     ...<2 lines>...
prefect-worker  |     )
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/websockets/asyncio/client.py", line 115, in handshake
prefect-worker  |     raise self.protocol.handshake_exc
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/websockets/client.py", line 327, in parse
prefect-worker  |     self.process_response(response)
prefect-worker  |     ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/websockets/client.py", line 144, in process_response
prefect-worker  |     raise InvalidStatus(response)
prefect-worker  | websockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 403
prefect-worker  | An exception occurred.
prefect-worker  | 09:07:01.104 | DEBUG   | prefect.profiles - Using profile 'ephemeral'
prefect-worker  | 09:07:01.606 | DEBUG   | prefect.runner - Starting runner...
prefect-worker  | 09:07:01.628 | DEBUG   | prefect.client - Connecting to API at https://prefect.ex-ample.com/api/
prefect-worker  | 09:07:01.629 | DEBUG   | prefect.events.clients - Reconnecting websocket connection.
prefect-worker  | 09:07:01.629 | DEBUG   | prefect.events.clients - Opening websocket connection.
prefect-worker  | 09:07:01.804 | DEBUG   | prefect.events.clients - Unable to connect to 'wss://prefect.ex-ample.com/api/events/in', will retry. Reason: server rejected WebSocket connection: HTTP 403
prefect-worker  | 09:07:01.804 | WARNING | prefect.events.clients - Unable to connect to 'wss://prefect.ex-ample.com/api/events/in'. Please check your network settings to ensure websocket connections to the API are allowed. Otherwise event data (including task run data) may be lost. Reason: server rejected WebSocket connection: HTTP 403. Set PREFECT_DEBUG_MODE=1 to see the full error.
prefect-worker  | Traceback (most recent call last):
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/prefect/cli/_utilities.py", line 44, in wrapper
prefect-worker  |     return fn(*args, **kwargs)
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/prefect/cli/_types.py", line 159, in sync_fn
prefect-worker  |     return asyncio.run(async_fn(*args, **kwargs))
prefect-worker  |            ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-worker  |   File "/usr/local/lib/python3.14/asyncio/runners.py", line 204, in run
prefect-worker  |     return runner.run(main)
prefect-worker  |            ~~~~~~~~~~^^^^^^
prefect-worker  |   File "/usr/local/lib/python3.14/asyncio/runners.py", line 127, in run
prefect-worker  |     return self._loop.run_until_complete(task)
prefect-worker  |            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
prefect-worker  |   File "/usr/local/lib/python3.14/asyncio/base_events.py", line 719, in run_until_complete
prefect-worker  |     return future.result()
prefect-worker  |            ~~~~~~~~~~~~~^^
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/prefect/cli/flow_run.py", line 676, in execute
prefect-worker  |     await runner.execute_flow_run(id)
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/prefect/runner/runner.py", line 598, in execute_flow_run
prefect-worker  |     async with context:
prefect-worker  |                ^^^^^^^
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/prefect/runner/runner.py", line 1553, in __aenter__
prefect-worker  |     await self._exit_stack.enter_async_context(self._events_client)
prefect-worker  |   File "/usr/local/lib/python3.14/contextlib.py", line 668, in enter_async_context
prefect-worker  |     result = await _enter(cm)
prefect-worker  |              ^^^^^^^^^^^^^^^^
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/prefect/events/clients.py", line 297, in __aenter__
prefect-worker  |     await self._reconnect()
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/prefect/events/clients.py", line 360, in _reconnect
prefect-worker  |     self._websocket = await self._connect.__aenter__()
prefect-worker  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/websockets/asyncio/client.py", line 590, in __aenter__
prefect-worker  |     return await self
prefect-worker  |            ^^^^^^^^^^
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/websockets/asyncio/client.py", line 546, in __await_impl__
prefect-worker  |     await self.connection.handshake(
prefect-worker  |     ...<2 lines>...
prefect-worker  |     )
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/websockets/asyncio/client.py", line 115, in handshake
prefect-worker  |     raise self.protocol.handshake_exc
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/websockets/client.py", line 327, in parse
prefect-worker  |     self.process_response(response)
prefect-worker  |     ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
prefect-worker  |   File "/usr/local/lib/python3.14/site-packages/websockets/client.py", line 144, in process_response
prefect-worker  |     raise InvalidStatus(response)
prefect-worker  | websockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 403
prefect-worker  | An exception occurred.
prefect-worker  | 09:21:48.399 | DEBUG   | prefect.profiles - Using profile 'ephemeral'
prefect-worker  | 09:21:48.893 | DEBUG   | prefect.runner - Starting runner...
prefect-worker  | 09:21:48.914 | DEBUG   | prefect.client - Connecting to API at https://prefect.ex-ample.com/api/
prefect-worker  | 09:21:48.915 | DEBUG   | prefect.events.clients - Reconnecting websocket connection.
prefect-worker  | 09:21:48.915 | DEBUG   | prefect.events.clients - Opening websocket connection.
prefect-worker  | 09:21:49.104 | DEBUG   | prefect.events.clients - Unable to connect to 'wss://prefect.ex-ample.com/api/events/in', will retry. Reason: server rejected WebSocket connection: HTTP 403
prefect-worker  | 06:21:49.744 | INFO    | prefect.workers.docker.bora - Docker container 'dazzling-woodpecker' has status 'removing'
prefect-worker  | 06:21:49.749 | INFO    | prefect.workers.docker.bora - Docker container 'dazzling-woodpecker' has status 'removing'
prefect-worker  | 06:21:49.882 | INFO    | prefect.flow_runs.worker - Reported flow run '666fa99e-b100-4ea3-9d46-d96d00b77808' as crashed: Flow run infrastructure exited with non-zero status code 1. Process exited with a general error. Check the flow run logs for an unhandled exception or assertion error.

[desertaxle]

Thanks for opening an issue @tonal! Do you have an auth string configured on your server? We made a change in 3.6.14 to require auth for inbound events when an auth string is configured, because previously, sending events bypassed auth. I can try to reproduce this issue once I know how your auth is configured.

[tonal]

@desertaxle Yes. We have an auth string configured on the server.

[desertaxle]

@tonal you will need to upgrade your client version then. The inbound events websocket before 3.6.14 ignored the server's authentication setup. We patched this hole in 3.6.14, but that requires upgrading your clients if authentication is enabled. Let me know if that is problematic for you, but most solutions to this will require poking holes in the authentication setup.