Missing task runs in flow runs.

[LukasJerabek]

Bug summary

We are missing task runs in our flow runs.

Our worker is at version 3.6.4. We started our server on 3.6.4 too and tried to raise servers version one by one. It starts happening on server version 3.6.7. We also tried 3.6.17, but the problem was still present and newer ones still doesn't seem to address it based on the changelogs, so didnt try any other versions.

This is the query that I use to confirm it in the database. When I look up these flow ids in UI I confirm that there are no tasks, but they actually ran on the worker (based on logs), but were not created in db.

SELECT *
FROM flow_run
WHERE NOT EXISTS (
SELECT 1
FROM task_run
WHERE task_run.flow_run_id = flow_run.id
)
and flow_run.state_type in ('COMPLETED')
and created between '2026-03-01' and '2026-12-12'
ORDER by flow_run.created desc

Unfortunately flows where it is missing are completely random. Tasks seem to run successfully, but they are not shown in the UI nor in the database. I just don't know how to investigate this any more, there are no clues, but I'd like to at least report it.

Version info

Version:              3.6.7
API version:          0.8.4
Python version:       3.12.13
Git commit:           ebfef643
Built:                Thu, Dec 18, 2025 07:54 PM
OS/Arch:              linux/x86_64
Profile:              ephemeral
Server type:          ephemeral
Pydantic version:     2.12.5
Server:
  Database:           postgresql
  PostgreSQL version: 16.13
Integrations:
  prefect-email:      0.4.2

Additional context

No response

[desertaxle]

Thanks for the issue @LukasJerabek! Do you see any warnings or errors in your flow run logs? It's possible that the state-change events aren't reaching the server. You can also check your server logs for warnings or errors indicating failed handling of task state change events.

[LukasJerabek]

No warnings or errors in flow run logs, the output is exactly the same as if the tasks were running normally.

Also nothing suspicious at all in server logs.

Well the events for such flow without any tasks look like this from earliest to latest:
flow run pending
flow run running
14x task run pending
14x task run running
14x task completed

For example this is how the first pending task event looks like:

{
  "id": "019cd72e-5d08-761e-9ae5-0e54116e2a61",
  "event": "prefect.task-run.Pending",
  "occurred": "2026-03-10T09:57:45.864Z",
  "payload": {
    "intended": {
      "to": "PENDING",
      "from": null
    },
    "task_run": {
      "name": "remote_hostname-eee",
      "tags": [],
      "labels": {},
      "task_key": "task_remote_hostname-2ba8195b",
      "run_count": 0,
      "dynamic_key": "eee3bb0b-ae5c-469e-9407-b2c1c93bf7b9",
      "task_inputs": {
        "machine": [],
        "username": [],
        "expected_hostname": []
      },
      "total_run_time": 0,
      "empirical_policy": {
        "retries": 2,
        "max_retries": 0,
        "retry_delay": 2,
        "retry_delay_seconds": 0
      },
      "flow_run_run_count": 0,
      "expected_start_time": "2026-03-10T09:57:45.864179Z"
    },
    "initial_state": null,
    "validated_state": {
      "data": null,
      "name": "Pending",
      "type": "PENDING",
      "message": "",
      "state_details": {}
    }
  },
  "received": "2026-03-10T09:57:46.122Z",
  "related": [
    {
      "prefect.resource.id": "prefect.flow-run.18ec1230-b576-40c8-b39d-5b7cd9d71937",
      "prefect.resource.name": "Hostnames",
      "prefect.resource.role": "flow-run"
    },
    {
      "prefect.resource.id": "prefect.flow.4d5dd47d-b98a-4b01-9708-7db932031ad1",
      "prefect.resource.name": "Hostname",
      "prefect.resource.role": "flow"
    },
    {
      "prefect.resource.id": "prefect.tag.<obfuscated>",
      "prefect.resource.role": "tag"
    },
  ],
  "resource": {
    "prefect.run-count": "0",
    "prefect.state-name": "Pending",
    "prefect.state-type": "PENDING",
    "prefect.resource.id": "prefect.task-run.019cd72e-5b6f-7a7a-9294-e399ac7cb3b3",
    "prefect.orchestration": "client",
    "prefect.resource.name": "remote_hostname-eee",
    "prefect.state-message": "",
    "prefect.state-timestamp": "2026-03-10T09:57:45.864179+00:00"
  }
}

if I try to show task run 019cd72e-5b6f-7a7a-9294-e399ac7cb3b3, it doesnt exist.

We will also try to dig deeper in the changes in 3.6.7 to check if something on our side couldn't break it.

[LukasJerabek]

Might really be our bug in the end, Id ask to reopen if we prove otherwise.

[LukasJerabek]

@desertaxle

So I have dispelled my suspicions around our side and Id like to reopen this. When checking the TaskRunRecorder, there is a new code between 3.6.6 and 3.6.7 in flush in this PR
https://github.com/PrefectHQ/prefect/pull/19855/changes
Couldn't the added code introduce some new occasional exceptions in the except Exception section, that raises before the logging and break whole TaskRunRecorder, for example in queue.put call?

Its just a suspicion, but Id still like to let this issue open.

[desertaxle]

@LukasJerabek what does your Prefect server setup look like? Are you running a single server or multiple replicas with Redis?

[LukasJerabek]

Its very simple setup, single server behind nginx with two workers.

[LukasJerabek]

No redis.

[LukasJerabek]

Just a note from playing around.

If I put raise ValueErrors into the flush function, flush_periodically, message_handler, then it didnt reproduce the same behavior, because it appeared in the server logs.

But when I put raise ValueError inside the start method of TaskRunRecorder or consumer function, then it behaves exactly same as we experience on our server and without any logs on the server.

This doesnt prove yet much, but its some clue at least... I suspect the TaskRunRecorder to fail silently for some reason. I might patch the 3.6.7 version with some try/except and more logging to confirm or dispute that something is raised there.

[LukasJerabek]

However that wont be easy for us, because we could yet reproduce that only in production...