Surface more internal state about offline entities

[matthew-white]

v2024.2 adds limited information to Frontend about offline entities. We're not sure how common various edge cases will be, so it's not clear yet how important it is to surface additional information. This issue is to track ideas to surface additional information if/when that seems useful. See also #688, which is about doing more to surface entity errors (whether those errors are about offline entities or otherwise).

Idea 1: Show the offline entities backlog

When an offline update is in the backlog, it hasn't been applied yet. It may be helpful to users (and us!) to indicate that there have been changes that have not been applied yet. We could add a lean version of the backlog. I wrote on Slack:

Maybe we could show an extra tab for the entity list when there’s something in the backlog. We don’t expect the backlog to be large, so we could probably render it all in a table without paging.

Each row in the table could link to the submission, link to the entity if it exists, etc.

@lognaturel has pointed out that there are two pretty different situations in which an offline update ends up in the backlog:

1. Submissions from an offline branch are received out of order. In that case, some of the submissions will be held in the backlog, but only until all the submissions from the branch have been received. The submissions will be held in the backlog only temporarily, during which the size of the backlog will fluctuate fairly quickly.
2. A submission from an offline branch is very late or missing entirely. In that case, any submission from later in the branch will be held in the backlog until 7 days have passed. Those submissions' stay in the backlog is less temporary, and the submissions may eventually be force-processed.

We may want to surface (2), but it doesn't seem particularly helpful to surface (1). With that in mind, when showing the backlog, we could filter out submissions that have been in the backlog for less than a day.

Idea 2: Show more on the entity detail page about force-processing and missing submissions

There are a couple of ideas in this area. They have to do with showing more on the entity detail page when force-processing has already taken place for at least one update.

• For each entity update in the feed, indicate if the update was force-processed out-of-order. This would give users greater visibility into the history of the entity.
• If an update was force-processed, and the previous update from the branch was never received, show a warning at the top of the page. This would tell the user that there may be data missing about the entity. There's a parallel with submissions missing a submission attachment, which we indicate on the submission detail page and in the submissions table.

Idea 3: Show the submission being processed and held in the backlog on the submission detail page

There is no indication that a submission was processed and held in the backlog vs. not yet being processed. Maybe we want to show something in the submission activity feed when this happens? The reason against this is there are two extremes for a submission going into the backlog: 1) it's there for a few seconds while waiting for the submissions to get in the right order, in which case we probably don't need to show this or 2) it's stuck for a long time, possibly until it gets force-applied by the backlog processing cron job.