Good practices for ingestion of devices

[mrsimpson]

tl:dr

How to actually make devices from the real world make an appearance in mds-core

Background

In an ideal world (ideal from an agency's perspective), MSPs would consume the agency api and push status changes to the agency, which triggers subsequent processes, such as compliance evaluation.
As per our experience, the real world is far from ideal:
Not all providers are willing to consume /agency, let alone in a resilient way. Therefore, polling data is necessary.
But (at least here in Germany), it's worse: not all providers offer an /events (or /status_changes) endpoint, but only expose snapshots of the current state via /vehicles or – worst case - GBFS. Edit: Forget about GBFS as worst case, there are providers only exposing custom formats 🤦

Challenges

Harmonized persistence and processing

Given the diversity of provided information, we need to transform data in such a way that its heritage is transparent to subsequent processes. There are basically two general options (which come to my mind)

1. Location-centric persistence: Given snapshots of locations or events, determine time slots at which a device was located at the same spot. When seeing the device in another location, start a new slot.
2. Event-centric persistence: Focus on the changes. With snapshots, determine a (fictional) change which could have produced the relocation for devices which appear in a new place.

We chose the location centric approach, since it's comparatively easy to handle when processing snapshots (which is what we've done most of the times) and simple to visualize. But it has a lot of downsides, particularly precision, but is also tricky wrt concurrency. With more and more providers exposing status changes, precision is getting better, but processing this transformed data (e. g. when determining compliance) is not ideal.

I can see the ingest-service being responsible for processing devices and that you went for fully processing events. While I appreciate this, I wonder whether I was missing out a piece on your code which transforms snapshots of locations into events which it feeds into the ingest-service.
Also, I have not seen a transformation of the events chain into s snapshot of locations in order to e. g. visualize a state on the UI. Or is this transformation a responsibility of the consumer?

Polling

Nobody likes polling. But at least we have to. There are various ways to do this, be it a simple deployment with a custom scheduler, Kubernetes Cron jobs or cloud infrastructure (e. g. Lambda, CloudWatch rules).
Is there a solution in/around mds-core which cares about this aspect?
If not: What are your experiences for good practices?

This might be a longer discussion (and in fact I am a bit hesitant to start this discussion as the first one here), but no doubt it's core to a challenge all agencies face, or isn't it?

Looking forward to reading from you,
Oliver

[janedotx]

Great questions. Currently, mds-core has no support for polling any sort of provider endpoint and is unlikely to in the near-term. This means we don't have as much guidance available on polling best practices as we might like, but what I can say is, is that the location-based polling is likely to be so lossy as to make use of policy and compliance infeasible. Policy and compliance assume that there's an unbroken stream of vehicle events.

[mrsimpson]

... the location-based polling is likely to be so lossy as to make use of policy and compliance infeasible.
💯!

At the same time, it is the (disappointing) reality that there's a number of MSPs that only expose GBFS-alike vehicle-status information.

Policy and compliance assume that there's an unbroken stream of vehicle events.

I'm currently thinking about implementing an events-generator events-guesser that transforms the diff in locations to events. Of course, this is lossy and it's not only the polling rate that influences quality: We'll never be able to determine why a particular vehicle changed its location.
If we transport the information about this imperfectionism in the device metadata, we could respect it when determining compliance. Of course, this still was far from perfect, but we could make this fact transparent to the cities (agencies) so that they could make MSPs offering services within their area provide events.