Centralized, generic, domain-agnostic API for event subscription and notification

[pjhac]

Good day and best wishes for 2024 !

Apologies in advance if I am not using the appropriate communication channel.

I would like to get the community’s feedback following a discussion I had with AT&T colleagues, where they expressed the need to have a “generic, centralized and domain-agnostic” event subscription & notification API, especially from an internal, operational network monitoring perspective.

So far in CAMARA each API has its own explicit event endpoint, for subscriptions to events generated specifically by/for the API “domain” (for example, zone entry/exit for DeviceLocation); same goes for DeviceStatus, QoD, and so on. This makes absolute sense and is perfectly fine when working on a few APIs (from a single application, for example), but when one needs to be able to quickly subscribe to several events from the network, this requires connecting to several endpoints, knowing each subscription detail, and so on.

With a single endpoint, using a generic event subscription format, it would become possible to subscribe to any event in the network, and receive a callback when the subscribed event(s) occurs. From the same place, it would be possible to subscribe for DeviceLocation events, DeviceStatus events… TMF 688 API allows something like this, where one provides a “query” used to filter the resources to get event notifications for (a possible format could be “event.devicestatus.roaming.type=RoamingOnEvent&event.msisdn=+12345678999” — please note these are examples only and not real use cases). Obviously, using the distinct CAMARA API endpoints would still be valid; the centralized, generic approach would be another way to do it, and complementary to the distinct APIs.

Behind the scene, this generic API could even make use of the distinct APIs for convenience (although this is implementation-specific); also note the query-URL type of filtering allows for flexibility, so any change in an API can be easily transcribed into the generic subscription format without the need to change the object format itself — this also allows any API, existing or future, to be supported since the “query” filter string is parsed. As said above, having the distinct, specific event subscription for separate APIs is still needed (so no change to existing APIs), it is only about bringing a new centralized endpoint to make network event management more convenient when many sources and network events are involved.

Should this be part of a baseline or made available as a new API is open for discussion, and I hope it makes sense.

Thanks in advance for your feedback

Best regards,
Pierre

[patrice-conil]

Hi @pjhac,
From a technical point of view, I agree with your approach, that is similar to the pub/sub systems we can find in the IoT ecosystem or in public data like Fiware.
On the other hand, I think it would be more difficult to track the "intentionality" of the requester if we had a single subscription system (maybe these are just prejudices and the 3-legged + scopes would do the trick).
Of course, the hardest part would be the versioning of this common API which would have to follow all the evolutions of all the APIs... how do we know which version of which API is served by this endpoint? Using a discovery endpoint?
WDYT ?

[pjhac]

Hello @patrice-conil,

Thank you for your message !

You are right, this is a pub/sub based system.

In my original post, I was presenting the possibility to make use of the distinct, existing CAMARA APIs (which I will call "higher-level" from now on) from the centralized endpoint. Now, the ability of doing the opposite also exists and is explained below, which hopefully may address better your points - again, it belongs to the implementation side of things, but I believe it has some advantages hopefully worth analyzing.

So, if this centralized API is deployed at an intermediate level, between higher-level APIs (where stand the other CAMARA APIs) and the network specific layers (at lower level), the advantages can be as follows:

• abstract the higher level APIs from the network-originated, low-level event notifications
• take advantage of the generic payload format of the generic API to support new queries coming from the higher-level API subscriptions; technically, this allows any change in higher-level API subscription parameters as they will be transformed into a generic (i.e. normalized) format (see later)
• work as a transparent relay for notification callbacks: as the higher level API will provide its own callback endpoint, the generic API, as intermediate layer, will simply relay the notification back
• provides a centralized system to collect / log subscription requests & notification callbacks, being used directly or as intermediate from higher-level API calls

The main challenge of this approach is to deal with the layer responsible of “normalizing” the payload from the original subscription; in an ideal world, it should be nothing else than flattening the original payload into a comma (or semicolon) separated query; as a simple example, a JSON payload like {“subscriptionDetail”: {“device”: {“phoneNumber”: “+19995554823”}, “type”: “roaming-status”}} would turn into “subscriptionDetail.device.phoneNumber=+19995554823,subscriptionDetail.type=roaming-status” (please note this is untested data, apologies for potential typos). This is fairly verbose, but still has the advantage of readability.

As of today, it can probably be fair considering companies started to implement CAMARA APIs as high-level APIs, so adding this generic API as an intermediate does not change anything for existing API consumers. On the other hand, if the intent is to use the existing APIs from the centralized API (as stated in my initial post), implementation-specific "denormalization" is technically all that is required, which is nothing more than rebuilding the JSON payload from the comma-separated string of key/value pairs (and since they contain a hierarchy using a dot separator, this is feasible).

Independently of the architectural choice (higher-level above centralized or centralized above existing higher-level), network operators can use the generic format to their needs without impacting others; it is also possible to expose this generic API to external consumers with proper access control. Additionally, discovery as you propose can help in all situations (if the generic API is used as an intermediate, the higher-level API can check if its payload parameters are supported; if the generic API is used above the existing APIs, it can ask the distinct API for supported payload parameters). Normalization and denormalization will handle the payloads, which should theoretically remove the alignment need on version, while discovery could indicate which parameters are supported or not, again independently of the architecture.

I hope I covered your questions and points.

Thanks again and best regards,
Pierre

[patrice-conil]

Hello @pjhac,
Thank you for the clarification.
I read too diagonally... and misunderstood your proposal.
I thought this was a single point of exposure for all of the Camara subscription APIs.
In fact, if I understood correctly, you propose to introduce a generic, weakly typed (domain agnostic) exposure point which hides the Camara or Network APIs.
As a developer using strongly typed languages, I'm not a fan of this kind of APIs ... because they introduce too much possibility of incompatibility between consumer and producer.
They also require manually validating all inputs...no possibility of delegating to the exposure framework.

Our current implementation choice is to position an intermediate layer between the Camara APIs and the Network APIs because the latter do not provide the APIs at the level we want.
This middle layer offers strongly typed APIs that are defined based on common Camara API needs. For example, device-location and geofencing could both call the 3gpp-event-monitoring API, retrieve the cell-ids and call a service that translates the cell-ids into GPS coordinates.
We prefer to expose a higher level LocationGPS API that performs these common processing operations that will be subscribed to by device-location and geofencing services.

Last I'm not sure to understand why you would normalize/denormalize Json into/from string.

Best regards
Patrice
​

[pjhac]

Hello @patrice-conil,

Thank you for your feedback, it's greatly appreciated.

You are correct about the "weakly typed domain agnostic API", this is indeed the intent.

As for input validation, it would have to be delegated to the systems that are to be called.

• If this API was to hide CAMARA APIs, there would be a need for the CAMARA APIs to validate the input, and relay back any error message. To be sure this is performed correctly, there is a need for a "normalization" layer in-between: not only it translates the generic, query type string into a valid JSON payload for the CAMARA API being called (and the opposite on the response), but it also ensures input validation is delegated appropriately (this could even include externally defined validation endpoints).
• If Network APIs were to be called instead from this generic API, same would occur.
  Additionally, if neither the CAMARA nor the Network APIs are able to do validation (for whatever reason), the intermediate "normalization" layer would have to take care of it. As some point, this "normalization" intermediate layer could look like the intermediate layer you are defining between Network and CAMARA APIs.
  Generally speaking, this approach of having an intermediate layer between exposed APIs and network APIs was used to be named Process Layer and is present in many API framework architectures, and if I understand you correctly, this is also where your intermediate layer is located, building upon 3gpp-event-monitoring.

As for the string/JSON (de)normalization, it was rather to support something like TMF688 "query" string format, but it could very well be a free-form JSON.

I think the main difference between this generic API and what you are saying is the strongly typed side of things. As a note, I am also a developer using strongly-typed languages and generally favors them whenever possible, but sometimes the amount of different parameters between APIs and objects in general (and over which you have no control as they are defined externally) becomes so huge that handling them ends up in similarly huge constructs where dozens of parameters become optional because one wants to supports them all (a kind of "union" of all parameters). I am not a fan of this either, but that is something that tends to grow in APIs (especially Network ones), somehow forcing us to have those intermediate layers to hide this complexity--and/or building weakly-typed/free-form payloads whose validation gets delegated to validation systems/APIs.

I still think there is room for convergence, for example using a specific event format that removes the weakly-typed aspect while keeping it compatible with Network and/or CAMARA APIs, but unfortunately the scope of genericity will be limited by the API family (it is probably fair to use latitude/longitude for everything that is related to location-related APIs, but will make little to no sense for other families).

Thanks again for your detailed feedback and interest in this generic API.

Best regards,
Pierre