Organizing OpenAPI endpoints together with configuration & map

[mbaudis]

This is the original statement by @jrambla from the (now closed) issue #30:

The current framework is using two different schema definition languages:

1. OpenAPI 3.0.2 for actual API endpoint definitions. To keep consistency with the Beacon v2 draft1+
2. Json Schema draft07, for any schema that is not an API endpoint. Version 'draft07' was chosen as it is the one supported by OpenAPI 3.0.2

The topic for discussion is, in the next version (minor or major):

1. Makes sense to keep OpenAPI? or could/should we go just for Json Schema?
2. Should we keep the compatibility between OA and JS? or could we update JS to the latest version as they are used quite independently in the Framework?

We discussed some of the associated issues at the BeaconAPI call on 2022-07-12, e.g.:

• OpenAPI is handy for others when wanting to learn about the API paths etc.
• the "all endpoints in one file" way (v1 and early v2 drafts) was unwieldy, but the current separate endpoints.json per entity requires a lot of diligence when changing things -> defies the "jut add & modify entity directory" concept for some part
• current Beacon implementers attending that call all do their custom code and do not make use of OpenAPI
• OpenAPI might be helpful in onboarding to Beacon networks - but this hasn't been shown
• a "perfect" solution might be to have a master definitions file & to bootstrap all the configuration, OpenAPI files from that - but do we really want to go there?

Discuss!

[jrambla]

Output from GA4GH Connect Ascona meeting in April'24 reaches the following agreement:

• OpenAPI (OA) is still valuable for developers or API consumers. This statement has been reinforced by recent conversation related to generating Beacon clients just by using the OpenAPI representation of a given beacon instance.
• Having the API definition in 2 parts (OA and Json Schema) is still odd and we must try to remove it.
• The preferred solution is to extend the Json Schema (and/or beacon configuration) with elements that could allow generating OA of a beacon instance just by using the informational endpoints.
• The latest needs engineering and we should check if it doable or not

[jrambla]

(Pasting from the conversation happening in issue #136 )

From Michael Baudis:

@jordi @datsirul While in principle it is great to spot & correct this, the overarching question we had previously was about maintaining the OpenAPI documents at all. In the Ascona developer session the conclusion was basically that:

the Beacon schema (esp. model) cannot be defined in OpenAPI (at least not reasonably)
OpenAPI documents (endpoints ...) replicate definitions done elsewhere (map ...)
the "fire up a UI from the specs" isn't really a use case for Beacon, besides testing scenarios
none of the implementers/devs actually make use of them
for tests we need dedicated tools
OpenAPI specifications can be generated
OpenAPI specs are misleading since many aspects of Beacon (parameter use & combination, aggregation, terminologies ...) need to be understood from other parts of the spec. and/or documentation
... just paraphrasing & YMMV on some of the arguments. But the longstanding question about ditching OpenAPI in one of the next releases should be answered (which it was) and documented as the path forward. OTOH it could be nice to keep OpenAPI docs as generated artifacts, but not as edited ones...

[jrambla]

(Pasting from issue #136 )
From Maria:

the Beacon schema (esp. model) cannot be defined in OpenAPI (at least not reasonably)
OpenAPI documents (endpoints ...) replicate definitions done elsewhere (map ...)
@mbaudis Those two points are not clear to me. While OpenAPI helps to define the Beacon Framework (endpoints, request, response, parameters, entities and types used), the internal data representation is not dictated by the standard and should be mapped to entities defined by standards. Or what is that "Beacon schema (esp.model)" are you referring and what cannot be defined?

[jrambla]

IMHO, the steps we are doing now, thanks to @datsirul, are in the same direction that we agreed to follow.
If we must be able to generate proper OpenAPI representations of beacon instances, we must detect which elements in the spec makes that complicated or how much tinkering the OpenAPI representation generators would need to apply.
I expect some blocking issues as Beacon spec is using the latest version of Json Schema and OpenAPI is using a slightly customized version of draft-07 (it was?)...
But, we should try to spot all the friction points and check if we have solutions for them.

[redmitry]

First of all thanks to @datsirul for his work on validating OpenAPI part. Two years ago I used OpenAPI to extract parameters for GET endpoints, but obviously POST is much more complicated.
Note that Beacon is a protocol, so without "generalizing" it, each OpenAPI endpoint will just describe the protocol with no differentiation on the data model behind the endpoint.

If we must be able to generate proper OpenAPI representations of beacon instances, we must detect which elements in the spec makes that complicated or how much tinkering the OpenAPI representation generators would need to apply.

Totally agree. I tried to autogenerate OpenAPI from the /configuration in Java Beacon Implementation and had no much success...
https://beacons.bsc.es/beacon/v2.0.0/individuals-openapi.json

I expect some blocking issues as Beacon spec is using the latest version of Json Schema and OpenAPI is using a slightly customized version of draft-07 (it was?)...

Although Beacon's schemas was updated to the "2020-12" they are still "draft-7"... I hardly see any new features...
https://json-schema.org/draft/2019-09/release-notes
https://json-schema.org/draft/2020-12/release-notes

For instance:
beaconConfigurationSchema.json#L8 in draft-7 anything along the $ref must be ignored, but "2019-09" - no. Anyway since entryTypeDefinition is an object - it is just an "artefact".

Another thing is that in "2019-09" "definitions" became "$def". But all validators understand both, even you explicitly define the version.

Ideally we would pass to the OpenAPI 3.1. Where do we specify OpenAPI version?

Kind regards,

Dmitry

[datsirul]

Thank you all for your work on this discussion. I understand that significant progress has been made on this topic, so I'm not trying to restart the conversation from scratch but rather to better understand the situation and see if we can help.

It seems that the current path presented is to potentially drop OpenAPI and replace it completely with JSON Schema. One of the reasons noted is OpenAPI’s schema limitations, so it would be great to get an example to understand the issue.

Another point I wanted to note is OpenAPI’s strong support for REST APIs, which helps ensure requests and responses are correctly formulated. JSON Schema, on the other hand, is primarily used for describing and validating JSON data, such as the content within Beacon requests and responses. If moving to JSON Schema requires manual adjustments to achieve what OpenAPI does automatically, it seems this might add complexity and increase the potential for issues and errors.

Additionally, OpenAPI offers several advantages, such as:

• Automatic client and server code generation with tools such as OpenAPI Generator.
• Improved testing, validation, and mocking with tools such as Swagger UI, Postman, and Prism.
• Automatic document generation with tools like Redoc.

Given these benefits, it’s important to carefully consider the implications before moving away from OpenAPI (at least for the RESTful parts).

Best regards,
Daniel

[jrambla]

The current agreement (Ascona, 2024) is to move from "OpenAPI first" to "extend the JSON Schema to allow generating OpenAPI from it" (a.k.a. as "OpenAPI generated".

See some reasons why in my post in this thread

In summary:

1. The specification should define commonalities to *all Beacon implementations"
2. OpenAPI should be describing a specific beacon instance that could implement just one endpoint and just one verb (e.g. POST) if they want so
3. Hence, we see OA as the solution for describing specific instances instead of trying to define the whole specification.