[WIP] Add a new rest api (v2) and web UI

[Chri-s]

I want to create a new web UI (using .net blazor webassembly, which doesn't seem to have as many dependency versioning problems as javascript frameworks) which can also do more then the existing web UI, for example edit partners and partnerships and querying the log.
The current web UI edits a partner by removing and adding a new one. That doesn't work if the partner is already used in a partnership, then the removal will fail.

For this I need a more powerful rest api, which is more "rest-like" and not based on the commands. This rest api would listen to /api/v2/ instead of /api/, so the two rest apis can co-exist.

I will upload the web UI to a public repository once it is more stable.

Would you be willing to accept a second rest API in OpenAS2?

Two more things:

1. Would it be okay for you if I change the type of partner from Map<String, Object> to Map<String, String> in XMLPartnershipFactory, BasePartnershipFactory and Partnership? That would remove some casts and I couldn't find any code which inserts anything else than a String and all placed which get data from the Map cast them to String.
2. I would like to add a second Map<String, String> for the partnership attributes and polling config in a partnership which stores the values from the <partnership> and <pollerConfig> elements before the variables like $properties.storageBaseDir$ and $attribute.sign$ were replaced. Then these values can be returned from the new rest API so that the user can change these values.
   Right now I can only return the replaced values, for example "signed-receipt-protocol=optional, pkcs7-signature; signed-receipt-micalg=optional, SHA512" for the as2_mdn_options. If the changes a partnership and edits the attribute sign, the change isn't reflected in the as_mdn_options because $attribute.sign$ was already replaced.

[uhurusurfa]

In response to your points above:

1. the change seems to be ok - please go ahead.
2. That makes sense. If you could share the intended API structure as I would like to standardise on a single API in the future and do away with the current API's so something that allows a common API for all future use by other plugins would be ideal.

[igwtech]

Hi, I agree with the initiative of separating the API from the Commands module. However, since we are potentially defining a new API contract that should remain somewhat stable over time to avoid breaking previous implementations (WebUI, Automations, etc), I would suggest we review the API definition more to include the desired features and security concerns. My recommendations would be: a) Not to pass the password over URL for private key export. Urls are not encrypted by SSL and can be leaked to proxies and other network devices which compromises the security of the password. b) Include methods for accessing messages received and sent in the Message Tracking module c) Include subscription methods to notify arrivals d) Include Poller configuration in the Add Partner method e) Include status and metrics endpoints for statistics and monitoring

…

On Tue, Aug 30, 2022 at 8:24 AM Christopher Broderick < ***@***.***> wrote: In response to your points above: 1. the change seems to be ok - please go ahead. 2. That makes sense. If you could share the intended API structure as I would like to standardise on a single API in the future and do away with the current API's so something that allows a common API for all future use by other plugins would be ideal. — Reply to this email directly, view it on GitHub <#296 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2QND2QABP6HBOKHZ3PNJDV3X4QZANCNFSM57ZSF5UQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Javier Muñoz CEO Ingeniería | Greicodex Software C.A. (+58) 212-7629120 <(+58)+212-7629120> ***@***.*** greicodex.com Av. Francisco Solano con Calle La Iglesia. Torre Centro Solano Plaza I. Piso 9, Oficina PH-A., Urb. Sabana Grande. Municipio Libertador. Dtto. Capital. Código postal 1050. [image: facebook] <http://facebook.com/greicodex> [image: twitter] <https://twitter.com/greicodex>

[Chri-s]

Hey,
thanks for your feedback :)

@uhurusurfa: For the first part (changing the partner to Map<String, String>) I will create a separate pull request. Then this change can be merged fast and the code bases for this don't differ for a longer time.

@igwtech: I'm also interested in a stable API and will push my changes (currently only removing the private key password from the url) in the next days.
The certificate part of my Web UI is also almost finished and will be published in some days.

a) Not to pass the password over URL for private key export. Urls are not
encrypted by SSL and can be leaked to proxies and other network devices
which compromises the security of the password.

This is already fixed in my local version. I wrote that part in the evening and the next day I noticed that it is not a good idea.

b) Include methods for accessing messages received and sent in the Message
Tracking module

That is already on my list, but not in the OpenAS2 rest API (see below).

c) Include subscription methods to notify arrivals

What do you mean with this? That OpenAS2 should be able to notify another server of arrivals or that it should notify browsers directly (e.g. via websockets)?

d) Include Poller configuration in the Add Partner method

Do you really mean the Add Partner method? In the partnerships.xml the poller config is defined in the partnership, not the partner.

e) Include status and metrics endpoints for statistics and monitoring

The statistics are part of my list, but not in the OpenAS2 rest API (see below).

My current approach to the new web UI (if you are also interested in that and not just the new rest API) is to have a .net 6 web API sitting between OpenAS2 and the browser. That web API runs on Linux and windows and can also be put in a docker container.

The reasons for this decision are:

1. I'm a .net developer for many years, but I don't have much experience with Java and the Java ecosystem.
2. My plan is to have multiple user accounts in the web UI with different permissions. Perhaps some users can only view log messages and nothing else while others can also update certificates.
   The authentication and authorization will be done by the .net web API, which checks the needed user roles and talks to the OpenAS2 rest API with the rest API credentials in config.xml.
3. Currently OpenAS2 supports any external database with a JDBC driver. This works for the INSERT and UPDATE statements of the message tracking. But I'm not sure if statistics and viewing the log can be solved in the same way because of different SQL dialects.
   As example: If you want to able to "page" through the message log, PostgreSQL uses a LIMIT <number of rows> OFFSET <start offset> clause to handle this. MySQL uses LIMIT <start offset>, <number of rows>. MS SQL Server uses OFFSET <start offset> ROWS FETCH NEXT <number of rows> ROWS ONLY.
   You would need to rewrite the queries for each database engine (or use something like Hibernate, if that is possible there - don't have experience with Hibernate).
   In the .net web API I would use Entity Framework core to handle the different database dialects.
4. I can easily use websockets via SignalR to show new arrivals in the web UI without polling the server. I don't know if something like that is possible in Java. And if it is, if it can be done without pulling to many new dependencies into the project.

My idea is to continue simultaneously on my web UI and the rest API. That way, the rest API already has a consumer and I can see if the API contract works for the use case. Instead of defining the rest API, implementing it, releasing it and then noticing that it can't be used easily.
I would push my changes to this pull request and be open for discussion and changes.

[igwtech]

Hi Chris, I agree with the approach of not bloating the OpenAS2 Rest API but it should be complete in the sense that it should allow you to have a low level management of the server.

On Tue, Aug 30, 2022 at 6:57 PM Chri-s ***@***.***> wrote: Hey, thanks for your feedback :) @uhurusurfa <https://github.com/uhurusurfa>: For the first part (changing the partner to Map<String, String>) I will create a separate pull request. Then this change can be merged fast and the code bases for this don't differ for a longer time. @igwtech <https://github.com/igwtech>: I'm also interested in a stable API and will push my changes (currently only removing the private key password from the url) in the next days. The certificate part of my Web UI is also almost finished and will be published in some days. a) Not to pass the password over URL for private key export. Urls are not encrypted by SSL and can be leaked to proxies and other network devices which compromises the security of the password. This is already fixed in my local version. I wrote that part in the evening and the next day I noticed that it is not a good idea.

Cool!

b) Include methods for accessing messages received and sent in the Message Tracking module That is already on my list, but not in the OpenAS2 rest API (see below).

Messages API, If you want to get information on a message (like status and such), keeping a low level API to get the Message Tracking data model on OpenAS2 will make it possible to improve the data model without breaking other projects that depend on it (they would just rely on the low level API)

c) Include subscription methods to notify arrivals What do you mean with this? That OpenAS2 should be able to notify another server of arrivals or that it should notify browsers directly (e.g. via websockets)?

Notify the API consumers (aka Other servers or Browsers). I was suggesting a way to notify external systems of low level events, thinking about Log Streams. Could be done with Websockets or Webhooks

d) Include Poller configuration in the Add Partner method Do you really mean the Add Partner method? In the partnerships.xml the poller config is defined in the partnership, not the partner.

I meant Add Partnership, my mistake.

e) Include status and metrics endpoints for statistics and monitoring The statistics are part of my list, but not in the OpenAS2 rest API (see below).

We should include data and statistics about the server itself. Like Inquiring the server for its active modules, uptime, configuration settings, etc.

My current approach to the new web UI (if you are also interested in that and not just the new rest API) is to have a .net 6 web API sitting between OpenAS2 and the browser. That web API runs on Linux and windows and can also be put in a docker container.

I'm interested, we have made changes to the REST api in our Repo that can't be merged back to the main branch. That not only includes a WebUI but also the possibility of managing the configuration thru a database instead of XML files and JMS message queues replacements for MessageStores and DirectoryPollers

The reasons for this decision are: 1. I'm a .net developer for many years, but I don't have much experience with Java and the Java ecosystem. That's great, as long as .NET is limited to separate projects like a

WebUI, having different languages will broaden the reach of people that can contribute to the OpenAS2 ecosystem.

1. My plan is to have multiple user accounts in the web UI with different permissions. Perhaps some users can only view log messages and nothing else while others can also update certificates. The authentication and authorization will be done by the .net web API, which checks the needed user roles and talks to the OpenAS2 rest API with the rest API credentials in config.xml. Agreed, that was our approach as well. In our production installations

proxy access to our OpenAS2 servers with NGINX and we handle end-user authentication via the NGINX Authentication module. We limit access to REST API Endpoints via Roles and Groups. The API credentials are baked in the NGINX config files.

1. Currently OpenAS2 supports any external database with a JDBC driver. This works for the INSERT and UPDATE statements of the message tracking. But I'm not sure if statistics and viewing the log can be solved in the same way because of different SQL dialects. As example: If you want to able to "page" through the message log, PostgreSQL uses a LIMIT <number of rows> OFFSET <start offset> clause to handle this. MySQL uses LIMIT <start offset>, <number of rows>. MS SQL Server uses OFFSET <start offset> ROWS FETCH NEXT <number of rows> ROWS ONLY. You would need to rewrite the queries for each database engine (or use something like Hibernate, if that is possible there - don't have experience with Hibernate). In the .net web API I would use Entity Framework core to handle the different database dialects. That's an interesting topic, that's why I was suggesting using the REST

API as an abstraction layer and solve the issue inside the OpenAS2 message tracking module with either an ORM or just supporting different specific dialects (PostgreSQL, MySQL & H2) internally via the current JDBC implementation.

1. I can easily use websockets via SignalR to show new arrivals in the web UI without polling the server. I don't know if something like that is possible in Java. And if it is, if it can be done without pulling to many new dependencies into the project. The legacy REST API (current one) depends on Jersey (JAX-RS) and I believe

that library supports websockets. In fact, the current HTTP server component for OpenAS2 is completely written from scratch and although it has holded up over the years, I don't trust it enough to expose my servers to the internet directly with it. I always put them behind a transparent reverse proxy.

My idea is to continue simultaneously on my web UI and the rest API. That way, the rest API already has a consumer and I can see if the API contract works for the use case. Instead of defining the rest API, implementing it, releasing it and then noticing that it can't be used easily. I would push my changes to this pull request and be open for discussion and changes.

Agreed, it is what makes sense.

— Reply to this email directly, view it on GitHub <#296 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2QNDYCBYIMYN7FCOYXRTTV32GUHANCNFSM57ZSF5UQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Higher level abstractions are good and I agree with your .NET 6 approach, but the basics (low level stuff) should be covered by the REST API. My opinion is that you should be able to configure and manage the complete server life cycle from the REST API itself. The first implementation relied on the Command interface as a crutch to keep backwards compatibility to the userspace interface (XML configs). Seems like newer releases are introducing more changes to userspace so it would be a good chance to review and fix some of its shortcomings

…

-- Javier Muñoz CEO Ingeniería | Greicodex Software C.A. (+58) 212-7629120 <(+58)+212-7629120> ***@***.*** greicodex.com Av. Francisco Solano con Calle La Iglesia. Torre Centro Solano Plaza I. Piso 9, Oficina PH-A., Urb. Sabana Grande. Municipio Libertador. Dtto. Capital. Código postal 1050. [image: facebook] <http://facebook.com/greicodex> [image: twitter] <https://twitter.com/greicodex>

[uhurusurfa]

I would like there to be a clear definition of what the API would be used for to avoid excessive bloat of the OpenAS2 code base to support functionality that is not really of great use to most people.
Remaining backwards compatible with components others have built over the years is important but of course if trying to remain backwards compatible results in messy and complicated code then remaining backwards compatibile does not make sense.
If a change is not backwards compatible then there needs to be a script to automate the upgrade from an older version.

Uses of the UI:

1. Core server configuration - not sure there is much value in providing a UI for this since it is something that is done once when deploying OpenAS2 and seldom if ever changed. If you want to do a UI for this, it can use the same mechanism as passing the openas2.properties.file parameter where the XMLSession class overrides defaults with values from a set of properties.
2. configuration of the partnerships - storing all the config in a DB should be relatively straightlforward since it is just name/value pairs with a link between partern and partnership objects. The XMLPartnership class can be enhanced to have a configuration property set to decide if it tries to load the partnership from the normal partnerships.xml or call a class that builds an XML doc from DB config so that all the rest of the code does not need to change much at all and can remain fully backwards compatible.
3. reviewing received and sent messages - this will provide the ability to query the message tracking DB in various ways to provide status and statistics.

The things that should not be part of the UI (and hence the API design) :

1. Logging - there are many logging tools out there that are designed to support log analysis and review and from my own perspective it makes little sense to add this into the API design. There is already support for Sentry in OpenAS2 and it is not difficult to add support for other cloud logging apps using the plugin mechanism currently supported by OpenAS2.
2. Events - again, the plugin capability of OpenAS2 supports have custom event modules that can write to message queues, send emails, other API's etc. I will need to write clear documentation on how to write a plugin and people can submit plugins for inclusion into OpenAS2. It is relatively simple to hook into the message module in the same way the DB Tracking module does to slectively send notifications of events to other API's/queues etc.

[Chri-s]

I published the current state of the webui: https://github.com/Chri-s/OpenAS2UI

Currently the certificate management is working and no authentication is used (I will add that later).

I thought it would be easy for you to see the current work if I publish OpenAS2 with the new rest API as a docker container (https://hub.docker.com/r/2chris/restapiv2) and also the Web UI (https://hub.docker.com/r/2chris/webuiv2).
I changed the OpenAS2 config in the docker image to disable the StreamCommandProcessor (which constantly throws exceptions in a docker container because no console is attached) and the RestCommandProcessor now listens on all IP addresses instead of localhost.
The docker images don't contain OpenAS2 in the name so that nobody finds them and uses them in production.

You can test the current development with this docker compose file:

version: "3.9"
services:
  openas2:
    image: 2chris/restapiv2:latest
    container_name: openas2
      
  openas2ui:
    image: 2chris/webuiv2:latest
    container_name: webui
    environment:
      OpenAS2__URL: http://openas2:8443/api/v2/
    ports:
    - 8080:80

Open http://localhost:8080/ and you see the web UI. Currently only the certificate part is working. You can browse the certificates, export the certificate and the certificate including the private key, import the certificates, delete them and see which partner is using the certificate.
I also added an expired and a soon expiring certificate in the docker container so that you can see how expiring certificates are shown.

I would continue with the partner and partnership configuration as I think we all agree that this should be included.

[Chri-s]

@uhurusurfa:

configuration of the partnerships - storing all the config in a DB should be relatively straightlforward since it is just name/value pairs with a link between partern and partnership objects. The XMLPartnership class can be enhanced to have a configuration property set to decide if it tries to load the partnership from the normal partnerships.xml or call a class that builds an XML doc from DB config so that all the rest of the code does not need to change much at all and can remain fully backwards compatible.

My approach would be to add a new interface EditablePartnershipFactory (as the project already has a RefreshablePartnershipFactory - btw. do you know why XMLPartnershipFactory doesn't implement it? XMLPartnershipFactory has the refresh() method) and XMLPartnershipFactory would implement it.
The interface would have methods like addPartner(Map<String, String> values), editPartner(Map<String, String> values) and deletePartner(string name) and the same for partnerships.
Then the implementing factory would be responsible for updating the stored data and saving it. Which means for the XMLPartnershipFactory that it would add a <partner> element with the attributes in addPartner(Map<String, String> values).

In my opinion this has 2 advantages:

1. The caller doesn't need to know how to add a partner to the xml Document. Today the AddPartnerCommand has to create an xml node for the partner, add the partner to the xml document of XMLPartnershipFactory and also call addPartner of XMLPartnershipFactory.
   If the new rest API adds a partner, it would have to do the same and more or less copy the code from AddPartnerCommand.
2. Today the AddPartnerCommand must cast the PartnershipFactory to XMLPartnershipFactory so that it can access the XmlDocument and add the nodes. If someone writes another PartnershipFactory which saves the configuration in the database and this factory doesn't extend XMLPartnershipFactory, the AddPartnerCommand would crash because the cast to XMLPartnershipFactory fails.
   With a new interface EditablePartnershipFactory, all commands can check if the PartnershipFactory implements it (like RefreshPartnershipsCommand does it with RefreshablePartnershipFactory) and call the needed methods on the interface. They don't need to know if it is a XMLPartnershipFactory or another factory.

There isn't much code which needs to be changed as today only the commands are directly using the type XMLPartnershipFactory. And they would become easier to read because they wouldn't change the xml Document any more.

[uhurusurfa]

@Chri-s - all sounds good to me. I have slowly but surely been cleaning up the code over the last decade but I know there is still a lot to do so feel free to raise separate PR's to fix/enhance pieces that need a rework that may make your implementation easier.