Should we use Mermaid for the visualisation?

[rocketstack-matt]

I've been experimenting trying to use Mermaid to visualise the CALM examples. My rationale for this is:

• Mermaid is already supported for visualisation in most common IDEs including IntelliJ and VS Code.
• There is a clear map between Mermaid flowchart syntax and the syntax used for CALM.

I've been able to produce the following diagram using Mermaid for the TraderX sample.

flowchart TB
    subgraph internal_bank_network [Internal Network: Bank ABC Internal Network]
        subgraph traderx_system [System: TraderX]
            web_client[Webclient: Web Client]
            web_gui_process[Service: Web GUI]
            position_service[Service: Position Service]
            traderx_db[(Database: TraderX DB)]
            reference_data_service[Service: Reference Data Service]
            trading_services[Service: Trading Services]
            trade_feed[Service: Trade Feed]
            trade_processor[Service: Trade Processor]
            accounts_service[Service: Accounts Service]
            people_service[Service: People Service]
        end
    end

    traderx_trader((Actor: Trader))
    user_directory[LDAP: User Directory]
    
    traderx_trader -->|interacts| web_client
    web_client -->|connects HTTPS OAuth2| web_gui_process
    web_gui_process -->|connects HTTPS SPNEGO| position_service
    position_service -->|connects JDBC Kerberos| traderx_db
    web_gui_process -->|connects HTTPS SPNEGO| reference_data_service
    web_gui_process -->|connects HTTPS SPNEGO| trading_services
    web_gui_process -->|connects WebSocket Kerberos| trade_feed
    trade_feed -->|connects AMQP Kerberos| trade_processor
    trade_processor -->|connects JDBC Kerberos| traderx_db
    web_gui_process -->|connects HTTPS SPNEGO| accounts_service
    web_gui_process -->|connects HTTPS SPNEGO| people_service
    people_service -->|connects LDAP Kerberos| user_directory

    %% Custom styles to indicate 'composed-of' relationship visually
    style traderx_system stroke-dasharray: 5 5

Loading

In this example I have not provided any custom styling which could help make it look even better and it has the advantage that we could generate the mermaid representation through IDE plugins and / or build time plugins which would mean the diagrams would stay in sync with the CALM manifests.

What do you think @aidanm3341 / @jpgough-ms / @Budlee

[aidanm3341]

Using a standard format like Mermaid definitely gives us a lot of benefits for free as you point out. It also has far better layout functionality than the library that I had initially used.

The main downside that I see is the lack of intractability. With the tool that we currently have, you can move the nodes around and get more information about them than what is just shown on the graph. The custom tool has the potential to dynamically change and create/delete nodes and relationships inside the visualisation itself (although this would require a fair bit of work I think).

To be honest, I think using mermaid would allow us to build something more useful and complete far quicker than the custom approach that I've taken so far. I'd definitely be keen to see the code that you wrote to convert from CALM to Mermaid.

[Budlee]

I really like this idea, I think this is a really nice approach to vitalise something that integrates easily as part of a VSCode/intelliJ plugin. I've not used Mermaid much myself though from what I have seen it seems very compelling.

I have used the Structurizer mermaid export which is also very useful (https://docs.structurizr.com/export/mermaid)

[simonbrowndotje]

I have used the Structurizer mermaid export which is also very useful

Glad you've found the exporter useful! I am watching CALM with interest, and I'd like to share some insights with you, because it may save you some pain, specifically related to generating "diagrams as code" formats (Mermaid, PlantUML, etc) from a rendering agnostic definition (in my case the Structurizr format, in yours CALM).

First of all, from my personal perspective, the perception that automatic layout is better than manual layout still persists and is very prevalent. Sure, it's convenient, but I'm not a fan. Architecture diagrams are about telling stories, and automatic layout typically hinders this by putting elements in odd positions, particularly as the diagrams grow in size.

The main downside that I see is the lack of intractability. With the tool that we currently have, you can move the nodes around

Correct. With the above in mind, and from the perspective of being a tool maintainer myself, I can say with some degree of confidence that the users of any CALM to Mermaid tooling will ask you the following questions:

• How do I align boxes A, B, and C on my diagrams?
• How do I make box B sit to the right of box A instead of underneath it?
• How do I prevent the labels on arrows from overlapping?
• etc

Automatic layout is a convenience, but it comes with some trade-offs. Mermaid and PlantUML understand this, and do provide features to make some of these things possible. For example, Mermaid and PlantUML both have ways to control things like arrow direction and length, which diagram authors will have access to when using those languages directly. Here's a good example. Users won't have that same capability when a tool is generating Mermaid on their behalf. Workarounds? I know teams who have forked my exporters and use element/relationship properties in the model to define view specific metadata to indicate, for example, that "this arrow should point to the right". It works, but it pollutes the architecture definition.

tl;dr ... generating Mermaid from a rendering tool agnostic architecture definition is trickier than it first seems.

[rocketstack-matt]

Hi @simonbrowndotje thanks for the input, very much appreciated.

Completely agree with the issues around auto-layout, certainly with the added complexities of CALM trying to model running systems I think any realistic architecture is quickly going to become hard to visualise especially with automated layout. Whilst we absolutely want a tool for being able to visualise CALM itself (and indeed author it) I think we're a long way from understanding the complexities of what we'll need to make that tool usable and this is really the first small step on that journey to help pattern authors.

It's also worth noting, that for communicating about architecture, we plan to map to C4 . . . @Budlee has been doing the initial thinking on this capability; we will need to adopt some form of convention or additional properties that would enable us to tag certain aspects of the CALM model to the corresponding layers in C4 and which views they would appear in.

We'd love to have you attend future meetups or on-site's we're running if you'd like to get into more details.

[simonbrowndotje]

We'd love to have you attend future meetups or on-site's we're running if you'd like to get into more details

Sure, I'm often passing through London - do you have a list of dates/events somewhere?

[rocketstack-matt]

Great, our monthly meetups are virtual and are the 4th Tuesday of each month (you can see the FINOS calendar here -https://www.finos.org/calendar) so the next is on the 26th March. We manage the meeting agenda in the devops-automation SIG repo, I tend to post it a couple of days before but you're welcome to post ahead suggestions for topics.

We actually have an in person session planned for next Tuesday, if you happen to be in London please drop you name on this issue!

[dc-ms]

As we move forward we should maintain backwards/forward compatibility with C4. Last I looked Mermaid had experimental support for C4.

[dc-ms]

@rocketstack-matt confirmed compatibility and raised the point of additional properties/tags for model mapping.