Few Qs

[cloudcompute]

Hi Spine Team

What is the current status of this event-sourcing framework, is it production-ready, if any project/application/company is using it in production?

Does it promote the Pure functions principle of functional programming i.e. do not include side-effects (like db, third-party api access) in your pure functions?

Which databases does it support, does it implement the Outbox table pattern (which is useful for writing the events to Kafka)?

Thank you.

[armiol]

Good day, @cloudcompute!

(We have migrated the issue you have created to Discussions section of our feedback repository, to make the conversation more convenient.)

What is the current status of this event-sourcing framework, is it production-ready, if any project/application/company is using it in production?

At the moment, Spine is used in production in several projects since 2016. One of them is Pure Photos, which is an AI-powered solution for image processing. Some of the projects built with Spine, were delivered to our customers as private-cloud and non-public services. And we also use Spine extensively to build our own tools, such as a tool for managing digital assets. Someday, they will become public services.

Just for the reference, mostly, the load handled by Spine-powered apps tops at about 100 req/sec. At the same time, the event stream size is relatively big, being billions of events emitted over the application lifespan.

All apps running on top of Spine in production are now built around Spine 1.x (in which 1.9.0 is the latest release). At the same time, core Spine team is working on Spine 2.0, by embracing the feedback from the teams. In 2.0 we somewhat rethink and simplify things for developers, migrate closer towards using Kotlin on both client- and server-side, and also automate more stuff by leveraging code generation from Protobuf definitions.

We expect to see Milestone 1 of Spine 2.0 later this year, and after that start creating an extensive documentation and promotion materials. Also, non-critical projects running on top of Spine 1.x will be migrated to 2.0-M1.

Does it promote the Pure functions principle of functional programming i.e. do not include side-effects (like db, third-party api access) in your pure functions?

Among other things, Spine is built with Hexagonal Architecture principles in mind. For instance, databases and other storage solutions are just so called "ports", meaning the business logic does not depend on it, and has no idea what is underneath. As a developer, one may plug anything into Storage port, as soon as the implementation complies with the contract, — and the business logic won't change at all.

In terms of programming an application with Spine, developers mostly deal with three things:

• Ubiquitous Language, and its definition as Protobuf messages,
• Discovery of domain events, commands, rejections, Bounded Contexts, etc., through techniques, such as #EventStorming,
• Implementing server- and client-sides of Bounded Contexts.

First two activities are free of coding per se, but from our experience, they are the most difficult things to master.

Server-side works usually imply implementing the command and event handlers, absolute majority of which we recommend to have side-effect free. When it comes to integrating with, say, some 3rd-party API, the recommended approach is to have a dedicated Process Manager, documented and tested, so that any unwanted incoming data does not break the domain logic (sort of an Anti-corruption layer, if you will). Here, I can go into more detail, if that's what interests you.

Which databases does it support, does it implement the Outbox table pattern (which is useful for writing the events to Kafka)?

Currently we have public implementations for Google Datastore (which is our choice of primary storage in production projects), RDBMS engines (we use MySQL and MariaDB, which are close relatives in terms of API), and we also experimented with Redis and Hazelcast, but we don't use them in production at the moment.

As to Kafka, several years ago we have created an adapter for it. However, with all things considered, it was not a good fit for us at that point.
Inside Spine runtime, all "signals" (that's our term for commands, events, and rejections) travel via respective buses, and developers may plug into those buses programmatically, and push anything to any destination including Kafka, if such is the need.

Now, the "Outbox" pattern. In short, yes, our implementation is similar to its idea, but over time it has grown into something bigger, providing numerous things: signal de-duplication, ordering, guaranteed one-time delivery, catch-up (or event replay for projections), etc.

Feel free to ask more questions, I will be happy to answer.

[cloudcompute]

The response is in detail, thanks.

Only one question.. As you said, the load handled by Spine-powered apps tops at about 100 req/sec, is the throughput not at a lower level?

[armiol]

@cloudcompute with that domain (media processing) we did not see any load higher than that. Probably, that's because it was "created" by human users, not machines. While surely that particular application was capable of handling more requests per second, there was never a time.

Spine-based apps are scaled horizontally, as it was always our goal, not to have any artificially created bottlenecks aka points-of-failure. In terms of system design, the upper boundary of available throughput is only restricted by "ports", meaning storage system underneath, and the number of nodes along with their bandwidth. However, if we speak of, for instance, Google Datastore, which we use extensively, it has its own throughput limits with the desired level of consistency. In our case, it was always the event streams underneath Aggregate entities, which storage demands a strong consistency, at least for the write-read operations performed by the same application node within a short period of time. And Datastore does provide such a consistency, but (CAP theorem!) there are limits on tx/sec allowed.

Of course, Spine users are not obligated to use Datastore. It is our choice in many cases, as in reality domain scenarios are asynchronous by nature, and the whole thing turns out well with "asynchronous", but auto-scaled storage like Datastore.

I personally wouldn't build a consistency-critical system (trading platform, for instance) purely top of Datastore, it's just not a proper tool for that.

[cloudcompute]

@armiol

I went through the code.. it is a very high-quality code. I'd certainly give it a try.

developers may plug into those buses programmatically, and push anything to any destination including Kafka

Is there any example? When is this write going to take place, would it be "after" an event is written to the event store db?

Thank you

[armiol]

@cloudcompute

We don't have a Kafka-specific example, but I have modified our Hello sample app by illustrating how to plug a standalone event subscriber into a Bounded Context.

In this example application, we have just one Bounded Context. And it is kept very simple, with just a single Process Manager, called Console.

The job of this Console is to handle a domain command called Print, and emit a domain event called Printed. Both of these messages are Protobuf types declared in commands.proto and events.proto files respectively.

When the application is launched, several things happen:

• a Bounded Context is assembled,
• a server part is started,
• client code sends Print command,
• Console Process Manager handles Print command, and emits Printed event.

I have just appended another destination for Printed events to the definition of this Bounded Context. It's a standalone subscriber to Printed events. Here is the changeset.

If you need the domain events to be put into any external system, such as Kafka, this @Subscribe-annotated method would be the place in which you write such a code.

Hope it helps.

Please feel free to ask further questions.

[cloudcompute]

Thanks @armiol

I will go through it. I want to write to Kafka only after the event(s) are successfully written to the Event store. I think this is out-of-scope of an event-sourcing library. I need to use a CDC solution like Debezium in order to fulfil this requirement.

There are many repositories in this org, there is one called web. Looks like it has an API that can be invoked from, for example, a browser. It is using Servlets though which is blocking (and not reactive). In which scenario does one use this web thing?

I am looking forward to use Spine in a Spring Boot application (microservice). How can I use it, is there a quick starter?

[armiol]

@cloudcompute

You have raised several points, I will comment on them one by one, briefly. And then let's discuss those that interest you more, in more detail.

1. We only send out the domain events after they are stored to Spine's EventStore. Speaking of which, it is also a pluggable thing, so one may use own implementation for storing the events.

2. Our web library consists of several things. But as to servlets, — and that's true for all other means of communication between any actor and Bounded Context — upon receiving a domain command, Spine sends so called "acknowledgement" synchronously, meaning the command message is valid, and has been accepted for handling. All other things happen asynchronously. Of course, one may configure Spine to act synchronously in some parts (i.e. for tests), but in general, it's all async.
   Returning to web, we rely onto Firebase RDB in this library, to allow users subscribe to domain events and-or entity state updates. It happens asynchronously.

3. We don't use Spring at all (we did, but then we found it was not fulfilling our needs). Instead, we just use Micronaut, or plain Servlets.
   If you want to use Spring Boot, nothing stops you from just assembling a Bounded Context within some singleton, and then interact with it programmatically.

Again, many more things to cover. But I hope that's a good start.