Building blocks to describe relationships between systems in an ECS

[jonesmz]

This is less of a thing where I have questions that need answers, and more of a thing where I think it would be useful to discuss the general idea.

Imagine you have some arbitrary application that is built around an ECS, such as EnTT. Imagine further that you want this application to take advantage of multiple CPU cores where it makes sense to do so.

If I understand the design of EnTT correctly, the following are true, or close enough to true for purposes of discussion anyway:

• EnTT does not attempt to synchronize across multiple threads, and requires users of EnTT to do that if they want it.
• Adding and removing components can re-arrange the underlying storage for all components of that component type in a way that precludes multi-threaded access.
• Reading / writing to unrelated components types can be done in parallel without synchronization (as far as EnTT is concerned, anyway)
• Reading a specific component from multiple threads simultaneously is fine.
• Writing to a specific component from multiple threads, or writing from one thread, and reading from another, requires external synchronization.

Keeping the above concerns in mind, even if they aren't 100% accurate, raises the following discussion points.

• A given System (e.g. some function somewhere) will conceptually have an exact list of Component Ids that it will try to access, either for read, or write, or both.
• Given some list of Component types, and the access type for those (read, write, readwrite), one could construct a map between a particular System, and the list of Systems that can be executed in parallel with it without risk of cross-thread conflicts.
• With the above point in mind, and assuming each system need be run only once, one could calculate an execution plan of these Systems such that the maximum amount of parallelism is achieved.
• We could further "optimize" the execution plan of the systems by taking into account whether any two particular systems have a dependency on each other for application specific purposes
  • e.g. in a game, you probably wouldn't want to apply velocity to position until after you've updated velocity to account for any acceleration forces
  • Or if you did want to flip the order, you at least want to do it in the same order every time.

As far as I know, EnTT does not try to provide any kind of facilities to make the above easier to achieve, either in terms of building blocks or algorithms, or in terms of debugging facilities to ensure correctness.

So what I'm hoping to discuss is:

• What would the interface look like for associating, at compile time or runtime, a list of Components that a particular system is intended to access?
• What would the interface look like for describing, at compile time or runtime, the execution order dependencies between systems?
• Could EnTT grow some debugging facilities that would help prevent invalid access?
• What would the interface for some kind of "Calculate an execution plan" look like?

For some additional motivation behind this topic, I'm looking to integrate a library like https://github.com/taskflow/taskflow with an EnTT based game engine, where I hope to be able to have the execution graph that's fed to taskflow every frame be calculated automatically using a declarative style interface. Obviously the reason why I would want this calculated automatically is because: Lazyness. Why make a human do something when an algorithm can do it for us?

I'm also hoping to help junior contributors to the game engine avoid screwing up the execution plan by mistake by adding a dependency on a component (e.g. using the component some how) without declaring it.

So maybe what a solution to this discussion topic looks like is some kind of template-metaprogramming registration for a system and it's components, and then a wrapper around EnTT that requires the system that's accessing the data to include the System's type (e.g. whatever was used to declare it in the system<->component dependency registry) in the call to EnTT to get the component(s) the system wants to access. This would allow either a compile-error, or a runtime assert, to inform the junior contributor that they goofed. Presumably this checking could be disabled for debug builds, or could be compile time only.

[skypjack]

EnTT does not attempt to synchronize across multiple threads, and requires users of EnTT to do that if they want it.

Yeah, mainly because you don't need to synchronize anything at all in many cases and with a good scheduling.

Adding and removing components can re-arrange the underlying storage for all components of that component type in a way that precludes multi-threaded access.

Yes and no. In theory, in-place deletion could be fully thread safe. Though, not sure it's worth it as a feature since deleting is a transparent operation that doesn't require the caller to know about the nature of the storage.

Reading / writing to unrelated components types can be done in parallel without synchronization (as far as EnTT is concerned, anyway)
Reading a specific component from multiple threads simultaneously is fine.

👍

Writing to a specific component from multiple threads, or writing from one thread, and reading from another, requires external synchronization.

Yeah, exactly, it boils down to accessing concurrently an instance of an object. It doesn't really depend on EnTT actually.

Keeping the above concerns in mind, even if they aren't 100% accurate, raises the following discussion points.
[...]
As far as I know, EnTT does not try to provide any kind of facilities to make the above easier to achieve, either in terms of building blocks or algorithms, or in terms of debugging facilities to ensure correctness.

Ehm, again, yes and no. There is a class called organizer you can look at in EnTT. It's a trivial tool meant more as a proof of concept than anything super solid to use but that works fine for the simplest cases.
It's not that hard to copy the same concepts and port them over to your own implementation eventually. Scheduling is a matter of constructing a graph and reworking it in the right way to avoid contention on the same resources.

What would the interface look like for associating, at compile time or runtime, a list of Components that a particular system is intended to access?

Technically speaking, if you pass a whole registry to a system, you can't easily limit the types of components it can access. On the other hand, if you only pass views to systems, it works (almost) out of the box, since the component types are part of their signatures with the right constness.
For example, view<T, const U> clearly tells you that we want to write T and read U. Additional resources can be expressed by the system or during registration in other ways. See the organizer for a working example.

What would the interface look like for describing, at compile time or runtime, the execution order dependencies between systems?

If registering T before U means giving T a higher priority than U, you're done. You don't need blocks nor phases nor anything else. With a proper way of setting global sync points (that literally define your sections eventually), you can just construct the graph from the system dependencies and make it safe to run then. When you want T to run after U necessarily, just set a shared resource and you've it.

Could EnTT grow some debugging facilities that would help prevent invalid access?

Not sure what kind of facilities would help but I'm open to discussing it, of course. Any ideas?

What would the interface for some kind of "Calculate an execution plan" look like?

See above. 🙂

For some additional motivation behind this topic [...]

Technically speaking, the organizer has been used (among the other things) with a similar library, that is enkiTs.
Long story short, the tool doesn't schedule anything for you. It just returns an execution graph to use with this kind of libraries. You then visit the graph and construct the flow in the target library. So, yeah, I know what you mean. 🙂