Multiple concurrent editors of schema?

[lapidus]

We're building something like a Headless CMS tool, and are considering EdgeDB.

The typical Headless CMS allows each tenant (in a multitenant setup) to flesh out their own schema (classes with properties and constraints).

In the ideal case, we could have a single instance of EdgeDB running and allow multiple editors of the schema. For example the users Anna and John might add organization-specific classes that inherit from similar abstract classes that we have configured. This would be really convenient as we could define abstractions that all users could benefit from.

But given that EdgeDB schema changes result in migrations (that take more than milliseconds to apply?), it doesn't seem feasible to cater to this scenario ...?

For example, take the scenario where there are 10 000 classes already defined in the database and a tenant (one of many) wants to change a single property of a class. If multiple tenants are working in the system, my concern is that it's too slow to give instant feedback, and it might also not handle concurrent edits very well?

I guess an option is to define something like a "class factory" (with a stable generic schema like ClassFactory.properties) that would allow users to create any type of class, but it kind of goes against the whole idea of EdgeDB where the specificity of the schema guarantees type safety, elegant introspection and more.

Questions:

1. Do you see any options to support this scenario currently?
2. Is there anything on the roadmap that would further enable a use case like this with 100s of concurrent editors of a schema?

Thank you!

[raddevon]

I talked with the team about this. You might consider using DDL directly instead of using migrations to make it faster and to help with the concurrency problem. Be prepared to handle concurrency errors with the DDL approach. This solution should be faster than running migrations and reasonably robust for concurrency.

We're taking an action to investigate if the error raised with concurrent DDL is retryable.

[lapidus]

Thank you @raddevon. I wasn't aware of the DDL option.

We're taking an action to #5546.

Is it correct to assume that:
a) DDL operations are generally much faster than SDL migrations?
b) 100s of users could potentially issue such DDL operations – but by default DDL would raise concurrency errors?
c) The issue you're pointing to #5546 is that DDL currently doesn't retry erroneous operations?
d) I could manually handle that case for now by putting erroneous requests in a queue?

Thank you for any clarifications!

[tailhook]

a) They are faster, since "SDL migrations" are schema diffing + running DDL (although, for someone reading this without context: edgedb migrate command just applies DDL, so it's not faster in the common usage scenario where diffing is going on during development). But the time it takes for specific DDL can be very different. So for slower DDLs the difference would be negligible, for faster DDLs difference is huge.
b) Technically yes, but they are serialized in the database anyways, so it might make sense to queue them in the first place
c) No, after investigation we figured out that DDLs retry by default. You might need more retries in case of 100s of users issuing such operation at the same time.
d) Yes. But you can also try using defaults. It looks like postgres does wait for the first transaction to commit before failing the subsequent one, so if there aren't too many concurrent ones it should work well. You can also try configuring delays (especially adding more randomness), and/or the number of retries.

[lapidus]

Thank you, excellent support @tailhook, @raddevon.
With this I think we could scale to quite a few concurrent users. If we need to go beyond that, I guess we need to setup multiple instances and let users work in their "namespaced instance" for the most part and only sync back to the "main/shared graph" when they really need to 🤔 Similar to how the "The Lightning Network" is a "layer 2" protocol layered on top of Bitcoin; putting some kind of cost on touching the more central parts of the graph.