Skip to content

Commit e28e5ce

Browse files
committed
Add exist tools compare
1 parent 76d9439 commit e28e5ce

File tree

1 file changed

+17
-2
lines changed

1 file changed

+17
-2
lines changed

010_replace_rabbitmq/readme.md

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -129,6 +129,21 @@ However, it is not trivial to implement all the communication needed from handsh
129129
The `kiwipy` in AiiDA servers as the role to provide the wrappered methods to talk between different components over RabbitMQ.
130130
Therefore, `kiwipy` is the interface for actioner and worker on bundle the operations to talk to RMQ, as the replacement, in `tatzelwurm` I should provide the python interface with methods ready to be used for sending and consuming certain type of MessagePack meesage.
131131

132+
## Other solution or exist tools?
133+
134+
It is discussed in https://github.com/aiidateam/AEP/pull/30#discussion_r813895745, better to look for tools that ready to use.
135+
What mentioned are [Apache Kafka](https://kafka.apache.org/), [faust-streaming](https://github.com/faust-streaming), [MQTT](https://mqtt.org/) and [Celery](https://docs.celeryq.dev/en/stable/getting-started/introduction.html).
136+
137+
- Apach Kafka: for event streaming, require service to start. (servirce required)
138+
- MQTT: is a messaging protocol in low bandwidth environments. (no queue system + overkill for low bandwidth)
139+
- faust-streaming: require Kafka as event streaming backend. (Kafka services required, it is like kiwipy but for Kafka)
140+
- Celery: this is a task queue. (For event streaming it requires a message broker, RMQ, Redis, SQS .., For persistent, it requires a results store.)
141+
142+
The `tatzelwurm` is tailored for AiiDA use cases specifically with message streaming/broke, lightwight queue system and persistent with KV store in disk in design.
143+
It is lightweight but will meet all the requirements for AiiDA.
144+
The tools above are for machines to machines scenario which required for huge message passing around millios of microservices and that is why those are overkill.
145+
We are not targeting to that but for AiiDA with required architecture as described above.
146+
132147
## Design details
133148

134149
In this section, I futher extend on the details of the new architecture.
@@ -509,8 +524,8 @@ Instead of go into details of comparing the Rust v.s. Python in async programmin
509524
I summarize following strong reasons drive me to make such decision:
510525

511526
- Performance wise, Python has GIL and therefore not able to provide real CPU parallelism. The bottleneck of AiiDA at the moment is from DB access, but it is not the excuse to compromise on performance for design.
512-
- Developers efficiency wise, I tried python from start, but I spend more time on figuring out how to frame stream to MassagePack in python even though I am more familiar with python. In python the mpsc and oneshot patterns which comes with tokio as commonly used patterns, has to implemented by hand with asyncio prime tools.
513-
- Use friendly wise, we want the future AiiDA is a `verdi` binary can download and run, without set up python environment. It is achievable with Rust.
527+
- Developers efficiency wise, I tried python from start, but I spend more time on figuring out how to frame stream to MassagePack in python even though I am more familiar with python. In python the mpsc and oneshot patterns which comes with tokio as commonly used patterns, but has to implemented by hand with asyncio prime tools.
528+
- User friendliness wise, we want the future AiiDA is a `verdi` binary can download and run, without set up python environment. It is achievable with Rust.
514529
- If the project done with python, if not me, there will be other people want to rewrite it in other language. If it writen in Rust, the only possibility within 10 years is rewriting it again in Rust.
515530
- With using Rust, [I believe I am able to finish this projet, not just start it](https://www.youtube.com/watch?v=Z3xPIYHKSoI).
516531

0 commit comments

Comments
 (0)