You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: 010_replace_rabbitmq/readme.md
+17-2Lines changed: 17 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -129,6 +129,21 @@ However, it is not trivial to implement all the communication needed from handsh
129
129
The `kiwipy` in AiiDA servers as the role to provide the wrappered methods to talk between different components over RabbitMQ.
130
130
Therefore, `kiwipy` is the interface for actioner and worker on bundle the operations to talk to RMQ, as the replacement, in `tatzelwurm` I should provide the python interface with methods ready to be used for sending and consuming certain type of MessagePack meesage.
131
131
132
+
## Other solution or exist tools?
133
+
134
+
It is discussed in https://github.com/aiidateam/AEP/pull/30#discussion_r813895745, better to look for tools that ready to use.
135
+
What mentioned are [Apache Kafka](https://kafka.apache.org/), [faust-streaming](https://github.com/faust-streaming), [MQTT](https://mqtt.org/) and [Celery](https://docs.celeryq.dev/en/stable/getting-started/introduction.html).
136
+
137
+
- Apach Kafka: for event streaming, require service to start. (servirce required)
138
+
- MQTT: is a messaging protocol in low bandwidth environments. (no queue system + overkill for low bandwidth)
139
+
- faust-streaming: require Kafka as event streaming backend. (Kafka services required, it is like kiwipy but for Kafka)
140
+
- Celery: this is a task queue. (For event streaming it requires a message broker, RMQ, Redis, SQS .., For persistent, it requires a results store.)
141
+
142
+
The `tatzelwurm` is tailored for AiiDA use cases specifically with message streaming/broke, lightwight queue system and persistent with KV store in disk in design.
143
+
It is lightweight but will meet all the requirements for AiiDA.
144
+
The tools above are for machines to machines scenario which required for huge message passing around millios of microservices and that is why those are overkill.
145
+
We are not targeting to that but for AiiDA with required architecture as described above.
146
+
132
147
## Design details
133
148
134
149
In this section, I futher extend on the details of the new architecture.
@@ -509,8 +524,8 @@ Instead of go into details of comparing the Rust v.s. Python in async programmin
509
524
I summarize following strong reasons drive me to make such decision:
510
525
511
526
- Performance wise, Python has GIL and therefore not able to provide real CPU parallelism. The bottleneck of AiiDA at the moment is from DB access, but it is not the excuse to compromise on performance for design.
512
-
- Developers efficiency wise, I tried python from start, but I spend more time on figuring out how to frame stream to MassagePack in python even though I am more familiar with python. In python the mpsc and oneshot patterns which comes with tokio as commonly used patterns, has to implemented by hand with asyncio prime tools.
513
-
-Use friendly wise, we want the future AiiDA is a `verdi` binary can download and run, without set up python environment. It is achievable with Rust.
527
+
- Developers efficiency wise, I tried python from start, but I spend more time on figuring out how to frame stream to MassagePack in python even though I am more familiar with python. In python the mpsc and oneshot patterns which comes with tokio as commonly used patterns, but has to implemented by hand with asyncio prime tools.
528
+
-User friendliness wise, we want the future AiiDA is a `verdi` binary can download and run, without set up python environment. It is achievable with Rust.
514
529
- If the project done with python, if not me, there will be other people want to rewrite it in other language. If it writen in Rust, the only possibility within 10 years is rewriting it again in Rust.
515
530
- With using Rust, [I believe I am able to finish this projet, not just start it](https://www.youtube.com/watch?v=Z3xPIYHKSoI).
0 commit comments