|
| 1 | += Waterfall |
| 2 | + |
| 3 | +image:https://github.com/robertluo/waterfall/actions/workflows/main.yml/badge.svg[CI,link=https://github.com/robertluo/waterfall/actions/workflows/main.yml] |
| 4 | +image:https://img.shields.io/clojars/v/io.github.robertluo/waterfall.svg[Clojars project,link=https://clojars.org/io.github.robertluo/waterfall] |
| 5 | +image:https://cljdoc.org/badge/io.github.robertluo/waterfall[cljdoc,link=https://cljdoc.org/d/io.github.robertluo/waterfall] |
| 6 | + |
| 7 | +== Rational |
| 8 | + |
| 9 | +Try to use Apache Kafka clients in idiomatic Clojure. |
| 10 | + |
| 11 | +Kafka has Java/Scala clients API, and programmers have no problem calling them directly in Clojure, so we do not need to "wrap" Java API. |
| 12 | + |
| 13 | +However, from a Clojure programmer's perspective, direct call Kafka Java API does not feel very good: |
| 14 | + |
| 15 | + - It is complex, with many interfaces, classes, and methods, which is not a good experience for a dynamic language. |
| 16 | + - It looks more like a framework or even frameworks. Users have to have a lot of background knowledge and implementation details. |
| 17 | + - Clojure programmers could use transducers to deal with data and do not need to use Kafka streams. |
| 18 | + |
| 19 | +Hence, Waterfall is an attempt at a minimalist library with additional optional batteries. |
| 20 | + |
| 21 | + - Only two core functions: `producer` and `consumer`. |
| 22 | + - Relying on the excellent https://github.com/clj-commons/manifold[manifold library], `producer`, and `consumer` is just the implementation of the Manifold stream. |
| 23 | + - Minimum necessary dependencies. |
| 24 | + - Clojure version SERDES, in `robertluo.waterfall.shape` namespace. Underlying, give Kafka API byte array SERDES. |
| 25 | + - Optional features require optional dependencies. For example, if you want to use nippy as SERDES, you put it in your classpath. |
| 26 | + |
| 27 | +== Development experience |
| 28 | + |
| 29 | +Because all functions have their schemas incorporated, you can get the best development experience if dependency `metosin/malli` is in your classpath. |
| 30 | + |
| 31 | +Put `(malli.dev/start!)` in your `user.clj` will https://github.com/metosin/malli/blob/master/docs/function-schemas.md#tldr[enable clj-kondo] to use the schemas when editing. |
| 32 | + |
| 33 | +## Single API |
| 34 | + |
| 35 | +Powered by https://github.com/robertluo/fun-map[fun-map], you can use one single API for accessing a Kafka cluster without any further knowledge: |
| 36 | + |
| 37 | + - https://cljdoc.org/d/io.github.robertluo/waterfall/CURRENT/api/robertluo.waterfall#kafka-cluster[`kafka-cluster`] |
| 38 | + |
| 39 | +You can see an example in link:notebook/easy.clj[this easy example notebook]. |
| 40 | + |
| 41 | +## API namespaces |
| 42 | + |
| 43 | +[cols="1,1"] |
| 44 | +|=== |
| 45 | +| namespace | Description |
| 46 | + |
| 47 | +| robertluo.waterfall | main API |
| 48 | +| robertluo.waterfall.shape | optional Clojure data shape transformation |
| 49 | +|=== |
| 50 | + |
| 51 | +## Files Description |
| 52 | + |
| 53 | +[cols="1,1"] |
| 54 | +|=== |
| 55 | +| Filename | Description |
| 56 | + |
| 57 | +| deps.edn | Clojure tools.deps configuration |
| 58 | +| tests.edn | kaocha test runner configuration |
| 59 | +| build.clj | Clojure tools.build building script |
| 60 | +| notebook | https://github.com/nextjournal/clerk[Clerk] notebooks to demostrate usage, use `clojure -M:dev:notebook` to use them |
| 61 | +|=== |
| 62 | + |
| 63 | +== https://unlicense.org[Unlicense] |
| 64 | + |
| 65 | +2022, Robertluo |
0 commit comments