Skip to content

Commit 5e3b302

Browse files
authored
Typo patch (#19)
* mv md to adoc * adoc format * fix typo in fn doc
1 parent 67a3695 commit 5e3b302

File tree

3 files changed

+74
-69
lines changed

3 files changed

+74
-69
lines changed

README.adoc

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
= Waterfall
2+
3+
image:https://github.com/robertluo/waterfall/actions/workflows/main.yml/badge.svg[CI,link=https://github.com/robertluo/waterfall/actions/workflows/main.yml]
4+
image:https://img.shields.io/clojars/v/io.github.robertluo/waterfall.svg[Clojars project,link=https://clojars.org/io.github.robertluo/waterfall]
5+
image:https://cljdoc.org/badge/io.github.robertluo/waterfall[cljdoc,link=https://cljdoc.org/d/io.github.robertluo/waterfall]
6+
7+
== Rational
8+
9+
Try to use Apache Kafka clients in idiomatic Clojure.
10+
11+
Kafka has Java/Scala clients API, and programmers have no problem calling them directly in Clojure, so we do not need to "wrap" Java API.
12+
13+
However, from a Clojure programmer's perspective, direct call Kafka Java API does not feel very good:
14+
15+
- It is complex, with many interfaces, classes, and methods, which is not a good experience for a dynamic language.
16+
- It looks more like a framework or even frameworks. Users have to have a lot of background knowledge and implementation details.
17+
- Clojure programmers could use transducers to deal with data and do not need to use Kafka streams.
18+
19+
Hence, Waterfall is an attempt at a minimalist library with additional optional batteries.
20+
21+
- Only two core functions: `producer` and `consumer`.
22+
- Relying on the excellent https://github.com/clj-commons/manifold[manifold library], `producer`, and `consumer` is just the implementation of the Manifold stream.
23+
- Minimum necessary dependencies.
24+
- Clojure version SERDES, in `robertluo.waterfall.shape` namespace. Underlying, give Kafka API byte array SERDES.
25+
- Optional features require optional dependencies. For example, if you want to use nippy as SERDES, you put it in your classpath.
26+
27+
== Development experience
28+
29+
Because all functions have their schemas incorporated, you can get the best development experience if dependency `metosin/malli` is in your classpath.
30+
31+
Put `(malli.dev/start!)` in your `user.clj` will https://github.com/metosin/malli/blob/master/docs/function-schemas.md#tldr[enable clj-kondo] to use the schemas when editing.
32+
33+
## Single API
34+
35+
Powered by https://github.com/robertluo/fun-map[fun-map], you can use one single API for accessing a Kafka cluster without any further knowledge:
36+
37+
- https://cljdoc.org/d/io.github.robertluo/waterfall/CURRENT/api/robertluo.waterfall#kafka-cluster[`kafka-cluster`]
38+
39+
You can see an example in link:notebook/easy.clj[this easy example notebook].
40+
41+
## API namespaces
42+
43+
[cols="1,1"]
44+
|===
45+
| namespace | Description
46+
47+
| robertluo.waterfall | main API
48+
| robertluo.waterfall.shape | optional Clojure data shape transformation
49+
|===
50+
51+
## Files Description
52+
53+
[cols="1,1"]
54+
|===
55+
| Filename | Description
56+
57+
| deps.edn | Clojure tools.deps configuration
58+
| tests.edn | kaocha test runner configuration
59+
| build.clj | Clojure tools.build building script
60+
| notebook | https://github.com/nextjournal/clerk[Clerk] notebooks to demostrate usage, use `clojure -M:dev:notebook` to use them
61+
|===
62+
63+
== https://unlicense.org[Unlicense]
64+
65+
2022, Robertluo

README.md

Lines changed: 0 additions & 59 deletions
This file was deleted.

src/robertluo/waterfall.clj

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,9 @@
33
(:require
44
[robertluo.waterfall
55
[core :as core]
6-
[util :as util]]
7-
[manifold.stream :as ms]
8-
[robertluo.waterfall.shape :as shape]
9-
[robertluo.waterfall.util :as util]))
6+
[util :as util]
7+
[shape :as shape]]
8+
[manifold.stream :as ms]))
109

1110
(def schema
1211
"Schema for waterfall"
@@ -86,13 +85,13 @@
8685
(defn kafka-cluster
8786
"returns a convient life-cycle-map of a kafka cluster. It requires:
8887
- `::nodes` kafka bootstrap servers
89-
- `::shapes` shape of data, example: [(shape/topic (constantly \"sentence\"))(shape/edn)(shape/value-only)]
88+
- `::shapes` shape of data, example: `[(shape/topic (constantly \"sentence\"))(shape/edn)(shape/value-only)]`
9089
9190
If you just use it to publish message,
9291
- optional `::producer-config` can specify additional kafka producer configuration.
9392
9493
If you want to consume from topics:
95-
- `::topics` the topic you want to subscribe to. example: [\"sentence\"]
94+
- `::topics` the topic you want to subscribe to. example: `[\"sentence\"]`
9695
- `::group-id` the group id for the message consumer
9796
- optional `::source-xform` is a transducer to process message before consuming
9897
- optional `::consumer-config` can specify additional kafka consumer configuration. With additions:
@@ -101,10 +100,10 @@
101100
102101
The returned map has different level of key-values let you use:
103102
- Highest level, no additional knowledge:
104-
- For consumer: `::consume` a function, a one-arity (each message) function as its arg, returns nil.
103+
- For consumer: `::consume` a function, a one-arity (each message) function as its arg, returns `nil`.
105104
- For producer:
106-
-`::put` a function with a message as its arg. e.g. ((::put return-map) {:a 3})
107-
-`::put-all` a function with message sequence as its arg
105+
-`::put!` a function with a message as its arg. e.g. `((::put return-map) {:a 3})`
106+
-`::put-all!` a function with message sequence as its arg
108107
- Mid level, if you need access of underlying manifold sink/source.
109108
- `::source` a manifold source.
110109
- `::sink` a manifold sink.
@@ -139,7 +138,7 @@
139138
(->> (comp (map (shape/deserializer shapes)) (or source-xform (map identity)))
140139
(xform-source consumer)))
141140
::put! (fnk [::sink] (partial ms/put! sink))
142-
::put-all (fnk [::sink] (partial ms/put-all! sink))
141+
::put-all! (fnk [::sink] (partial ms/put-all! sink))
143142
::consume (fnk [::source] #(ms/consume % source))})
144143
kafka-conf-map)
145144
(throw (ClassNotFoundException. "Need io.github.robertluo/fun-map library in the classpath"))))

0 commit comments

Comments
 (0)