Reducing coupling with kafka-journal

[grzegorz-bielski]

As you may know from kafka-flow's overview, the library is tightly coupled with kafka-journal. But in reality, only the sources from the com.evolutiongaming.kafka.flow.journal package are involved in reading the journal.

kafka-flow is already used by many internal Evo apps to a great success, and a lot of them are reading generic kafka topics instead ofkafka-journal's journal.

Direct dependency on kafka-journal brings a lot of unnecessary, transient dependencies to classpath like akka, making the library much harder to use than it should be, and the path for migration to Scala 3 unclear. In many cases kafka-journal is just treated as a source of util classes that are not strictly related to the kafka-journal domain.

My suggestion would be to move code that is shared between kafka-journal and kafka-flow into separate libraries and modules.

In particular, we can:

1. Ensure binary and source compatibility is checked:

• in kafka-flow: Add sbt-version-policy #593
• in scassandra: Add sbt-version-policy scassandra#250
• in skafka: Add sbt-version-policy skafka#420

2. kafka related

• Use ConsumerRecord and ConsumerRecords from skafka directly instead of ConsRecord and ConsRecords aliases
• Extract to skafka / remove / internalize:
  • PartitionOffset
  • KafkaHealthCheck
  • SkafkaHelper
  • HeaderToTuple
  • TupleToHeader

3. cassandra related

• Extract to scassandra / remove / internalize:
  • CassandraSession
  • CassandraHealthCheck
  • Segments
  • CassandraHelper

4. Extract to separate kafka-flow-journal and kafka-flow-journal-cassandramodules:

• all *Journal* stuff with com.evolutiongaming.kafka.flow.journal from core and persistence-cassandra
  In the interim stage, these module could directly depend on the kafka-journal, but ideally types like SeqNr or ConsRecord would be extracted to some kafka-journal-api, which would be a shared dependency between kafka-journal and kafka-flow-journal.

5. Extract to cats-helper or other helper libs or just remove:

• RandomIdOf
• FromBytes
• ToBytes
• FromAttempt
• Fail - we could make a helper (extension) method for JournalError(a).raiseError and just use ApplicativeThrowable directly instead
• FromJsResult (play-json util)