Migration from Akka to Pekko - jackson serialization error

[kaa-dev-35]

Hey,

I'm migrating my microservice from Akka 2.6.19 to Pekko 1.1.0-M1. I use Play, CQRS, and Durable State Persistent Actor. However, I'm encountering an issue with Jackson serialization.

Here are some details about my setup:

Commands:

trait CborSerializable
protected trait Message extends CborSerializable
sealed trait Command extends Message
sealed trait Result extends Message
final case class Create(..., created: Instant, replyTo: ActorRef[Result]) extends Command

build.sbt:

scalaVersion := "2.13.14"

lazy val PekkoVersion = "1.1.0-M1"
lazy val PekkoHttpVersion = "1.1.0-M1"
lazy val PekkoManagementVersion = "1.1.0-M1"

libraryDependencies ++= Seq(
  guice,
  ws,
  clusterSharding,
  "org.apache.pekko"  %%  "pekko-connectors-kafka"  % "1.1.0-M1",
  "org.postgresql"  % "postgresql"  % "42.7.3",
  "org.playframework" %%  "play-slick"  % "6.1.1",
  // Play uses Pekko HTTP
  "org.apache.pekko"  %%  "pekko-http" % PekkoHttpVersion,
  "org.apache.pekko"  %%  "pekko-http-spray-json" % PekkoHttpVersion,
  // Cluster formation
  "org.apache.pekko"  %%  "pekko-cluster-typed" % PekkoVersion,
  "org.apache.pekko"  %%  "pekko-cluster-sharding-typed" % PekkoVersion,
  "org.apache.pekko"  %%  "pekko-serialization-jackson" % PekkoVersion,
  "org.apache.pekko"  %%  "pekko-discovery" % PekkoVersion,
  "org.apache.pekko"  %%  "pekko-management-cluster-http" % PekkoManagementVersion,
  "org.apache.pekko"  %%  "pekko-management-cluster-bootstrap"  % PekkoManagementVersion,
  "org.apache.pekko"  %%  "pekko-discovery-kubernetes-api"  % PekkoManagementVersion,
  // Persistence
  "org.apache.pekko" %% "pekko-persistence-typed" % PekkoVersion,
  "org.apache.pekko" %% "pekko-persistence-query" % PekkoVersion,
  "org.apache.pekko" %% "pekko-persistence-jdbc" % "1.1.0-M1",
  // For projections
  // "org.apache.pekko" %% "pekko-projection-durable-state" % "1.0.0",
  // "org.apache.pekko" %% "pekko-projection-slick" % "1.0.0",
  "org.scalatestplus.play" %% "scalatestplus-play" % "7.0.1" % Test
)

plugins.sbt:

addSbtPlugin("org.playframework" % "sbt-plugin" % "3.0.4")
addSbtPlugin("org.foundweekends.giter8" % "sbt-giter8-scaffold" % "0.16.2")
addSbtPlugin("io.github.davidmweber" % "flyway-sbt" % "7.4.0")

application.conf:

pekko {
  actor {
    provider = cluster
    serialization-bindings {
      "shared.CborSerializable" = jackson-cbor
    }
  }
  cluster {
    sharding {
      number-of-shards = 100
    }
    shutdown-after-unsuccessful-join-seed-nodes = 60s
    downing-provider-class = "org.apache.pekko.cluster.sbr.SplitBrainResolverProvider"
  }
}

Issue:

In Akka, everything worked fine, but in Pekko, I get the following error:

2024-07-04 08:45:02 ERROR core.application.ServiceImpl  Internal error. Create.
com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Java 8 date/time type `java.time.Instant` not supported by default:
add Module "com.fasterxml.jackson.datatype:jackson-datatype-jsr310" to enable handling

According to the default configuration, the module is added:

pekko.serialization.jackson {
  # The Jackson JSON serializer will register these modules.
  jackson-modules += "org.apache.pekko.serialization.jackson.PekkoJacksonModule"
  # PekkoTypedJacksonModule optionally included if pekko-actor-typed is in classpath
  jackson-modules += "org.apache.pekko.serialization.jackson.PekkoTypedJacksonModule"
  # PekkoStreamsModule optionally included if pekko-streams is in classpath
  jackson-modules += "org.apache.pekko.serialization.jackson.PekkoStreamJacksonModule"
  jackson-modules += "com.fasterxml.jackson.module.paramnames.ParameterNamesModule"
  jackson-modules += "com.fasterxml.jackson.datatype.jdk8.Jdk8Module"
  jackson-modules += "com.fasterxml.jackson.datatype.jsr310.JavaTimeModule"
  jackson-modules += "com.fasterxml.jackson.module.scala.DefaultScalaModule"
}

Despite this, the error persists. I would greatly appreciate any suggestions or guidance on resolving this issue.

Thanks!

[pjfanning]

Could you try Pekko 1.0.3? It is closer to Akka 2.6 than Pekko 1.1 is.
If this works with Pekko 1.0.3 and not Pekko 1.1.0-M1, we can raise a bug.

[pjfanning]

You would probably be better off using 1.0.x versions of all the Pekko modules. 1.1.0-M1 is an experimental set of releases. (M is for milestone).

[kaa-dev-35]

The problem that led me to switch to 1.1.0-M1 was with the "pekko-persistence-jdbc" % "1.0.0" dependency. It relies on an older version of Slick (3.3.3), while "org.playframework" %% "play-slick" % "6.1.1" depends on the latest Slick version. I finally changed the play-slick dependency to an older version from Typesafe. The project compiled and started, but I got the same serialization error.

application.conf:

scalaVersion := "2.13.14"

ThisBuild / evictionErrorLevel := Level.Info

lazy val PekkoVersion = "1.0.2"
lazy val PekkoHttpVersion = "1.0.1"
lazy val PekkoManagementVersion = "1.0.0"

libraryDependencies ++= Seq(
  guice,
  ws,
  clusterSharding,
  "org.apache.pekko"  %%  "pekko-connectors-kafka"  % "1.0.0",
  "org.postgresql"  % "postgresql"  % "42.7.3",
  //"org.playframework" %%  "play-slick"  % "6.1.1",
  "com.typesafe.play" %% "play-slick" % "5.0.2",
  "org.apache.pekko"  %%  "pekko-http" % PekkoHttpVersion,
  "org.apache.pekko"  %%  "pekko-http-spray-json" % PekkoHttpVersion,
  //"org.apache.pekko"  %%  "pekko-cluster-typed" % PekkoVersion,
  //"org.apache.pekko"  %%  "pekko-cluster-sharding-typed" % PekkoVersion,
  //"org.apache.pekko"  %%  "pekko-serialization-jackson" % PekkoVersion,
  "org.apache.pekko"  %%  "pekko-discovery" % PekkoVersion,
  "org.apache.pekko"  %%  "pekko-management-cluster-http" % PekkoManagementVersion,
  "org.apache.pekko"  %%  "pekko-management-cluster-bootstrap"  % PekkoManagementVersion,
  "org.apache.pekko"  %%  "pekko-discovery-kubernetes-api"  % PekkoManagementVersion,
  "org.apache.pekko" %% "pekko-persistence-typed" % PekkoVersion,
  "org.apache.pekko" %% "pekko-persistence-query" % PekkoVersion,
  "org.apache.pekko" %% "pekko-persistence-jdbc" % "1.0.0",
  // For projections
  //"org.apache.pekko" %% "pekko-projection-durable-state" % "1.0.0",
  //"org.apache.pekko" %% "pekko-projection-slick" % "1.0.0",
  "org.scalatestplus.play" %% "scalatestplus-play" % "7.0.1" % Test
)

[pjfanning]

Can you enable debug logging?

There is debug logging showing which Jackson modules get added.

https://github.com/apache/pekko/blob/main/serialization-jackson/src/main/scala/org/apache/pekko/serialization/jackson/JacksonObjectMapperProvider.scala#L286

[kaa-dev-35]

2024-07-04 15:54:19.543
2024-07-04 12:54:19 DEBUG o.a.p.s.j.JacksonObjectMapperProvider$  Registered Jackson module [com.fasterxml.jackson.module.scala.DefaultScalaModule]

2024-07-04 15:54:19.543
2024-07-04 12:54:19 DEBUG o.a.p.s.j.JacksonObjectMapperProvider$  Registered Jackson module [com.fasterxml.jackson.datatype.jsr310.JavaTimeModule]

2024-07-04 15:54:19.543
2024-07-04 12:54:19 DEBUG o.a.p.s.j.JacksonObjectMapperProvider$  Registered Jackson module [com.fasterxml.jackson.datatype.jdk8.Jdk8Module]

2024-07-04 15:54:19.543
2024-07-04 12:54:19 DEBUG o.a.p.s.j.JacksonObjectMapperProvider$  Registered Jackson module [com.fasterxml.jackson.module.paramnames.ParameterNamesModule]

2024-07-04 15:54:19.543
2024-07-04 12:54:19 DEBUG o.a.p.s.j.JacksonObjectMapperProvider$  Registered Jackson module [org.apache.pekko.serialization.jackson.PekkoStreamJacksonModule]

2024-07-04 15:54:19.543
2024-07-04 12:54:19 DEBUG o.a.p.s.j.JacksonObjectMapperProvider$  Registered Jackson module [org.apache.pekko.serialization.jackson.PekkoTypedJacksonModule]

2024-07-04 15:54:19.542
2024-07-04 12:54:19 DEBUG o.a.p.s.j.JacksonObjectMapperProvider$  Registered Jackson module [org.apache.pekko.serialization.jackson.PekkoJacksonModule]

[pjfanning]

Since I don't have a reproducible case - I can only speculate. Is it at all possible that the Jackson serialization is not being done with the pekko-serialization-jackson ObjectMapper? Your logging shows ERROR core.application.ServiceImpl Internal error. Create.

[kaa-dev-35]

The Create command is indirectly marked as CborSerializable via the Message trait. The documentation suggests this declaration and it works in Akka. ObjectMapper's initialization happens under the hood, so I'm not entirely sure how it works. :)

I'll try to mark it directly.

[pjfanning]

Can you even get the full stack trace for when com.fasterxml.jackson.databind.exc.InvalidDefinitionException happens?

[kaa-dev-35]

Thank you greatly for the idea! The problem lied in my custom serialization for publishing binary data to Kafka.

private val objectMapper = new ObjectMapper(new CBORFactory()) with ScalaObjectMapper
objectMapper.registerModule(DefaultScalaModule)
objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)

This worked with the previous version but doesn't work now.

[pjfanning]

I'll close this since it appears to be unrelated to Pekko. Try registering the extra module(s) on your ObjectMapper.

[kaa-dev-35]

Yes, Thank you! I'll fix it

[kaa-dev-35]

Solved.

val objectMapper = CBORMapper.builder()
    .addModule(new ParameterNamesModule())
    .addModule(new Jdk8Module())
    .addModule(new JavaTimeModule())
    .addModule(DefaultScalaModule)
    .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
    .build() :: ClassTagExtensions