support confluent's data contracts - ruleSet and metadata fields in subject

[fijolekProjects]

Hi team!
Recently Confluent added data-contracts feature to their schema-registry.
There are 2 new fields in subject entity: ruleSet and metadata. See more details in links below:
https://docs.confluent.io/platform/current/schema-registry/develop/api.html#post--subjects-(string-%20subject)

https://docs.confluent.io/platform/7.7/schema-registry/fundamentals/data-contracts.html

ruleSet is really interesting as you can specify row-level validation rules for your data. Using Confluent Java dependencies like:

io.confluent:kafka-schema-rules:7.6.1
io.confluent:kafka-avro-serializer:7.6.1

kafka serializer, before producing a message, uses validation rules from schema registry subject, and applies those rules in runtime. If validation passes then data is produced to the topic - if not, then data is sent for example to DLQ.

Sample schema with schema rules in confluent schema-registry looks like this:

{
  "schema": "{\"type\":\"record\",\"name\":\"OutboxOrder\",\"namespace\":\"com.example\",\"doc\":\"Order avro message\",\"fields\":[{\"name\":\"id\",\"type\":[\"null\",\"string\"],\"doc\":\"Order Id\",\"default\":null},{\"name\":\"store_id\",\"type\":\"string\"},{\"name\":\"person\",\"type\":{\"type\":\"record\",\"name\":\"OutboxPerson\",\"doc\":\"Person avro message\",\"fields\":[{\"name\":\"first_name\",\"type\":\"string\",\"doc\":\"The person first name - mandatory\"},{\"name\":\"last_name\",\"type\":[\"null\",\"string\"],\"doc\":\"The person last name - optional\",\"default\":null},{\"name\":\"age\",\"type\":[\"null\",\"int\"],\"doc\":\"The person age\",\"default\":null}]},\"doc\":\"The person making the order\"},{\"name\":\"product_id\",\"type\":\"long\"},{\"name\":\"timestamp\",\"type\":{\"type\":\"long\",\"logicalType\":\"timestamp-millis\"},\"doc\":\"Original timestamp of the event\"}]}",
  "ruleSet": {
    "domainRules": [
      {
        "name": "checkPersonAge",
        "kind": "CONDITION",
        "mode": "WRITE",
        "type": "CEL",
        "params": {
          "dlq.topic": "bad-data"
        },
        "expr": "message.person.age > 18",
        "onFailure": "DLQ",
        "disabled": false
      }
    ]
  }
}

Are there any plans to add similar support for Apicurio to make it fully compatible with new schema-registry releases?

Thanks!

[EricWittmann]

We do always want to make sure that our ccompat API is as compliant with the latest features of Confluent's schema registry as possible. However, we have not yet had the opportunity to scope out the data contract functionality they added somewhat recently.

We do have a GH issue tracking it though, if you want to watch it:

#4901

Who actually does the validation? Is it the SerDes or something else?

It's worth noting that field level validation and encryption is something that's being worked on within the Strimzi / AMQ Streams communities as well I believe. There is a proxy based solution ( https://kroxylicious.io/ ) that I'm aware of that might be of interest.

I wonder if @tombentley would have any thoughts in this area...