Add support of references/dependencies for Protobuf schema type (DEPRECATED)

[libretto]

What it does

These changes add basic support of references/dependencies for protobuf schema type.
This code contains much common code which allows adding support of references/dependencies to other types.
Limitations:
The compatibility check does not support references/dependencies.
Serialization/Deserialization of protobuf does not support references/dependencies.

References: #195

UPDATE: This PR already supports compatibility check (schema compare).

[jjaakola-aiven]

I am getting 500 error with following request flow:

#!/bin/bash

NO_REF='{"schemaType":"PROTOBUF","schema":"syntax = \"proto3\";\n\nmessage NoReference {\n    string name = 1;\n}\n"}'
curl -H "Content-Type: application/json" localhost:8081/subjects/sub1/versions -d "$NO_REF"

WITH_FIRST_REF='{"schemaType":"PROTOBUF","schema":"syntax = \"proto3\";\n\nimport \"NoReference.proto\";\n\nmessage WithReference {\n    string name = 1;\n    NoReference ref = 2;\n}","references":[{"name":"NoReference.proto","subject":"sub1","version":1}]}'
curl -H "Content-Type: application/json" localhost:8081/subjects/sub2/versions -d "$WITH_FIRST_REF"

NO_REF_SECOND='{"schemaType":"PROTOBUF","schema":"syntax = \"proto3\";\n\nmessage NoReferenceTwo {\n    string name = 1;\n}\n"}'
curl -H "Content-Type: application/json" localhost:8081/subjects/sub3/versions -d "$NO_REF_SECOND"

ADD_NEW_REF_IN_SUB2='{"schemaType":"PROTOBUF","schema":"syntax = \"proto3\";\n\nimport \"NoReference.proto\";\nimport \"NoReferenceTwo.proto\";\n\nmessage WithReference {\n    string name = 1;\n    NoReference ref = 2;\n    NoReferenceTwo refTwo = 3;\n}\n","references":[{"name":"NoReference.proto","subject":"sub1","version":1},{"name":"NoReferenceTwo.proto","subject":"sub3","version":1}]}'
curl -H "Content-Type: application/json" localhost:8081/subjects/sub2/versions -d "$ADD_NEW_REF_IN_SUB2"

The usage of well known types should be demonstrated in tests and with the REST proxy.

[jjaakola-aiven]

@libretto I will merge this to feature branch in Karapace and I will add some tests and changes on top of this. Please comment on the open questions, especially the need of .proto files.

Following work here: https://github.com/aiven/karapace/commits/protobuf-schema-references-support
Pull request #473 for visibility and commenting.

[libretto]

I just working on fixing issues. I just debugging them. Also, I will add a compatibility check to dependencies support as soon as one of fixes requires it.

[jjaakola-aiven]

@libretto some of the issues are resolved in PR #473. Please do not update to this PR.

[libretto]

@libretto some of the issues are resolved in PR #473. Please do not update to this PR.

@jjaakola-aiven all things which You resolved was resolved in my code too but i do not push it to GitHub while tests are not successful.
Now all tests are successful and I push it there. The next step is testing and polishing it.

[libretto]

@jjaakola-aiven latest code from https://github.com/instaclustr/karapace/tree/pr2_clear works ok. So we could add Your tests there to see if it works. If something does not work let me fix the bugs.

[libretto]

@jjaakola-aiven When I open this PR I notice in the description its limitations and You know that I still working on it. I do not understand why You go to refactor and change code that is under my development without any discussion or requests for changes. I see You change the target branch for this PR to the draft branch and PR it to Main... But Your draft branch does not contain my latest code which includes fixes and recursive compatibility check which is already released in my code. The question is what I must do? Will You add this functionality by yourself so my branch and this PR are not required anymore so I may close my PR and will continue my next task on top of Your code? Or You will get my latest code and add Your changes to it so You will have all the functionality and fixes which I already implemented with Your changes?

[libretto]

@jjaakola-aiven i added there more tests from my branch which show bugs in dependencies compatibility algorithm which You implemented by self. And now I try debugging code which You offer there.

[jjaakola-aiven]

@libretto Pick the #473 as a base, add missing features, tests, corrections on top of it. That will serve as the feature branch to land the profobuf reference support to Karapace.

[libretto]

@jjaakola-aiven please check and merge with Your branch

[sujayinstaclustr]

@jjaakola-aiven can you confirm if there is anything that needs to be done, we are waiting for you to approve this PR for merge.

[libretto]

@jjaakola-aiven I see red text with 1 change request but all change requests were resolved weeks ago. I fixed all bugs which we talk about there. But this PR was not accepted. Why?

[RyanSkraba]

Hello @libretto, I just wanted to keep you in the loop -- I'll be taking a closer look at this PR and #504 in order to get this moving for the next release of Karapace. Thanks for all the work so far!

I'm catching up a bit on this and the the work being done in #195