Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix CI #3

Merged
merged 3 commits into from
Nov 1, 2023
Merged

Fix CI #3

merged 3 commits into from
Nov 1, 2023

Conversation

mgibowski
Copy link
Contributor

I have noticed CI failures, so I decided to fix that.

As seen on kafka-client project, I think it's more convenient to have a docker-compose.yml file, and start its services in CI, rather than using Github Actions-native services. The advantage is that the same docker configuration can be started locally to run the tests.

Furthermore:

  • The Postgres service in docker-compose.yml doesn't start automatically and is activated with a profile with-postgres. I imagine developers may often have native Postgres already running (at least that's the case for me).
  • Kafka container uses the current latest official kafka broker image from confluence without zookeper. Kraft is currently the recommended way to run kafka, and should also be faster to boot.
  • I extended sleep time in the test, as the test was failing on Github Actions with the previous value.

Example run on my fork:
https://github.com/mgibowski/kafkaesque/actions/runs/6707420753/job/18225921988

v0idpwn
v0idpwn approved these changes Nov 1, 2023
View reviewed changes
Copy link
Owner

@v0idpwn v0idpwn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good but one of the tests is failing. I assume that, for some reason, the setup failed to create the topic?

@mgibowski
Copy link
Contributor Author

So :bord.create_topics/3 returns: {:error, :invalid_config}.

I was able to reproduce it on my machine by running docker-compose down && docker-compose up -d.
I think it might be some schema registry config issue that I missed because docker was using some previous container on my machine and on my fork. Will try to fix that.

@mgibowski
Copy link
Contributor Author

So the issues is (from Kafka logs):

INVALID_CONFIG (Unknown topic config name: confluent.value.schema.validation)

According to Stack Overflow this option is only available on Confluent Cloud and not Open Source Kafka.

Should the tests (and possibly functionality) be adopted accordingly?

@mgibowski
Copy link
Contributor Author

The tests were also failing due to topics being compacted (which looks related to #1):

org.apache.kafka.common.InvalidRecordException: One or more records have been rejected due to 1 record errors in total, and only showing the first three errors at most: [RecordError(batchIndex=0, message='Compacted topic cannot accept message without key in topic partition integration_test_topic-0')]

Seem to run now correctly, after I removed all topic configs (https://github.com/mgibowski/kafkaesque/actions/runs/6720572630/job/18264440463#logs).

BTW, at work we use Redpanda and didn't encounter that issue.

@v0idpwn v0idpwn merged commit 42e0076 into v0idpwn:master Nov 1, 2023
1 check passed
@v0idpwn
Copy link
Owner

v0idpwn commented Nov 1, 2023

Thanks a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@v0idpwn v0idpwn v0idpwn approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mgibowski @v0idpwn @mgibowski-andjaro