Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add schema backends #47

Merged
merged 2 commits into from
Jan 17, 2020
Merged

Add schema backends #47

merged 2 commits into from
Jan 17, 2020

Conversation

dorner
Copy link
Member

@dorner dorner commented Jan 2, 2020

This introduces the concept of schema backends, allowing us to define different backends which we can swap in.

There are several main benefits of this:

  • The TestHelpers module does a lot of inconvenient stubbing which makes it very hard to debug. Decoupling this from testing allows us to write a separate "test backend" which doesn't involve RSpec stubbing.
  • Currently, it is difficult to run Deimos in development mode without a schema registry running. Separating "local encoding" to a different backend allows us to use this backend in development and skip the registry entirely.
  • Feedback from the initial release of Deimos questioned why it was being tied to Avro as a standard. This allows any schema definition to be used and is no longer Avro-specific.

Note that as part of this change, to keep consistency with the new SchemaBackends::Base class, I renamed PublishBackend to Backends::Base.

This also fixes #33, #10, part of #16, and part of #13.

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Unit tests, as well as ran locally both with and without schema registry. Still needs to be tested on a live system.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

@dorner dorner requested a review from colinmroberts January 2, 2020 20:18
@dorner dorner force-pushed the schema-backends branch 2 times, most recently from b28c10a to e360563 Compare January 13, 2020 15:04
colinmroberts
colinmroberts requested changes Jan 15, 2020
View reviewed changes
Copy link
Collaborator

@colinmroberts colinmroberts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

lib/deimos/active_record_consumer.rb Outdated
false
end

if column.type == :datetime && is_integer
return Time.zone.strptime(val.to_s, '%s')
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you refactor is_integer, can probably use Time.at and skip some of the string->int->string->int conversion here

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh good catch - probably could have done that before :)

lib/deimos/message.rb Outdated
@@ -18,23 +18,23 @@ def initialize(payload, producer, topic: nil, key: nil, partition_key: nil)

# Add message_id and timestamp default values if they are in the
# schema and don't already have values.
# @param schema [Avro::Schema]
def add_fields(schema)
# @param fields [Array<Name>] existing fields in the schema.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be Array<String> existing field names in the schema.

lib/deimos/schema_backends/avro_base.rb Outdated
'type' => 'record',
'name' => name,
'namespace' => @namespace,
'doc' => "Key for #{@namespace}.#{@schema}",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if the docs should have some indication it was autogenerated from Deimos

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't see why not :)

lib/deimos/schema_backends/avro_base.rb Outdated
# @param schema [String]
# @return [String]
def _key_schema_name(schema)
"#{schema.gsub('-value', '')}_key"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't -key more conventional? Or is there some other reason we're mixing _ and -

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I remember going back and forth with this a lot. Although the schema registry can register '-value' as a subject, '-value' is not a valid Avro schema name. This was one of the reasons that triggered this: apache/avro#732

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I think I can get rid of the gsub here because the schema wouldn't have -value in it to begin with - it would be in the subject but not the name.

lib/deimos/schema_backends/avro_schema_coercer.rb Outdated
Comment on lines 69 to 73
begin
true if Integer(val)
rescue StandardError
false
end
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indentation is wonky here and in _is_float_string

lib/deimos/schema_backends/base.rb Outdated
end

# Encode a message key. To be defined by subclass.
# @param key [String] the value to use as the key.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be String|Hash?

lib/deimos/test_helpers.rb Outdated
end
end

# Stub all already-loaded producers and consumers for unit testing purposes.
# :nodoc:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mark these with YARD @deprecated

spec/publish_backend_spec.rb
@@ -1,6 +1,6 @@
# frozen_string_literal: true

RSpec.describe Deimos::PublishBackend do
RSpec.describe Deimos::Backends::Base do
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May be worth renaming the file to make it easier to find

@dorner
Copy link
Member Author

dorner commented Jan 15, 2020

@colinmroberts changes made

@dorner dorner merged commit 6675a1d into master Jan 17, 2020
@dorner dorner deleted the schema-backends branch January 17, 2020 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@colinmroberts colinmroberts colinmroberts approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Upgrade to Avro 1.9
2 participants
@dorner @colinmroberts