Skip to content

Commit

Permalink
Do not remove fields which do not fit in the schema
Browse files Browse the repository at this point in the history
  • Loading branch information
martin-village committed Feb 26, 2024
1 parent 0d0ff1e commit aa05ba8
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion singer/transform.py
Original file line number Diff line number Diff line change
Expand Up @@ -214,12 +214,16 @@ def _transform_object(self, data, schema, path, pattern_properties):
successes.append(success)
result[key] = subdata
else:
# Customization: if no schema is specified, insert the raw data
# This will happen if the data source changes their data model
result[key] = value

# track that field has been removed because it wasn't
# found in the schema. This likely indicates some problem
# with discovery but rather than failing the run because
# new data was added we'd rather continue the sync and
# allow customers to indicate that they want the new data.
self.removed.add(".".join(map(str, path + [key])))
# self.removed.add(".".join(map(str, path + [key])))

return all(successes), result

Expand Down

0 comments on commit aa05ba8

Please sign in to comment.