The penguin-datalayer-core is the validation engine for the penguin-datalayer modules that belongs to raft-suite ecosystem created by DP6 in order to guarantee the Data Quality of the Data Engineering projects implemented on our clients, which is achieved through monitoring and automated data pipelines to guarantee credibility, consistency and availability in the planning, collection and data maintenance steps.
The JSON Schema is a structure that allow the validation of JSON documents. This structure is used in the project because it allows the declaration of expected data formats in the data layer.
The following Data Types are currently supported within this module:
- String
- Number
- Boolean
- Object
- Array
The following validation rules are accepted:
- Enum (Equals): Should be used to validate the parity between the schema value versus the value sent to the data layer;
- Pattern (Regex - String): Create regular expressions to validate the values of each key;
- minItems (Array): Validates the minimum itens contained in an array;
- Required: Should be used when a given key is required in the data layer
The following structure is a JSON Schema example:
{
"$schema": "",
"title": "Schema example",
"array": {
"$id": "#/properties/schema",
"type": "array",
"items": [
{
"type": "object",
"properties": {
"event": {
"type": "string",
"enum": ["test"]
},
"key1": {
"type": "object",
"properties": {
"key1_sub1": {
"type": "number",
"enum": [10]
},
"key1_sub2": {
"type": "string",
"pattern": "teste|test|.*"
},
"key1_sub3": {
"type": "string",
"enum": ["production"]
},
"key1_sub4": {
"type": "boolean",
"enum": "desktop|mobile|msite"
}
},
"required": ["key1_sub1", "key1_sub2", "key1_sub3", "key1_sub4"]
}
},
"required": ["event"]
}
]
}
}DP6 owns an automation that generates a structured validation schema.
Pull requests welcome! We would love some help to evolve this module. Feel free to search for open issues. If there's a new feature or bug, please open a new issue, so our team can follow up.
It will only be accepted contributions that follows the below requirements:
DP6 Koopa-troopa Team
e-mail: mailto:koopas@dp6.com.br
