From c3347d3382be2b4cd273646bb52e43ba878e45ed Mon Sep 17 00:00:00 2001 From: Emily Cheng Date: Thu, 21 Nov 2024 13:45:07 -0800 Subject: [PATCH] Reformat code block tables (#28) * Reformat html tags * Fix yaml spacing * More fixups --- ...data-quality-rule-definition-guidelines.md | 308 ++++++++---------- 1 file changed, 142 insertions(+), 166 deletions(-) diff --git a/docs/data-quality-rule-definition-guidelines.md b/docs/data-quality-rule-definition-guidelines.md index b228241..95dbd33 100644 --- a/docs/data-quality-rule-definition-guidelines.md +++ b/docs/data-quality-rule-definition-guidelines.md @@ -27,11 +27,12 @@ The form validator uses QC rules defined as JSON or YAML data objects to check d - + + + +
YAML Rule Definition JSON Rule Definition When Validating YAML Rule DefinitionJSON Rule DefinitionWhen Validating
- -```yaml +

 ptid:
   type: integer
   required: true
@@ -41,11 +42,10 @@ birthmo:
   required: true
   min: 1
   max: 12
-```
+
 - -```json +

 {
     "ptid": {
         "type": "integer",
@@ -58,16 +58,16 @@ birthmo:
         "max": 12
     }
 }
-```
+
 - -```python +

 {"ptid": 101, "birthmo": 12}    # passes
 {"ptid": 102, "birthmo": 15}    # fails
 {"ptid": 103}                   # fails
-```
+
## Validation Rules @@ -78,13 +78,15 @@ Keywords frequently used in UDS rules are described in the table below: - + + + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + +
Keyword Description Example KeywordDescriptionExample
allowed Specify the list of allowed values. Validation will fail if any other value is given in the data record. allowedSpecify the list of allowed values. Validation will fail if any other value is given in the data record. - -```python +

 schema = {
     "limit": {
         "type": "integer",
@@ -94,16 +96,14 @@ schema = {
 
 data = {"limit": 10}    # passes
 data = {"limit": 20}    # fails
-```
-
+
forbidden Specify the list of forbidden values. Validation will fail if values in this list are included in the record. forbiddenSpecify the list of forbidden values. Validation will fail if values in this list are included in the record. - -```python +

 schema = {
     "user": {
         "type": "string",
@@ -113,16 +113,14 @@ schema = {
 
 data = {"user": "admin"}    # passes
 data = {"user": "viewer"}   # fails
-```
-
+
min, max Minimum and maximum value allowed (only applicable to object types which support comparison operations like integers or floats). Each keyword can be used independently. Use together to define a range. min, maxMinimum and maximum value allowed (only applicable to object types which support comparison operations like integers or floats). Each keyword can be used independently. Use together to define a range. - -```python +

 schema = {
     "length": {
         "type": "float",
@@ -133,16 +131,14 @@ schema = {
 
 data = {"length": 14}       # passes
 data = {"length": 20.8}     # fails
-```
-
+
nullable If set to "true", the field value is allowed to be empty. This rule will be checked on every field, regardless of if it's defined or not. The rule's constraints defaults it to "false". In other words, if neither `nullable` nor `required` are set, the field is required. nullableIf set to "true", the field value is allowed to be empty. This rule will be checked on every field, regardless of if it's defined or not. The rule's constraints defaults it to "false". In other words, if neither `nullable` nor `required` are set, the field is required. - -```python +

 schema = {
     "country": {
         "type": "string",
@@ -160,16 +156,14 @@ schema = {
 }
 
 data = {"country": ""}      # fails
-```
-
+
required If set to "true", the field is mandatory. Validation will fail when it is missing. requiredIf set to "true", the field is mandatory. Validation will fail when it is missing. - -```python +

 schema = {
     "name": {
         "type": "string",
@@ -184,16 +178,14 @@ schema = {
 data = {"name": "Steve", "age": 50}     # passes
 data = {"name": "Debby"}                # passes
 data = {"age": 40}                      # fails
-```
-
+
type Data type allowed for the value. See the Cerberus documentation for the list of type names. If multiple types are allowed, you can specify the types as a list. typeData type allowed for the value. See the Cerberus documentation for the list of type names. If multiple types are allowed, you can specify the types as a list. - -```python +

 schema = {
     "limit": {
         "type": "integer"
@@ -212,16 +204,14 @@ schema = {
 data = {"limit": 10}        # passes
 data = {"limit": 11.5}      # passes
 data = {"limit": "one"}     # fails
-```
-
+
anyof Allows to define different sets of rules to validate against, supplied in a list of dicts. Field will be considered valid if any of the provided constraints validates the field. anyofAllows to define different sets of rules to validate against, supplied in a list of dicts. Field will be considered valid if any of the provided constraints validates the field. - -```python +

 schema = {
     "age": {
         "type": "integer",
@@ -240,27 +230,24 @@ schema = {
 data = {"age": 40}      # passes
 data = {"age": 999}     # passes
 data = {"age": 200}     # fails
-```
-
+
regex Regex to validate against; only valid for string values. regexRegex to validate against; only valid for string values. - -```python +

 schema = {
     "email": {
         "type": "string",
-        "regex": "^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$ "
+        "regex": "^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$"
     }
 }
 
-data = {"email": "john@example.com"}            # passes
+data = {"email": "john\@example.com"}            # passes
 data = {"email": "john_at_example_dot_com"}     # fails
-```
-
+
@@ -313,11 +300,12 @@ The rule definition for `compare_with` should follow the following format: - + + + - +
YAML Rule Definition JSON Rule Definition When Validating YAML Rule DefinitionJSON Rule DefinitionWhen Validating
- -```yaml +

 birthyr:
   type: integer
   required: true
@@ -326,11 +314,10 @@ birthyr:
     base: current_year
     adjustment: 15
     op: "-"
-```
+
 - -```json +

 {
     "birthyr": {
         "type": "integer",
@@ -343,16 +330,15 @@ birthyr:
         }
     }
 }
-```
+
 - -```python +

 {"birthyr": 1995}   # passes
 {"birthyr": 2030}   # fails until 2045
-```
+
**Absolute Value Example** @@ -361,12 +347,12 @@ birthyr: - + + + - - +
YAML Rule Definition JSON Rule Definition When Validating YAML Rule DefinitionJSON Rule DefinitionWhen Validating
- -```yaml +

 waist1:
   type: float
   required: true
@@ -376,13 +362,12 @@ waist1:
     adjustment: 0.5
     op: abs
 waist2:
-    type: float
-    required: true
-```
+  type: float
+  required: true
+
 - -```json +

 {
         "waist1": {
             "type": "float",
@@ -399,16 +384,15 @@ waist2:
             "required": true
         }
     }
-```
+
 - -```python +

 {'waist1': 5, 'waist2': 5.25}   # passes
 {'waist1': 5, 'waist2': 4.4}    # fails
-```
+
### compare_age @@ -451,11 +435,12 @@ The rule definition for `compare_age` should follow the following format: - + + + - +
YAML Rule Definition JSON Rule Definition When Validating YAML Rule DefinitionJSON Rule DefinitionWhen Validating
- -```yaml +

 frmdate:
   type: string
   formatting: date
@@ -464,22 +449,18 @@ frmdate:
     birth_year: birthyr
     birth_month: birthmo
     compare_to: behage
-
 birthmo:
   type: integer
   min: 1
   max: 12
-
 birthyr:
   type: integer
-
 behage:
   type: integer
-```
+
 - -```json +

 {
     "frmdate": {
         "type": "string",
@@ -503,16 +484,15 @@ behage:
         "type": "integer"
     }
 }
-```
+
 - -```python +

 {'frmdate': '2024/02/02', 'birthmo': 6, 'birthyr': 1950, 'behage': 50}  # passes
 {'frmdate': '2024/02/02', 'birthmo': 1, 'birthyr': 2024, 'behage': 50}  # fails
-```
+
@@ -563,7 +543,7 @@ The rule definition for `compatibility` follows the following format: One additional nuance is the evaluation against `None`/null values. Because Cerberus always evaluates `"nullable": False` by default, the application of a subschema in this case must explicitly set `"nullable": True` if the attributes evaluate or result in null values. For example -``` +```python # if case: If PARENTVAR is blank or 88, then VAR1 must be blank "if": { "parentvar": { @@ -614,30 +594,29 @@ If field `incntmod` (primary contact mode with participant) is 6, then field `in - + + + +
YAML Rule Definition JSON Rule Definition When Validating YAML Rule DefinitionJSON Rule DefinitionWhen Validating
- -```yaml +

 incntmod:
-  type: integer
+type: integer
   required: true
-
 incntmdx:
   type: integer
   nullable: true
   compatibility:
     - if:
-        incntmod:
-          allowed:
-            - 6
+      incntmod:
+        allowed:
+          - 6
       then:
         nullable: false
-```
+
 - -```json +

 {
     "incntmod": {
         "type": "integer",
@@ -660,49 +639,48 @@ incntmdx:
         ]
     }
 }
-```
+
 - -```python +

 {"incntmod": 1, "incntmdx": None}   # passes
 {"incntmod": 6, "incntmdx": 1}      # passes
 {"incntmod": 6, "incntmdx": None}   # fails
-```
+
This rule can also be used to define the "if not, then" case. For this, we use `forbidden` instead of `allowed`. So if field `incntmod` (primary contact mode with participant) is NOT 6, then field `incntmdx` (specify primary contact mode with participant) must be blank. - +
- + + + +
YAML Rule Definition JSON Rule Definition When Validating YAML Rule DefinitionJSON Rule DefinitionWhen Validating
- -```yaml +

 incntmod:
   type: integer
   required: true
-
 incntmdx:
   type: string
   nullable: true
   compatibility:
-    - if:
-        incntmod:
-          forbidden:
-            - 6
+    - if: null
+      incntmod:
+        forbidden:
+          - 6
       then:
         nullable: true
         filled: false
-```
+
 - -```json +

 {
     "incntmod": {
         "type": "integer",
@@ -726,17 +704,17 @@ incntmdx:
         ]
     }
 }
-```
+
 - -```python +

 {"incntmod": 1, "incntmdx": None}   # passes
 {"incntmod": 6, "incntmdx": 1}      # fails
 {"incntmod": 6, "incntmdx": None}   # passes
 {"incntmod": 1, "incntmdx": 1}      # fails
-```
+
@@ -767,19 +745,18 @@ One of `var1`, `var2`, or `var3` must be `1`. - + + + +
YAML Rule Definition JSON Rule Definition When Validating YAML Rule DefinitionJSON Rule DefinitionWhen Validating
- -```yaml +

 var1:
   type: integer
   nullable: true
-
 var2:
   type: integer
   nullable: true
-
 var3:
   type: integer
   nullable: true
@@ -787,19 +764,18 @@ var3:
     formula:
       or:
         - ==:
-            - 1
-            - var: var1
+          - 1
+          - var: var1
         - ==:
-            - 1
-            - var: var2
+          - 1
+          - var: var2
         - ==:
-            - 1
-            - var: var3
-```
+          - 1
+          - var: var3
+
 - -```json +

 {
     "var1": {
         "type": "integer",
@@ -829,16 +805,16 @@ var3:
         }
     }
 }
-```
+
 - -```python +

 {"var1": 1, "var2": 1, "var3": 1}            # passes
 {"var1": 1, "var2": None, "var3": None}      # passes
 {"var1": None, "var2": None, "var3": None}   # fails
-```
+
#### Custom Operations @@ -911,27 +887,27 @@ If field `taxes` (difficulty with taxes, business, and other papers) is 0 (norma - + + + +
YAML Rule Definition JSON Rule Definition When Validating YAML Rule DefinitionJSON Rule DefinitionWhen Validating
- -```yaml +

 taxes:
   type: integer
   temporalrules:
     - previous:
-        taxes:
-            allowed:
-            - 0
+      taxes:
+        allowed:
+          - 0
       current:
         taxes:
-            forbidden:
+          forbidden:
             - 8
-```
+
 - -```json +

 {
     "taxes": {
         "type": "integer",
@@ -951,18 +927,18 @@ taxes:
         ]
     }
 }
-```
+
 - -```python +

 # assume this record already exists
 # {"visit_date": 1, "taxes": 0}
 
 {"visit_date": 2, "taxes": 1}   # passes
 {"visit_date": 2, "taxes": 8}   # fails
-```
+
### compute_gds @@ -983,7 +959,7 @@ The rule definition for `compute_gds` should follow the following format: Custom rule defined to check whether a given Drug ID is valid RXCUI code. -This function uses `check_with` rule from Cerberus. Rule definition should be in the following format: +This function uses the `check_with` rule from Cerberus. Rule definition should be in the following format: ```json { @@ -992,4 +968,4 @@ This function uses `check_with` rule from Cerberus. Rule definition should be in } } ``` -*To validate `rxnorm`, validator should have a `Datastore` instance which implements `is_valid_rxcui` function which will check whether the given rxnormid value is valid RXCUI code* \ No newline at end of file +> **NOTE**: To validate `rxnorm`, validator should have a `Datastore` instance which implements the `is_valid_rxcui` function which will check if the given rxnormid value is a valid RXCUI code