Skip to content

Commit

Permalink
Return current field value when using transformations and returnUnmat…
Browse files Browse the repository at this point in the history
…ched
  • Loading branch information
pipliggins committed Oct 4, 2024
1 parent c08289f commit ca88239
Show file tree
Hide file tree
Showing 4 changed files with 11 additions and 4 deletions.
5 changes: 3 additions & 2 deletions adtl/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -117,9 +117,10 @@ def get_value_unhashed(row: StrDict, rule: Rule, ctx: Context = None) -> Any:
)
except AdtlTransformationWarning as e:
if ctx and ctx.get("returnUnmatched"):
return e.args[0]
else:
warnings.warn(str(e), AdtlTransformationWarning)
return value
else:
logging.error(str(e))
return None
return value
if value == "":
Expand Down
1 change: 1 addition & 0 deletions adtl/transformations.py
Original file line number Diff line number Diff line change
Expand Up @@ -347,6 +347,7 @@ def splitDate(
AdtlTransformationWarning,
stacklevel=2,
)
return None


def startYear(
Expand Down
7 changes: 6 additions & 1 deletion docs/specification.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,12 @@ if not present in a datafile, following the same syntax as `fieldPattern` key.
with either "date_" / "_date" in the field name or that have format date
set in the JSON schema
* **returnUnmatched**: Returns all values that are not able to be converted
according to the provided rules and formats. For fields with [value mappings](#field-with-value-mapping), it is equivalent to using `ignoreMissingKeys`. Fields using [data transformation functions](#data-transformations-(apply)) will return a string describing the failed function and input values.
according to the provided rules and formats. For fields with [value mappings](#field-with-value-mapping), it is equivalent to using `ignoreMissingKeys`. Fields using [data transformation functions](#data-transformations-(apply)) will issue a warning to the
terminal describing the error in the transformation. Transformations requiring multiple
parameters will only return the current field value that was not transformed.
> :warning: This is likely to return columns with non-matching datatypes. External json
validation may fail, as will attempting to use the `--parquet` option to save outputs as
parquet files (which required a consistent type down each column).

## Validation

Expand Down
2 changes: 1 addition & 1 deletion tests/__snapshots__/test_parser.ambr
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@
# name: test_return_unmapped
'''
age,date_death,date_of_birth,diabetes_type,has_smoking,pregnancy_birth_weight_kg,subject_id
fifteen,"Could not construct date from: year=2023, month=11, day=80",2023-11-20,No matches found for: 'no diabetes',today,eight,1
fifteen,2023,2023-11-20,no diabetes,today,eight,1

'''
# ---
Expand Down

0 comments on commit ca88239

Please sign in to comment.