light processing on over threshold errors, and keep track of record count #330
Conversation
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||
| @@ -199,7 +205,6 @@ def validate_batch_csv( | |||
| ) | |||
| yield results | |||
|
|
|||
| print("Processing other logic errors") | |||
| for validation_results, _ in validate_chunks( | |||
There was a problem hiding this comment.
I say we take out any of the none validate lf/parquet stuff because I think we've proven the batch csv approach isn't the way to go. Either here or in a new story.
There was a problem hiding this comment.
let's deal with it in another story; I think I replicated parts of that code, and abstracted out to what we have now for lazyframes; the thought is to still be able to do cli csv, but it would then fall into using the lazyframe batch processing.
| validation_results = format_findings( | ||
| validation_results, schema.name.value, checks | ||
| ) | ||
|
|
||
| error_counts, warning_counts = get_scope_counts(validation_results) | ||
| results = ValidationResults( | ||
| record_count=df.height, | ||
| error_counts=error_counts, |
There was a problem hiding this comment.
What is the intended use of this?
There was a problem hiding this comment.
gonna guess u meant to comment on the record_count part, it's so the aggregator knows the total records of the submitted file, so we don't do that processing in the API, lessening the load there, and truly just have the API upload the file, and nothing else.
| register_schema = get_register_schema(context) | ||
| validation_results = validate(register_schema, pl.DataFrame({"uid": all_uids}), 0) | ||
| validation_results = validate(register_schema, pl.DataFrame({"uid": all_uids}), 0, True) |
There was a problem hiding this comment.
I know this will probably never happen, but say a bank submits a sblar with over 1mil entries and they accidentally use the same UID for every row. We'd end up with register errors beyond our max limit, as well moving on to processing other logic errors beyond the max limit. Doesn't need to take place here but think we should look at treating these the same.
There was a problem hiding this comment.
maybe, although this part isn't doing it on the submitted lf/df; it's on the information we already gathered from the previous pass (syntax errors), so it's a lot less intensive, and it's currently not counting towards the limit.
No description provided.