Skip to content

Commit

Permalink
fix: change overwrite detection to also allow .meta files
Browse files Browse the repository at this point in the history
Our ndjson format's new overwrite warning only looked for .ndjson
files but we can also write .meta files for NLP jobs. So allow
that sort of file too.
  • Loading branch information
mikix committed Oct 30, 2023
1 parent a9cdbd4 commit 53d243e
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 1 deletion.
2 changes: 1 addition & 1 deletion cumulus_etl/formats/batched_files.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ def _confirm_no_unknown_files_exist(self, folder: str) -> None:
except FileNotFoundError:
return # folder doesn't exist, we're good!

allowed_pattern = re.compile(rf"{self.dbname}\.[0-9]+\.{self.suffix}")
allowed_pattern = re.compile(rf"{self.dbname}\.[0-9]+\.({self.suffix}|meta)")
if not all(map(allowed_pattern.fullmatch, filenames)):
errors.fatal(
f"There are unexpected files in the output folder '{folder}'.\n"
Expand Down
1 change: 1 addition & 0 deletions tests/formats/test_ndjson.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ def store(
(None, True),
([], True),
(["condition.1234.ndjson", "condition.22.ndjson"], True),
(["condition.000.meta"], True),
(["condition.ndjson"], False),
(["condition.000.parquet"], False),
(["patient.000.ndjson"], False),
Expand Down

0 comments on commit 53d243e

Please sign in to comment.