Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

convert claims hospital and quidel to chunk backfill into month instead of 28 days #2071

Open
aysim319 opened this issue Oct 29, 2024 · 0 comments
Assignees
Labels
data quality Missing data, weird data, broken data

Comments

@aysim319
Copy link
Contributor

The current behavior for claims hospital and quidel is that the source data is written to a parquet file and merged every 28 days, when a patch needs to be done, the parquet files also need to be patched which involves significant increase in complexity. (multiple files needs to be modified and the process is hard to automate as is)

The current process poses a problem in maintainability. The backfill is a system parallel to the indicator that requires separate processes to maintain. Furthermore, the specific format that the parquet files are created and updated significantly increases the complexity of patching. Given the current process when there is an outage, multiple files that are originally unrelated to the outage dates have to be modified in a cascading fashion. As a result, extra time and manual intervention needs to be taken for patching signals.

@aysim319 aysim319 added the data quality Missing data, weird data, broken data label Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data quality Missing data, weird data, broken data
Projects
None yet
Development

No branches or pull requests

2 participants