Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify NRIS-EPD importer to support a full and daily import #1266

Open
sggerard opened this issue Aug 6, 2024 · 0 comments
Open

Modify NRIS-EPD importer to support a full and daily import #1266

sggerard opened this issue Aug 6, 2024 · 0 comments
Labels
Roadmap To help with idnetifying on a planning roadmap Task

Comments

@sggerard
Copy link
Contributor

sggerard commented Aug 6, 2024

Describe the task
Modify the NRIS-EPD importer to support both a full import and a smaller dataset daily import.

Purpose
The amount of data the NRIS-EPD importer needs to process keeps growing larger and larger, causing the importer to take longer and longer. (Currently around 10 hours). This is not sustainable, consumes unnecessary resource and increases the likelihood of failure.

Acceptance Criteria

  • Daily import job is quick and handles all the most recent changes (Process the latest year's worth of data)
  • A full run takes place at a regular occurrence (weekly or monthly) to ensure all records are up to date

Additional context

  • Could potentially add a way to manually import a record (i.e by ID)
  • Change is approved by the EPD team and Robin.
@sggerard sggerard added the Task label Aug 6, 2024
@Keegnan Keegnan added the Roadmap To help with idnetifying on a planning roadmap label Nov 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Roadmap To help with idnetifying on a planning roadmap Task
Projects
None yet
Development

No branches or pull requests

2 participants