New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatic data collection system #1

Open

7 tasks

ValterAro opened this issue Jul 9, 2024 · 0 comments

ValterAro commented Jul 9, 2024 •

edited by PaulaMerle

Loading

AS A product manager
I WANT TO have an automatic system for retrieving data
SO THAT data can be collected without manual interference

AC:

Possibility to add and configure new API connections Setting up API connections #2
Possibility to add new websites for scraping Setting up web scraping #3
Metadata should be added to each file showing when the data was acquired and from where Metadata #4
Data should be automatically uploaded to the specified data storage

For automated renewal

The technical solution should use CronManager (https://github.com/buerokratt/CronManager) for triggering periodical work?
Before downloading, data source must be compared to the allready retrieved file from the same data source in the data storage. Data will be renewed only in case there are any changes in the data.
The period of data renewal is adjustable

ValterAro mentioned this issue

Clean the acquired data #6

Open

6 tasks

PaulaMerle changed the title ~~Automatic data scraping system~~ Automatic data update system

PaulaMerle changed the title ~~Automatic data update system~~ Automatic data collection system

PaulaMerle transferred this issue from buerokratt/Common-Services

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment