Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

create local data store for database data #48

Open
jmlondon opened this issue Jun 27, 2023 · 2 comments
Open

create local data store for database data #48

jmlondon opened this issue Jun 27, 2023 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@jmlondon
Copy link
Collaborator

Many data components of this analysis rely on a pull from an internal PostgreSQL database. This has two implications:

  1. the data on the database can change/update and impact downstream analysis and results along with future reproducibility
  2. there's no option to version control the data

To resolve this, the plan is to create local data stores within the repository as either *.csv or *.parquet files

@jmlondon jmlondon added the enhancement New feature or request label Jun 27, 2023
@jmlondon jmlondon added this to the Submit Revisions to PeerJ milestone Jun 27, 2023
@jmlondon jmlondon self-assigned this Jun 27, 2023
@jmlondon
Copy link
Collaborator Author

going to explore managing this data store with the piggyback package

https://cran.r-project.org/web/packages/piggyback/vignettes/intro.html

@jmlondon
Copy link
Collaborator Author

my plan for this is to manage 3 repos related to this analysis/manuscript:

  1. berchukseals-haulout (this repo) will house the manuscript and core analysis/model fitting data. data will be pulled from the (new) berchukseals-data repo
  2. berchukseals-fits will house the final model fits as releases that can be easily downloaded for us in other analyses
  3. berchukseals-data will house the raw data and data processing scripts that involve interaction with the internal NOAA/PEP database. data products will be available as releases

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant