Abstract file system with local and S3-based implementations

[jsstevenson]

Description

Currently wags-tails operates on a local file system. However, we could possibly abstract that out to a more general notion of a file directory, and provide alternate implementations (like S3-based).

Use case

Our proposed cloud data management workflow includes the use of an S3 bucket for primary source data, since that data is shared between multiple workflows and downloading it takes time (plus, it'd be nice for later reproducibility to retain a local copy). It'd be nice to have wags-tails check against this bucket directly rather than wrapping it somehow.

Acceptance criteria

Some kind of file abstraction that

• Passes all existing tests
• Supports configuration of S3 bucket location
• Checks for availability of latest file in an S3 bucket subdirectory
• Copies or streams a newly acquired file to an S3 bucket

[github-actions]

This issue is stale because it has been open 135 days with no activity. This issue will be closed if no further activity occurs in 14 days.