sda-pipeline is part of NeIC Sensitive Data Archive and implements the components required for data submission.
It can be used as part of a Federated EGA or as a isolated Sensitive Data Archive.
sda-pipeline was built with support for both S3 and POSIX storage.
Recommended provisioning method for production is:
- on a ,
kubernetes clusterusing the helm chart;
For local development/testing see instructions in dev_utils folder.
| Component | Role |
|---|---|
| intercept | The intercept service relays message between the queue provided from the federated service and local queues. (Required only for Federated EGA use case) |
| ingest | The ingest service accepts messages for files uploaded to the inbox, registers the files in the database with their headers, and stores them header-stripped in the archive storage. |
| verify | The verify service reads and decrypts ingested files from the archive storage and sends accession requests. |
| finalize | The finalize command accepts messages with accessionIDs for ingested files and registers them in the database. |
| mapper | The mapper service register mapping of accessionIDs (IDs for files) to datasetIDs. |
| Component | Role |
|---|---|
| broker | Package containing communication with Message Broker https://github.com/neicnordic/sda-mq |
| config | Package for managing configuration. |
| database | Provides functionalities for using the database, providing high level functions https://github.com/neicnordic/sda-db. |
| storage | Provides interface for storage areas such as a regular file system or as a S3 object store. |
sda-pipeline documentation can be found at: https://neicnordic.github.io/sda-pipeline/pkg/sda-pipeline/
NeIC Sensitive Data Archive documentation can be found at: https://neic-sda.readthedocs.io/en/latest/
We happily accepts contributions. Please see our contributing documentation for some tips on getting started.