storage: update existing dataset to store workflow outputs

[mbjones]

Update an existing dataset by adding new objects from the workflow, and serializing both metadata and objects as a new version. This involves:

1. for each object, create SystemMetadata, and upload it and the object with a unique PID
2. add metadata to each object to EML metadata doc, update new version
3. add objects to data package resource map, update new version

These can be accomplished singly using atomic API operations, or using a more integrated client library like d1_python or rdataone that can do some of these steps for you.

[rmarow]

relates to what was discussed on 12/8 in standup https://qgreenland-net.github.io/notes/2025-12-08_dev-standup.html

first attempt could be to update an existing data set.

[rmarow]

@rushirajnenuji will share details on how to get started on dev server

[rushirajnenuji]

Thank you @rmarow for taking this on! 🙌🏻

This could be our very first iteration for the publish feature and I'll recommend to just focus on the shell recipe to keep things simple. (viz-workflows usually have a lot of output artifacts and we might use a different approach, for now we can leave them on PVCs).

The flow I imagine for the simplest case could be:

1. fetch file object (could be from an existing ADC dataset)
2. recipe execution
3. write output back to same dataset (write access required)

Sharing some pointers related to DataONE and ADC.

• DataONE API documentation
• CLI example for creating a new dataset
• d1_Python package
  • related examples
• prior related issue/s: QGreenland-Net/.github#1,

recommended settings to use for dev/testing:

Feel free to use either of the instances for testing:

ADC Test instance:
MetacatUI hosted at: https://test.arcticdata.io
base_url: `https://test.arcticdata.io/metacat/d1/mn`
cn_base_url: `https://cn-stage.test.dataone.org/cn`

KNB Dev instance:
MetacatUI hosted at: https://dev.nceas.ucsb.edu
base_url: `https://dev.nceas.ucsb.edu/knb/d1/mn`
cn_base_url: `https://cn-stage.test.dataone.org/cn`

Please let me know if you have questions, happy to jump on and work together! Thanks again!