feat(nf-tower): add dataset:// FileSystem provider

[edmundmiller]

Summary

NIO FileSystemProvider for dataset:// URIs in the nf-tower plugin. Resolves Seqera Platform dataset references to their backing cloud storage paths transparently — pipelines use file('dataset://my-samplesheet') with zero code changes.

Replaces the rejected Channel.fromDataset() approach (seqeralabs/nextflow#19) with a platform-agnostic file system abstraction.

Resolution flow

file('dataset://my-samplesheet')
  → FileHelper.asPath() → DatasetPathFactory.parseUri()
  → DatasetFileSystemProvider → DatasetPath (lazy)
  → DatasetResolver: GET /datasets → ID → GET /versions → cloud URL
  → FileHelper.asPath('s3://bucket/data.csv') → delegate to cloud FS

Components

File	Purpose
DatasetFileSystemProvider.java	NIO SPI provider, delegates I/O to resolved cloud path
DatasetFileSystem.java	Minimal read-only FileSystem
DatasetPath.java	Path impl with lazy resolution, parses dataset://name?version=N
DatasetResolver.groovy	Platform API client (HttpClient + Bearer auth)
DatasetPathFactory.groovy	FileSystemPathFactory extension point

Phase 1 scope

• Read-only — write ops throw ReadOnlyFileSystemException
• Works with file(), Channel.fromPath(), nf-schema samplesheetToList()
• Config from existing tower.* settings (endpoint, accessToken, workspaceId)

Tests

• Unit tests: DatasetPath parsing, provider scheme/read-only enforcement, factory URI matching
• WireMock integration tests: full resolve → read flow with mocked Platform API, version selection, caching, auth headers
• All existing nf-tower tests pass unchanged

Related

• Supersedes seqeralabs/nextflow#19 (Channel.fromDataset)

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	475c1c1
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/699f9799fefc6300087f1050

[adamrtalbot]

Neat idea but I dislike the name dataset://, which feels too generic. But I can't think of a better one 🤔.

This will remove the requirement for an ephemeral URI for the dataset when running the pipeline and make it easier to make datasets a general data store.

[bentsherman]

I think it would be better to do seqera:// and add some qualifiers to the URI, for example:

seqera://<org>/<workspace>/datasets/<dataset>

Maybe some of these can be omitted. But the extra scoping would keep the URI open to future extensions, like accessing data links.

[jordeu]

I like Ben's suggestion; it allows future extensions.

Currently we have an experimental Fusion version that maps multiple Seqera resources to a path like this:

/fusion/seqera/
├── <organization>/
│   └── <workspace>/
│       ├── pipelines/                        # read-only
│       │   └── <name>.<id>.json
│       ├── runs/                             # read-only
│       │   ├── running/
│       │   ├── failed/
│       │   └── all/
│       │       └── <name>.<workflowId>/
│       │           ├── workflow.json
│       │           └── tasks/
│       │               └── <name>.<taskId>/
│       │                   ├── stdout.log
│       │                   └── stderr.log
│       ├── compute-envs/                     # read-only
│       │   └── <name>.<id>.json
│       ├── credentials/                      # read-only
│       │   └── <name>.<id>.json
│       ├── datasets/                         # read-only
│       │   └── <name>.<id>/
│       │       ├── metadata.json
│       │       ├── latest.csv
│       │       └── versions/
│       │           └── v<N>.csv
│       ├── datarepos/                        # read/write
│       │   └── <link-name>/
│       │       └── <cloud-storage-contents>...
│       └── studios/                          # read-only
│           └── <name>.<sessionId>/
│               ├── studio.json
│               └── checkpoints/
│                   └── <name>.<id>.json

I think that this new filesystem provider, even if it's only used for datasets now, can be extended to fetch other platform resources in the future.