Data Access Credentials

[briandoconnor]

Overview

This pull request updates the Workflow Execution Service (WES) OpenAPI specification (starting with #226) to enhance functionality in several key ways without creating breaking changes (hopefully). Key changes include 1) adding POST endpoints to support authentication/authorizing with a GA4GH Passport rather than a bearer token, 2) providing credentials for a WES server to access DRS (or other) inputs, and 3) for providing back credentials for the output of the workflow run to be accessible. Claude Code was used to generate some of these changes.

Related issues

• Data access credentials #18
• Add a workflow_params_drs_headers field to RunWorkflow endpoint #131
• Updates now that GA4GH AAI standard is confirmed #156

Related Standards

This implementation aligns with:

• GA4GH Passport Specification
• GA4GH DRS Specification for
  data access patterns
• WES 1.1.0 Specification
  for backward compatibility

Related PR

The feature branch behind this PR is based on feature/issue-176-wes-params and not develop since I was building off of the structured input/output. See #226

eLwazi-hosted GA4GH Hackathon

The eLwazi hosted GA4GH hackathon 7/28-8/1 is working on this issue given the need by various groups attending the session. For more info, see the agenda.

Built Documentation

The human-readable documentation: https://ga4gh.github.io/workflow-execution-service-schemas/preview/feature/issue-18-data-access-creds/docs/index.html

Issues/questions for discussion

• See security and implementation sections below...
• Do passport visas need to evolve to be useful for using passports as an AuthN/Z token for making WES requests? Right now, I know of Passports being used with visas that specifically talk about data access but the changes made in this PR open up using Passports for compute.
• Is it better to have a POST /runs/list endpoint instead of overloading POST /runs based on what request body schema is used?
• In previous conversations, the idea of delegating tokens to a WES service brought up security concerns. I think it's a valid point however we see examples with Driver projects of Passports being used in this way... retrieved by one service from the passport broken and handed to another service for data access. So it's a concern but I think we need to weigh the positives with negatives here and realize one security profile doesn't meet every implementers needs.
• do we need to make it more explicit in the schema that supporting passports is optional? A service may choose to not implement passports at all and they are still a valid WES. Do we need an error code. For example:
  • 405 Method Not Allowed - The endpoint exists, but the HTTP method (GET, POST, etc.) is
    not supported... useful for the POST endpoints on existing paths.
  • 501 Not Implemented - The endpoint is defined but not yet implemented by the server. Maybe this is the response for POST /runs when trying to list runs with a passport when that's not implemented?
• Do we need something in service-info that says this server supports POSTs w/ Passports or not?
• Do we need something in service-info to say the WES server supports delegated credentials? And more generally, the structured inputs/outputs.
• What's required? Does every WES 1.2 server need to support DRS+HTTP URLs with auth for inputs? For outputs the server isn't really mandated to use DRS or even HTTP outputs.
• in this PR, outputs are deposited (potentially) in a location and credentials (if applicable) are handed back to the WES caller. However, do we want a mode additionally where the submitter tells WES where they want the outputs to be written to (along with credentials to push data there)?

Authentication/Authorization for WES Requests

I followed the same pattern here as used in DRS, making POST methods for existing endpoints that now support including a passport array in the body. These mirror the existing GET endpoints that use a bearer token.

Making Requests

Here's an example showing a bearer token request vs. the Passport in body request:

#  Bearer Token (existing):
  curl -H "Authorization: Bearer <token>" https://wes.example.com/ga4gh/wes/v1/runs

#  GA4GH Passport (new):
  curl -X POST https://wes.example.com/ga4gh/wes/v1/runs/passport \
    -H "Content-Type: application/json" \
    -d '{"passport": ["<jwt1>", "<jwt2>"]}'

POST and /runs endpoint

This approach makes the /runs endpoint complicated. It feels overloaded since it's doing multiple things:

• if you do a GET /runs with a bearer token it's a list of runs
• if you do a POST /runs with a bearer token (or Passport in the body) and either application/json body schema of type PassportRunRequest or multipart/form-data you get a new workflow run
• if you do a POST /runs with a passport in the body using application/json body schema of type RunListRequest then you get a list of runs

So it might be more clear if we have a POST /runs/list endpoint instead... so /runs is less overloaded.

Sending Credentials

In addition to adding support for Passports in WES requests (as an alternative to bearer tokens), this PR includes syntax extensions to allow for providing credentials to access inputs and pass back credentials so callers can access outputs.

Data Inputs

Here's how a client would structure a workflow submission:

  {
    "workflow_type": "WDL",
    "workflow_type_version": "1.0",
    "workflow_url": "https://example.com/workflow.wdl",

    "credentials": {
      "tcga_passport": {
        "type": "passport",
        "passport": ["eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1N..."]
      },
      "s3_token": {
        "type": "bearer_token",
        "token": "aws-token-here..."
      },
      "api_access": {
        "type": "api_key",
        "key": "abc123",
        "header_name": "X-Custom-Key"
      }
    },

    "workflow_unified_params": {
      "version": "1.0",
      "parameters": {
        "input_bam": {
          "type": "File",
          "value": "drs://drs.example.org/abc123",
          "file_metadata": {
            "credential_id": "tcga_passport",
            "size": 1000000,
            "checksum": "sha256:abc123..."
          }
        },
        "reference_genome": {
          "type": "File",
          "value": "s3://bucket/reference.fa",
          "file_metadata": {
            "credential_id": "s3_token"
          }
        },
        "public_annotation": {
          "type": "File",
          "value": "https://public.example.com/annotation.bed"
          // No credential_id needed for public files
        }
      }
    }
  }

Data Outputs

Now GetRunLog responses can include credentials for accessing outputs:

{
    "run_id": "workflow-123",
    "state": "COMPLETE",

    "output_credentials": {
      "s3_output_access": {
        "type": "bearer_token",
        "token": "aws-s3-token-456..."
      },
      "restricted_results": {
        "type": "passport",
        "passport": ["eyJ0eXAiOiJKV1QiLCJhbGci..."]
      }
    },

    "structured_outputs": {
      "version": "1.0",
      "outputs": {
        "result_bam": {
          "class": "File",
          "location": "s3://results-bucket/output.bam",
          "credential_id": "s3_output_access",
          "size": 2000000,
          "checksum": "sha256:def456..."
        },
        "analysis_report": {
          "class": "File",
          "location": "drs://restricted.example.org/report123",
          "credential_id": "restricted_results",
          "format": "application/pdf"
        },
        "public_summary": {
          "class": "File",
          "location": "https://public.example.com/summary.txt"
          // No credential_id needed for public files
        }
      }
    }
  }

Security Considerations

• Credentials are only included in responses when the requesting client is authorized to
  access the outputs. We need to really think about how this works for Passports since typically these are issued through an OIDC flow and not just handed around. API keys and bearer tokens are a little easier since it would be possible to issue temporary/limited scoped tokens.
• Large passport JWTs are defined once and referenced multiple times to minimize request/response size... this should help with performance issues (less a security concern)
• Credential scope should be limited to specific file resources, not broad system access... helps to address the concern about passing around powerful tokens. If possible, Passports passed in for inputs access could be downscoped to just include visas relevant to the files in the inputs.
• It's unclear if WES servers should validate passport signatures and visa claims before using the passports for inputs access. I suspect not but something to discuss
• This implementation assumes you can pass your passport/token/api key to another system and have that system use it on your behalf. This would work fine in the DRS world except if there's an additional trust layer implemented (as some Driver Projects are discussing). In this case, the WES server might not be trusted by the DRS server and, even with having a correct Passport, might be denied access to files. This is a fluid topic, we don't have a clear trust mechanism but if trust mechanisms do get implemented (either in the specs or outside) it could affect the model presented in this PR of being able to pass around credentials.

Implementation Notes

• Passport Support: WES servers MAY implement passport authentication (not required)
• Credential Types: Servers SHOULD support at least bearer tokens, MAY support
  passport credentials
• Delegated Credentials: Servers MUST support DRS and HTTP URLs for delegated credentials
• Backward Compatibility: All existing authentication methods MUST continue to work. The current inputs/multipart/form-data and output approaches need to continue to work. So a WES 1.1 client should be able to work with a WES 1.2 server without modification.
• Error Handling: Servers should return appropriate HTTP status codes for credential
  failures.

[Copilot]

Pull Request Overview

This PR enhances the Workflow Execution Service (WES) OpenAPI specification to version 1.2.0, adding GA4GH Passport authentication support and data access credential management capabilities. The changes maintain backward compatibility while enabling workflow engines to securely access input data and provide credentials for output access.

• Adds POST endpoints with GA4GH Passport authentication support alongside existing bearer token authentication
• Introduces structured credential management for input and output data access via DRS and other services
• Implements unified parameter format for workflow language-agnostic parameter passing
• Expands workflow engine support beyond CWL/WDL to include Nextflow and Snakemake

Comments suppressed due to low confidence (1)

openapi/workflow_execution_service.openapi.yaml:7

• The updated logo URL path may not exist. The original path used 'ga4gh-theme' while the new path uses 'ga4gh/dist/assets/svg/logos/'. Verify this URL is accessible and correct.

    url: 'https://www.ga4gh.org/wp-content/themes/ga4gh/dist/assets/svg/logos/logo-full-color.svg'

[Copilot]

The 'x-in: body' extension in PassportAuth security scheme is non-standard OpenAPI. Standard security schemes don't support body-based authentication. This could cause issues with code generation tools and API documentation renderers.

Suggested change

	x-in: body
	bearerFormat: JWT
	description:
	A valid GA4GH Passport must be passed in the body of an HTTP POST request as a passports[] array.
	bearerFormat: JWT
	description:
	A valid GA4GH Passport must be included in the request body as a passports[] array for relevant endpoints.

[Copilot]

The workflow_params field description states it's 'OPTIONAL' but then says 'Either workflow_params or workflow_unified_params must be provided'. This creates confusion about whether the field is truly optional. Consider clarifying that one of the two parameter fields is required.

[Copilot]

There's inconsistent capitalization for 'Snakemake'. In line 13, it's written as 'Snakemake' but in line 1380 and elsewhere it appears as 'Snakemake'. The correct spelling is 'Snakemake' (not 'SnakeMake').

[Copilot]

The 'required' array for multipart/form-data schema only includes workflow_type, workflow_type_version, and workflow_url, but doesn't include the passport field when using passport authentication. This could allow requests without proper authentication to be considered valid.

Suggested change

	required:
	required:
	- passport

[uniqueg]

Thanks a lot for this effort, @briandoconnor!

But I need to be honest here: I always had a strong dislike for the POST Passport approach in DRS!

Apart from the security concerns which you already mentioned (sooner or later, someone will end up logging the request - or, in this case, also the response - bodies), I think reserving POST /objects for this use case was a bad idea (and embarrassed our perfectly fine, useful and semantically sound custom POST /objects endpoint). Please don't let it be a precedent!

I guess there are cases where breaking REST semantics (and implications following from that, e.g., with respect to caching and idempotency) is an acceptable tradeoff. But in the case of passing around Passports, it just seems like a bad design decision on top of a bad design decision to me:

Ever since the April Connect in Switzerland, it was abundantly discussed and, IIRC, predominantly agreed with, that a generic mechanism by which assertions/visas could be provided back to a clearing house upon presentation of a small Passport would constitute a cleaner design for Passports themselves, as well as for any APIs consuming them. Have sentiments changed since? If so, do you know what's the basis for the change of heart?

Besides, I think if credentials need to be passed around in the cloud, the state of the art seems to be the use of secret stores. In particular, HashiCorp Vault's API is open sourced and is a de factor industry standard. Wouldn't it be better to explore an approach like that, which could be reused for all such cases (data access, compute access, tool access credentials in DRS, TRS, WES, TES etc.), rather than complicating schemas and duplicating endpoints across a whole bunch of API products individually?

[vinjana]

Sorry for interfering, but I don't think it is a wise decision to adopt a "de facto industry standard" that is owned by a company for an open standard like the GA4GH standards are. There are numerous cases, where such companies arbitrarily changed their license (Akka, Anaconda, and you can probably find more) and suddenly users had to adapt. A standard should be grounded on a stronger foundation than the good will of a company. Sure, in both mentioned examples there were workarounds (Akka->Pekko, Anaconda->Mamba & Co.), but that does not mean they were trivial (the Anaconda licensing issue forced us to do lots of tests to ensure our workflows were backwards compatible). While such a change may be trivial for a standard -- just write "use the new fork of the old tool" -- the situation may be very different for the implementers and the operating institutions.

The Hashicorp Vault license says

You may make production use of the Licensed Work, provided Your use does not include offering the Licensed Work to third parties on a hosted or embedded basis in order to compete with HashiCorp's paid version(s) of the Licensed Work.

This sounds vague and arbitrary. If Hashicorp decides to make their money in a different way that overlaps with the use of their tool in GA4GH we have to change the standard. This is the current license. So, I interpret that as that even for the current vault version a normal fork, like with Akka->Pekko, would not even be a legally solid option. I'm not a lawyer, sure, so I could be completely wrong.

[uniqueg]

Sorry for interfering, but I don't think it is a wise decision to adopt a "de facto industry standard" that is owned by a company for an open standard like the GA4GH standards are. There are numerous cases, where such companies arbitrarily changed their license (Akka, Anaconda, and you can probably find more) and suddenly users had to adapt. A standard should be grounded on a stronger foundation than the good will of a company. Sure, in both mentioned examples there were workarounds (Akka->Pekko, Anaconda->Mamba & Co.), but that does not mean they were trivial (the Anaconda licensing issue forced us to do lots of tests to ensure our workflows were backwards compatible). While such a change may be trivial for a standard -- just write "use the new fork of the old tool" -- the situation may be very different for the implementers and the operating institutions.

The Hashicorp Vault license says

You may make production use of the Licensed Work, provided Your use does not include offering the Licensed Work to third parties on a hosted or embedded basis in order to compete with HashiCorp's paid version(s) of the Licensed Work.

This sounds vague and arbitrary. If Hashicorp decides to make their money in a different way that overlaps with the use of their tool in GA4GH we have to change the standard. This is the current license. So, I interpret that as that even for the current vault version a normal fork, like with Akka->Pekko, would not even be a legally solid option. I'm not a lawyer, sure, so I could be completely wrong.

You are not interfering, @vinjana - every voice is important!

I am not generally disagreeing with your important point, but I would like to clarify two points:

• I wasn't necessarily saying that the GA4GH should recommend a particular secret store API for sharing credentials, but rather that it's worth checking how similar use cases are handled by the wider community, especially given that passing secrets around in (multi-)cloud environments is not a problem that is particularly close to GA4GH's core focus (and expertise). What the outcome of such research and discussions will be, who knows? Maybe a document defining a generic process (e.g., "use a secret store") with examples, the recommendation of a specific API (such as HashiCorp's Vault API), a new API specification, or something entirely different (like this PR)...
• Unlike the HashiCorp's Vault implementation, the actual API is (still) released under a permissive license approved by the OSI, MSL-2.0, to be precise (see here). The same goes for much of the tooling that has been built around Vault, both by HashiCorp and others.

I would also like to note that there were previous discussions within the GA4GH community around potentially recommending existing standards in favor of drafting new ones, in order to avoid this situation. Personally, I think that makes a lot of sense, especially in areas that are not intrinsically related to life sciences, genomics, healthcare etc. AFAIK, no official decision has been made or policy drafted to define rules around recommending external standards, e.g., with respect to the licensing terms, governance, revising recommendations etc. In this regard, the Vault API may be as good an example as we might get (at least in the Cloud WS) as a driver for such a discussion. FWIW, I wholeheartedly agree with you that we should probably not recommend an API spec or other standard that is not published under an OSI-approved license.

[patmagee]

I can understand the challenge that this PR is trying to solve, however I fall in line with @uniqueg opinion here. I think breaking REST semantics is essentially a bad idea that will make the API harder to implement but also much less intuitive for the end user. IN particular I think making endpoints polymorphic makes writing clients in certain languages harder and also makes machine Interoperability harder IMO. If we really want to support POST then we may want to propose a totally separate RPC based api where these type of semantics are more accepted.

Additionally, If Passports are simply not usable within the context of a RESTful API service, it feels like we should be rethinking passports (similar to what @uniqueg mentioned as well), instead of propagating a bad design paradigm. Internally we use token-exchange extensively which dramatically reduces the size of tokens that we pass around. Additionally, I know clouds have essentially the same thing, they typically call them things like "Workload Identity Federation". These are not vender specific but largely are using token exchange under the hood

• GCP
• AWS
• Azure

These are not vendor specific solutions and I would highly urge us to evaluate standards that are being developed by the broader software industry as they become available. Passport was initially created before things like token-exchange were adopted, but maybe we should revisit (not to use necessarily, just for inspiration.

Fundamentally, this PR is trying to achieve granting the engine access to data from a federated source using a passport or some other access credential. I think that is a hard problem but maybe for a different reason.

Unlike TES, which is low level and can actually define how a single task interacts with underlying data (Ie it can write the command to use the credential), WES is a high level abstraction that in almost all cases sits in front of an engine. Any functionality we specify WES needs to fall into two categories of support

1. Something the engines are willing to adopt
2. Something the WES Server can do itself before handing off to the WES engine.

Data access is a hard one. I have done it in a similar way that is proposed here in a WES service. Essentially you end up needing to copy the data to a bucket from within your server or have some other workflow / batch job run. While this is technically doable (i did it), it was slow, error prone and caused data duplication. I could not read the data in-place once in the workflow.

On the flip side, we could not actually do this at the engine level because none of the engines supported this type of resolution and we are not an engine author ourselves. Additionally, most engines (at least for WDL) do not have a way of handling secrets, which basically means passing in sensitive information to the workflow itself should be avoided.

I don't think this means we cannot do it. BUT I do think we often are overlooking the key element here... the engine. A lot of WES requires the feature set provided by the engines

[patmagee]

I do not think we should be embedding output credentials in the outputs, ideally we should NEVER have to return credentials but at most use something like a signed URL.

There are a few concerns here

1. Much less secure and exposes access credentials. I think returning DRS ids and then having the caller call each individually is moderately better, but even that pattern is not ideal
2. The engine may not be able to generate output credentials
3. Output credentials can be difficult to produce or lock down scope. IE a passport, bearertoken, api key could allow the user to access much more then JUST the outputs of the current run. A lot of DRS apis returned signed URls but these would be expensive to produce for a large workflow

[patmagee]

The more I think about the problem of Passport credentials, the more I realize we are actually on a very troubling path that is taking us away from established security best practices and current trends in the software industry.

Putting aside the challenges of overloading endpoints and breaking restful semantics, using a POST body to send credentials is problematic in the general case

• The body of a post request is much more likely to be logged then the headers
• Easily exposed in error messages
• Monitoring tools (ie APM) may capture the values in the body
• Passes through multiple layers unencrypted: Once TLS is terminated the body is plaintext and the credentials are usable by anyone who wants to see them. Headers are also plain-text, so this just means that the attack surface has ballooned
• This now becomes easy to store in the DB accidentally and send back in the RunLog response

Additionally, the industry is largely going passwordless or putting a strong emphasis on platform provided credentials. There is a broad recognition that passing around ANY credential is typically a weak leek in the system so the best way to mitigate that is to simply not provide the raw credential to the user. Many clouds will now provide Credentials to their VM's without you having to make them yourselves, allowing the user to assign permissions to the VM's identity instead of passing in a particular credential. Additionally, most clouds have rolled out Workload Identity Federation allowing you to do OAuth2 token-exchange dramatically reducing the attack surface by only passing around a single credential that can be exchange by trusted client for the credentials needed to access other resources. Introducing a new approach where we are running the opposite direction to this feels like it has the potential for some serious mixups

I also think there is a general problem in how we are representing passport in general. The credentials that are being passed in are meant for OTHER services and are not meant for the WES service. This introduces a variety of challenges:

• Replay: Hopefully those credentials are short lived (although due to the nature of workflows, they likely are not), but they can be used by the current engine, or passed around without any difficulty. It imposes the same problem that
• No validation: The credentials are meant for other services but we are just passing them around and not validating that they were actually intended to be sent to the WES service. In the OAUTH world this would come from validating that 1) the token has the appropriate audience and 2) comes from a trusted issuer and 3) is cryptographically verifiable. There are other assertions (ie scope, or custom ones) that can be made
• No Trust: This is a decentralized system and that part I understand, however I do not think we should be advocating for obliterating trust entirely. I think the server (WES) should have SOME sort of relationship with each target OAUTH server (ideally as an OAuth client). This would open the door for the Passport tokens to be VERY short lived, targeted tokens that are specifically meant for the WES client which the WES client can then exchange with the targeted issuer for a longer lived credential. This approach would actually eliminate the problem of eavesdroppers stealing the credential if tro exchange it the WES client needed to use it's client secret.

I think we need a broader discussion including security experts about how we should implement passport. My Gut feeling as that the current trajectory is taking is farther and farther away from established best practices and I would strongly advocate for deviating too far from them.