From c4f4e3b892b7d6f3e897df75b40d9ad012a5e81b Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Tue, 29 Jul 2025 11:39:28 +0200 Subject: [PATCH 01/19] Changes generated by Claude:Key Changes Made: 1. Modified workflow_params description to indicate it's required unless workflow_unified_params is provided 2. Added workflow_unified_params field with the hybrid structure we discussed 3. Enhanced file metadata support with optional fields for size, checksum, secondary files, format, and modification time 4. Added comprehensive validation constraints for different parameter types 5. Validated the schema - no OpenAPI validation issues detected Key Features of the Implementation: - Version field for format evolution (default: 1.0) - Rich file metadata (size, checksum, secondary_files, format, last_modified) - Comprehensive validation constraints (min/max, length, pattern, enum, array limits) - Type-safe parameter definitions with clear enums - Backward compatibility - existing workflow_params still works - Precedence handling - workflow_unified_params takes precedence when provided --- .../workflow_execution_service.openapi.yaml | 91 ++++++++++++++++++- 1 file changed, 90 insertions(+), 1 deletion(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index 025b7d56..bfcdbf9c 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -742,9 +742,98 @@ components: workflow_params: type: object description: >- - REQUIRED + REQUIRED (unless workflow_unified_params provided) The workflow run parameterizations (JSON encoded), including input and output file locations + workflow_unified_params: + type: object + description: >- + OPTIONAL + + Unified parameter format that can be converted to workflow-language-specific format. + If provided, takes precedence over workflow_params. WES implementations should + convert these to the appropriate native format for the specified workflow_type. + properties: + version: + type: string + description: Version of the unified parameter format + default: "1.0" + parameters: + type: object + additionalProperties: + type: object + properties: + type: + type: string + enum: ["String", "Integer", "Float", "Boolean", "File", "Directory", "Array", "Object"] + description: Parameter type + value: + description: Parameter value (type depends on 'type' field) + element_type: + type: string + description: For Array types, specifies the type of array elements + file_metadata: + type: object + description: Optional metadata for File and Directory types + properties: + size: + type: integer + description: File size in bytes + format: int64 + checksum: + type: string + description: File checksum in format 'algorithm:hash' (e.g., 'sha256:abc123...') + secondary_files: + type: array + items: + type: string + description: Related files such as indexes or companion files + format: + type: string + description: File format or MIME type + last_modified: + type: string + description: Last modification time in ISO 8601 format + description: + type: string + description: Human-readable parameter description + optional: + type: boolean + default: false + description: Whether this parameter is optional + default: + description: Default value if parameter not provided (type depends on 'type' field) + constraints: + type: object + description: Validation constraints for the parameter value + properties: + min: + description: Minimum value (for numeric types) + max: + description: Maximum value (for numeric types) + min_length: + type: integer + description: Minimum length (for String types) + max_length: + type: integer + description: Maximum length (for String types) + pattern: + type: string + description: Regular expression pattern (for String types) + enum: + type: array + items: + description: Allowed value (type matches parameter type) + description: List of allowed values + min_items: + type: integer + description: Minimum number of items (for Array types) + max_items: + type: integer + description: Maximum number of items (for Array types) + required: + - type + - value workflow_type: type: string description: >- From 69408c1d463ec52a0000dd24b96eb16729250a70 Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Tue, 29 Jul 2025 14:11:36 +0200 Subject: [PATCH 02/19] Improvements from Claude Code to leverage RunRequest in POST Key Improvements Made: 1. Dual Content Type Support - application/json (Recommended): Uses the proper RunRequest model object - multipart/form-data (Legacy): Maintains backward compatibility for file uploads 2. Proper Model Usage - JSON requests now use $ref: '#/components/schemas/RunRequest' - Leverages all the rich typing and validation from the RunRequest schema - Supports both workflow_params and workflow_unified_params 3. Enhanced Documentation - Clear guidance on when to use each content type - Explains file handling differences between formats - Documents the new unified parameter format - Security considerations for file uploads 4. Better Developer Experience - OpenAPI tooling can generate proper client code for JSON requests - Type safety with structured objects instead of string parsing - Validation happens automatically with the model schema - Consistency across the API Usage Examples: Preferred JSON format: POST /runs Content-Type: application/json { "workflow_type": "CWL", "workflow_type_version": "v1.0", "workflow_url": "https://example.com/workflow.cwl", "workflow_unified_params": { "version": "1.0", "parameters": { "input_file": { "type": "File", "value": "gs://bucket/input.fastq", "file_metadata": { "size": 1073741824, "checksum": "sha256:abc123..." } } } } } Legacy multipart format (when file uploads needed): POST /runs Content-Type: multipart/form-data workflow_type: CWL workflow_unified_params: {"version":"1.0","parameters":{...}} workflow_attachment: [binary file data] --- .../workflow_execution_service.openapi.yaml | 52 +++++++++++++++---- 1 file changed, 41 insertions(+), 11 deletions(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index bfcdbf9c..dffe4acd 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -238,25 +238,40 @@ paths: description: >- This endpoint creates a new workflow run and returns a `RunId` to monitor its progress. - The `workflow_attachment` array may be used to upload files that are required to execute the workflow, including the primary workflow, tools imported by the workflow, other files referenced by the workflow, or files which are part of the input. The implementation should stage these files to a temporary directory and execute the workflow from there. These parts must have a Content-Disposition header with a "filename" provided for each part. Filenames may include subdirectories, but must not include references to parent directories with '..' -- implementations should guard against maliciously constructed filenames. + **Content Types Supported:** - The `workflow_url` is either an absolute URL to a workflow file that is accessible by the WES endpoint, or a relative URL corresponding to one of the files attached using `workflow_attachment`. + 1. **`application/json`** (Recommended): Submit a complete `RunRequest` object with structured data and type validation. - The `workflow_params` JSON object specifies input parameters, such as input files. The exact format of the JSON object depends on the conventions of the workflow language being used. Input files should either be absolute URLs, or relative URLs corresponding to files uploaded using `workflow_attachment`. The WES endpoint must understand and be able to access URLs supplied in the input. This is implementation specific. + 2. **`multipart/form-data`**: Legacy format for cases requiring file uploads. Fields are JSON-encoded strings that mirror the `RunRequest` structure. - The `workflow_type` is the type of workflow language and must be "CWL" or "WDL" currently (or another alternative supported by this WES instance). + **File Handling:** - The `workflow_type_version` is the version of the workflow language submitted and must be one supported by this WES instance. + - **JSON requests**: Files are referenced by URL or path. Use `workflow_unified_params` with File objects for rich metadata. - The `workflow_engine` is the engine that supports the workflow_type and must be supported by this WES instance. - - The `workflow_engine_version` is the version of workflow engine and must be supported by this WES instance. + - **Multipart requests**: The `workflow_attachment` array uploads files required for workflow execution. Files are staged to a temporary directory with Content-Disposition headers containing filenames. Subdirectories are allowed but parent directory references ('..') are prohibited for security. + + **Parameter Formats:** + + - **`workflow_params`**: Language-specific parameters (CWL, WDL, Nextflow, SnakeMake format) + + - **`workflow_unified_params`**: Universal parameter format that WES converts to the target workflow language + + **Required Fields:** - See the `RunRequest` documentation for details about other fields. + - `workflow_type`: Workflow language ("CWL", "WDL", "Nextflow", "SnakeMake", or other supported types) + + - `workflow_type_version`: Version of the workflow language supported by this WES instance + + - `workflow_url`: Absolute URL to workflow file, or relative path to uploaded attachment + + See the `RunRequest` schema documentation for complete field descriptions and validation rules. operationId: RunWorkflow parameters: [ ] requestBody: content: + application/json: + schema: + $ref: '#/components/schemas/RunRequest' multipart/form-data: encoding: { } schema: @@ -264,27 +279,42 @@ paths: properties: workflow_params: type: string + description: JSON-encoded workflow parameters (alternative to using RunRequest model) + workflow_unified_params: + type: string + description: JSON-encoded unified workflow parameters (alternative to using RunRequest model) workflow_type: type: string + description: Workflow descriptor type (CWL, WDL, etc.) workflow_type_version: type: string + description: Version of the workflow descriptor type tags: type: string + description: JSON-encoded key-value tags for the workflow run workflow_engine: type: string + description: Workflow engine to use workflow_engine_version: type: string + description: Version of the workflow engine workflow_engine_parameters: type: string + description: JSON-encoded engine-specific parameters workflow_url: type: string + description: URL to the workflow descriptor file workflow_attachment: type: array items: type: string format: binary - description: '' - required: false + description: Files to be staged for workflow execution + required: + - workflow_type + - workflow_type_version + - workflow_url + required: true responses: 200: description: '' From ba700b1e3ba6b41f98b01ebe539840ace5f7fb57 Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Tue, 29 Jul 2025 15:04:51 +0200 Subject: [PATCH 03/19] working on formating of markdown --- openapi/workflow_execution_service.openapi.yaml | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index dffe4acd..872e6c3f 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -14,10 +14,10 @@ info: Key features of the API: - + can request that a workflow be run - + can pass parameters to that workflow (e.g. input files, cmdline arguments) - + can get information about running workflows (e.g. status, errors, output file locations) - + can cancel a running workflow + - can request that a workflow be run + - can pass parameters to that workflow (e.g. input files, cmdline arguments) + - can get information about running workflows (e.g. status, errors, output file locations) + - can cancel a running workflow servers: - url: https://{defaultHost}/{basePath}/{apiVersion} variables: @@ -38,8 +38,8 @@ tags: Use cases include: - + "Bring your code to the data": a researcher who has built their own custom analysis can submit it to run on a dataset owned by an external organization, instead of having to make a copy of the data - + Best-practices pipelines: a researcher who maintains their own controlled data environment can find useful workflows in a shared directory (e.g. Dockstore.org), and run them over their data + - "Bring your code to the data": a researcher who has built their own custom analysis can submit it to run on a dataset owned by an external organization, instead of having to make a copy of the data + - Best-practices pipelines: a researcher who maintains their own controlled data environment can find useful workflows in a shared directory (e.g. Dockstore.org), and run them over their data ## Standards @@ -235,7 +235,7 @@ paths: tags: - Workflow Runs summary: RunWorkflow - description: >- + description: | This endpoint creates a new workflow run and returns a `RunId` to monitor its progress. **Content Types Supported:** From 079ee33a82f94b15f4c5cfa7bc9391bbb6cf7120 Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Tue, 29 Jul 2025 16:06:06 +0200 Subject: [PATCH 04/19] working on cleaning up the documentation --- .../workflow_execution_service.openapi.yaml | 50 +++++++++---------- 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index 872e6c3f..f2c0a858 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -5,7 +5,7 @@ info: version: '1.1.0' x-logo: url: 'https://www.ga4gh.org/wp-content/themes/ga4gh-theme/gfx/GA-logo-horizontal-tag-RGB.svg' - description: > + description: | *Run standard workflows on workflow execution platforms in a platform-agnostic way.* ## Executive Summary @@ -15,14 +15,14 @@ info: Key features of the API: - can request that a workflow be run - - can pass parameters to that workflow (e.g. input files, cmdline arguments) + - can pass parameters to that workflow (e.g. input files, cmdline arguments) both workflow-type specific as well as a generalized parameter format - can get information about running workflows (e.g. status, errors, output file locations) - can cancel a running workflow servers: - url: https://{defaultHost}/{basePath}/{apiVersion} variables: defaultHost: - default: www.example.com + default: wes.example.com basePath: default: ga4gh/wes apiVersion: @@ -45,12 +45,12 @@ tags: The WES API specification is written in OpenAPI and embodies a RESTful service philosophy. It uses JSON in requests and responses and standard HTTP/HTTPS for information transport. - name: Authorization & Authentication - description: > + description: | Users must supply credentials that establish their identity and authorization in order to use a WES endpoint. We recommend that WES implementations use an OAuth2 [**bearer token**](https://oauth.net/2/bearer-tokens/), although they can choose other mechanisms if appropriate. WES callers can use the `auth_instructions_url` from the [**`service-info` endpoint**](https://ga4gh.github.io/workflow-execution-service-schemas/#/WorkflowExecutionService/GetServiceInfo) to learn how to obtain and use a bearer token for a particular implementation. The WES implementation is responsible for checking that a user is authorized to submit workflow run requests. The particular authorization policy is up to the WES implementer. - Systems like WES need to also address the ability to pass credentials with jobs for input and output access. In the current version of WES, the passing of credentials to authenticate and authorize access to inputs and outputs, as well as mandates about necessary file transfer protocols to support, are out of scope. However, parallel work on the Data Object Service is addressing ways to pass around access credentials with data object references, opening up the possibility that a future version of WES will provide concrete mechanisms for workflow runs to access data using credentials different than those used for WES. This is a work in progress and support of DOS in WES will be added in a future release of WES. + Systems like WES need to also address the ability to pass credentials with jobs for input and output access. In the current version of WES, the passing of credentials to authenticate and authorize access to inputs and outputs, as well as mandates about necessary file transfer protocols to support, are out of scope. However, parallel work on the GA4GH Data Repository Service is addressing ways to pass around access credentials with data object references, opening up the possibility that a future version of WES will provide concrete mechanisms for workflow runs to access data using credentials different than those used for WES. This is a work in progress and support of DRS in WES will be added in a future release of WES. - name: serviceinfo_model x-displayName: ServiceInfo description: | @@ -240,31 +240,31 @@ paths: **Content Types Supported:** - 1. **`application/json`** (Recommended): Submit a complete `RunRequest` object with structured data and type validation. + 1. **`application/json`** (Recommended): Submit a complete `RunRequest` object with structured data and type validation. The advantage of this approach is full schema validation and OpenAPI tooling support along with workflow_unified_params object support (see below for more info). Disadvantage is that it does not support multiple workflow file uploads directly, which may be required for multi-file workflows. - 2. **`multipart/form-data`**: Legacy format for cases requiring file uploads. Fields are JSON-encoded strings that mirror the `RunRequest` structure. + 2. **`multipart/form-data`**: Submit for cases requiring workflow file uploads. Fields are JSON-encoded strings that mirror the `RunRequest` structure. Advantage is you can upload one or more workflow files directly while the disadvantage is you need to be careful in formating the string encoded arguments correctly (multipart form submissions only support strings hence the limitation). **File Handling:** - - **JSON requests**: Files are referenced by URL or path. Use `workflow_unified_params` with File objects for rich metadata. + - **JSON requests**: Workflow file is referenced by URL or path. Use `workflow_unified_params` with File objects for rich metadata and generic parameterization. - **Multipart requests**: The `workflow_attachment` array uploads files required for workflow execution. Files are staged to a temporary directory with Content-Disposition headers containing filenames. Subdirectories are allowed but parent directory references ('..') are prohibited for security. **Parameter Formats:** - - **`workflow_params`**: Language-specific parameters (CWL, WDL, Nextflow, SnakeMake format) + - **`workflow_params`**: Language-specific parameters (CWL, WDL, Nextflow, SnakeMake format). This field is used to send the native parameterization file used by the workflow engine. - - **`workflow_unified_params`**: Universal parameter format that WES converts to the target workflow language + - **`workflow_unified_params`**: Universal parameter format that WES converts to the target workflow language. This allows for a more generic way to pass parameters that can be converted to the appropriate native format for the specified workflow_type. This is an optional field, but if provided, it takes precedence over `workflow_params`. **Required Fields:** - - `workflow_type`: Workflow language ("CWL", "WDL", "Nextflow", "SnakeMake", or other supported types) + - `workflow_type`: Workflow language ("CWL", "WDL", "Nextflow", "SnakeMake", or another alternative supported by this WES instance, see service-info) - - `workflow_type_version`: Version of the workflow language supported by this WES instance + - `workflow_type_version`: The workflow descriptor type version, must be one supported by this WES instance - `workflow_url`: Absolute URL to workflow file, or relative path to uploaded attachment - See the `RunRequest` schema documentation for complete field descriptions and validation rules. + See the `RunRequest` schema documentation for complete field descriptions and validation rules when encoding as strings in the multipart/form-data request. operationId: RunWorkflow parameters: [ ] requestBody: @@ -279,31 +279,31 @@ paths: properties: workflow_params: type: string - description: JSON-encoded workflow parameters (alternative to using RunRequest model) + description: JSON-encoded workflow parameters (see RunRequest for how to encode) workflow_unified_params: type: string - description: JSON-encoded unified workflow parameters (alternative to using RunRequest model) + description: JSON-encoded unified workflow parameters (see RunRequest for how to encode) workflow_type: type: string - description: Workflow descriptor type (CWL, WDL, etc.) + description: Workflow descriptor type (CWL, WDL, Nextflow, SnakeMake, etc. see service-info for supported types) workflow_type_version: type: string - description: Version of the workflow descriptor type + description: The workflow descriptor type version, must be one supported by this WES instance tags: type: string - description: JSON-encoded key-value tags for the workflow run + description: JSON-encoded key-value tags for the workflow run (see RunRequest for how to encode) workflow_engine: type: string - description: Workflow engine to use + description: The workflow engine, must be one supported by this WES instance. Required if workflow_engine_version is provided. workflow_engine_version: type: string - description: Version of the workflow engine + description: The workflow engine version, must be one supported by this WES instance. If workflow_engine is provided, but workflow_engine_version is not, servers can make no assumptions with regard to the engine version the WES instance uses to process the request if that WES instance supports multiple versions of the requested engine. workflow_engine_parameters: type: string - description: JSON-encoded engine-specific parameters + description: JSON-encoded engine-specific parameters (see RunRequest for how to encode) workflow_url: type: string - description: URL to the workflow descriptor file + description: The workflow document. When workflow_attachment is used to attach files, the workflow_url may be a relative path to one of the attachments. workflow_attachment: type: array items: @@ -701,7 +701,7 @@ components: - CANCELING - PREEMPTED type: string - description: > + description: | State can take any of the following values: + UNKNOWN: The state of the task is unknown. This provides a safe default for messages where this field is missing, for example, so that a missing field does not accidentally imply that the state is QUEUED. @@ -869,7 +869,7 @@ components: description: >- REQUIRED - The workflow descriptor type, must be "CWL" or "WDL" currently (or another alternative supported by this WES instance) + The workflow descriptor type, must be "CWL", "WDL", "Nextflow", or "SnakeMake" currently (or another alternative supported by this WES instance) workflow_type_version: type: string description: >- @@ -899,7 +899,7 @@ components: description: >- REQUIRED - The workflow CWL or WDL document. When `workflow_attachments` is used to attach files, the `workflow_url` may be a relative path to one of the attachments. + The workflow document. When `workflow_attachments` is used to attach files, the `workflow_url` may be a relative path to one of the attachments. description: >- To execute a workflow, send a run request including all the details needed to begin downloading From 3b45af110e58d0f4fcdb3b2186e3f3eaa21fc97d Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Tue, 29 Jul 2025 16:10:20 +0200 Subject: [PATCH 05/19] working on docs --- openapi/workflow_execution_service.openapi.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index f2c0a858..ace25605 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -2,7 +2,7 @@ openapi: 3.0.0 info: title: Workflow Execution Service contact: {} - version: '1.1.0' + version: '1.2.0' x-logo: url: 'https://www.ga4gh.org/wp-content/themes/ga4gh-theme/gfx/GA-logo-horizontal-tag-RGB.svg' description: | @@ -10,7 +10,7 @@ info: ## Executive Summary - The Workflow Execution Service (WES) API provides a standard way for users to submit workflow requests to workflow execution systems, and to monitor their execution. This API lets users run a single workflow (currently [**CWL**](https://www.commonwl.org/) or [**WDL**](http://www.openwdl.org/) formatted workflows, other types may be supported in the future) on multiple different platforms, clouds, and environments. + The Workflow Execution Service (WES) API provides a standard way for users to submit workflow requests to workflow execution systems, and to monitor their execution. This API lets users run a single workflow (currently [CWL](https://www.commonwl.org/), [WDL](http://www.openwdl.org/), [Nextflow](https://www.nextflow.io/), and [SnakeMake](https://snakemake.readthedocs.io/en/stable/) formatted workflows, other types may be supported in the future) on multiple different platforms, clouds, and environments. Key features of the API: From ad8ed26739730bde1d1655e056f533e61f528aa8 Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Tue, 29 Jul 2025 16:19:18 +0200 Subject: [PATCH 06/19] cleanup of docs --- openapi/workflow_execution_service.openapi.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index ace25605..234c7ea9 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -254,7 +254,7 @@ paths: - **`workflow_params`**: Language-specific parameters (CWL, WDL, Nextflow, SnakeMake format). This field is used to send the native parameterization file used by the workflow engine. - - **`workflow_unified_params`**: Universal parameter format that WES converts to the target workflow language. This allows for a more generic way to pass parameters that can be converted to the appropriate native format for the specified workflow_type. This is an optional field, but if provided, it takes precedence over `workflow_params`. + - **`workflow_unified_params`**: Universal parameter format that WES server implementation converts to the target workflow language. This allows for a more generic way to pass parameters that can be converted to the appropriate native format for the specified workflow_type. This is an optional field, but if provided, it takes precedence over `workflow_params`. **Required Fields:** From 913d425067e9f92930f4933b70fe41266086d781 Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Tue, 29 Jul 2025 16:48:49 +0200 Subject: [PATCH 07/19] Changes Made: 1. Updated RunLog schema - Added structured_outputs field alongside the existing outputs 2. Added WorkflowOutputs schema - Main container for structured outputs with version and metadata 3. Added OutputObject schema - Flexible output type supporting Files, Directories, Arrays, and primitives 4. Added documentation tags - Both schemas appear in the Models section of the API docs Key Features Implemented: WorkflowOutputs Schema: - Version field for format evolution - Named outputs with rich metadata - Workflow-level metadata (execution ID, timing, resource usage) - Provenance tracking (engine, version, status) OutputObject Schema: - Type system - File, Directory, Array, String, Integer, Float, Boolean - File metadata - location, size, checksum, format, basename - Provenance - source task, command, creation time - Secondary files - Associated files like indexes - Array support - Collections of outputs - Content embedding - Small file contents can be included Backward Compatibility: - Existing outputs field remains unchanged (marked as "legacy format") - structured_outputs is optional - implementations can provide either or both - No breaking changes to existing API consumers --- .../workflow_execution_service.openapi.yaml | 129 +++++++++++++++++- 1 file changed, 126 insertions(+), 3 deletions(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index 234c7ea9..3e5b4e14 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -107,6 +107,14 @@ tags: x-displayName: WorkflowEngineVersion description: | + - name: workflowoutputs_model + x-displayName: WorkflowOutputs + description: | + + - name: outputobject_model + x-displayName: OutputObject + description: | + x-tagGroups: - name: Overview tags: @@ -133,6 +141,8 @@ x-tagGroups: - defaultworkflowengineparameter_model - workflowtypeversion_model - workflowengineversion_model + - workflowoutputs_model + - outputobject_model paths: /service-info: get: @@ -939,7 +949,10 @@ components: specification (2.0.0) outputs: type: object - description: The outputs from the workflow run. + description: The outputs from the workflow run (legacy format) + structured_outputs: + $ref: '#/components/schemas/WorkflowOutputs' + description: Structured workflow outputs with rich metadata and type safety Log: title: Log type: object @@ -1094,8 +1107,118 @@ components: description: The integer representing the HTTP status code (e.g. 200, 404). format: int32 description: An object that can optionally include information about the error. - - + WorkflowOutputs: + title: WorkflowOutputs + type: object + description: Structured workflow outputs with rich metadata and provenance information + properties: + version: + type: string + description: Version of the structured outputs format + default: "1.0" + outputs: + type: object + description: Named workflow outputs with structured metadata + additionalProperties: + $ref: '#/components/schemas/OutputObject' + workflow_metadata: + type: object + description: Overall workflow execution metadata + properties: + execution_id: + type: string + description: Unique identifier for this workflow execution + status: + $ref: '#/components/schemas/State' + start_time: + type: string + description: Workflow start time in ISO 8601 format + end_time: + type: string + description: Workflow completion time in ISO 8601 format + engine: + type: string + description: Workflow engine used (cromwell, toil, cwltool, nextflow, snakemake) + engine_version: + type: string + description: Version of the workflow engine + resource_usage: + type: object + description: Resource consumption statistics + properties: + cpu_hours: + type: number + description: Total CPU hours consumed + memory_gb_hours: + type: number + description: Total memory GB-hours consumed + disk_gb: + type: number + description: Total disk space used in GB + OutputObject: + title: OutputObject + type: object + description: A structured output object with metadata and provenance information + properties: + class: + type: string + enum: ["File", "Directory", "Array", "String", "Integer", "Float", "Boolean"] + description: Type of the output object + value: + description: Output value (type depends on class field) + location: + type: string + description: Absolute path or URL to the output file/directory (for File/Directory class) + basename: + type: string + description: Filename without directory path (for File/Directory class) + size: + type: integer + format: int64 + description: Size in bytes (for File/Directory class) + checksum: + type: string + description: File integrity hash in format 'algorithm:hash' (e.g., 'sha256:abc123...') + format: + type: string + description: MIME type or format identifier (e.g., EDAM ontology reference) + items: + type: array + items: + $ref: '#/components/schemas/OutputObject' + description: Array elements (for Array class) + metadata: + type: object + description: Additional metadata about the output + properties: + created: + type: string + description: Creation timestamp in ISO 8601 format + source_task: + type: string + description: Task/step that generated this output + task_id: + type: string + description: Unique identifier of the generating task + command: + type: array + items: + type: string + description: Command line that generated this output + description: + type: string + description: Human-readable description of the output + secondary_files: + type: array + items: + $ref: '#/components/schemas/OutputObject' + description: Associated files (e.g., index files, companion files) + contents: + type: string + description: File contents for small files (≤64 KiB) + required: + - class + - value From 96612f8128c0a17f2852a8954d023eaa1e6754ec Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Wed, 30 Jul 2025 11:41:45 +0200 Subject: [PATCH 08/19] adding additional_workflow_urls which allows you to send multiple secondary workflow URLs beyond the primary workflow --- .../workflow_execution_service.openapi.yaml | 29 ++++++++++++++++--- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index 3e5b4e14..50d9950f 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -250,15 +250,15 @@ paths: **Content Types Supported:** - 1. **`application/json`** (Recommended): Submit a complete `RunRequest` object with structured data and type validation. The advantage of this approach is full schema validation and OpenAPI tooling support along with workflow_unified_params object support (see below for more info). Disadvantage is that it does not support multiple workflow file uploads directly, which may be required for multi-file workflows. + 1. **`application/json`** (Recommended for single workflows submitted by URL): Submit a complete `RunRequest` object with structured data and type validation. The advantage of this approach is the schema is explicitly defined by OpenAPI syntax (vs. just strings in multipart-form) along with workflow_unified_params object support (see below for more info). Disadvantage is that it does not support workflow file uploads directly, which may be required for multi-file workflows. - 2. **`multipart/form-data`**: Submit for cases requiring workflow file uploads. Fields are JSON-encoded strings that mirror the `RunRequest` structure. Advantage is you can upload one or more workflow files directly while the disadvantage is you need to be careful in formating the string encoded arguments correctly (multipart form submissions only support strings hence the limitation). + 2. **`multipart/form-data`**: Submit for cases requiring workflow file uploads (rather than just a URL). Fields are JSON-encoded strings that mirror the `RunRequest` structure. Advantage is you can upload one or more workflow files directly while the disadvantage is you need to be careful in formating the JSON-ended string arguments correctly (multipart form submissions only support strings hence the limitation). **File Handling:** - **JSON requests**: Workflow file is referenced by URL or path. Use `workflow_unified_params` with File objects for rich metadata and generic parameterization. - - **Multipart requests**: The `workflow_attachment` array uploads files required for workflow execution. Files are staged to a temporary directory with Content-Disposition headers containing filenames. Subdirectories are allowed but parent directory references ('..') are prohibited for security. + - **Multipart requests**: The `workflow_attachment` array uploads files required for workflow execution. Files are staged to a temporary directory with Content-Disposition headers containing filenames. Subdirectories are allowed but parent directory references ('..') are prohibited for security purposes. **Parameter Formats:** @@ -314,12 +314,15 @@ paths: workflow_url: type: string description: The workflow document. When workflow_attachment is used to attach files, the workflow_url may be a relative path to one of the attachments. + additional_workflow_urls: + type: string + description: JSON-encoded array of additional workflow URLs (see RunRequest for how to encode) workflow_attachment: type: array items: type: string format: binary - description: Files to be staged for workflow execution + description: Files to be staged for workflow execution. You set the filename/path using the Content-Disposition header in the multipart form submission. For example 'Content-Disposition: form-data; name="workflow_attachment"; filename="workflows/helper.wdl"'. The files are staged to a temporary directory and can be referenced by relative path in the workflow_url. required: - workflow_type - workflow_type_version @@ -910,6 +913,24 @@ components: REQUIRED The workflow document. When `workflow_attachments` is used to attach files, the `workflow_url` may be a relative path to one of the attachments. + additional_workflow_urls: + type: array + items: + type: object + properties: + url: + type: string + description: URL to the workflow file (absolute URL or relative to uploaded attachments) + filename: + type: string + description: Target filename, possibly including relative path, where this file should be placed e.g. 'workflows/helper.wdl'. Subdirectories are allowed but parent directory references ('..') are prohibited for security purposes. + required: + - url + - filename + description: >- + OPTIONAL + + Additional workflow files required for execution (e.g., imported workflows, subworkflows, or dependencies). Each entry specifies both the source URL and target file path for precise file placement. description: >- To execute a workflow, send a run request including all the details needed to begin downloading From 4e7d6034d3d11753f68528aed598b7bce67230e6 Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Wed, 30 Jul 2025 11:47:42 +0200 Subject: [PATCH 09/19] updated the logo --- openapi/workflow_execution_service.openapi.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index 50d9950f..26acc486 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -4,7 +4,7 @@ info: contact: {} version: '1.2.0' x-logo: - url: 'https://www.ga4gh.org/wp-content/themes/ga4gh-theme/gfx/GA-logo-horizontal-tag-RGB.svg' + url: 'https://www.ga4gh.org/wp-content/themes/ga4gh/dist/assets/svg/logos/logo-full-color.svg' description: | *Run standard workflows on workflow execution platforms in a platform-agnostic way.* @@ -262,7 +262,7 @@ paths: **Parameter Formats:** - - **`workflow_params`**: Language-specific parameters (CWL, WDL, Nextflow, SnakeMake format). This field is used to send the native parameterization file used by the workflow engine. + - **`workflow_params`**: Language-specific parameters (CWL, WDL, Nextflow, SnakeMake format). This field is used to send the native parameterization file (encoded as a string) used by the workflow engine. - **`workflow_unified_params`**: Universal parameter format that WES server implementation converts to the target workflow language. This allows for a more generic way to pass parameters that can be converted to the appropriate native format for the specified workflow_type. This is an optional field, but if provided, it takes precedence over `workflow_params`. From fd09f4513ff9298d9411d14ed0be8be20b9e57fc Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Wed, 30 Jul 2025 11:54:36 +0200 Subject: [PATCH 10/19] updated the logo --- openapi/workflow_execution_service.openapi.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index 26acc486..1b14ce42 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -10,14 +10,15 @@ info: ## Executive Summary - The Workflow Execution Service (WES) API provides a standard way for users to submit workflow requests to workflow execution systems, and to monitor their execution. This API lets users run a single workflow (currently [CWL](https://www.commonwl.org/), [WDL](http://www.openwdl.org/), [Nextflow](https://www.nextflow.io/), and [SnakeMake](https://snakemake.readthedocs.io/en/stable/) formatted workflows, other types may be supported in the future) on multiple different platforms, clouds, and environments. + The Workflow Execution Service (WES) API provides a standard way for users to submit workflow requests to workflow execution systems, and to monitor their execution. This API lets users run a single workflow (commonly [CWL](https://www.commonwl.org/), [WDL](http://www.openwdl.org/), [Nextflow](https://www.nextflow.io/), and [SnakeMake](https://snakemake.readthedocs.io/en/stable/) formatted workflows, WES servers describe what they support via service-info) on multiple different platforms, clouds, and environments. Key features of the API: - can request that a workflow be run - - can pass parameters to that workflow (e.g. input files, cmdline arguments) both workflow-type specific as well as a generalized parameter format + - can pass parameters to that workflow (e.g. input files, cmdline arguments) via both workflow-type specific parameter formats as well as a generalized parameter JSON format - can get information about running workflows (e.g. status, errors, output file locations) - can cancel a running workflow + - can retrieve logs and outputs file locations from a workflow run servers: - url: https://{defaultHost}/{basePath}/{apiVersion} variables: From 3ccc4d5dc6fd26f264851d6e944c1c127537fac4 Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Wed, 30 Jul 2025 12:37:13 +0200 Subject: [PATCH 11/19] updates to documentation for clarity --- .../workflow_execution_service.openapi.yaml | 52 ++++++++++++------- 1 file changed, 33 insertions(+), 19 deletions(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index 1b14ce42..9788e537 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -10,7 +10,7 @@ info: ## Executive Summary - The Workflow Execution Service (WES) API provides a standard way for users to submit workflow requests to workflow execution systems, and to monitor their execution. This API lets users run a single workflow (commonly [CWL](https://www.commonwl.org/), [WDL](http://www.openwdl.org/), [Nextflow](https://www.nextflow.io/), and [SnakeMake](https://snakemake.readthedocs.io/en/stable/) formatted workflows, WES servers describe what they support via service-info) on multiple different platforms, clouds, and environments. + The Workflow Execution Service (WES) API provides a standard way for users to submit workflow requests to workflow execution systems, and to monitor their execution. This API lets users run a single workflow (commonly [CWL](https://www.commonwl.org/), [WDL](http://www.openwdl.org/), [Nextflow](https://www.nextflow.io/), and [Snakemake](https://Snakemake.readthedocs.io/en/stable/) formatted workflows, WES servers describe what they support via service-info) on multiple different platforms, clouds, and environments. Key features of the API: @@ -35,23 +35,23 @@ tags: description: Submit and monitor workflows in a WES environment - name: Introduction description: | - This document describes the WES API and provides details on the specific endpoints, request formats, and response. It is intended to provide key information for developers of WES-compatible services as well as clients that will call these WES services. + This document describes the WES API and provides details on the specific endpoints, request formats, and responses. It is intended to provide key information for developers of WES-compatible services as well as clients that will call these WES services. Use cases include: - - "Bring your code to the data": a researcher who has built their own custom analysis can submit it to run on a dataset owned by an external organization, instead of having to make a copy of the data - - Best-practices pipelines: a researcher who maintains their own controlled data environment can find useful workflows in a shared directory (e.g. Dockstore.org), and run them over their data - + - "Bring your code to the data": a researcher who has built their own custom analysis workflow can submit it to run on a dataset owned by an external organization, instead of having to download a copy of the data for local analysis. + - Best-practices pipelines: a researcher who maintains their own controlled data environment can find useful workflows in a shared repository (e.g. [Dockstore.org](https://dockstore.org)), and run them on their data + ## Standards - The WES API specification is written in OpenAPI and embodies a RESTful service philosophy. It uses JSON in requests and responses and standard HTTP/HTTPS for information transport. + The WES API specification is written in OpenAPI and embodies a RESTful service philosophy. It uses JSON in requests and responses (where applicable) and standard HTTP/HTTPS for information transport. - name: Authorization & Authentication description: | Users must supply credentials that establish their identity and authorization in order to use a WES endpoint. We recommend that WES implementations use an OAuth2 [**bearer token**](https://oauth.net/2/bearer-tokens/), although they can choose other mechanisms if appropriate. WES callers can use the `auth_instructions_url` from the [**`service-info` endpoint**](https://ga4gh.github.io/workflow-execution-service-schemas/#/WorkflowExecutionService/GetServiceInfo) to learn how to obtain and use a bearer token for a particular implementation. The WES implementation is responsible for checking that a user is authorized to submit workflow run requests. The particular authorization policy is up to the WES implementer. - Systems like WES need to also address the ability to pass credentials with jobs for input and output access. In the current version of WES, the passing of credentials to authenticate and authorize access to inputs and outputs, as well as mandates about necessary file transfer protocols to support, are out of scope. However, parallel work on the GA4GH Data Repository Service is addressing ways to pass around access credentials with data object references, opening up the possibility that a future version of WES will provide concrete mechanisms for workflow runs to access data using credentials different than those used for WES. This is a work in progress and support of DRS in WES will be added in a future release of WES. + Systems like WES need to also address the ability to pass credentials with jobs for input and output access. In the current version of WES, the passing of credentials to authenticate and authorize access to inputs and outputs, as well as mandates about necessary file transfer protocols to support, are out of scope. However, parallel work on the [GA4GH Data Repository Service](https://ga4gh.github.io/data-repository-service-schemas/) is addressing ways to pass around access credentials with data object references, opening up the possibility that a future version of WES will provide concrete mechanisms for workflow runs to access data using credentials different than those used for WES calls. This is a work in progress and support of DRS in WES will be added in a future release of WES. - name: serviceinfo_model x-displayName: ServiceInfo description: | @@ -150,7 +150,7 @@ paths: tags: - Service Info summary: GetServiceInfo - description: May include information related (but not limited to) the workflow descriptor formats, versions supported, the WES API versions supported, and information about general service availability. + description: Includes information related, but not limited to, the workflow descriptor formats, workflow engines, versions supported, the WES API versions supported, and information about general service availability and access. operationId: GetServiceInfo parameters: [ ] responses: @@ -251,25 +251,25 @@ paths: **Content Types Supported:** - 1. **`application/json`** (Recommended for single workflows submitted by URL): Submit a complete `RunRequest` object with structured data and type validation. The advantage of this approach is the schema is explicitly defined by OpenAPI syntax (vs. just strings in multipart-form) along with workflow_unified_params object support (see below for more info). Disadvantage is that it does not support workflow file uploads directly, which may be required for multi-file workflows. + 1. **`application/json`** (Recommended for single workflows submitted by URL): Submit a complete `RunRequest` object with structured data and type validation. The advantage of this approach is the schema is explicitly defined by OpenAPI syntax (vs. just strings in multipart-form) along with workflow_unified_params object support (see below for more info). Disadvantage is that it does not support workflow file uploads directly, which may be required for multi-file workflows not available as URLs. 2. **`multipart/form-data`**: Submit for cases requiring workflow file uploads (rather than just a URL). Fields are JSON-encoded strings that mirror the `RunRequest` structure. Advantage is you can upload one or more workflow files directly while the disadvantage is you need to be careful in formating the JSON-ended string arguments correctly (multipart form submissions only support strings hence the limitation). **File Handling:** - - **JSON requests**: Workflow file is referenced by URL or path. Use `workflow_unified_params` with File objects for rich metadata and generic parameterization. + - **`application/json` requests**: Workflow file is referenced by URL or path. Use `workflow_unified_params` with File objects for rich metadata and generic parameterization. - - **Multipart requests**: The `workflow_attachment` array uploads files required for workflow execution. Files are staged to a temporary directory with Content-Disposition headers containing filenames. Subdirectories are allowed but parent directory references ('..') are prohibited for security purposes. + - **`multipart/form-data` requests**: The `workflow_attachment` array uploads files required for workflow execution. Files are staged to a temporary directory with Content-Disposition headers containing filenames. Subdirectories are allowed but parent directory references ('..') are prohibited for security purposes. **Parameter Formats:** - - **`workflow_params`**: Language-specific parameters (CWL, WDL, Nextflow, SnakeMake format). This field is used to send the native parameterization file (encoded as a string) used by the workflow engine. + - **`workflow_params`**: Language-specific parameters (CWL, WDL, Nextflow, Snakemake format). This field is used to send the native parameterization file (encoded as a string) used by the workflow engine. - **`workflow_unified_params`**: Universal parameter format that WES server implementation converts to the target workflow language. This allows for a more generic way to pass parameters that can be converted to the appropriate native format for the specified workflow_type. This is an optional field, but if provided, it takes precedence over `workflow_params`. **Required Fields:** - - `workflow_type`: Workflow language ("CWL", "WDL", "Nextflow", "SnakeMake", or another alternative supported by this WES instance, see service-info) + - `workflow_type`: Workflow language ("CWL", "WDL", "Nextflow", "Snakemake", or another alternative supported by this WES instance, see service-info) - `workflow_type_version`: The workflow descriptor type version, must be one supported by this WES instance @@ -296,7 +296,7 @@ paths: description: JSON-encoded unified workflow parameters (see RunRequest for how to encode) workflow_type: type: string - description: Workflow descriptor type (CWL, WDL, Nextflow, SnakeMake, etc. see service-info for supported types) + description: Workflow descriptor type must be "CWL", "WDL", "Nextflow", or "Snakemake" currently (or another alternative supported by this WES instance, see service-info) workflow_type_version: type: string description: The workflow descriptor type version, must be one supported by this WES instance @@ -662,6 +662,10 @@ components: items: type: string description: The filesystem protocols supported by this service, currently these may include common protocols using the terms 'http', 'https', 'sftp', 's3', 'gs', 'file', or 'synapse', but others are possible and the terms beyond these core protocols are currently not fixed. This section reports those protocols (either common or not) supported by this WES service. + workflow_engine: + type: object + additionalProperties: + $ref: '#/components/schemas/WorkflowEngine' workflow_engine_versions: type: object additionalProperties: @@ -683,7 +687,7 @@ components: type: object additionalProperties: type: string - description: A message containing useful information about the running service, including supported versions and default settings. + description: Tags containing useful information about the WES service. required: - workflow_type_versions - supported_wes_versions @@ -883,7 +887,7 @@ components: description: >- REQUIRED - The workflow descriptor type, must be "CWL", "WDL", "Nextflow", or "SnakeMake" currently (or another alternative supported by this WES instance) + The workflow descriptor type, must be "CWL", "WDL", "Nextflow", or "Snakemake" currently (or another alternative supported by this WES instance, see service-info) workflow_type_version: type: string description: >- @@ -1040,7 +1044,7 @@ components: type: array items: type: string - description: an array of one or more acceptable types for the `workflow_type` + description: an array of one or more acceptable types for the `workflow_type`, must be "CWL", "WDL", "Nextflow", or "Snakemake" currently (or another alternative supported by this WES instance, see service-info) description: Available workflow types supported by a given instance of the service. TaskListResponse: title: TaskListResponse @@ -1096,8 +1100,18 @@ components: type: array items: type: string - description: An array of one or more acceptable engines versions for the `workflow_engine` + description: An array of one or more acceptable engines versions for the `workflow_engine`. Since a server may support multiple engines and version, the recommendation is to encode this array as '_' for clarity. description: Available workflow engine versions supported by a given instance of the service. + WorkflowEngine: + title: WorkflowEngine + type: object + properties: + workflow_engine: + type: array + items: + type: string + description: An array of one or more acceptable workflow engines. Since a server may support multiple engines and version, the recommendation is to encode the workflow_engine_version array as '_' where workflow_engine values match this array for clarity. + description: Available workflow engines supported by a given instance of the service. RunListResponse: title: RunListResponse type: object @@ -1160,7 +1174,7 @@ components: description: Workflow completion time in ISO 8601 format engine: type: string - description: Workflow engine used (cromwell, toil, cwltool, nextflow, snakemake) + description: Workflow engine used (cromwell, toil, cwltool, nextflow, Snakemake) engine_version: type: string description: Version of the workflow engine From e9a7eed27081e61b3410a9c00521067cb15fef58 Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Wed, 30 Jul 2025 12:40:05 +0200 Subject: [PATCH 12/19] fixing syntax errors --- openapi/workflow_execution_service.openapi.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index 9788e537..906a660e 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -323,7 +323,8 @@ paths: items: type: string format: binary - description: Files to be staged for workflow execution. You set the filename/path using the Content-Disposition header in the multipart form submission. For example 'Content-Disposition: form-data; name="workflow_attachment"; filename="workflows/helper.wdl"'. The files are staged to a temporary directory and can be referenced by relative path in the workflow_url. + description: >- + Files to be staged for workflow execution. You set the filename/path using the Content-Disposition header in the multipart form submission. For example 'Content-Disposition: form-data; name="workflow_attachment"; filename="workflows/helper.wdl"'. The files are staged to a temporary directory and can be referenced by relative path in the workflow_url. required: - workflow_type - workflow_type_version @@ -1102,7 +1103,7 @@ components: type: string description: An array of one or more acceptable engines versions for the `workflow_engine`. Since a server may support multiple engines and version, the recommendation is to encode this array as '_' for clarity. description: Available workflow engine versions supported by a given instance of the service. - WorkflowEngine: + WorkflowEngine: title: WorkflowEngine type: object properties: From 7915cb12157a89f3a387c7b0e3402f2a3263a6fb Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Wed, 30 Jul 2025 12:57:39 +0200 Subject: [PATCH 13/19] updates to documentation for clarity --- .../workflow_execution_service.openapi.yaml | 30 ++++++++++++++----- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index 906a660e..f135639f 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -257,15 +257,15 @@ paths: **File Handling:** - - **`application/json` requests**: Workflow file is referenced by URL or path. Use `workflow_unified_params` with File objects for rich metadata and generic parameterization. + - **`application/json` requests**: Workflow file is referenced by URL or path via `workflow_url` with additional workflow files specified by `additional_workflow_urls`. Use `workflow_unified_params` for generic workflow parameterization (recommended when possible) or `workflow_params` for JSON-string encoded native workflow params passing. - - **`multipart/form-data` requests**: The `workflow_attachment` array uploads files required for workflow execution. Files are staged to a temporary directory with Content-Disposition headers containing filenames. Subdirectories are allowed but parent directory references ('..') are prohibited for security purposes. + - **`multipart/form-data` requests**: The `workflow_attachment` array uploads files required for workflow execution. Files are staged to a temporary directory with Content-Disposition headers containing filenames. Subdirectories are allowed but parent directory references ('..') are prohibited for security purposes. Use `workflow_unified_params` for generic workflow parameterization (recommended when possible) or `workflow_params` for JSON-string encoded native workflow params passing. **Parameter Formats:** - - **`workflow_params`**: Language-specific parameters (CWL, WDL, Nextflow, Snakemake format). This field is used to send the native parameterization file (encoded as a string) used by the workflow engine. + - **`workflow_params`**: Language-specific parameters (CWL, WDL, Nextflow, Snakemake format). This field is used to send the native parameterization file (encoded as a JSON string) used by the workflow engine. - - **`workflow_unified_params`**: Universal parameter format that WES server implementation converts to the target workflow language. This allows for a more generic way to pass parameters that can be converted to the appropriate native format for the specified workflow_type. This is an optional field, but if provided, it takes precedence over `workflow_params`. + - **`workflow_unified_params`**: Universal parameter format that WES server implementation converts to the target workflow language. This allows for a more generic way to pass parameters that can be converted to the appropriate native format for the specified workflow_type. This is an optional field (and the recommended way to pass params). If provided, it takes precedence over `workflow_params`. **Required Fields:** @@ -649,6 +649,10 @@ components: - $ref: 'https://raw.githubusercontent.com/ga4gh-discovery/ga4gh-service-info/v1.0.0/service-info.yaml#/components/schemas/Service' - type: object properties: + workflow_type: + type: object + additionalProperties: + $ref: '#/components/schemas/WorkflowType' workflow_type_versions: type: object additionalProperties: @@ -1037,6 +1041,16 @@ components: type: string description: The stringified version of the default parameter. e.g. "2.45". description: A message that allows one to describe default parameters for a workflow engine. + WorkflowType: + title: WorkflowType + type: object + properties: + workflow_type: + type: array + items: + type: string + description: an array of one or more acceptable types for the `workflow_type`, must be "CWL", "WDL", "Nextflow", or "Snakemake" currently (or another alternative supported by this WES instance, see service-info) + description: Available workflow types supported by a given instance of the service. WorkflowTypeVersion: title: WorkflowTypeVersion type: object @@ -1045,8 +1059,8 @@ components: type: array items: type: string - description: an array of one or more acceptable types for the `workflow_type`, must be "CWL", "WDL", "Nextflow", or "Snakemake" currently (or another alternative supported by this WES instance, see service-info) - description: Available workflow types supported by a given instance of the service. + description: an array of one or more acceptable type and versions for the `workflow_type`. Use the convension `_` to encode this array for clarity. For example, a WES service supporting CWL 1.0 and WDL 1.1 would encode this as `["CWL_1.0", "WDL_1.1"]`. + description: Available workflow type versions supported by a given instance of the service. TaskListResponse: title: TaskListResponse type: object @@ -1101,7 +1115,7 @@ components: type: array items: type: string - description: An array of one or more acceptable engines versions for the `workflow_engine`. Since a server may support multiple engines and version, the recommendation is to encode this array as '_' for clarity. + description: An array of one or more acceptable engines versions for the `workflow_engine`. Since a server may support multiple engines and version, the recommendation is to encode this array as `_` for clarity. description: Available workflow engine versions supported by a given instance of the service. WorkflowEngine: title: WorkflowEngine @@ -1111,7 +1125,7 @@ components: type: array items: type: string - description: An array of one or more acceptable workflow engines. Since a server may support multiple engines and version, the recommendation is to encode the workflow_engine_version array as '_' where workflow_engine values match this array for clarity. + description: An array of one or more acceptable workflow engines. Since a server may support multiple engines and version, the recommendation is to encode the workflow_engine_version array as `_` where `workflow_engine` values match this array for clarity. description: Available workflow engines supported by a given instance of the service. RunListResponse: title: RunListResponse From e32a75b085dea8baa6b0db0310f2e5105caca644 Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Wed, 30 Jul 2025 13:25:45 +0200 Subject: [PATCH 14/19] updates to documentation for clarity --- .../workflow_execution_service.openapi.yaml | 38 +++++++++++-------- 1 file changed, 22 insertions(+), 16 deletions(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index f135639f..6b3e8375 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -251,9 +251,9 @@ paths: **Content Types Supported:** - 1. **`application/json`** (Recommended for single workflows submitted by URL): Submit a complete `RunRequest` object with structured data and type validation. The advantage of this approach is the schema is explicitly defined by OpenAPI syntax (vs. just strings in multipart-form) along with workflow_unified_params object support (see below for more info). Disadvantage is that it does not support workflow file uploads directly, which may be required for multi-file workflows not available as URLs. + 1. **`application/json`** (Recommended for workflows submitted by URLs): Submit a complete `RunRequest` object with structured data and type validation. The advantage of this approach is the schema is explicitly defined by OpenAPI syntax (vs. just strings in multipart-form) along with workflow_unified_params object support (see below for more info). Disadvantage is that it does not support workflow file uploads directly, which may be required for multi-file workflows not available as URLs. - 2. **`multipart/form-data`**: Submit for cases requiring workflow file uploads (rather than just a URL). Fields are JSON-encoded strings that mirror the `RunRequest` structure. Advantage is you can upload one or more workflow files directly while the disadvantage is you need to be careful in formating the JSON-ended string arguments correctly (multipart form submissions only support strings hence the limitation). + 2. **`multipart/form-data`**: Submit for cases requiring workflow file uploads (rather than just a URL). Fields are JSON-encoded strings that mirror the `RunRequest` structure. Advantage is you can upload one or more workflow files directly while the disadvantage is you need to be careful in formating the JSON-encoded string arguments correctly (multipart form submissions only support strings hence the limitation). **File Handling:** @@ -265,15 +265,15 @@ paths: - **`workflow_params`**: Language-specific parameters (CWL, WDL, Nextflow, Snakemake format). This field is used to send the native parameterization file (encoded as a JSON string) used by the workflow engine. - - **`workflow_unified_params`**: Universal parameter format that WES server implementation converts to the target workflow language. This allows for a more generic way to pass parameters that can be converted to the appropriate native format for the specified workflow_type. This is an optional field (and the recommended way to pass params). If provided, it takes precedence over `workflow_params`. + - **`workflow_unified_params`**: Universal parameter format that the WES server implementation converts to the target workflow language parameterization format. This allows for a more generic way to pass parameters that can be converted to the appropriate native format for the specified `workflow_type`. This is an optional field (and the recommended way to pass params). If provided, it takes precedence over `workflow_params`. **Required Fields:** - `workflow_type`: Workflow language ("CWL", "WDL", "Nextflow", "Snakemake", or another alternative supported by this WES instance, see service-info) - - `workflow_type_version`: The workflow descriptor type version, must be one supported by this WES instance + - `workflow_type_version`: The workflow descriptor type version, must be one supported by this WES instance, see service-info - - `workflow_url`: Absolute URL to workflow file, or relative path to uploaded attachment + - `workflow_url`: Absolute URL to primary workflow file, or relative path to uploaded attachment See the `RunRequest` schema documentation for complete field descriptions and validation rules when encoding as strings in the multipart/form-data request. operationId: RunWorkflow @@ -290,19 +290,19 @@ paths: properties: workflow_params: type: string - description: JSON-encoded workflow parameters (see RunRequest for how to encode) + description: JSON-encoded native workflow parameters (see RunRequest for how to encode) workflow_unified_params: type: string - description: JSON-encoded unified workflow parameters (see RunRequest for how to encode) + description: JSON-encoded universal workflow parameters (see RunRequest for how to encode) workflow_type: type: string description: Workflow descriptor type must be "CWL", "WDL", "Nextflow", or "Snakemake" currently (or another alternative supported by this WES instance, see service-info) workflow_type_version: type: string - description: The workflow descriptor type version, must be one supported by this WES instance + description: The workflow descriptor type version, must be one supported by this WES instance, see service-info tags: type: string - description: JSON-encoded key-value tags for the workflow run (see RunRequest for how to encode) + description: JSON-encoded object containing key-value tags for the workflow run (e.g., '{"experiment":"rna-seq","user":"john","sample_id":"SAMPLE_001"}') workflow_engine: type: string description: The workflow engine, must be one supported by this WES instance. Required if workflow_engine_version is provided. @@ -892,17 +892,21 @@ components: description: >- REQUIRED - The workflow descriptor type, must be "CWL", "WDL", "Nextflow", or "Snakemake" currently (or another alternative supported by this WES instance, see service-info) + Workflow descriptor type, must be "CWL", "WDL", "Nextflow", or "Snakemake" currently (or another alternative supported by this WES instance, see service-info for supported types) workflow_type_version: type: string description: >- REQUIRED - The workflow descriptor type version, must be one supported by this WES instance + The workflow descriptor type version, must be one supported by this WES instance (see service-info for supported versions) tags: type: object additionalProperties: type: string + description: >- + OPTIONAL + + Arbitrary key-value tags to associate with the workflow run for organization, tracking, or metadata purposes. Examples include experiment names, user identifiers, sample IDs, or project codes. workflow_engine_parameters: type: object additionalProperties: @@ -910,19 +914,21 @@ components: workflow_engine: type: string description: >- - The workflow engine, must be one supported by this WES instance. Required if workflow_engine_version is provided. + OPTIONAL + + The workflow engine, must be one supported by this WES instance (see service-info for supported engines). Required if workflow_engine_version is provided. workflow_engine_version: type: string description: >- - The workflow engine version, must be one supported by this WES instance. - If workflow_engine is provided, but workflow_engine_version is not, servers can make no assumptions with regard to the engine version the WES instance uses to process the request if - that WES instance supports multiple versions of the requested engine. + OPTIONAL + + The workflow engine version, must be one supported by this WES instance (see service-info for supported engine versions). If workflow_engine is provided, but workflow_engine_version is not, servers can make no assumptions with regard to the engine version the WES instance uses to process the request if that WES instance supports multiple versions of the requested engine. workflow_url: type: string description: >- REQUIRED - The workflow document. When `workflow_attachments` is used to attach files, the `workflow_url` may be a relative path to one of the attachments. + The primary workflow document. When `workflow_attachments` is used to attach files, the `workflow_url` may be a relative path to one of the attachments that is the primary workflow to be executed. additional_workflow_urls: type: array items: From 486f505118064253511e40ce66bc1ceab6387a99 Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Wed, 30 Jul 2025 14:15:53 +0200 Subject: [PATCH 15/19] simplified outputs object --- .../workflow_execution_service.openapi.yaml | 34 ------------------- 1 file changed, 34 deletions(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index 6b3e8375..5ecd04cd 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -1178,40 +1178,6 @@ components: description: Named workflow outputs with structured metadata additionalProperties: $ref: '#/components/schemas/OutputObject' - workflow_metadata: - type: object - description: Overall workflow execution metadata - properties: - execution_id: - type: string - description: Unique identifier for this workflow execution - status: - $ref: '#/components/schemas/State' - start_time: - type: string - description: Workflow start time in ISO 8601 format - end_time: - type: string - description: Workflow completion time in ISO 8601 format - engine: - type: string - description: Workflow engine used (cromwell, toil, cwltool, nextflow, Snakemake) - engine_version: - type: string - description: Version of the workflow engine - resource_usage: - type: object - description: Resource consumption statistics - properties: - cpu_hours: - type: number - description: Total CPU hours consumed - memory_gb_hours: - type: number - description: Total memory GB-hours consumed - disk_gb: - type: number - description: Total disk space used in GB OutputObject: title: OutputObject type: object From 8916c58a7b7cf6e022fd8a55a332aa6835d5fc96 Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Wed, 30 Jul 2025 14:37:00 +0200 Subject: [PATCH 16/19] Update openapi/workflow_execution_service.openapi.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- openapi/workflow_execution_service.openapi.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index 5ecd04cd..33caec8b 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -1065,7 +1065,7 @@ components: type: array items: type: string - description: an array of one or more acceptable type and versions for the `workflow_type`. Use the convension `_` to encode this array for clarity. For example, a WES service supporting CWL 1.0 and WDL 1.1 would encode this as `["CWL_1.0", "WDL_1.1"]`. + description: an array of one or more acceptable type and versions for the `workflow_type`. Use the convention `_` to encode this array for clarity. For example, a WES service supporting CWL 1.0 and WDL 1.1 would encode this as `["CWL_1.0", "WDL_1.1"]`. description: Available workflow type versions supported by a given instance of the service. TaskListResponse: title: TaskListResponse From d3397cad8f2587aa38696a6911b686c31aa33520 Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Wed, 30 Jul 2025 14:38:32 +0200 Subject: [PATCH 17/19] fixed spelling --- openapi/workflow_execution_service.openapi.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index 5ecd04cd..387cad1b 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -253,7 +253,7 @@ paths: 1. **`application/json`** (Recommended for workflows submitted by URLs): Submit a complete `RunRequest` object with structured data and type validation. The advantage of this approach is the schema is explicitly defined by OpenAPI syntax (vs. just strings in multipart-form) along with workflow_unified_params object support (see below for more info). Disadvantage is that it does not support workflow file uploads directly, which may be required for multi-file workflows not available as URLs. - 2. **`multipart/form-data`**: Submit for cases requiring workflow file uploads (rather than just a URL). Fields are JSON-encoded strings that mirror the `RunRequest` structure. Advantage is you can upload one or more workflow files directly while the disadvantage is you need to be careful in formating the JSON-encoded string arguments correctly (multipart form submissions only support strings hence the limitation). + 2. **`multipart/form-data`**: Submit for cases requiring workflow file uploads (rather than just a URL). Fields are JSON-encoded strings that mirror the `RunRequest` structure. Advantage is you can upload one or more workflow files directly while the disadvantage is you need to be careful in formatting the JSON-encoded string arguments correctly (multipart form submissions only support strings hence the limitation). **File Handling:** From cb8361ffa2e3080063592c88ffce7549205d206b Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Wed, 30 Jul 2025 14:42:44 +0200 Subject: [PATCH 18/19] Update openapi/workflow_execution_service.openapi.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- openapi/workflow_execution_service.openapi.yaml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index 64b297eb..56024a1a 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -795,9 +795,10 @@ components: workflow_params: type: object description: >- - REQUIRED (unless workflow_unified_params provided) + OPTIONAL - The workflow run parameterizations (JSON encoded), including input and output file locations + The workflow run parameterizations (JSON encoded), including input and output file locations. + Either `workflow_params` or `workflow_unified_params` must be provided, but not both. workflow_unified_params: type: object description: >- From a7b81196f68140323fbbf8cd44c04a158ace58a7 Mon Sep 17 00:00:00 2001 From: Brian O'Connor Date: Wed, 30 Jul 2025 14:43:43 +0200 Subject: [PATCH 19/19] updated docs --- openapi/workflow_execution_service.openapi.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/openapi/workflow_execution_service.openapi.yaml b/openapi/workflow_execution_service.openapi.yaml index 56024a1a..aecf9fc7 100644 --- a/openapi/workflow_execution_service.openapi.yaml +++ b/openapi/workflow_execution_service.openapi.yaml @@ -798,7 +798,7 @@ components: OPTIONAL The workflow run parameterizations (JSON encoded), including input and output file locations. - Either `workflow_params` or `workflow_unified_params` must be provided, but not both. + Either `workflow_params` or `workflow_unified_params` must be provided, `workflow_unified_params` takes precedence. workflow_unified_params: type: object description: >-