Skip to content

Conversation

edmundmiller
Copy link
Contributor

Problem

Addresses two critical issues identified in the AWSMegatests workflow failures:

  1. S3 Publishing Failures: workflow failing with S3 copy errors for large files
  2. Configuration Casting Errors:

Root Causes

S3 Publishing Failures

  • Missing S3 permissions for multipart upload operations (required for files >5GB)
  • Current policy lacked: , , ,

Configuration Casting Errors

  • Closures in publishDir configuration being cast as strings
  • Problematic closures: , ,

Solution

Enhanced S3 Permissions

  • Added multipart upload permissions to TowerForge S3 IAM policy
  • Enables robust handling of large file publishing operations
  • Updated policy version hash to trigger compute environment recreation

Fixed publishDir Configuration

  • Removed problematic closures from publishDir tags in all environment configs
  • Kept only static string values that don't require runtime evaluation
  • Applied across CPU, GPU, and ARM compute environment configurations

Files Changed

    • Enhanced S3 IAM policy with multipart upload permissions
    • Fixed publishDir tags configuration across all environments

Testing

  • Pulumi preview confirmed S3 policy updates and compute environment recreation
  • All three compute environments will be replaced with updated configurations
  • Policy version hash updated from to

Impact

  • Resolves S3 publishing failures for large files (>5GB)
  • Eliminates configuration casting errors in Nextflow workflows
  • Ensures robust S3 operations with proper multipart upload support

Fixes the workflow failure reported by Seqera AI.

@edmundmiller edmundmiller requested review from maxulysse and a team as code owners September 22, 2025 14:47
@edmundmiller edmundmiller self-assigned this Sep 22, 2025
…ing errors

- Add multipart upload permissions to S3 IAM policy for large files >5GB
- Remove problematic closures from publishDir tags configuration
- Update policy version hash to trigger compute environment recreation
- Fixes workflow failures with S3 copy operations and casting errors
@edmundmiller edmundmiller force-pushed the fix/s3-publishdir-casting-errors branch from fd8aad0 to 99078ba Compare September 22, 2025 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants