Helm chart for Argo + Github Actions + Nextflow runner

[bwalsh]

Goals

• K8s-native orchestration via Argo Workflows running Nextflow (k8s executor).
• Multi-tenant isolation by namespace, RBAC, quotas, and network policies.
• Triggers from internal Git servers and/or GitHub Actions → Argo Events.
• Results and cache on Amazon S3 (or S3-compatible on-prem), surfaced in Argo UI and CI artifacts.

---

High-Level Architecture (Multi-Tenant)

flowchart TB
  subgraph Git[Tenant Git Servers & GitHub Actions]
    GH[GitHub Actions per-tenant workflows]
    GS[Internal Git servers GHES/GitLab/Gitea]
  end

  GH -->|Webhook / API| AE[Argo Events Gateway per-tenant or shared]
  GS -->|Webhook / Polling| AE

  subgraph ARGO[Cluster: Argo Control Plane namespace: argo]
    AE --> SENS[Argo Sensors]
    SENS -->|submit| WFCTRL[Argo Workflows Controller]
    UI[Argo Server SSO + RBAC]
  end

  subgraph TENANT_A[Namespace: wf-teamA]
    WFT_A[WorkflowTemplate: nextflow-runner]
    SA_A[(ServiceAccount: nextflow-launcher)]
    CFG_A[ConfigMap: nextflow.config k8s profile]
    SEC_A[(Secret: S3 creds / WI)]
    NF_A[Nextflow Executor Pods]
  end

  subgraph TENANT_B[Namespace: wf-teamB]
    WFT_B[WorkflowTemplate: nextflow-runner]
    SA_B[(ServiceAccount: nextflow-launcher)]
    CFG_B[ConfigMap: nextflow.config]
    SEC_B[(Secret: S3 creds / WI)]
    NF_B[Nextflow Executor Pods]
  end

  WFCTRL -->|launch| WFT_A
  WFCTRL -->|launch| WFT_B
  WFT_A --> NF_A
  WFT_B --> NF_B

  subgraph S3[Amazon S3 Multi-tenant buckets/prefixes]
    CACHE[Nextflow workDir cache]
    ART[Reports & results: report.html, timeline.html, trace.txt]
  end

  NF_A -->|read/write| S3
  NF_B -->|read/write| S3
  UI -->|view| WFCTRL

Loading

Key multi-tenant controls

• One namespace per team: wf-<team>, each with own SA/RBAC/Quotas/NetworkPolicies.
• Argo Server SSO + RBAC: users only see their namespace.
• S3 buckets or strict prefixes per tenant + lifecycle rules.
• GitHub Actions can trigger Argo (webhook → Argo Events) or call Argo API directly with a short-lived token.

---

Milestones

M1 — Control Plane & Guardrails

• Install Argo Workflows + Argo Events into argo.
• Configure Argo Server SSO (OIDC) and Argo RBAC (map IdP groups → namespaces).
• Baseline Pod Security and default deny NetworkPolicy cluster-wide.
• Prepare S3: per-tenant bucket (or prefix), IAM policies, lifecycle (cache expiry).

M2 — Tenant Blueprint (repeatable)

Create per tenant:

• Namespace wf-<team>.
• ServiceAccount nextflow-launcher + namespace-scoped Role/RoleBinding (pods/jobs + argo resources).
• ResourceQuota/LimitRange, NetworkPolicies (egress only to Git, registry, S3, DNS).
• ConfigMap nextflow-config (k8s profile).
• Secret for S3 (or workload identity).

RBAC (namespace-scoped)

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nextflow-launcher
  namespace: wf-teamA
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: nextflow-executor
  namespace: wf-teamA
rules:
  - apiGroups: ["", "batch"]
    resources: ["pods","pods/log","pods/status","jobs"]
    verbs: ["create","get","list","watch","delete","patch"]
  - apiGroups: ["argoproj.io"]
    resources: ["workflows","workflowtemplates","cronworkflows"]
    verbs: ["create","get","list","watch","delete","patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: nextflow-executor-binding
  namespace: wf-teamA
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nextflow-executor
subjects:
  - kind: ServiceAccount
    name: nextflow-launcher
    namespace: wf-teamA

Nextflow k8s profile (uses S3)

apiVersion: v1
kind: ConfigMap
metadata:
  name: nextflow-config
  namespace: wf-teamA
data:
  nextflow.config: |
    profiles {
      k8s {
        process.executor = 'k8s'
        workDir = System.getenv('NF_WORKDIR') ?: 's3://teamA-nf-workdir'
        k8s {
          namespace = 'wf-teamA'
          serviceAccount = 'nextflow-launcher'
          pod = [
            imagePullPolicy: 'IfNotPresent',
            resources: [ limits: [ cpu: '4', memory: '8Gi' ], requests: [ cpu: '1', memory: '2Gi' ] ]
          ]
        }
        aws.client.bucketRegion = System.getenv('AWS_REGION') ?: 'us-west-2'
        // For S3-compatible endpoints, also set aws.client.endpoint & pathStyleAccess
      }
    }

M3 — Artifact Surfacing (Argo ↔ S3)

• Configure ArtifactRepositoryRef so Argo links HTML reports while Nextflow uses S3 as workDir.
• Option A: one global default in argo namespace (simple).
• Option B: per-tenant artifact repo (strict separation).

apiVersion: v1
kind: ConfigMap
metadata:
  name: artifact-repositories
  namespace: argo
data:
  default-v1: |
    s3:
      bucket: teamA-argo-artifacts
      endpoint: s3.amazonaws.com
      region: us-west-2
      accessKeySecret: { name: teamA-s3, key: accessKey }
      secretKeySecret: { name: teamA-s3, key: secretKey }
    archiveLogs: true

M4 — WorkflowTemplate (Nextflow runner, reusable)

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: nextflow-runner
  namespace: wf-teamA
spec:
  entrypoint: run
  arguments:
    parameters:
      - { name: pipeline, value: "./" }
      - { name: params,   value: "" }
  templates:
    - name: run
      container:
        image: ghcr.io/yourorg/nf-runtime:jdk17
        command: ["/bin/bash","-lc"]
        args:
          - >-
            set -euo pipefail;
            export NXF_HOME="/workspace/.nxf";
            cp /config/nextflow.config ./nextflow.config || true;
            if [[ "{{workflow.parameters.pipeline}}" == "./" ]]; then
              PIPE="./";
            else
              PIPE="-r {{workflow.parameters.pipeline}}";
            fi;
            nextflow run $PIPE -profile k8s {{workflow.parameters.params}} \
              -with-report report.html -with-timeline timeline.html -with-trace trace.txt;
      volumeMounts:
        - name: cfg
          mountPath: /config
      volumes:
        - name: cfg
          configMap: { name: nextflow-config }
      env:
        - name: AWS_REGION
          valueFrom: { secretKeyRef: { name: s3-credentials, key: AWS_REGION } }
        - name: AWS_ACCESS_KEY_ID
          valueFrom: { secretKeyRef: { name: s3-credentials, key: AWS_ACCESS_KEY_ID } }
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom: { secretKeyRef: { name: s3-credentials, key: AWS_SECRET_ACCESS_KEY } }
        - name: NF_WORKDIR
          valueFrom: { secretKeyRef: { name: s3-credentials, key: NF_WORKDIR } }
        # For S3-compatible endpoints, add S3_ENDPOINT + pathStyleAccess in config
      outputs:
        artifacts:
          - { name: report,   path: /workspace/report.html }
          - { name: timeline, path: /workspace/timeline.html }
          - { name: trace,    path: /workspace/trace.txt }
      serviceAccountName: nextflow-launcher

M5 — Triggers

A) GitHub Actions → Argo (Nextflow-based Actions emphasized)

• Preferred: GHA workflow sends a webhook (signed secret) → Argo Events.
• Alternative: GHA calls Argo Server API (OIDC token or robot user) to submit a Workflow from WorkflowTemplate.

GHA example (dispatch → Argo API)

name: trigger-argo-nextflow
on:
  workflow_dispatch:
    inputs:
      pipeline: { description: "Path or repo", default: "./", required: true }
      params:   { description: "Nextflow params", required: false, default: "" }

jobs:
  submit:
    runs-on: ubuntu-latest
    steps:
      - name: Submit to Argo (tenant A)
        env:
          ARGO_ADDR: https://argo.example.org
          ARGO_TOKEN: ${{ secrets.ARGO_TEAM_A_BEARER }}
        run: |
          cat > wf.json <<'JSON'
          {
            "namespace": "wf-teamA",
            "resourceKind": "Workflow",
            "resource": {
              "apiVersion": "argoproj.io/v1alpha1",
              "kind": "Workflow",
              "metadata": { "generateName": "nf-" },
              "spec": {
                "workflowTemplateRef": { "name": "nextflow-runner" },
                "arguments": {
                  "parameters": [
                    { "name": "pipeline", "value": "${{ github.event.inputs.pipeline }}" },
                    { "name": "params",   "value": "${{ github.event.inputs.params }}" }
                  ]
                }
              }
            }
          }
          JSON
          curl -sS -X POST "$ARGO_ADDR/api/v1/workflows/wf-teamA/submit" \
            -H "Authorization: Bearer $ARGO_TOKEN" \
            -H "Content-Type: application/json" \
            --data-binary @wf.json

B) Argo Events (webhook) from Git servers

• EventSource exposes /webhook/<repo>; Sensor parameterizes pipeline & params and submits from WorkflowTemplate.
• Polling fallback: CronJob that git ls-remote and fires a local event when refs change.

M6 — Ad-Hoc & Scheduled

• Ad-hoc: Argo UI “Submit” (RBAC isolates per namespace), or argo submit --from workflowtemplate/… -p ….
• Scheduled: CronWorkflow per tenant:

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: nightly-nf
  namespace: wf-teamA
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: "Forbid"
  workflowSpec:
    workflowTemplateRef: { name: nextflow-runner }
    arguments:
      parameters:
        - { name: pipeline, value: "./pipelines/rnaseq" }
        - { name: params,   value: "--reads 's3://teamA/raw/*.fastq.gz'" }

M7 — Results & Observability

• S3 is source of truth: s3://teamA-nf-workdir/<pipeline>/<run-id>/…
• Argo UI shows HTML reports (report.html, timeline.html, trace.txt) via artifacts.
• Centralize logs to Loki/ELK with labels: tenant, pipeline, run_id, commit, trigger=gha|webhook.
• S3 lifecycle: expire cache/work dirs after N days; retain final outputs per policy.

---

Multi-Tenant Checklist

• Namespaces: one per team; no shared runtime.
• RBAC: namespace-scoped Roles; Argo RBAC maps IdP groups → only their namespace.
• NetworkPolicies: default deny; egress allowlist (Git, registry, S3, DNS).
• Quotas: per namespace CPU/MEM/POD caps; optional PriorityClasses.
• S3 segmentation: buckets per tenant or prefixes with IAM boundaries.
• Supply chain: signed/pinned images; OPA/Gatekeeper allow-list.
• Secrets: External Secrets / workload identity; never mount cluster-wide creds.

---

Acceptance

• Ad-hoc run (UI + CLI) and GitHub Actions-triggered run both succeed.
• Nextflow executor pods stay within tenant namespace; no cross-tenant access.
• Artifacts visible in Argo UI; S3 paths follow tenant/run-ID convention.
• Quota breach is contained to the tenant; other tenants unaffected.
• Lifecycle rules reclaim cache without deleting retained outputs.

---

[bwalsh]

Roadmap with high-level epics and a 6-sprint plan tailored to an on-prem, multi-tenant K8s cluster using Argo Workflows/Events, S3, Nextflow, and GitHub Actions → Argo triggers.

Epics (Very High Level)

1. E1 — Control Plane Bootstrap

• Stand up Argo Workflows, Argo Events, Argo Server (SSO/RBAC).
• Base cluster guardrails (Pod Security, default-deny NetworkPolicies).
• DoD: Authenticated users can log in to Argo; baseline policies enforced.

2. E2 — Multi-Tenant Framework & Isolation

• Namespace per team, SA/RBAC, ResourceQuota/LimitRange, NetworkPolicies.
• Tenant bootstrap script (idempotent).
• DoD: New tenants can be provisioned in <10 min with isolation verified.

3. E3 — Storage & Artifacts on S3

• Per-tenant S3 buckets/prefixes, IAM boundaries, lifecycle rules.
• Argo ArtifactRepositoryRef (global or per-tenant).
• DoD: Nextflow reports/artifacts visible in Argo UI; cache lifecycle works.

4. E4 — Nextflow Runtime & Templates

• Hardened Nextflow runtime image (JDK17 + nf + kubectl + awscli).
• Per-tenant nextflow.config (k8s executor), reusable WorkflowTemplate.
• DoD: Hello-world DSL2 pipeline runs successfully via template.

5. E5 — Triggers & CI Integration

• Argo Events webhook path for internal Git; polling fallback (CronJob) if needed.
• GitHub Actions → Argo submit (bearer token or webhook) with parameters.
• DoD: Manual run, webhook run, and GHA-triggered run all work end-to-end.

6. E6 — Observability & FinOps

• Centralized logs (Loki/ELK), labels/annotations (tenant, pipeline, run_id).
• Basic Grafana boards: runs/day, success rate, P95 duration, resource usage.
• DoD: Operators can diagnose failures in <5 min; dashboards populated.

7. E7 — Security & Compliance

• Secrets via External Secrets or workload identity; image signing/allow-lists (OPA/Gatekeeper).
• Argo RBAC tied to IdP groups; audit logging.
• DoD: CIS-ish checks pass; no cross-tenant data/perm leakage in tests.

8. E8 — Scale, Resilience & DR

• Load tests (concurrency/pod counts), failure injection (node/pod kill, S3 auth revoke).
• Quota behavior validated; retry/cleanup policies tuned.
• DoD: Meets agreed SLOs; zero leaked pods; documented recovery steps.

9. E9 — Enablement & Docs

• Tenant runbook, operator runbook, “Ad-hoc Submit” quickstart, CI examples.
• DoD: A new team self-onboards and runs a pipeline without platform help.

---

Sprints (6 × two-weeks)

Sprint 1 — Control Plane & Guardrails (E1)

Goals

• Deploy Argo Workflows/Events + Argo Server with OIDC SSO.
• Apply baseline Pod Security + default-deny NetworkPolicies (cluster-level).
  Deliverables
• Helm values/manifests; SSO config; Argo RBAC scaffold.
  Exit Criteria
• AuthN/AuthZ works; non-authorized users can’t see workloads.

Sprint 2 — Tenant Framework (E2)

Goals

• Build tenant bootstrap (namespace, SA/RBAC, quotas/limits, NetPols).
• Create two pilot tenants (Team A/B).
  Deliverables
• bootstrap/tenant.sh (idempotent), sample YAMLs, smoke tests.
  Exit Criteria
• Team A cannot access Team B objects/UI; quotas enforced.

Sprint 3 — S3 & Artifacts (E3)

Goals

• Create per-tenant S3 buckets/prefixes, IAM policies, lifecycle.
• Wire Argo ArtifactRepositoryRef (global or per-tenant).
  Deliverables
• S3 policy docs, lifecycle JSON, Argo artifact ConfigMap(s).
  Exit Criteria
• NF reports (report.html, timeline.html, trace.txt) visible in Argo UI; cache aging works.

Sprint 4 — Nextflow Runtime & Templates (E4)

Goals

• Build/publish hardened NF image; tenant nextflow.config + WorkflowTemplate.
• Run hello-world DSL2 pipeline in each pilot tenant.
  Deliverables
• Image, WorkflowTemplate nextflow-runner, sample pipeline repo.
  Exit Criteria
• Successful runs in A/B; NF executor pods restricted to tenant namespaces.

Sprint 5 — Triggers (Argo Events + GitHub Actions) (E5)

Goals

• Argo Events webhook from internal Git (per-repo route + Sensor param mapping).
• GitHub Actions → Argo “submit with inputs” workflow for ad-hoc Nextflow runs.
  Deliverables
• EventSource/Sensor manifests; GHA workflow; token/secret handling.
  Exit Criteria
• Three green paths: (1) manual UI submit, (2) webhook on push, (3) GHA manual dispatch.

Sprint 6 — Observability, Security Hardening, Scale (E6–E8–E9)

Goals

• Logging/metrics dashboards; traceable labels.
• Secrets via ESO/WI; image allow-list; basic OPA policies.
• Load & failure tests; finalize runbooks and tenant self-service docs.
  Deliverables
• Grafana boards, alert rules, OPA policies, enablement docs.
  Exit Criteria
• Operators triage in <5 min; ten concurrent runs succeed; failure injections behave; docs enable a new team to self-onboard.

---

Cross-Cutting Acceptance (applies by end of Sprint 6)

• Multi-tenant isolation proven (RBAC/NetPol/Quotas/S3 IAM).
• Nextflow + Argo standard template usable by any tenant.
• CI integration: GitHub Actions trigger path operational with parameters.
• S3 lifecycle prevents cache bloat; artifacts discoverable.
• Runbooks & Quickstarts allow self-service.

---

Risks & Mitigations (brief)

• Secret sprawl → External Secrets/Workload Identity; least-privilege IAM.
• Cross-tenant leakage → Automated validation tests per release; OPA policies.
• Cost/cluster saturation → Quotas/PriorityClasses; dashboards + alerts.
• Tool drift → Pin image/versions; CI that runs a smoke NF workflow on changes.

[bwalsh]

epics.csv

Tips to import

Create a new GitHub Project (Projects → New).

Add any custom fields you want (e.g., Iteration for Sprint, Estimate, Type).

Click … → Import items and upload this CSV.

After import, bulk-edit to map Sprint to your Iteration field (or keep as text), and consider converting Epics into parent items via a “Parent”/“Epic” custom field or by adding an “Epic” label and using views/filters.

[bwalsh]

fastest, lightest-weight POC path that still proves: Argo → Nextflow on K8s → S3 artifacts, with a nod to multi-tenant and a minimal GitHub Actions trigger. You can do this in a single afternoon.

---

POC goals (keep it tiny)

• One namespace: wf-poc
• Argo Workflows only (skip Argo Events/SSO/RBAC complexity for now)
• Single ServiceAccount + Role scoped to that namespace
• One WorkflowTemplate that runs a tiny Nextflow DSL2 pipeline
• S3 for workDir + upload HTML reports as Argo artifacts
• Manual submit first; optional GitHub Actions trigger via a one-liner curl

---

Step 1 — Create namespace + minimal RBAC

# wf-poc.yaml
apiVersion: v1
kind: Namespace
metadata: { name: wf-poc }

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nextflow-launcher
  namespace: wf-poc

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: nextflow-executor
  namespace: wf-poc
rules:
  - apiGroups: ["", "batch"]
    resources: ["pods","pods/log","pods/status","jobs"]
    verbs: ["create","get","list","watch","delete","patch"]
  - apiGroups: ["argoproj.io"]
    resources: ["workflows","workflowtemplates"]
    verbs: ["create","get","list","watch","delete","patch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: nextflow-executor-binding
  namespace: wf-poc
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nextflow-executor
subjects:
  - kind: ServiceAccount
    name: nextflow-launcher
    namespace: wf-poc

Apply:

kubectl apply -f wf-poc.yaml

---

Step 2 — Install Argo Workflows (controller + UI)

(Use defaults; no SSO yet. You’ll port-forward to the UI.)

kubectl create ns argo
helm repo add argo https://argoproj.github.io/argo-helm
helm upgrade --install argo argo/argo-workflows -n argo
kubectl -n argo rollout status deploy/argo-workflows-server
kubectl -n argo rollout status deploy/argo-workflows-workflow-controller

# UI (local):
kubectl -n argo port-forward svc/argo-workflows-server 2746:2746
# Open http://localhost:2746

---

Step 3 — S3 credentials + Nextflow config (super small)

# s3-and-nf-config.yaml
apiVersion: v1
kind: Secret
metadata:
  name: s3-credentials
  namespace: wf-poc
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: "<your-ak>"
  AWS_SECRET_ACCESS_KEY: "<your-sk>"
  AWS_REGION: "us-west-2"
  NF_WORKDIR: "s3://your-poc-bucket/nf-workdir"

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nextflow-config
  namespace: wf-poc
data:
  nextflow.config: |
    profiles {
      k8s {
        process.executor = 'k8s'
        workDir = System.getenv('NF_WORKDIR')
        k8s {
          namespace = 'wf-poc'
          serviceAccount = 'nextflow-launcher'
          pod = [
            imagePullPolicy: 'IfNotPresent',
            resources: [ limits: [ cpu: '2', memory: '4Gi' ], requests: [ cpu: '0.5', memory: '1Gi' ] ]
          ]
        }
        aws.client.bucketRegion = System.getenv('AWS_REGION')
      }
    }

Apply:

kubectl apply -f s3-and-nf-config.yaml

---

Step 4 — One WorkflowTemplate that runs Nextflow

Use a public Nextflow image (no kubectl needed). This template writes report.html, timeline.html, trace.txt so Argo can surface them.

# wf-template-nextflow.yaml
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: nextflow-runner
  namespace: wf-poc
spec:
  entrypoint: run
  arguments:
    parameters:
      - name: pipeline
        value: "./"         # or remote: 'nextflow-io/rnaseq-nf'
      - name: params
        value: ""           # e.g. "--reads 's3://bucket/*.fastq.gz'"
  templates:
    - name: run
      container:
        image: nextflow/nextflow:24.04.4
        command: ["/bin/bash","-lc"]
        args:
          - >-
            set -euo pipefail;
            export NXF_HOME="/workspace/.nxf";
            cp /config/nextflow.config ./nextflow.config || true;
            if [[ "{{workflow.parameters.pipeline}}" == "./" ]]; then
              # tiny inline pipeline for the POC
              cat > main.nf <<'NF'
              nextflow.enable.dsl=2
              process HELLO {
                output: path 'hello.txt'
                """
                echo "hello nextflow + argo" > hello.txt
                """
              }
              workflow { HELLO() }
              NF
              PIPE="./"
            else
              PIPE="-r {{workflow.parameters.pipeline}}"
            fi;
            nextflow run $PIPE -profile k8s {{workflow.parameters.params}} \
              -with-report report.html -with-timeline timeline.html -with-trace trace.txt;
      volumeMounts:
        - name: cfg
          mountPath: /config
      volumes:
        - name: cfg
          configMap: { name: nextflow-config }
      env:
        - name: AWS_REGION
          valueFrom: { secretKeyRef: { name: s3-credentials, key: AWS_REGION } }
        - name: AWS_ACCESS_KEY_ID
          valueFrom: { secretKeyRef: { name: s3-credentials, key: AWS_ACCESS_KEY_ID } }
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom: { secretKeyRef: { name: s3-credentials, key: AWS_SECRET_ACCESS_KEY } }
        - name: NF_WORKDIR
          valueFrom: { secretKeyRef: { name: s3-credentials, key: NF_WORKDIR } }
      outputs:
        artifacts:
          - { name: report,   path: /workspace/report.html }
          - { name: timeline, path: /workspace/timeline.html }
          - { name: trace,    path: /workspace/trace.txt }
      serviceAccountName: nextflow-launcher

Apply:

kubectl apply -f wf-template-nextflow.yaml

---

Step 5 — Run it (manual, zero extras)

Option A: via Argo UI

• Open http://localhost:2746/ → Submit → From template → nextflow-runner → Submit.

Option B: via CLI

argo -n wf-poc template list
argo -n wf-poc submit --from workflowtemplate/nextflow-runner
argo -n wf-poc get @latest
argo -n wf-poc logs @latest

You’ll see:

• A short Nextflow launcher pod (this template).
• Nextflow executor pods in wf-poc.
• Reports downloadable from the Argo UI (Artifacts tab).
• Files in s3://your-poc-bucket/nf-workdir/...

---

Step 6 — (Optional) Easiest GitHub Actions → Argo trigger

Skip Argo Events for the POC. Use a simple Actions workflow that POSTs to Argo’s API. Easiest path: temporarily expose the Argo Server with a NodePort or port-forward from a small jump host.

Actions workflow (manual dispatch):

name: trigger-argo-poc
on:
  workflow_dispatch:
    inputs:
      pipeline:
        description: "Path or repo (./ or nextflow-io/rnaseq-nf)"
        required: true
        default: "./"
      params:
        description: "Nextflow params"
        required: false
        default: ""

jobs:
  submit:
    runs-on: ubuntu-latest
    steps:
      - name: Submit to Argo
        env:
          ARGO_ADDR: ${{ secrets.ARGO_ADDR }}           # e.g., https://argo.mycorp.local
          ARGO_TOKEN: ${{ secrets.ARGO_BEARER }}        # 'Bearer <token>'; use a short-lived token or basic auth for POC
          PIPELINE: ${{ inputs.pipeline }}
          PARAMS: ${{ inputs.params }}
        run: |
          cat > wf.json <<JSON
          {
            "namespace": "wf-poc",
            "resourceKind": "Workflow",
            "resource": {
              "apiVersion": "argoproj.io/v1alpha1",
              "kind": "Workflow",
              "metadata": { "generateName": "nf-" },
              "spec": {
                "workflowTemplateRef": { "name": "nextflow-runner" },
                "arguments": {
                  "parameters": [
                    { "name": "pipeline", "value": "$PIPELINE" },
                    { "name": "params",   "value": "$PARAMS" }
                  ]
                }
              }
            }
          }
          JSON
          curl -sS -X POST "$ARGO_ADDR/api/v1/workflows/wf-poc/submit" \
            -H "Authorization: $ARGO_TOKEN" \
            -H "Content-Type: application/json" \
            --data-binary @wf.json

For the POC, use a temporary Bearer token from argo auth token or enable basic auth. Lock this down later (OIDC, Ingress, mTLS).

---

Why this is the lightest path

• No Argo Events (remove a whole controller + webhook/IAM work).
• No SSO/RBAC wiring yet (one namespace, one SA).
• No custom images (use official nextflow/nextflow).
• No External Secrets (just one K8s Secret).
• Single template and a tiny inline pipeline.

You still prove:

1. Argo can launch Nextflow in K8s.
2. Nextflow uses S3 workDir and generates reports.
3. Manual + CI trigger both work.

---

When you’re ready to harden

• Add namespaces per team, quotas, and NetworkPolicies.
• Move to Argo Events (webhooks) and SSO with Argo RBAC.
• Swap K8s Secret → External Secrets (or Workload Identity).
• Create a hardened Nextflow runtime image and pin versions.
• Add lifecycle on S3 prefixes for cache control.

[bwalsh]

Repo & GitHub Actions Guide for Nextflow + Argo (Multi-Tenant)

This doc shows where to put your Nextflow pipelines in a Git repo and how to author GitHub Actions that trigger runs in your on-prem cluster (via Argo Workflows). It assumes:

• Argo Workflows is installed in-cluster.
• Each tenant has a namespace like wf-<team>.
• Nextflow uses the k8s executor and writes to S3 (NF_WORKDIR).

---

1) Recommended Repo Layout

Use a mono-repo layout that supports multiple pipelines, shared modules, and CI.

your-repo/
├─ pipelines/
│  ├─ rnaseq/
│  │  ├─ main.nf
│  │  ├─ nextflow.config          # pipeline-local overrides (optional)
│  │  ├─ modules/                 # DSL2 modules for rnaseq
│  │  └─ conf/                    # params profiles, e.g., conf/poc.config
│  ├─ atacseq/
│  │  └─ ...
│  └─ ... (more pipelines)
├─ modules/                        # shared DSL2 modules across pipelines
│  ├─ qc/
│  ├─ align/
│  └─ utils/
├─ containers/                     # Dockerfiles or references to pinned images
│  ├─ rnaseq.Dockerfile
│  └─ base.Dockerfile
├─ config/
│  ├─ nextflow-k8s.template.conf   # template k8s profile (no secrets)
│  ├─ params/                      # sample params JSON/YAML used in CI
│  │  ├─ rnaseq.small.json
│  │  └─ atacseq.small.json
│  └─ tenants/                     # non-secret tenant mappings consumed by CI
│     ├─ teamA.env                 # AWS_REGION, NF_WORKDIR (no keys)
│     └─ teamB.env
├─ testdata/                       # tiny test files for smoke tests
├─ docs/
│  └─ usage.md
└─ .github/
   └─ workflows/
      ├─ nf-manual-dispatch.yml            # GH Actions → Argo (ad-hoc)
      ├─ nf-on-push.yml                    # GH Actions → Argo (on branch push)
      └─ lint-and-validate.yml             # static checks (schema, nf -validate)

Why this structure?

• pipelines/* keeps each workflow self-contained but allows shared modules/.
• config/tenants/*.env lets you select tenant in a workflow without storing secrets in git.
• .github/workflows/* centralizes CI/CD.

---

2) Authoring Nextflow Pipelines

Minimal pipelines/rnaseq/main.nf

nextflow.enable.dsl=2

workflow {
  HELLO()
}

process HELLO {
  output:
    path 'hello.txt'
  """
  echo "hello rnaseq" > hello.txt
  """
}

Pipeline-local nextflow.config (optional)

profiles {
  default {
    // local defaults for dev; CI will inject k8s profile & NF_WORKDIR
  }
}

The k8s profile and S3 settings live outside the repo (cluster ConfigMap/Argo template). Keep the repo free of secrets.

---

3) What Goes into GitHub (vs. the Cluster)

• In GitHub (this repo):

  • Pipelines, modules, sample params, non-secret tenant env files, GitHub Actions workflows.

• In the Cluster (Argo & K8s):

  • WorkflowTemplate nextflow-runner (per tenant) that mounts the k8s Nextflow config.
  • Tenant ServiceAccount/RBAC and S3 credentials (K8s Secret or Workload Identity).
  • Argo artifact repository config.

This split keeps secrets and cluster specifics out of git.

---

4) GitHub Actions: Ad-Hoc Manual Dispatch (→ Argo)

Create .github/workflows/nf-manual-dispatch.yml. This workflow does not run Nextflow in GitHub; it submits a Workflow to Argo in the target tenant namespace using Argo’s HTTP API.

name: nextflow-ad-hoc
on:
  workflow_dispatch:
    inputs:
      tenant:
        description: "Target tenant (e.g., teamA, teamB)"
        required: true
        default: "teamA"
      pipeline:
        description: "Path under pipelines/ or external repo (e.g., pipelines/rnaseq or nextflow-io/rnaseq-nf)"
        required: true
        default: "pipelines/rnaseq"
      params:
        description: "Nextflow params (e.g., --reads 's3://bucket/*.fastq.gz')"
        required: false
        default: ""

jobs:
  submit:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout (optional—only needed for referencing repo paths)
        uses: actions/checkout@v4

      - name: Load tenant mapping (non-secret)
        run: |
          set -euo pipefail
          TENANT="${{ inputs.tenant }}"
          case "$TENANT" in
            teamA) NS="wf-teamA"; ARGO_ADDR="${{ vars.ARGO_ADDR_TEAMA }}";;
            teamB) NS="wf-teamB"; ARGO_ADDR="${{ vars.ARGO_ADDR_TEAMB }}";;
            *) echo "Unknown tenant: $TENANT"; exit 1;;
          esac
          echo "NS=$NS" >> $GITHUB_ENV
          echo "ARGO_ADDR=$ARGO_ADDR" >> $GITHUB_ENV

      - name: Submit to Argo
        env:
          ARGO_TOKEN: ${{ secrets.ARGO_BEARER }}     # Bearer token usable for the chosen tenant
          PIPELINE: ${{ inputs.pipeline }}
          PARAMS:   ${{ inputs.params }}
        run: |
          cat > wf.json <<JSON
          {
            "namespace": "${NS}",
            "resourceKind": "Workflow",
            "resource": {
              "apiVersion": "argoproj.io/v1alpha1",
              "kind": "Workflow",
              "metadata": { "generateName": "nf-" },
              "spec": {
                "workflowTemplateRef": { "name": "nextflow-runner" },
                "arguments": {
                  "parameters": [
                    { "name": "pipeline", "value": "${PIPELINE}" },
                    { "name": "params",   "value": "${PARAMS}" }
                  ]
                }
              }
            }
          }
          JSON
          curl -sS -X POST "$ARGO_ADDR/api/v1/workflows/${NS}/submit" \
            -H "Authorization: Bearer $ARGO_TOKEN" \
            -H "Content-Type: application/json" \
            --data-binary @wf.json

Where to put secrets & vars

• ARGO_BEARER → Actions secret (per repo or org), scoped to a minimal RBAC token.
• ARGO_ADDR_TEAMA, ARGO_ADDR_TEAMB → Actions variables (non-secret URLs).
• Optionally add a matrix to fan out across tenants.

---

5) GitHub Actions: Auto-Run on Push (→ Argo)

.github/workflows/nf-on-push.yml triggers on changes under a pipeline path and sends an Argo submit (same pattern).

name: nextflow-on-push
on:
  push:
    branches: [ main ]
    paths:
      - "pipelines/rnaseq/**"

jobs:
  submit:
    if: ${{ github.repository_owner == 'your-org' }}  # optional guard
    runs-on: ubuntu-latest
    steps:
      - name: Determine tenant & params
        run: |
          echo "NS=wf-teamA" >> $GITHUB_ENV
          echo "PIPELINE=pipelines/rnaseq" >> $GITHUB_ENV
          echo "PARAMS=--reads 's3://teamA/dev/*.fastq.gz'" >> $GITHUB_ENV
          echo "ARGO_ADDR=${{ vars.ARGO_ADDR_TEAMA }}" >> $GITHUB_ENV

      - name: Submit to Argo
        env:
          ARGO_TOKEN: ${{ secrets.ARGO_BEARER }}
        run: |
          cat > wf.json <<JSON
          {
            "namespace": "${NS}",
            "resourceKind": "Workflow",
            "resource": {
              "apiVersion": "argoproj.io/v1alpha1",
              "kind": "Workflow",
              "metadata": { "generateName": "nf-" },
              "spec": {
                "workflowTemplateRef": { "name": "nextflow-runner" },
                "arguments": {
                  "parameters": [
                    { "name": "pipeline", "value": "${PIPELINE}" },
                    { "name": "params",   "value": "${PARAMS}" }
                  ]
                }
              }
            }
          }
          JSON
          curl -sS -X POST "$ARGO_ADDR/api/v1/workflows/${NS}/submit" \
            -H "Authorization: Bearer $ARGO_TOKEN" \
            -H "Content-Type: application/json" \
            --data-binary @wf.json

---

6) Parameterization Patterns

• Direct string via inputs.params:
  --reads 's3://bucket/*.fastq.gz' --genome hg38
• Params file in repo (config/params/rnaseq.small.json):
  -params-file config/params/rnaseq.small.json
• Branch-based: choose different params per branch (dev/stage/prod) inside the Action steps.

---

7) Multi-Tenant Conventions

• Namespace = tenant (e.g., wf-teamA), referenced in Actions by the tenant input or mapping.
• Per-tenant WorkflowTemplate (same name: nextflow-runner) mounted with the tenant’s k8s Nextflow config.
• S3 pathing by tenant in k8s config:
  NF_WORKDIR=s3://teamA-nf-workdir/<pipeline>/<run-id>/…
• Keep secrets out of git. Only non-secret tenant mappings live in config/tenants/*.env (or rely entirely on Actions variables).

---

8) Optional: Lint & Validate in CI

Before submitting to Argo, you can add a quick syntax check:

name: lint-and-validate
on: [pull_request]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Nextflow validate
        uses: nextflow-io/gh-action-nextflow@v1
        with:
          args: "validate pipelines/rnaseq"

---

9) Developer Workflow (TL;DR)

1. Create a new pipeline under pipelines/<name>/ with main.nf.
2. Commit tiny test data under testdata/ and sample params under config/params/.
3. Run locally (optional): nextflow run pipelines/rnaseq -profile docker -params-file config/params/rnaseq.small.json
4. Open a PR → CI validates.
5. Manual run: Actions → “Run workflow” → pick tenant, pipeline, params.
6. Results: Argo UI shows Artifacts (report.html, timeline.html, trace.txt) and logs; S3 has the full workDir/results.

---

10) Security Notes

• Use the smallest-scope Argo bearer token; ideally one per tenant.
• Store tokens in Actions Secrets. Put non-secret endpoints in Actions Variables.
• Do not commit AWS keys or NF_WORKDIR values; these belong to K8s Secrets (cluster side).
• Pin container image digests for reproducibility; sign images if possible.

[bwalsh]

Implementation of quickstart: Got a minimal namespace running that can run nextflow
nextflow-argo-poc.zip

[bwalsh]

OIDC-capable (Fence):

• issuer: https://calypr-dev.ohsu.edu/user
• authorization_endpoint: …/oauth2/authorize
• token_endpoint: …/oauth2/token
• jwks_uri: …/.well-known/jwks
• userinfo_endpoint: …/user/

Below are drop-in configs to let Argo CD and Argo Workflows validate tokens via this issuer. I’m giving you a minimal, works-now setup first, then an optional tweak to surface your custom claims (e.g., context.user.name) as the username and groups.

---

1) Argo CD — native OIDC

Minimal (works now)

Uses standard OIDC; username falls back to sub. Add RBAC as needed.

# argocd-cm
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  oidc.config: |
    name: Calypr
    issuer: https://calypr-dev.ohsu.edu/user
    clientID: argocd
    clientSecret: $oidc.argocd.clientSecret
    requestedScopes: ["openid","profile","email"]

# argocd-secret
apiVersion: v1
kind: Secret
metadata:
  name: argocd-secret
  namespace: argocd
type: Opaque
stringData:
  oidc.argocd.clientSecret: "<ARGOCD_CLIENT_SECRET>"

# argocd-rbac-cm (example RBAC)
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.csv: |
    p, role:readonly, applications, get, */*, allow
    g, argo-viewers, role:readonly
  scopes: "[]"

Restart Argo CD server after changes:

kubectl -n argocd rollout restart deploy/argocd-server

Login: browse to Argo CD → Login with Calypr.

---

Optional: surface your custom claims as username/groups

Your tokens expose context.user.name (email-ish) and no built-in groups. If your issuer can add a groups claim, great — Argo CD can use it directly. Otherwise, keep RBAC broad or add groups later.

# argocd-cm (with claim mapping)
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  oidc.config: |
    name: Calypr
    issuer: https://calypr-dev.ohsu.edu/user
    clientID: argocd
    clientSecret: $oidc.argocd.clientSecret
    requestedScopes: ["openid","profile","email","groups"]
    # Prefer custom username claim if present (falls back to sub otherwise)
    usernameClaim: context.user.name
    # If your issuer adds a 'groups' array, uncomment the next line:
    # groupsClaim: groups

And RBAC that keys on groups (once available):

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  scopes: "[groups]"
  policy.csv: |
    g, argo-admins, role:admin
    g, argo-viewers, role:readonly

If your IdP cannot emit groups, you can still map specific users to roles using Argo CD’s user-to-role bindings (with context.user.name), but that’s less scalable.

---

2) Argo Workflows (Argo Server) — SSO

Enable SSO in the server’s config and use the same issuer. This validates tokens via the issuer’s JWKS and maps claims to username/groups.

# workflow-controller-configmap (SSO block is read by the Argo Server)
apiVersion: v1
kind: ConfigMap
metadata:
  name: workflow-controller-configmap
  namespace: argo
data:
  sso: |
    issuer: https://calypr-dev.ohsu.edu/user
    clientId:
      name: argo-server-sso
      key: client-id
    clientSecret:
      name: argo-server-sso
      key: client-secret
    redirectUrl: https://argo.<your-domain>/oauth2/callback
    scopes:
      - openid
      - profile
      - email
      - groups
    rbac:
      enabled: true
    # Optional claim mapping (works on recent Argo):
    usernameClaim: context.user.name
    # groupsClaim: groups

# client credentials for the Argo Server
apiVersion: v1
kind: Secret
metadata:
  name: argo-server-sso
  namespace: argo
type: Opaque
stringData:
  client-id: argo-workflows
  client-secret: "<ARGO_WORKFLOWS_CLIENT_SECRET>"

Start the server with SSO:

• Helm values (or args): server.extraArgs: ["--auth-mode=sso"]

Restart server & controller:

kubectl -n argo rollout restart deploy/argo-workflows-server
kubectl -n argo rollout status deploy/argo-workflows-server

Authorization inside workflow namespaces still relies on Kubernetes RBAC (and, if rbac.enabled: true, Argo can also use the groups claim). Create RoleBindings that grant your users/groups rights in wf-* namespaces.

---

Notes & gotchas

• Your discovery doc lists id_token_signing_alg_values_supported: RS256 and a JWKS URL — perfect; both Argo apps will auto-refresh keys.

• The discovery doesn’t advertise email/groups as supported claims. If you want those, add claim/Scope mapping in your issuer:

  • Map context.user.name → email (and/or preferred_username)
  • Emit a groups array for RBAC (“argo-admins”, “argo-viewers”, …)

• The access token you pasted has audiences like openid, user, etc. For UI/API login, Argo expects a normal OIDC flow that yields an ID token with aud=<client_id> (argocd / argo-workflows). Don’t pass the raw access token to the UI; let the OIDC login happen.

• Time sync matters: ensure cluster nodes’ clocks are correct (iat/exp).

---

Quick validation

1. Browse to Argo CD → choose Calypr → login succeeds.
2. argocd account get-user-info shows your subject (and groups once mapped).
3. Browse to Argo Workflows → login via SSO → you can see/submit in allowed namespaces.

[bwalsh]

The Arborist /user/user endpoint gives you everything you need to call your own authz API per request and then let Argo/Argo CD trust the result.

Below are two clean, working patterns. Pick one.

---

Option 1 — Tiny “authz-adapter” + NGINX Ingress auth

Put a small sidecar service in your cluster that:

1. Receives the incoming request from the Ingress auth hook,
2. Forwards the user’s bearer token to GET https://calypr-dev.ohsu.edu/user/user,
3. Interprets the Arborist JSON,
4. Returns 200 with headers when authorized, or 403 otherwise.

Adapter logic (example policy)

• If active != true → deny
• If authz["/services/workflow/gen3-workflow"] contains an object with "method": "create" → grant runner group
• Else grant viewer group

You can evolve this mapping however you like (path+service+method → groups).

Minimal adapter (Python/Flask)

# app.py
from flask import Flask, request, jsonify, make_response
import requests, time, os

ISSUER_BASE = "https://calypr-dev.ohsu.edu/user"
USERINFO = f"{ISSUER_BASE}/user"
TIMEOUT = 3
app = Flask(__name__)

def decide_groups(doc):
    groups = []
    if not doc.get("active"):
        return groups
    authz = doc.get("authz", {})
    gw = authz.get("/services/workflow/gen3-workflow", [])
    if any(x.get("method") == "create" for x in gw):
        groups.append("argo-runner")
    # everyone active gets viewer
    groups.append("argo-viewer")
    return groups

@app.route("/check", methods=["GET"])
def check():
    tok = request.headers.get("Authorization", "")
    if not tok.startswith("Bearer "):
        return make_response("no token", 401)
    try:
        r = requests.get(USERINFO, headers={"Authorization": tok}, timeout=TIMEOUT)
        if r.status_code != 200:
            return make_response("authz fetch failed", 401)
        doc = r.json()
        if not doc.get("active"):
            return make_response("inactive", 403)
        email = doc.get("email") or doc.get("name") or doc.get("username")
        groups = decide_groups(doc)
        if not groups:
            return make_response("no groups", 403)
        # Pass identity to upstream
        resp = make_response("", 200)
        resp.headers["X-Auth-Request-User"] = email or "unknown"
        resp.headers["X-Auth-Request-Email"] = email or ""
        resp.headers["X-Auth-Request-Groups"] = ",".join(groups)
        return resp
    except Exception:
        return make_response("error", 401)

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

Containerize (or just run as a simple Deployment). Then wire Argo CD and Argo Workflows Ingresses to use it:

NGINX Ingress annotations

metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-url: "http://authz-adapter.security.svc.cluster.local:8080/check"
    nginx.ingress.kubernetes.io/auth-method: GET
    nginx.ingress.kubernetes.io/auth-snippet: |
      proxy_set_header Authorization $http_authorization;
      proxy_set_header X-Original-URI $request_uri;
      proxy_set_header X-Original-Method $request_method;
    # Forward identity headers from adapter to app
    nginx.ingress.kubernetes.io/auth-response-headers: "X-Auth-Request-User,X-Auth-Request-Email,X-Auth-Request-Groups"

Do this on both the argocd-server Ingress and the argo-workflows-server Ingress.

Use the groups inside Argo

Argo CD (map groups to roles):

# argocd-rbac-cm
data:
  scopes: "[X-Auth-Request-Groups]"
  policy.csv: |
    g, argo-runner, role:admin        # or a custom role with create/sync
    g, argo-viewer, role:readonly

Argo Workflows (namespace RBAC):

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: wf-runner, namespace: wf-poc }
subjects:
- kind: Group
  name: argo-runner
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argo-wf-submit-view   # define with get/list/watch on wfs, submit, pods/log, etc.

This gives you dynamic, per-request authorization with no changes to your issuer. The adapter can grow to encode your full matrix (e.g., per-program/project paths in the JSON → distinct groups).

---

Option 2 — Native OIDC + enrich token at login (no proxy)

If you can modify the issuer:

1. At login, have the issuer call GET /user/user,
2. Transform that authz into OIDC groups (e.g., put users with "gen3-workflow":"create" into argo-runner),
3. Emit email and groups in the ID token.

Then configure:

Argo CD

data:
  oidc.config: |
    issuer: https://calypr-dev.ohsu.edu/user
    clientID: argocd
    clientSecret: $oidc.argocd.clientSecret
    requestedScopes: ["openid","profile","email","groups"]
    usernameClaim: email
    groupsClaim: groups
---
data:
  scopes: "[groups]"
  policy.csv: |
    g, argo-runner, role:admin
    g, argo-viewer, role:readonly

Argo Workflows

data:
  sso: |
    issuer: https://calypr-dev.ohsu.edu/user
    clientId: { name: argo-server-sso, key: client-id }
    clientSecret: { name: argo-server-sso, key: client-secret }
    redirectUrl: https://argo.example.com/oauth2/callback
    scopes: [openid, profile, email, groups]
    rbac: { enabled: true }
    usernameClaim: email
    groupsClaim: groups

This is simpler to operate (no proxy), but depends on your IdP being able to map your authz JSON into token claims.

---

Which to choose?

• Need dynamic decisions now? → Option 1 (adapter + Ingress auth).
• Can change issuer claims? → Option 2 (native OIDC with groups).

---

Quick smoke test (Option 1)

1. Deploy the adapter (Service authz-adapter in ns security).
2. Add the Ingress annotations to argocd-server and argo-workflows-server.
3. Hit Argo CD URL with a valid Authorization: Bearer <access_token> header (e.g., via browser if your IdP already sets it or with a test curl against /api/version). You should see 200 only when /user/user returns active:true.
4. In Argo CD, verify your user appears and carries X-Auth-Request-Groups; RBAC should reflect it.

[lbeckman314]

Next Steps 🌀

Tip

Potential next steps to be run (for self):

• Create new K8s cluster in Exastack (i.e. as a "control/system cluster")
• Deploy argo-helm chart to this cluster
• Connect apps + services from other cluster (e.g. CALYPR-dev)