Skip to content

test: add integration test suite for CloudFormation template validation #742

@VirtueMe

Description

@VirtueMe

Summary

The current test suite is entirely unit-level — all CloudFormation compilation is tested by inspecting generated JSON/YAML objects. This means a whole class of bugs can pass unit tests but still fail at deploy time.

The problem

CloudFormation-specific issues like circular resource dependencies, invalid ARN references, or misconfigured IAM policies are only caught when CloudFormation actually tries to resolve the template. No amount of unit testing can reliably catch these, because the bug only manifests when the full dependency graph is evaluated.

A concrete example: #470 (circular dependency when using !Ref to a Lambda function as a Task state resource). The logic looks reasonable in isolation but CloudFormation rejects the generated template. We cannot confidently fix — or even confirm — this bug without being able to deploy a real template.

Proposed solution

Use Serverless Compose to orchestrate a set of minimal fixture services, each representing a specific edge case or known bug scenario.

Structure:

fixtures/
  serverless-compose.yml       # orchestrates all fixtures
  circular-dependency/
    serverless.yml             # reproduces #470
  notifications-policy-merge/
    serverless.yml             # reproduces #275
  lambda-arn-iam/
    serverless.yml             # reproduces #302
  ...

Running the full suite:

sls compose deploy

Targeting a single fixture:

sls compose deploy --service circular-dependency

Teardown:

sls compose remove

Each fixture is a real serverless.yml that documents the exact scenario — self-explanatory to anyone reading the repo.

CI setup

CI runs the fixtures against LocalStack in a Docker container. The GitHub Actions workflow pattern is already proven — see BANCS-Norway/serverless-offline-sns — integration.yml as the reference implementation.

This keeps CI self-contained, fast, and free — no AWS costs, no credential management.

Why now

Several open bugs are difficult to reproduce, fix, or verify without this infrastructure. Fixing them without integration tests risks introducing regressions or shipping fixes that don't actually solve the problem.

Related issues

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions