-
Notifications
You must be signed in to change notification settings - Fork 218
test: add integration test suite for CloudFormation template validation #742
Description
Summary
The current test suite is entirely unit-level — all CloudFormation compilation is tested by inspecting generated JSON/YAML objects. This means a whole class of bugs can pass unit tests but still fail at deploy time.
The problem
CloudFormation-specific issues like circular resource dependencies, invalid ARN references, or misconfigured IAM policies are only caught when CloudFormation actually tries to resolve the template. No amount of unit testing can reliably catch these, because the bug only manifests when the full dependency graph is evaluated.
A concrete example: #470 (circular dependency when using !Ref to a Lambda function as a Task state resource). The logic looks reasonable in isolation but CloudFormation rejects the generated template. We cannot confidently fix — or even confirm — this bug without being able to deploy a real template.
Proposed solution
Use Serverless Compose to orchestrate a set of minimal fixture services, each representing a specific edge case or known bug scenario.
Structure:
fixtures/
serverless-compose.yml # orchestrates all fixtures
circular-dependency/
serverless.yml # reproduces #470
notifications-policy-merge/
serverless.yml # reproduces #275
lambda-arn-iam/
serverless.yml # reproduces #302
...
Running the full suite:
sls compose deployTargeting a single fixture:
sls compose deploy --service circular-dependencyTeardown:
sls compose removeEach fixture is a real serverless.yml that documents the exact scenario — self-explanatory to anyone reading the repo.
CI setup
CI runs the fixtures against LocalStack in a Docker container. The GitHub Actions workflow pattern is already proven — see BANCS-Norway/serverless-offline-sns — integration.yml as the reference implementation.
This keeps CI self-contained, fast, and free — no AWS costs, no credential management.
Why now
Several open bugs are difficult to reproduce, fix, or verify without this infrastructure. Fixing them without integration tests risks introducing regressions or shipping fixes that don't actually solve the problem.
Related issues
- Circular dependency calling intrinsic function #470 — circular dependency with
!Reflambda resource - Wrong policy state machine generation for lambda arn #302 — wrong IAM policy for lambda ARN with
Fn::Sub - Using notifications overwrites exisiting resouce permissions. #275 — notifications overwrite existing SNS/SQS policy resource instead of merging statements