feat(ci): implement tiered CI testing strategy#1311
Merged
Conversation
Replace full E2E on every PR with three test tiers: - PR validation: lint + unit + smoke E2E (chromium only, ~5 min) - Weekly health: full E2E, all 6 browser projects (Mon 6am UTC) - Release gate: full E2E before GitHub Release creation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
CodeAnt AI is reviewing your PR. Thanks for using CodeAnt! 🎉We're free for open-source projects. if you're enjoying it, help us grow by sharing. Share on X · |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review infoConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThis PR refactors GitHub Actions test workflows to separate full end-to-end tests (scheduled weekly) from smoke tests (run on pull requests), introduces a new Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant GitHub as GitHub Actions
participant FullTest as Full-Test Job
participant Release as Create-Release Job
participant Issue as GitHub Issues API
User->>GitHub: Trigger release workflow (with/without skip_tests)
alt skip_tests = 'true'
GitHub->>Release: Skip full-test, proceed to create-release
Release->>Release: Emit warning: Tests skipped
else skip_tests = 'false' (default)
GitHub->>FullTest: Run full-test job (trigger test-full.yml)
FullTest->>FullTest: Checkout, setup Node, run linting<br/>Run E2E tests, upload artifacts
alt Tests Pass
FullTest->>GitHub: result = 'success'
GitHub->>Release: Proceed (needs.full-test.result = 'success')
else Tests Fail
FullTest->>Issue: Create/comment issue with<br/>test failure details & run link
FullTest->>GitHub: result = 'failure'
GitHub->>Release: Block release
end
end
Release->>Release: Create release
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
codeant-ai
bot
added
the
size:L
Contributor
Nitpicks 🔍
|
Contributor
|
CodeAnt AI finished reviewing your PR. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/workflows/release.yml:
- Around line 7-12: Add a visible annotation when the emergency hotfix escape
hatch (workflow_dispatch input skip_tests) is used by updating the
create-release job to emit a warning or summary if tests were bypassed;
specifically detect the test gate outcome (e.g., check needs.full-test.result ==
'skipped' or the job result that indicates skip) and run a step that uses GitHub
Actions logging (warning/notice) or adds a workflow summary entry stating
"Release created with skip_tests=true - full E2E tests were bypassed" so
reviewers and auditors can easily see when the safety gate was ignored.
In `@package.json`:
- Around line 22-23: The package.json contains duplicate scripts
"test:e2e:ci-smoke" and "test:e2e:smoke" that both run "playwright test --config
playwright.smoke.config.ts"; either consolidate them into a single script
(remove one and update references to use the remaining script) or make the
CI-specific script explicit by setting the CI environment variable and/or
SMOKE_TEST_URL when invoking Playwright (e.g., have "test:e2e:ci-smoke" set
CI=true or provide SMOKE_TEST_URL before calling the same command) so the intent
is clear and maintenance overhead is reduced; update any CI/job definitions or
README references to the chosen script name accordingly.
In `@playwright.smoke.config.ts`:
- Around line 43-51: Add a webServer timeout to the conditional webServer config
so builds that hang fail fast; inside the object returned when smokeTestUrl is
falsy (the webServer object that currently contains command, port,
reuseExistingServer), add a timeout property (milliseconds) — e.g., 120000 — to
limit how long "command: 'npm run build && npm run preview'" can run before
Playwright fails the setup.
ℹ️ Review info
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (5)
.github/workflows/release.yml.github/workflows/test-full.yml.github/workflows/test.ymlpackage.jsonplaywright.smoke.config.ts
Add skip_tests warning annotation to release workflow, consolidate duplicate ci-smoke/smoke scripts, and add webServer timeout to prevent hung builds. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
User description
Summary
Implements tiered CI testing to cut PR feedback loops from ~20 min to ~5 min (#1276):
workflow_dispatchfor hotfixesChanges
playwright.smoke.config.ts— conditionalwebServer(local build when noSMOKE_TEST_URL, remote URL when set); addedbasic-workflow.spec.tsto test matchpackage.json— addedtest:e2e:ci-smokescript.github/workflows/test.yml— removedpush: maintrigger, chromium-only, smoke E2E.github/workflows/test-full.yml— new weekly + callable full E2E with dedup issue notification.github/workflows/release.yml— added full E2E gate withskip_testshotfix escapeSibling issue alignment
deploy-prod.yml(chore(release): harden deploy-prod trigger and tag/ref guards #1279),claude.yml(chore(security): tighten Claude workflow trust boundary and permissions #1280),code-health.yml(chore(metrics): replace weekly code-health issue spam with rolling reports #1281), oroctocov.yml(chore(metrics): make Octocov publication non-mutating and failure-visible #1282)automated+ci-failurelabels compatible with chore(metrics): replace weekly code-health issue spam with rolling reports #1281's rolling reports patternTest plan
npm run test:e2e:ci-smokepasses locally (9 tests, 28s, chromium only)workflow_dispatchontest-full.ymlruns all 6 browser projectsskip_tests: truebypasses E2E gateCloses #1276
🤖 Generated with Claude Code
CodeAnt-AI Description
Run fast smoke tests on PRs, weekly full cross-browser E2E, and a gated full-test release workflow
What Changed
Impact
✅ Faster PR feedback✅ Fewer full E2E runs consuming CI minutes✅ Clearer release gating with an emergency bypass💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.