Enable scale tests with 20 pods #4179
Open
+251
−96
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
There was a problem hiding this comment.
Pull request overview
This PR increases the scale test capacity from 15 to 20 pods while improving test reliability through better timeout handling, retry logic, and error reporting. The changes also include infrastructure updates to support the increased pod density and fixes to RBAC propagation issues in storage account setup.
Key Changes
- Scale test now creates 20 pods (10 per cluster) instead of 15 (8+7 split)
- Reduced deletion timeouts from 30-60 attempts to 10 attempts with consistent 20s command timeouts
- Added RBAC propagation verification for storage accounts with retry logic
- Changed default deployment region from centraluseuap to eastus2
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| test/integration/swiftv2/longRunningCluster/datapath_scale_test.go | Increases pod count to 20 (10 per cluster), updates timeouts to 50 minutes, and improves error handling with deferred verification |
| test/integration/swiftv2/longRunningCluster/datapath.go | Adds Reservations and Namespace fields to TestResources struct with default values |
| test/integration/swiftv2/helpers/az_helpers.go | Reduces deletion retry attempts from 30-60 to 10, adds consistent command timeouts, and new deletion error types |
| test/integration/manifests/swiftv2/long-running-cluster/pod-with-device-plugin.yaml | New manifest file defining pod template with device plugin resource requests for vnet-nic |
| hack/aks/Makefile | Adds PODS_PER_NODE variable to aks-nic-secondary-count tag for node pool configuration |
| .pipelines/swiftv2-long-running/template/long-running-pipeline-template.yaml | Adds ScaleTest job, updates storage account RBAC assignment to single account, and adds ScaleTest to DeleteTestResources dependencies |
| .pipelines/swiftv2-long-running/scripts/manage_storage_rbac.sh | Adds RBAC propagation verification with SAS token generation retry logic |
| .pipelines/swiftv2-long-running/scripts/create_storage.sh | Implements retry logic with exponential backoff for blob uploads |
| .pipelines/swiftv2-long-running/scripts/create_aks.sh | Defines PODS_PER_NODE=7 constant and passes it to Makefile |
| .pipelines/swiftv2-long-running/pipeline.yaml | Changes default region from centraluseuap to eastus2 |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
test/integration/swiftv2/longRunningCluster/datapath_scale_test.go
Outdated
Show resolved
Hide resolved
.pipelines/swiftv2-long-running/template/long-running-pipeline-template.yaml
Show resolved
Hide resolved
sivakami-projects
force-pushed
the
sv2-scale-test-improvements
branch
from
d208b89 to
5821b8a
Compare
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: sivakami-projects <126191544+sivakami-projects@users.noreply.github.com>
sivakami-projects
force-pushed
the
sv2-scale-test-improvements
branch
from
5821b8a to
eadef4b
Compare
Contributor
Author
|
/azp run Azure Container Networking PR |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reason for Change:
This PR increases the scale test capacity from 15 to 20 pods while improving test reliability through better timeout handling, retry logic, and error reporting. The changes also include infrastructure updates to support the increased pod density and fixes to RBAC propagation issues in storage account setup.
Key Changes
Scale test now creates 20 pods (10 per cluster) instead of 15 (8+7 split)
Reduced deletion timeouts from 30-60 attempts to 10 attempts with consistent 20s command timeouts
Added RBAC propagation verification for storage accounts with retry logic
Changed scheduled test region from centraluseuap to eastus2
Issue Fixed:
Requirements:
Notes: