Squashed commit of the following:

commit ce5c12b Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Thu Oct 19 08:16:35 2023 -0700 ci: Replace manual install of kubectl with ADO KubectlInstaller task (#2307) * ci: remove kubectl install * ci: add KubectlInstaller for kubectl commit 86e5e13 Author: rjdenney <105380463+rjdenney@users.noreply.github.com> Date: Tue Oct 17 19:41:00 2023 -0400 Adding dualstack overlay support to azure-ipam plugin for Cilium (#2239) * dualstack cilium changes * remove comment * addressing comments and adding unit tests commit 0b45d15 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Tue Oct 17 08:57:42 2023 -0700 ci: Enable multiple runs with the same commitID (#2292) * ci: add unique value based on pipeline runtime Signed-off-by: John Payne <89417863+jpayne3506@users.noreply.github.com> * ci: adjust cluster names * ci: hange commitID on load-test pipeline --------- Signed-off-by: John Payne <89417863+jpayne3506@users.noreply.github.com> commit e62e44e Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Sat Oct 14 18:51:23 2023 -0700 revert: Cilium load test RBAC (#2291) revert: cilium manifest directories Signed-off-by: John Payne <89417863+jpayne3506@users.noreply.github.com> commit 73d8bfb Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Fri Oct 13 14:06:59 2023 -0700 ci: Add NPM arm64 to CNI Load Test Pipeline (#2289) commit ae37d40 Author: Paul Yu <129891899+paulyufan2@users.noreply.github.com> Date: Fri Oct 13 15:08:58 2023 -0400 [CNI] zap logger for platform package (#2233) * zap logger for platform package commit a3ec127 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Thu Oct 12 20:22:35 2023 -0700 ci: Update CNI Release Test pipeline (#2283) * ci: update restart node template * ci: parameterize dropgz image commit 6ddc44c Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Thu Oct 12 16:57:44 2023 -0700 ci: Add control through environment variables for CNI Load Test (#2277) * ci: conditional run logic * add: SCENARIO env var control * ci: change RUN to CNI * ci: move SCENARIO to job level * fix: change env vars to unique values * ci: condition logic and description commit 1190646 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Wed Oct 11 10:31:15 2023 -0700 revert: Latest NPM for integration test (#2263) commit 131cf7a Author: aggarwal0009 <127549148+aggarwal0009@users.noreply.github.com> Date: Mon Oct 9 11:43:01 2023 -0700 Fix: Update pni scope to namespaced (#2282) update pni scope to namespaced commit f46a430 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Mon Oct 9 11:40:53 2023 -0700 ci: Parameterize CNS image repository (#2280) ci: Parameterize CNS image repo commit 5866205 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Fri Oct 6 14:03:00 2023 -0700 fix: change artifact naming convention (#2278) * fix: matching template artifacts * test: jobName Parameter * ci: capture all test namespaces commit f4dd79c Author: Hunter Gregory <42728408+huntergregory@users.noreply.github.com> Date: Thu Oct 5 07:20:25 2023 -0700 test(capz): [WIN-NPM] support containerd 1.7 filesystem (#2267) * style: whitespace in all NPM yamls * fix(capz): first try for containerd 1.7 filesystem change * fix(capz): remove kubeconfig arg and add comment commit 728dbb3 Author: Paul Yu <129891899+paulyufan2@users.noreply.github.com> Date: Wed Oct 4 23:07:07 2023 -0400 ci: v4overlay conformance test cases (#2274) v4overlay conformance test cases commit 7a5cb5e Author: Paul Johnston <35265851+pjohnst5@users.noreply.github.com> Date: Wed Oct 4 18:58:14 2023 -0600 chore: install ip-masq-agent as part of overlay cns scenarios (#2273) commit f061370 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Wed Oct 4 13:44:35 2023 -0700 ci: Agent pool default set at stage level (#2272) commit ae0c08c Author: Paul Yu <129891899+paulyufan2@users.noreply.github.com> Date: Wed Oct 4 14:43:46 2023 -0400 [CNI] zap logger telemetry package (#2266) * zap logger telemetry package commit 64f01b2 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Tue Oct 3 16:51:12 2023 -0700 ci: Add log template to PR and Load Test Pipeline (#2264) * Initial commit * add: log paths and label to priv. daemonset * add: log template * add: log template to load-test yamls * remove: kubeconfig calls * add: capture failed pods on job failure * add: Linux state files * add: Windows state files * style: change terminal output * add: log template to PR pipeline * fix: rebase * style: add comments to log-template * chore: Addressing Comments * add: sub-directories * ci: Only call log-template on fail for PR commit ad3329a Author: Paul Johnston <35265851+pjohnst5@users.noreply.github.com> Date: Tue Oct 3 12:16:37 2023 -0600 chore: must* functions should panic (#2268) commit 35a6f89 Author: Paul Yu <129891899+paulyufan2@users.noreply.github.com> Date: Mon Oct 2 18:22:42 2023 -0400 ci: dualstack overlay windows test cases (#2262) * dualstack overlay windows test cases commit e874736 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Fri Sep 29 14:28:43 2023 -0700 ci: Add HNS restart to Windows CNIv1 in Load Test Pipeline (#2261) * add: HNS restart template * add: re-run all tests after HNS restart * style: add provided test coverage to template commit 4423a94 Author: jshr-w <144164353+jshr-w@users.noreply.github.com> Date: Thu Sep 28 20:17:36 2023 -0700 ci: add CNIv2 testing for different proc/OS (#2230) * ci: proc/os load tests * ci: update deprecated param * ci: update naming * ci: add other proc/os load tests * ci: add other proc/os load tests * remove ubuntu18 test * remove redundant test * fix template call * add infiniband test, arm binary * fix rdma node name, build * update RDMA node count for quota * force pipeline permissions * undo force permissions * remove hardcoding * add os as parameter for cniv1 * set default osSKU * merge variable name change * set default OSSKU * set default OSSKU --------- Signed-off-by: jshr-w <144164353+jshr-w@users.noreply.github.com> commit 9200af8 Author: Timothy J. Raymond <timraymond@users.noreply.github.com> Date: Thu Sep 28 15:36:15 2023 -0400 Add OpenAPI doucmentation to CNS (#1461) Add initial swagger documentation for CNS This adds documentation for a few of the endpoints in CNS. It's not exhaustive, but it's a place to start. commit 78a577c Author: Paul Yu <129891899+paulyufan2@users.noreply.github.com> Date: Wed Sep 27 19:36:48 2023 -0400 v4overlay windows test cases (#2187) * feat: adding in v4overlay windows tests * chore: address feedback * fix: addressing feedback --------- Co-authored-by: Paul Johnston <johnstonpaul801@gmail.com> commit a3e6682 Author: Evan Baker <rbtr@users.noreply.github.com> Date: Wed Sep 27 18:29:39 2023 -0500 submodule dependency updates (#2242) Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> commit f0021b5 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Wed Sep 27 13:23:52 2023 -0700 fix: cluster template env vars (#2258) * fix: cluster templates * cherry-pick: 0c8106a * chore: addressing comments commit 982b794 Author: Jaeryn <13284103+jaer-tsun@users.noreply.github.com> Date: Wed Sep 27 11:49:43 2023 -0700 feat: update cns network container contract for swift 2 (#2250) Co-authored-by: Jaeryn <tsch@microsoft.com> Co-authored-by: Quang Nguyen <nguyenquang@microsoft.com> commit 6251874 Author: Evan Baker <rbtr@users.noreply.github.com> Date: Tue Sep 26 18:18:16 2023 -0500 fix: set cluster auto-upgrade and node auto-upgrade in the hackfile (#2253) Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> commit 0abdc9d Author: Vipul Singh <vipul21sept@gmail.com> Date: Tue Sep 26 14:53:56 2023 -0700 ci: removing the submodules pipeline as PR pipeline covers the checks (#2251) commit e4cefac Author: Evan Baker <rbtr@users.noreply.github.com> Date: Tue Sep 26 16:53:18 2023 -0500 Create initial contrbuting guides (#2244) * start creating contrbuting guides Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> * add CLA instructions Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> --------- Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> commit 53d114c Author: Vipul Singh <vipul21sept@gmail.com> Date: Tue Sep 26 13:33:58 2023 -0700 fix: skipping the k8s e2e unsupported service conformance test (#2255) commit f28fe96 Author: Hunter Gregory <42728408+huntergregory@users.noreply.github.com> Date: Mon Sep 25 17:11:52 2023 -0700 test: [NPM] skip cyclonus for test dir (#2247) Signed-off-by: Hunter Gregory <42728408+huntergregory@users.noreply.github.com> commit beeb66a Author: Paul Johnston <35265851+pjohnst5@users.noreply.github.com> Date: Mon Sep 25 15:40:26 2023 -0600 feat: installing windows cns if windows nodes present (#2246) commit 4943198 Author: Paul Yu <129891899+paulyufan2@users.noreply.github.com> Date: Mon Sep 25 11:59:56 2023 -0400 [CNI] zap logger migration for store package (#2231) * zap logger migration for store package commit 3c6bb62 Author: Evan Baker <rbtr@users.noreply.github.com> Date: Fri Sep 22 17:07:27 2023 -0700 proposal: redesign of CNS IPAM (#2013) proposal: redesign of cns ipam Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> commit 22bc8c7 Author: Rajvi <107083915+rajvinar@users.noreply.github.com> Date: Fri Sep 22 16:02:57 2023 -0700 fixing config.channelmode to execute go routine for Mellanox (#2240) Co-authored-by: Ashvin Deodhar <asdeodha@microsoft.com> commit 5fa9eda Author: rjdenney <105380463+rjdenney@users.noreply.github.com> Date: Fri Sep 22 17:44:37 2023 -0400 Adding overlay v4/dualstack conflists to Windows Dropgz (#2224) adding overlay v4/dualstack conflists to Windows commit 8760107 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Fri Sep 22 13:52:31 2023 -0700 ci: Add CNIv1 Linux to Load Test Pipeline (#2241) * refactor: single cniv1 template * Add: handle multiple nodepools * Add: linux cniv1 validate * ci: move NPM|CNI integration * ci: Update subscription used * addressing comments commit ec519a5 Author: Paul Johnston <35265851+pjohnst5@users.noreply.github.com> Date: Fri Sep 22 10:18:30 2023 -0600 chore: refactor load test suite (#2229) commit a61940d Author: Ashish Nair <nairashu@gmail.com> Date: Thu Sep 21 19:50:29 2023 -0700 feat: Consume the NCStatus to be able to append subnet is full error to Pod IP reservation failures (#2202) * Added Network Conatiner Status to include the latest error code for a Network Container * Updated the crd to have the Status field included into the Network Container * Updated the names and added Status and ErrorText as two fields in NC Status * Fixed the casing and json values for these variables * Propagated the NC Status inside the CNS and IPAM Monitor pool states * Fixed the lint error of missing comma * Saved the updated NC Status into the CNS statefile * Updated the IP assignment to check and error out subnet is Full when there are no more available IPs for CNS to assign * Fixed a minor compilation issue * Fixed lint failures * Fixed lint failures * Removed the reference from the metastate of the ipam monitor * Added Update Success and Update Failed statuses to the NC Status to be able to clearly indicate response status inside the NNC from DNC-RC * Updated the error to use errors pkg instead of fmt * Updating the cns reconcillation logic to skip if there is a failure updating the NC and there are no IPs allocated for the NC * Handled PR comments: * Updated the code to have the NC status be part of the error directly so that it can be consumed by containerD and cx can perform actions on it. * Code update to not use dynamic slices. * Removed the logic which handled 0 IPs allocated to NNC in CNS reconcile Signed-off-by: GitHub <noreply@github.com> * Addressed the PR comment which helped delete a block of code to store ncIDs and also added more error codes to the NCStatus --------- Signed-off-by: GitHub <noreply@github.com> commit 1b22180 Author: Camryn Lee <31013536+camrynl@users.noreply.github.com> Date: Thu Sep 21 07:45:29 2023 -0700 CNI async delete after ReleaseIPAddress (#2232) add second check for releaseIP failure Co-authored-by: Quang Nguyen <nguyenquang@microsoft.com> commit 028e162 Author: Evan Baker <rbtr@users.noreply.github.com> Date: Wed Sep 20 14:58:21 2023 -0700 migrate to patch for NNC spec updates (#2188) migrate to patch for nnc spec updates commit 95cc2d6 Author: Evan Baker <rbtr@users.noreply.github.com> Date: Wed Sep 20 14:36:30 2023 -0700 chore: consolidate core dependencies upgrade (#2213) * ci: bump actions/checkout from 3 to 4 Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v3...v4) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * chore: root dependency updates Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> commit 2791885 Author: Camryn Lee <31013536+camrynl@users.noreply.github.com> Date: Wed Sep 20 10:28:03 2023 -0700 feat: CNS/CNI async delete changes for azure-ipam (#2201) * azure-ipam changes for async delete * cilium cnsconfig change for tests * address comments: update config value and add log line * matching ipam changes to #2183 * include containerID in log msg * update addFile args * return on failure to add file * update go.mod commit c2c59db Author: Paul Johnston <35265851+pjohnst5@users.noreply.github.com> Date: Tue Sep 19 22:05:47 2023 -0600 chore: making installation of CNS daemonset shared (#2227) commit bad286b Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Tue Sep 19 18:40:51 2023 -0700 ci: Enable dropgz-test in PR pipeline (#2116) commit 246fee4 Author: Paul Yu <129891899+paulyufan2@users.noreply.github.com> Date: Tue Sep 19 19:02:28 2023 -0400 [CNI] restruct CNI logger (#2226) * small change for zap log restruct commit 4772008 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Tue Sep 19 12:37:05 2023 -0700 ci: Add CNIv2 Linux to Load Test pipeline (#2141) * Initial Commit * Add sleep for swift cluster * Change NPM|CNI Integration * Addressing comments * Add: NPM continueOnError * Add: Generate logs for NPM * Change NPM Linux branch - long sleep 10s * refactor: linux validate * fix: rebase * Add: maxSkew for noop deployments * Add: Capture improper node restart * Add: Restart CNS case for Cilium * Addressing Comments * Add: Restart CNS template commit 4fa3bf4 Author: Camryn Lee <31013536+camrynl@users.noreply.github.com> Date: Mon Sep 18 21:04:38 2023 -0700 feat: CNS/CNI async pod delete (#2183) * initial changes for cni/cns delete deadlock * add logs and set watcher path * working fswatcher, removing extra debug lines * watcher changes for azure-ipam * remove additional logger from fsnotify and address comments * /deleteIDs directory as part of cnsconfig * add feature flag for async delete * adds some unit test + remove changes for azure-ipam(split pr, dependency conflicts) * update ut * update uts * swift configmap update * fix configmap for test * addressing comments * fix lint * adding cause to connection error struct * connectionerr lint * addressing comments, change watchfs to watcher method * add ctx to releaseIP func * log containerID in failure to add watcher, exit select if context is cancelled * fix logs in network.go after rebase * catch release ip error in invoker_cns.go * retry on failure to release ip * lint fix * rework asyncdelete watcher Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> * include podinterfaceID in file for releaseIP * close file before delete --------- Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> Co-authored-by: Evan Baker <rbtr@users.noreply.github.com> commit 44dc74e Author: Paul Yu <129891899+paulyufan2@users.noreply.github.com> Date: Fri Sep 15 20:14:44 2023 -0400 [CNI] Migrate network and platform package logging to zap (#2209) * network package zap logger * add zaplogger for platform commit c9adf9a Author: rjdenney <105380463+rjdenney@users.noreply.github.com> Date: Fri Sep 15 18:51:13 2023 -0400 Publishing our Windows dropgz version in linux dropgz manifest (#2218) * publishing windows dropgz * adding 2019 * adding os versions to test * fix arm * removing 2019 cni dropgz test commit 7d479b3 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Wed Sep 13 11:00:58 2023 -0700 revert: makefile naming for cluster creation (#2214) * revert: cilium naming * Addressing Comments commit d01947f Author: rjdenney <105380463+rjdenney@users.noreply.github.com> Date: Tue Sep 12 11:49:17 2023 -0400 fix: updating CNI to v1.5.11 for dropgz (#2211) commit c281674 Author: Ramiro <64089641+ramiro-gamarra@users.noreply.github.com> Date: Fri Sep 8 16:15:45 2023 -0700 fix: serviceAccount must be specified at pod spec level in dual stack goldpinger deployments (#2208) service account needs to be specified at the pod spec level, not the container commit 61da686 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Fri Sep 8 14:27:16 2023 -0700 ci: Increase timeout of datapath tests (#2206) commit 210fe86 Author: Ashish Nair <nairashu@gmail.com> Date: Fri Sep 8 11:39:17 2023 -0700 Feat: Created a Network Container Status Section to be updated with the latest error code (#2193) * Added Network Conatiner Status to include the latest error code for a Network Container * Updated the crd to have the Status field included into the Network Container * Updated the names and added Status and ErrorText as two fields in NC Status * Fixed the casing and json values for these variables * Added error code to the NC Status and removed the latest prefix from the varibale names * Removed the timestamp variable from the NC Status * Moved the Status object inside the NC Status to be able to accurately define the status of each NC for the node * Changed to having an enum representing the NC Status which DNC-RC will update after inferring the error and CNS can use this field to propagate and NCRequest failures * Made the validation of the new enum optional to keep it backward compatible commit a90a77e Author: Paul Yu <129891899+paulyufan2@users.noreply.github.com> Date: Fri Sep 8 14:36:19 2023 -0400 restruct zap logger for CNI (#2184) * restruct zap logger commit 666f36c Author: Ramiro <64089641+ramiro-gamarra@users.noreply.github.com> Date: Thu Sep 7 20:39:08 2023 -0700 CNS - Ensuring no stale NCs during NNC reconcile (#2192) * ensuring no stale ncs during nnc reconcile * only save state if mutated * ensuring we only remove stale ncs if none of their ips are assigned commit 1360e02 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Wed Sep 6 18:00:52 2023 -0700 ci: Changing Service Connection for PR pipeline (#2153) * ci: change service connection * add: change ACN PR azureSubscription commit 23e37aa Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Wed Sep 6 11:43:24 2023 -0700 ci: Increase vmSize for dualstack overlay (#2203) fix: Increase vmSize for dualstack overlay Signed-off-by: John Payne <89417863+jpayne3506@users.noreply.github.com> commit e102891 Author: Hunter Gregory <42728408+huntergregory@users.noreply.github.com> Date: Fri Sep 1 11:25:14 2023 -0700 test(cyclonus): [WIN-NPM] fix consistent failure from not sleeping long enough (#2174) * test(cyclonus): [WIN-NPM] fix consistent failure from not sleeping long enough * build: fix syntax * fix: specify namespace in kubectl * fix: wait timeout=5m Signed-off-by: Hunter Gregory <42728408+huntergregory@users.noreply.github.com> --------- Signed-off-by: Hunter Gregory <42728408+huntergregory@users.noreply.github.com> commit 3f313ad Author: Evan Baker <rbtr@users.noreply.github.com> Date: Wed Aug 30 15:54:48 2023 -0500 fix: add OS=windows to manifest-add args (#2194) due to a change in buildx, we need to explicitly pull the windows variant of the windows image that we have previously built. Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> commit 0b743ac Author: Matthew Long <61910737+thatmattlong@users.noreply.github.com> Date: Mon Aug 28 16:11:12 2023 -0700 fix: don't delete an existing conflist (#2115) commit 1116f7e Author: aggarwal0009 <127549148+aggarwal0009@users.noreply.github.com> Date: Mon Aug 28 12:46:55 2023 -0700 Update MTPNC scope (#2186) * update mtpnc scope * update mtpnc scope to Namespaced commit 9ace4b7 Author: Evan Baker <rbtr@users.noreply.github.com> Date: Mon Aug 28 14:45:36 2023 -0500 change total IPs and add secondary IP metric (#2172) * change total IPs and add secondary IP metric Updates the Total IPs metrics to include the NC Primary IP in the total. Adds a Secondary IPs metric which holds the value that the Total IPs previously held: NC Secondary IPs known to CNS which could be used by Pods. Signed-off-by: GitHub <noreply@github.com> * update help wording on IPAM metrics Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> * reword PrimaryIP metric help Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> --------- Signed-off-by: GitHub <noreply@github.com> Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> commit 1adf24a Author: Evan Baker <rbtr@users.noreply.github.com> Date: Mon Aug 28 13:50:01 2023 -0500 Add MTPNC reconciler for cache population in Swift V2 (#2164) add mtpnc watcher Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> commit c93109a Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Fri Aug 25 17:47:05 2023 -0700 ci: Bypass Dualstack E2E on cluster creation failure (#2185) * test: continue on fail * test: continueOnError * test: succeed cluster creation * revert commented job commit d0559f8 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Fri Aug 25 10:00:19 2023 -0700 ci: Improve CreateValidator and restartNetworkCmd reliability (#2181) * ci: Improve CreateValidator and restartNetworkCmd reliability * lint fix * Addressing comments commit 2a468c4 Author: rjdenney <105380463+rjdenney@users.noreply.github.com> Date: Fri Aug 25 12:29:32 2023 -0400 fix: updating CNI to v1.5.10 for dropgz (#2168) change CNI version to v1.5.10 for new dropgz commit 216cc23 Author: aggarwal0009 <127549148+aggarwal0009@users.noreply.github.com> Date: Fri Aug 25 09:08:14 2023 -0700 update swiftv2 crds scope (#2176) commit e789e04 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Thu Aug 24 10:55:23 2023 -0700 ci: update CNS|dropgz version (#2169) * ci: update CNS|dropgz version * Revert cns daemonset to acnpublic commit b1c2508 Author: Paul Yu <129891899+paulyufan2@users.noreply.github.com> Date: Wed Aug 23 21:49:27 2023 -0400 ci: add node and CNS restart test case in dualstack (#2135) * add noderestart test cases commit e5d97bb Author: Diego Becerra <47841864+debecerra@users.noreply.github.com> Date: Wed Aug 23 15:53:46 2023 -0700 Add NotFound error handling to NMAgent client (#2163) This PR makes some minor changes to the NMAgent client package as a part of a larger work item. This commit adds a NotFound method to the error returned by the NMAgent client. It also adds special handling to treat a 400 BadRequest as a NotFound when returned by the DeleteNetwork API call, since this case should be interpreted as a NotFound by the caller. commit 35c6833 Author: aggarwal0009 <127549148+aggarwal0009@users.noreply.github.com> Date: Wed Aug 23 14:22:58 2023 -0700 Ankaggar/consolidate crds (#2171) * organise related swiftv2 CRDs in same GV package * fix linter error * update crdgen * consolidate external swiftv2 crds * fix manifests * more changes * update crdgen * resolve merge conflicts * fix crdgen check failure * further flatten swiftv2 crds * fix UT failure commit e767b15 Author: Quang Nguyen <nguyenquang@microsoft.com> Date: Wed Aug 23 16:04:05 2023 -0400 feat: CNS writes SWIFT conflist (#2110) * cns writes swift conflist * gofumpt ed * var naming * add swift scenario to switch commit 06e3877 Author: Evan Baker <rbtr@users.noreply.github.com> Date: Wed Aug 23 10:58:52 2023 -0500 Upgrade controller-runtime (#2162) * upgrade controller-runtime Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> * update build tools and regen crds Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> * fix import conflict Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> --------- Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> commit b5440dd Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Tue Aug 22 16:46:49 2023 -0700 fix: Change cluster name for cilium nightly pipeline (#2167) commit ef30552 Author: John Payne <89417863+jpayne3506@users.noreply.github.com> Date: Mon Aug 21 16:55:35 2023 -0700 fix: Change argument in load-test template (#2136) * fix: Change argument in load-test template * Addressing Comments commit ec9d41e Author: Paul Yu <129891899+paulyufan2@users.noreply.github.com> Date: Mon Aug 21 15:33:10 2023 -0400 hotfix for adding duplicated routes (#2161) commit 1b2a04a Author: Evan Baker <rbtr@users.noreply.github.com> Date: Mon Aug 21 11:56:43 2023 -0500 feat: stub CNS Pod watcher (#2112) * feat: cns watches pods Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> * indirect pod reconcile for more dynamic behavior Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> --------- Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> commit d17079b Author: Evan Baker <rbtr@users.noreply.github.com> Date: Fri Aug 18 16:49:32 2023 -0500 docs: proposal for async pod delete handling (#2138) * docs: proposal for async pod delete handling Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> * Update docs/feature/async-delete/readme.md Co-authored-by: Tyler Lloyd <tyler@ikq.io> Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> * reword based on PR feedback Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> --------- Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> Co-authored-by: Tyler Lloyd <tyler@ikq.io> commit 626e16c Author: Paul Yu <129891899+paulyufan2@users.noreply.github.com> Date: Fri Aug 18 15:24:55 2023 -0400 ci: add node and CNS restart test case for v4overlay cluster (#2152) * add node restart test case for v4overlay cluster commit 0062d04 Author: tamilmani1989 <tamanoha@microsoft.com> Date: Fri Aug 18 09:14:20 2023 -0700 fix: Remove NLM_F_EXCL flag from Netlink Delete call (#2150) * fix: Remove unix.NLM_F_EXCL from Netlink Delete Route api call unix.NLM_F_EXCL is not expected to set in netlink delete route calls. It's no-op in older kernel and didnt return error. From kernel 5.19+, new flag NLM_F_BULK was defined with same value and serves a purpose in delete route call. This changes breaks azure cni and netlink calls fails in 5.19 kernel and onwards. The fix is to remove setting unix.NLM_F_EXCL in netlink delete route request. * fix: Remove unix.NLM_F_EXCL from Netlink Delete Route api call unix.NLM_F_EXCL is not expected to set in netlink delete route calls. It's no-op in older kernel and didnt return error. From kernel 5.19+, new flag NLM_F_BULK was defined with same value and serves a purpose in delete route call. This changes breaks azure cni and netlink calls fails in 5.19 kernel and onwards. The fix is to remove setting unix.NLM_F_EXCL in netlink delete route request. * Add unit tests for netlink add/delete address and add/delete routes commit ba689eb Author: Evan Baker <rbtr@users.noreply.github.com> Date: Thu Aug 17 14:09:56 2023 -0500 add swift v2 config based on node label (#2144) * add swift v2 config based on node label Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> * add tentative swiftv2 label Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> --------- Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> commit 50422dc Author: aggarwal0009 <127549148+aggarwal0009@users.noreply.github.com> Date: Thu Aug 17 11:54:18 2023 -0700 [Multitenancy]: Add PodNetwork field to MTPNC spec (#2151) * Add PN to MTPNC spec * add annotation * remove omitempty tag commit 4aecfd6 Author: Vipul Singh <vipul21sept@gmail.com> Date: Thu Aug 17 10:31:43 2023 -0700 ci: add packages for submodule trigger (#2154) commit 9a3c50c Author: Behzad Mirkhanzadeh <behzadm@microsoft.com> Date: Wed Aug 16 17:16:55 2023 -0700 perf: 🚀 Increase CNI Lock Timeout to 30 seconds for Linux AKS. (#2101) * Increase Linux Lock Timout to 30 seconds. * addressing the comment by adding a new lock constant for Linux. # Conflicts: # .pipelines/cni/cilium/nightly-release-test.yml
Azure · Oct 19, 2023 · 11193b4 · 11193b4
1 parent 84ccc1a
commit 11193b4
Show file tree

Hide file tree

Showing 263 changed files with 8,732 additions and 5,559 deletions.
diff --git a/.github/workflows/codeql.yaml b/.github/workflows/codeql.yaml
@@ -24,7 +24,7 @@ jobs:
         language: [ 'go' ]
     steps:
     - name: Checkout repository
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
     - name: Initialize CodeQL
       uses: github/codeql-action/init@v2
       with:

diff --git a/.github/workflows/crdgen.yaml b/.github/workflows/crdgen.yaml
@@ -16,7 +16,7 @@ jobs:
     name: CRDs are Generated
     runs-on: ${{ matrix.os }}
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
       with:
         fetch-depth: 0
     - uses: actions/setup-go@v4
@@ -26,13 +26,7 @@ jobs:
       run: make -C crd/nodenetworkconfig
     - name: Regenerate MultitenantNetworkContainer CRD
       run: make -C crd/multitenantnetworkcontainer
-    - name: Regenerate PodNetworkInstance CRD
-      run: make -C crd/external/podnetworkinstance
-    - name: Regenerate PodNetwork CRD
-      run: make -C crd/external/podnetwork
-    - name: Regenerate NodeInfo CRD
-      run: make -C crd/nodeinfo
-    - name: Regenerate MultitenantPodNetworkConfig CRD
-      run: make -C crd/multitenantpodnetworkconfig
+    - name: Regenerate Multitenancy CRDs
+      run: make -C crd/multitenancy
     - name: Fail if the tree is dirty
       run: test -z "$(git status --porcelain)"
diff --git a/.github/workflows/cyclonus-netpol-extended-nightly-test.yaml b/.github/workflows/cyclonus-netpol-extended-nightly-test.yaml
@@ -22,7 +22,7 @@ jobs:
           ]
     steps:
       - name: Checkout
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - uses: actions/setup-go@v4
         with:

diff --git a/.github/workflows/cyclonus-netpol-test.yaml b/.github/workflows/cyclonus-netpol-test.yaml
@@ -8,8 +8,7 @@ on:
   pull_request:
     paths:
       - 'npm/**'
-      - 'github/**'
-      - 'test/**'
+      - '.github/**'
   schedule:
     # run once a day at midnight
     - cron: '0 0 * * *'
@@ -29,7 +28,7 @@ jobs:
           ]
     steps:
       - name: Checkout
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       - uses: actions/setup-go@v4
         with:

diff --git a/.github/workflows/golangci.yaml b/.github/workflows/golangci.yaml
@@ -20,7 +20,7 @@ jobs:
     - uses: actions/setup-go@v4
       with:
         go-version: ${{ matrix.go-version }} 
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
       with:
         fetch-depth: 0
     - name: golangci-lint

diff --git a/.pipelines/cni/cilium/cilium-overlay-load-test-template.yaml b/.pipelines/cni/cilium/cilium-overlay-load-test-template.yaml
@@ -1,17 +1,24 @@
 parameters:
   dependsOn: ""
   name: "cilium"
-  clusterType: "cilium-overlay-up"
+  clusterType: "overlay-byocni-nokubeproxy-up"
   clusterName: "cilium-overlay"
   nodeCount: 10
   vmSize: "Standard_DS4_v2"
+  testDropgz: true
+  os: "linux"
 
-
+# Condition confirms that:
+# Previous job has reported Succeeded. Previous job is currently setup which controls variable assignment and we are dependent on its success.
+# CONTROL_CNI either contains 'cniv1' or 'all'. It is not case sensitive
 stages:
-
-  - stage: createAKScluster
-    dependsOn: ${{ parameters.dependsOn }}
-    displayName: "AKS Cluster with Cilium"
+  - stage: create_${{ parameters.name }}
+    condition: and( succeeded(), or( contains(variables.CONTROL_CNI, 'cilium') , contains(variables.CONTROL_CNI, 'all') ) )
+    variables:
+      commitID: $[ stagedependencies.setup.env.outputs['SetEnvVars.commitID'] ]
+    dependsOn:
+      - setup
+    displayName: "Create Cluster - ${{ parameters.clusterName }}"
     jobs:
       - job: create_aks_cluster_with_${{ parameters.name }}
         pool:
@@ -20,105 +27,109 @@ stages:
           - template: ../load-test-templates/create-cluster-template.yaml
             parameters:
               clusterType: ${{ parameters.clusterType }}
-              clusterName: ${{ parameters.clusterName }}
+              clusterName: ${{ parameters.clusterName }}-$(commitID)
               nodeCount: ${{ parameters.nodeCount }}
               vmSize: ${{ parameters.vmSize }}
-  - stage: install_cilium
-    dependsOn: createAKScluster
-    displayName: "Install Cilium on AKS Overlay"
+              region: $(LOCATION)
+
+# Conditions for below E2E test scenarios confirm that:
+# Pipeline has not been canceled and that the previous job has reports anything other than failure(Succeeded, SuccededWithIssues, Skipped). Previous job is declared by dependsOn:
+# CONTROL_SCENARIO either contains 'all' or its respective scenario 'npm', 'scaleTest', 'restartNode', 'restartCNS'. It is not case sensitive
+# Ex. CONTROL_SCENARIO = "restartnode SCALETEST" will only run Scale Test and Restart Test.
+  - stage: ${{ parameters.name }}
+    variables:
+      commitID: $[ stagedependencies.setup.env.outputs['SetEnvVars.commitID'] ]
+      cnsVersion: $[ stagedependencies.setup.env.outputs['SetEnvVars.cnsVersion'] ]
+      dropgzVersion: $[ stagedependencies.setup.env.outputs['SetEnvVars.dropgzVersion'] ]
+    pool:
+      name: "$(BUILD_POOL_NAME_DEFAULT)"
+    dependsOn:
+      - create_${{ parameters.name }}
+      - setup
+    displayName: "Cilium Test - ${{ parameters.name }}"
     jobs:
       - job: deploy_cilium_components
-        pool:
-          name: "$(BUILD_POOL_NAME_DEFAULT)"
         steps:
           - task: AzureCLI@1
             displayName: "Install Cilium, CNS, and ip-masq-agent"
             inputs:
-              azureSubscription: $(TEST_SUB_SERVICE_CONNECTION)
+              azureSubscription: $(BUILD_VALIDATIONS_SERVICE_CONNECTION)
               scriptLocation: "inlineScript"
               scriptType: "bash"
               addSpnToEnvironment: true
               inlineScript: |
                 set -ex
                 az extension add --name aks-preview
-                make -C ./hack/aks set-kubeconf AZCLI=az CLUSTER=${{ parameters.clusterName }}-$(make revision)
+                make -C ./hack/aks set-kubeconf AZCLI=az CLUSTER=${{ parameters.clusterName }}-$(commitID)
                 ls -lah
                 pwd
                 kubectl cluster-info
                 kubectl get po -owide -A
-                echo "Deploy Azure-CNS"
-                kubectl apply -f test/integration/manifests/cilium/cns-write-ovly.yaml
+
                 echo "deploy Cilium ConfigMap"
                 kubectl apply -f cilium/configmap.yaml
                 kubectl apply -f test/integration/manifests/cilium/cilium-config.yaml
                 echo "install Cilium onto Overlay Cluster"
                 kubectl apply -f test/integration/manifests/cilium/cilium-agent
                 kubectl apply -f test/integration/manifests/cilium/cilium-operator
+                echo "install Cilium ${CILIUM_VERSION_TAG} onto Overlay Cluster"
                 # Passes Cilium image to daemonset and deployment
                 envsubst '${CILIUM_VERSION_TAG},${CILIUM_IMAGE_REGISTRY}' < test/integration/manifests/cilium/daemonset.yaml | kubectl apply -f -
                 envsubst '${CILIUM_VERSION_TAG},${CILIUM_IMAGE_REGISTRY}' < test/integration/manifests/cilium/deployment.yaml | kubectl apply -f -
                 kubectl get po -owide -A
-                echo "deploy ip-masq-agent for overlay"
-                kubectl create -f test/integration/manifests/ip-masq-agent/ip-masq-agent.yaml --validate=false
-                cd test/integration/manifests/ip-masq-agent/
-                kubectl create configmap config-custom.yaml
-                kubectl create configmap config-reconcile.yaml
-                cd ../../../..
+
+                echo "Deploy Azure-CNS"
+                if [ -z $(DROP_GZ_VERSION) ]; then
+                  sudo -E env "PATH=$PATH" make test-integration CNS_VERSION=$(cnsVersion) CNI_DROPGZ_VERSION=$(dropgzVersion) INSTALL_CNS=true INSTALL_AZURE_CNI_OVERLAY=true TEST_DROPGZ=${{ parameters.testDropgz }} CNS_IMAGE_REPO=$(CNS_IMAGE_REPO)
+                else
+                  sudo -E env "PATH=$PATH" make test-integration CNS_VERSION=$(cnsVersion) CNI_DROPGZ_VERSION=$(DROP_GZ_VERSION) INSTALL_CNS=true INSTALL_AZURE_CNI_OVERLAY=true TEST_DROPGZ=${{ parameters.testDropgz }} CNS_IMAGE_REPO=$(CNS_IMAGE_REPO)
+                fi
                 kubectl get po -owide -A
-  - stage: pod_deployment
-    dependsOn: install_cilium
-    displayName: "Pod Deployment"
-    jobs:
       - job: deploy_pods
-        pool:
-          name: "$(BUILD_POOL_NAME_DEFAULT)"
+        condition: and( and( not(canceled()), not(failed()) ), or( contains(variables.CONTROL_SCENARIO, 'scaleTest') , contains(variables.CONTROL_SCENARIO, 'all') ) )
+        displayName: "Scale Test"
+        dependsOn: deploy_cilium_components
         steps:
           - template: ../load-test-templates/pod-deployment-template.yaml
             parameters:
-              clusterName: ${{ parameters.clusterName }}
-              scaleup: 2400
+              clusterName: ${{ parameters.clusterName }}-$(commitID)
+              scaleup: ${SCALEUP_CILIUM}
               os: linux
-  - stage: validate_state
-    dependsOn: pod_deployment
-    displayName: "Validate State"
-    jobs:
-      - job: validate_state
-        pool:
-          name: "$(BUILD_POOL_NAME_DEFAULT)"
-        steps:
+              iterations: ${ITERATIONS_CILIUM}
+              nodeCount: ${{ parameters.nodeCount }}
+              cni: cilium
           - template: ../load-test-templates/validate-state-template.yaml
             parameters:
-              clusterName: ${{ parameters.clusterName }}
-  - stage: restart_nodes
-    dependsOn: validate_state
-    displayName: "Restart Node"
-    jobs:
+              clusterName: ${{ parameters.clusterName }}-$(commitID)
       - job: restart_nodes
-        pool:
-          name: "$(BUILD_POOL_NAME_DEFAULT)"
+        condition: and( and( not(canceled()), not(failed()) ), or( contains(variables.CONTROL_SCENARIO, 'restartNode') , contains(variables.CONTROL_SCENARIO, 'all') ) )
+        displayName: "Restart Test"
+        dependsOn: deploy_pods
         steps:
           - template: ../load-test-templates/restart-node-template.yaml
             parameters:
-              clusterName: ${{ parameters.clusterName }}
-  - stage: validate_restart_state
-    dependsOn: restart_nodes
-    displayName: "Validate Restart State"
-    jobs:
-      - job: validate_restart_state
-        pool:
-          name: "$(BUILD_POOL_NAME_DEFAULT)"
-        steps:
+              clusterName: ${{ parameters.clusterName }}-$(commitID)
+              os: ${{ parameters.os }}
+              cni: cilium
           - template: ../load-test-templates/validate-state-template.yaml
             parameters:
-              clusterName: ${{ parameters.clusterName }}
+              clusterName: ${{ parameters.clusterName }}-$(commitID)
               restartCase: "true"
-  - stage: connectivity_tests
-    dependsOn: validate_restart_state
-    displayName: "Connectivity Tests"
-    jobs:
+      - job: restart_cns
+        condition: and( and( not(canceled()), not(failed()) ), or( contains(variables.CONTROL_SCENARIO, 'restartCNS') , contains(variables.CONTROL_SCENARIO, 'all') ) )
+        displayName: "Restart and Validate CNS"
+        dependsOn: restart_nodes
+        steps:
+          - template: ../load-test-templates/restart-cns-template.yaml
+            parameters:
+              clusterName: ${{ parameters.clusterName }}-$(commitID)
+              os: ${{ parameters.os }}
+              scaleup: ${SCALEUP_CILIUM}
+              nodeCount: ${{ parameters.nodeCount }}
+              cni: cilium
       - job: cni_tests
-        pool:
-          name: "$(BUILD_POOL_NAME_DEFAULT)"
+        displayName: "Cilium Test"
+        dependsOn: restart_cns
         steps:
           - script: |
               echo "install cilium CLI"
@@ -132,13 +143,13 @@ stages:
             displayName: "Install Cilium CLI"
           - task: AzureCLI@1
             inputs:
-              azureSubscription: $(TEST_SUB_SERVICE_CONNECTION)
+              azureSubscription: $(BUILD_VALIDATIONS_SERVICE_CONNECTION)
               scriptLocation: "inlineScript"
               scriptType: "bash"
               addSpnToEnvironment: true
               inlineScript: |
                 set -ex
-                make -C ./hack/aks set-kubeconf AZCLI=az CLUSTER=${{ parameters.clusterName }}-$(make revision)
+                make -C ./hack/aks set-kubeconf AZCLI=az CLUSTER=${{ parameters.clusterName }}-$(commitID)
             name: "GetCluster"
             displayName: "Get AKS Cluster"
           - script: |
@@ -147,33 +158,18 @@ stages:
             retryCountOnTaskFailure: 6
             name: "CiliumConnectivityTests"
             displayName: "Run Cilium Connectivity Tests"
-  - stage: delete
-    displayName: "Delete Resources"
-    dependsOn:
-      - connectivity_tests
-    jobs:
-      - job: delete_resources
-        pool:
-          name: "$(BUILD_POOL_NAME_DEFAULT)"
+      - job: failedE2ELogs
+        displayName: "Failure Logs"
+        dependsOn:
+          - deploy_cilium_components
+          - deploy_pods
+          - restart_nodes
+          - restart_cns
+          - cni_tests
+        condition: failed()
         steps:
-          - task: AzureCLI@1
-            inputs:
-              azureSubscription: $(TEST_SUB_SERVICE_CONNECTION)
-              scriptLocation: "inlineScript"
-              scriptType: "bash"
-              addSpnToEnvironment: true
-              inlineScript: |
-                set -ex
-                if [ "$(DELETE_RESOURCES)" ]
-                then
-                  echo "Deleting Cluster and resource group"
-                  make -C ./hack/aks set-kubeconf AZCLI=az CLUSTER=${{ parameters.clusterName }}-$(make revision)
-                  make -C ./hack/aks azcfg AZCLI=az REGION=$(LOCATION)
-                  make -C ./hack/aks down AZCLI=az REGION=$(LOCATION) SUB=$(SUBSCRIPTION_ID) CLUSTER=${{ parameters.clusterName }}-$(make revision)
-                  echo "Cluster and resources down"
-                else
-                  echo "Deletion of resources is False"
-                fi
-            name: "CleanUpCluster"
-            displayName: "Cleanup cluster"
-    condition: always()
+          - template: ../../templates/log-template.yaml
+            parameters:
+              clusterName: ${{ parameters.clusterName }}-$(commitID)
+              os: linux
+              cni: cilium
diff --git a/.pipelines/cni/cilium/nightly-release-test.yml b/.pipelines/cni/cilium/nightly-release-test.yml
@@ -70,7 +70,7 @@ stages:
         parameters:
           name: cilium_overlay_nightly
           displayName: Cilium on AKS Overlay
-          clusterType: cilium-overlay-up
+          clusterType: overlay-byocni-nokubeproxy-up
           clusterName: ciliumnightly-$(commitID)
           vmSize: Standard_B2ms
           k8sVersion: ""
@@ -88,3 +88,14 @@ stages:
               name: "cilium_nightly"
               testDropgz: ""
               clusterName: ciliumnightly-$(commitID)
+      - job: logs
+        displayName: "Failure Logs"
+        dependsOn:
+          - cilium_nightly
+        condition: failed()
+        steps:
+          - template: ../../templates/log-template.yaml
+            parameters:
+              clusterName: ciliumnightly-$(commitID)
+              os: linux
+              cni: cilium
diff --git a/.pipelines/cni/k8s-e2e/k8s-e2e-job-template.yaml b/.pipelines/cni/k8s-e2e/k8s-e2e-job-template.yaml
@@ -6,8 +6,8 @@ parameters:
 
 
 jobs:
-  - job: cni_k8se2e
-    displayName: "CNI k8s E2E"
+  - job: CNI_${{ parameters.os }}
+    displayName: CNI k8s E2E ${{ parameters.os }}
     dependsOn: ${{ parameters.dependsOn }}
     pool:
       name: $(BUILD_POOL_NAME_DEFAULT)

diff --git a/.pipelines/cni/k8s-e2e/k8s-e2e-step-template.yaml b/.pipelines/cni/k8s-e2e/k8s-e2e-step-template.yaml
@@ -28,6 +28,12 @@ steps:
         kubectl taint nodes -l kubernetes.azure.com/mode=system node-role.kubernetes.io/control-plane:NoSchedule
       fi
 
+      # Taint Windows nodes so that Linux tests do not run on them
+      if ${{ lower(eq(parameters.os, 'linux')) }}
+      then
+        kubectl taint nodes -l kubernetes.azure.com/mode=user node-role.kubernetes.io/control-plane:NoSchedule
+      fi
+
       # Depreciating flags. Change once k8s minimum version supported is > 1.24
       # nodes -> procs
       # flakeAttempts -> flake-attempts
@@ -49,5 +55,11 @@ steps:
       then
         kubectl taint nodes -l kubernetes.azure.com/mode=system node-role.kubernetes.io/control-plane:NoSchedule-
       fi
+
+      # Untaint Windows nodes once testing is complete
+      if ${{ lower(eq(parameters.os, 'linux')) }}
+      then
+        kubectl taint nodes -l kubernetes.azure.com/mode=user node-role.kubernetes.io/control-plane:NoSchedule-
+      fi
     name: ${{ parameters.name }}
     displayName: k8s E2E - ${{ parameters.testName }}