Automated test plan

[edipascale]

No description provided.

[Frostman]

@edipascale could you please make a separate PR with the things refactoring in the testing.go so we can get it merged quickly and you'll not have to rebase

[github-actions]

Test Results

16 tests   11 ✅  1h 31m 40s ⏱️
 3 suites   0 💤
 1 files     5 ❌

For more details on these failures, see this check.

Results for commit d6f53c0.

♻️ This comment has been updated with latest results.

[Frostman]

Great job! I wasn't digging much into each test case, mainly looking for potential issues, unclear behavior, and general Go code issues.

Few notes for fixing in future PRs:

• I have a few questions about making sure that tests that are suitable for the VLAB will run
• failed cleanups in defer may leave fabric in a semi-broken state
• make SKIPed (b/c some requirements are missing) tests not FAIL (let's discuss)
• overall report in the could be a bit more clear (prev item probably contributes a lot to it)

[edipascale]

@Frostman I think I've addressed all of your comments. A couple of standing issues:

• a run now takes ~2h5m in env-1. I tried cutting corners but short of disabling iperfs (which we agreed we cannot afford to do) the only other option I can think of is to skip entire tests. There is an extended flag which is set to false by default, but currently it only disables curls and some extra steps in a few tests. Infra is already there to skip entire tests based on that flag, if desired.
• vlab release-tests in the ci have been enabled using the same label trick, but based on many runs on my local machine I had to disable several tests, i.e. all failover ones, the static external and all of those with "restrictions". Then there are those that won't run because of the lack of externals, and the final result is that there's only a couple of tests left.
• even on real envs, tests seem to be quite brittle. I've had a few runs with no issues, but many more with one/two random failures, typically on a ping check, pretty much on any of the tests with no discernible pattern. I've also seen one failure with a slightly below target iperf speed (8490 vs 8500). Unless I missed something, we are already waiting for switches to be ready with an "applied for" timer of at least 30 seconds in all instances where I've observed failures, but it doesn't seem to be enough. I could try to increase the timer further, but that would make the runs even longer, and regardless I'm not sure where we should set the acceptance bar or how long is reasonable to wait for

Feel free to have a look and/or let me know if you want to discuss this over a call!