The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2658866	dotnet-runtime	Send tests to Helix (Unix)	Log
2658653	dotnet-dotnet	🛡 1ES PT: AntiMalware MDE Linux Scanner (Binary) (1ES PT)	Log
2658678	dotnet-runtime	Send job to Helix (Unix)	Log
973019	dotnet/runtime	Send tests to Helix (Unix)	Log
2658520	dotnet-runtime	Send job to Helix (Unix)	Log
973020	dotnet/runtime	Send tests to Helix (Windows)	Log
973024	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113294
2658421	dotnet-runtime	Send job to Helix (Unix)	Log
972953	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113008
972957	dotnet/aspnetcore	Run E2E tests	Log	dotnet/aspnetcore#60751
972796	dotnet/runtime	Send tests to Helix (Unix)	Log	dotnet/runtime#113288
972915	dotnet/runtime	AOT compile CoreCLR tests	Log	dotnet/runtime#111967
972879	dotnet/runtime	Checkout dotnet/runtime@refs/pull/111967/merge to s	Log	dotnet/runtime#111967
972861	dotnet/runtime	Send to Helix	Log
2658276	dotnet-dotnet	Build	Log
972742	dotnet/performance	Component Detection	Log	dotnet/performance#4748
2658179	dotnet-sdk	Run Tests in Helix	Log	#48181
972237	dotnet/runtime	Send tests to Helix (Unix)	Log
2658130	dotnet-roslyn	Test	Log
2658132	dotnet-roslyn	Test	Log
2658098	dotnet-runtime	Send job to Helix (Unix)	Log
2658102	dotnet-sdk	Run AoT Tests	Log	#47786
2658101	dotnet-sdk	Run AoT Tests	Log	#47785
2658077	dotnet-roslyn	Test	Log
2658054	dotnet-sdk	Run TestBuild Tests	Log	#47785
972492	dotnet/aspnetcore	Run E2E tests	Log	dotnet/aspnetcore#60410
972445	dotnet/roslyn	Run Integration Tests	Log
972197	dotnet/runtime	Send to Helix	Log
972274	dotnet/runtime	Build product	Log	dotnet/runtime#113248
972222	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#112997
972211	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113141
972238	dotnet/runtime	Send tests to Helix (Windows)	Log
2657881	dotnet-sdk	Run AoT Tests in Helix	Log	#48182
972152	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#111934
972196	dotnet/runtime	Send to Helix	Log
972130	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113267
972189	dotnet/runtime	Send job to Helix (Unix)	Log
972135	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113266
972117	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113237
972113	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113257
2657825	dotnet-dotnet	🛡 Guardian: Publish SDL Logs to Artifacts	Log
2657837	dotnet-runtime	Send job to Helix (Unix)	Log
972051	dotnet/runtime	Send to Helix	Log
972029	dotnet/runtime	Send tests to Helix (Windows)	Log	dotnet/runtime#113242
2657830	dotnet-dotnet-buildtools-prereqs-docker	Build Images (Authenticated)	Log
2657726	dotnet-runtime	NuGetAuthenticate	Log
2657723	dotnet-dotnet-docker	Secure Supply Chain Analysis (auto-injected by policy)	Log
2657715	dotnet-runtime	Initialize containers	Log
2657704	dotnet-runtime	Publish AOT Mono Artifacts	Log
971631	dotnet/runtime	Send tests to Helix (Unix)	Log
2657657	dotnet-dotnet-buildtools-prereqs-docker	Build Images (Authenticated)	Log
2657635	dotnet-runtime	Send job to Helix (Unix)	Log
2657560	dotnet-runtime	Send Utf8JsonWriterFuzzer to OneFuzz	Log
2657470	dotnet-dotnet	Build (centOSStreamContainer)	Log
971565	dotnet/aspnetcore	Run E2E tests	Log	dotnet/aspnetcore#60410
2657333	dotnet-sdk	Run AoT Tests	Log	#47785
2657334	dotnet-sdk	Run AoT Tests	Log	#47786
2657326	dotnet-runtime	Send job to Helix (Unix)	Log
2657361	dotnet-dotnet-monitor	Build	Log
2657363	dotnet-sdk	Run AoT Tests in Helix	Log	#48182
2657362	dotnet-sdk	Run AoT Tests in Helix	Log	#48183
2657364	dotnet-sdk	Run Tests in Helix	Log	#48181
2657324	dotnet-sdk	🛡 Guardian: AntiMalware Scanner (Source)	Log	#47785
2657295	dotnet-sdk	Run Tests in Helix	Log	#48181
2657296	dotnet-sdk	Run Tests in Helix	Log	#48183
2657294	dotnet-sdk	Run Tests in Helix	Log	#48182
2657247	dotnet-sdk	Build	Log
971364	dotnet/aspnetcore	Run E2E tests	Log
971108	dotnet/runtime	Send to Helix	Log
2657012	dotnet-runtime	Send job to Helix (Unix)	Log
971111	dotnet/aspire	Run helix tests	Log
2656969	dotnet-runtime	🛡 Guardian: AntiMalware Scanner (Binary)	Log
2656965	dotnet-runtime	Install MicroBuild plugin	Log
2656950	dotnet-dotnet	🛡 Guardian: ARMory	Log
970994	dotnet/aspnetcore	Run E2E tests	Log	dotnet/aspnetcore#60785
970898	dotnet/aspire	Run helix tests	Log
2656902	dotnet-dotnet	Run Tests	Log
2656866	dotnet-dotnet	Build	Log
970787	dotnet/machinelearning	Build	Log	dotnet/machinelearning#7410
2656786	dotnet-runtime	Send job to Helix (Unix)	Log
970495	dotnet/runtime	Send tests to Helix (Unix)	Log
970636	dotnet/runtime	Send to Helix	Log
970567	dotnet/runtime	Copy native test components to test output folder	Log	dotnet/runtime#113206
970577	dotnet/sdk	Build	Log	dotnet/sdk#47307
2656635	dotnet-runtime	Send job to Helix (Unix)	Log
970457	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113008
969963	dotnet/runtime	Send tests to Helix (Unix)	Log
2656322	dotnet-runtime	Send job to Helix (Unix)	Log
2656244	dotnet-runtime	Build product (android)	Log
2656196	dotnet-runtime	🛡 Guardian: AntiMalware Scanner (Binary)	Log
2656147	dotnet-arcade	Validate Hosted Pool Information (1ES PT)	Log
2656104	dotnet-aspnetcore	Build ARM64 Installers	Log
969909	dotnet/aspnetcore	Run E2E tests	Log	dotnet/aspnetcore#60410
2656123	dotnet-runtime	🛡 Generate SBoM Manifest	Log
2656125	dotnet-dotnet	Build	Log
2656099	dotnet-runtime	Send job to Helix (Unix)	Log
2656110	dotnet-crank	Build and Publish	Log
2656056	dotnet-dotnet-buildtools-prereqs-docker	Auto-baselining mode and SDL pre-checks (1ES PT)	Log
969557	dotnet/aspnetcore	Run E2E tests	Log	dotnet/aspnetcore#60751
2655980	dotnet-runtime	Initialize containers	Log
Displaying 100 of 1197 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
4	133	1138

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)