Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Production - [Alerting] Android emulator failure rate alert #11605

Closed
dotnet-eng-status bot opened this issue Nov 12, 2022 · 12 comments
Closed

Production - [Alerting] Android emulator failure rate alert #11605

dotnet-eng-status bot opened this issue Nov 12, 2022 · 12 comments
Assignees
@premun
Labels
Active Alert Issues from Grafana alerts that are now active Critical Grafana Alert Issues opened by Grafana Ops - First Responder Production Tied to the Production environment (as opposed to Staging)

Comments

@dotnet-eng-status
Copy link

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=API open / a001OKF} 90
  • FailureRate {Machine=API open / a001OKG} 87
  • FailureRate {Machine=API open / a001OKH} 93
  • FailureRate {Machine=API open / a001OKI} 88
  • FailureRate {Machine=API open / a001OKK} 93
  • FailureRate {Machine=API open / a001OKN} 83
  • FailureRate {Machine=API open / a001OKO} 83
  • FailureRate {Machine=API open / a001OKW} 83
  • FailureRate {Machine=API open / a001OL7} 85
  • FailureRate {Machine=API open / a001OL9} 83
  • FailureRate {Machine=API open / a001OLA} 83
  • FailureRate {Machine=API open / a001OLB} 92
  • FailureRate {Machine=API open / a001OLD} 100
  • FailureRate {Machine=API open / a001OLE} 100
  • FailureRate {Machine=API open / a001OLF} 100
  • FailureRate {Machine=API open / a001OLG} 100
  • FailureRate {Machine=API open / a001OLH} 100
  • FailureRate {Machine=API open / a001OLI} 100
  • FailureRate {Machine=API open / a001OLJ} 100
  • FailureRate {Machine=API open / a001OLK} 100
  • FailureRate {Machine=API open / a001OLL} 100
  • FailureRate {Machine=API open / a001OLM} 100
  • FailureRate {Machine=API open / a001OLN} 100
  • FailureRate {Machine=API open / a001OLO} 100
  • FailureRate {Machine=API open / a001OLP} 100
  • FailureRate {Machine=API open / a001OLQ} 100
  • FailureRate {Machine=API open / a001OLR} 100
  • FailureRate {Machine=API open / a001OLS} 100
  • FailureRate {Machine=API open / a001OLT} 100
  • FailureRate {Machine=API open / a001OLU} 100
  • FailureRate {Machine=API open / a001OLV} 100
  • FailureRate {Machine=API open / a001OLY} 100
  • FailureRate {Machine=API open / a001OM1} 100

Go to rule

@dotnet/dnceng, please investigate

Automation information below, do not change

Grafana-Automated-Alert-Id-e38f14fe3367451d8de43da6e2453fdd
@dotnet-eng-status dotnet-eng-status bot added Active Alert Issues from Grafana alerts that are now active Critical Ops - First Responder Grafana Alert Issues opened by Grafana Production Tied to the Production environment (as opposed to Staging) labels Nov 12, 2022
@dotnet-eng-status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=API open / a001OKI} 92
  • FailureRate {Machine=API open / a001OKK} 81
  • FailureRate {Machine=API open / a001OL7} 88
  • FailureRate {Machine=API open / a001OLF} 100
  • FailureRate {Machine=API open / a001OLH} 100
  • FailureRate {Machine=API open / a001OLO} 100
  • FailureRate {Machine=API open / a001OLP} 90
  • FailureRate {Machine=API open / a001OLQ} 100
  • FailureRate {Machine=API open / a001OLU} 84
  • FailureRate {Machine=API open / a001OLV} 100
  • FailureRate {Machine=API open / a001OLZ} 92
  • FailureRate {Machine=API open / a001OM1} 100
  • FailureRate {Machine=API open / a001OMB} 94

Go to rule

3 similar comments
@dotnet-eng-status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=API open / a001OKI} 92
  • FailureRate {Machine=API open / a001OKK} 81
  • FailureRate {Machine=API open / a001OL7} 88
  • FailureRate {Machine=API open / a001OLF} 100
  • FailureRate {Machine=API open / a001OLH} 100
  • FailureRate {Machine=API open / a001OLO} 100
  • FailureRate {Machine=API open / a001OLP} 90
  • FailureRate {Machine=API open / a001OLQ} 100
  • FailureRate {Machine=API open / a001OLU} 84
  • FailureRate {Machine=API open / a001OLV} 100
  • FailureRate {Machine=API open / a001OLZ} 92
  • FailureRate {Machine=API open / a001OM1} 100
  • FailureRate {Machine=API open / a001OMB} 94

Go to rule

@dotnet-eng-status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=API open / a001OKI} 92
  • FailureRate {Machine=API open / a001OKK} 81
  • FailureRate {Machine=API open / a001OL7} 88
  • FailureRate {Machine=API open / a001OLF} 100
  • FailureRate {Machine=API open / a001OLH} 100
  • FailureRate {Machine=API open / a001OLO} 100
  • FailureRate {Machine=API open / a001OLP} 90
  • FailureRate {Machine=API open / a001OLQ} 100
  • FailureRate {Machine=API open / a001OLU} 84
  • FailureRate {Machine=API open / a001OLV} 100
  • FailureRate {Machine=API open / a001OLZ} 92
  • FailureRate {Machine=API open / a001OM1} 100
  • FailureRate {Machine=API open / a001OMB} 94

Go to rule

@dotnet-eng-status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=API open / a001OKI} 92
  • FailureRate {Machine=API open / a001OKK} 81
  • FailureRate {Machine=API open / a001OL7} 88
  • FailureRate {Machine=API open / a001OLF} 100
  • FailureRate {Machine=API open / a001OLH} 100
  • FailureRate {Machine=API open / a001OLO} 100
  • FailureRate {Machine=API open / a001OLP} 90
  • FailureRate {Machine=API open / a001OLQ} 100
  • FailureRate {Machine=API open / a001OLU} 84
  • FailureRate {Machine=API open / a001OLV} 100
  • FailureRate {Machine=API open / a001OLZ} 92
  • FailureRate {Machine=API open / a001OM1} 100
  • FailureRate {Machine=API open / a001OMB} 94

Go to rule

@premun premun self-assigned this Nov 14, 2022
@premun
Copy link
Member

premun commented Nov 14, 2022

A massive spike happened during the weekend. I will check whether this comes from a specific PR

image

@premun
Copy link
Member

premun commented Nov 14, 2022 •
edited
Loading

These are the sources of the problematic work items

Source JobName Pipeline Stage count_
ci/public/dotnet/runtime/refs/heads/release/7.0 5ec835ed-bbe2-437a-8a00-07f9af9f9d45 runtime-extra-platforms build_Android_x64_Release_AllSubsets_Mono_RuntimeTests_Interp 1904
ci/public/dotnet/runtime/refs/heads/release/7.0 15af78cc-43ec-4ff4-8c24-45e3d49f08f4 runtime-extra-platforms build_Android_x64_Release_AllSubsets_Mono_RuntimeTests 1770
ci/public/dotnet/runtime/refs/heads/release/7.0 f45ab1d6-cd98-4738-836b-9af4778e2b4c runtime-extra-platforms build_Android_x64_Release_AllSubsets_Mono 451
ci/public/dotnet/runtime/refs/heads/release/7.0 39504e54-9ed6-433f-aedf-31999837ce70 runtime-extra-platforms build_Android_arm64_Release_AllSubsets_Mono 442

The run: https://dev.azure.com/dnceng-public/public/_build/results?buildId=81161&view=results

The query:

Metrics
| where EventType == "_MobileDeviceOperation" and MetricName == "ExitCode" and Timestamp > now()-3d
| extend Dimensions = parse_json(Dimensions)
| extend
    ExitCode = MetricValue,
    Command = tostring(Dimensions.command),
    Platform = tostring(Dimensions.platform),
    Target = tostring(Dimensions.target)
| where Platform == "android" and ExitCode == 80 and Target != ""
| join kind=inner Jobs on JobId
| extend Props = parse_json(Properties)
| extend
    Stage=tostring(Props['System.PhaseName']),
    Pipeline=tostring(Props.DefinitionName)    
| summarize count() by Source, JobName, Pipeline, Stage
| order by count_ desc

@premun
Copy link
Member

premun commented Nov 14, 2022

Doesn't seem like an infra issue. The exit code is APP_CRASH and the logs hint the same:

[04:13:01] info: Running instrumentation class net.dot.MonoRunner took 20.2076418 seconds
[04:13:01] dbug: Exit code: 0
                 Std out:
                 INSTRUMENTATION_RESULT: shortMsg=Process crashed.
                 INSTRUMENTATION_CODE: 0
                 
                 
                 
[04:13:01] info: Short message:
                 Process crashed.
[04:13:01] fail: No value for 'return-code' provided in instrumentation result. This may indicate a crashed test (see log)

@premun
Copy link
Member

premun commented Nov 14, 2022

@akoeplinger I might have found something possibly interesting to look into. There was a spike on Friday->Saturday night, Redmond evening, when the runtime-extra-platforms run for the release/7.0 branch failed quite spectacularly. We should watch the next build but something might be broken a lot.

This alert never fires and there are no infra failures by the 3rd attempt for Android emulators (like ever). Literally, this line graph is not a line graph as there are no data points except for Saturday:
image
(also see screenshots above)

All work items end with APP_CRASH with logs as shown above.

@akoeplinger
Copy link
Member

akoeplinger commented Nov 14, 2022 •
edited
Loading

I checked a few of these failures and they all crash with this in the adb log:

11-11 18:46:17.959 19813 19830 D DOTNET  : ((null) warning) Process terminated.
11-11 18:46:17.959 19813 19830 D DOTNET  : ((null) warning) Encountered infinite recursion while looking up resource 'Arg_EntryPointNotFoundException' in System.Private.CoreLib. Verify the installation of .NET is complete and does not need repairing, and that the state of the process has not become corrupted.
11-11 18:46:17.959 19813 19830 F libc    : Fatal signal 6 (SIGABRT), code -1 (SI_QUEUE) in tid 19830 (.dot.MonoRunner), pid 19813 (ot.Common.Tests)

@akoeplinger
Copy link
Member

This is almost certainly because of dotnet/runtime#78018
We don't seem to see this issue in main though, so probably some 7.0 specific issue.

@akoeplinger
Copy link
Member

@premun I've opened a PR to revert the problematic change, thanks for catching it!

@premun premun closed this as completed Nov 14, 2022
@tkapin
Copy link
Member

tkapin commented Nov 14, 2022

Thanks for investigating this @premun and taking action @akoeplinger. Also, happy to see our mobile telemetry helping to identify a product issue.

@premun premun mentioned this issue Nov 14, 2022
Closed
@premun premun mentioned this issue Dec 5, 2022
Closed
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@premun premun

Labels
Active Alert Issues from Grafana alerts that are now active Critical Grafana Alert Issues opened by Grafana Ops - First Responder Production Tied to the Production environment (as opposed to Staging)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@akoeplinger @premun @tkapin