Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Production - [Alerting] Android devices disconnected #4846

Closed
dotnet-eng-status bot opened this issue Jan 28, 2025 · 19 comments
Closed

Production - [Alerting] Android devices disconnected #4846

dotnet-eng-status bot opened this issue Jan 28, 2025 · 19 comments
Assignees
@meghnave
Labels
Critical Grafana Alert Issues opened by Grafana Inactive Alert Issues from Grafana alerts that are now "OK" Ops - First Responder Production Tied to the Production environment (as opposed to Staging)

Comments

@dotnet-eng-status
Copy link

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=DNCENGWIN-029} 100

Go to rule

@dotnet/dnceng, @dotnet/prodconsvcs, please investigate

Automation information below, do not change

Grafana-Automated-Alert-Id-35f560112f7a4bfabf9fd69bc1bd76fa
@dotnet-eng-status dotnet-eng-status bot added Active Alert Issues from Grafana alerts that are now active Critical Grafana Alert Issues opened by Grafana Ops - First Responder Production Tied to the Production environment (as opposed to Staging) labels Jan 28, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=DNCENGWIN-026} 100
  • FailureRate {Machine=DNCENGWIN-029} 100
  • FailureRate {Machine=DNCENGWIN-135} 100
  • FailureRate {Machine=DNCENGWIN-136} 100

Go to rule

3 similar comments
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=DNCENGWIN-026} 100
  • FailureRate {Machine=DNCENGWIN-029} 100
  • FailureRate {Machine=DNCENGWIN-135} 100
  • FailureRate {Machine=DNCENGWIN-136} 100

Go to rule

@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=DNCENGWIN-026} 100
  • FailureRate {Machine=DNCENGWIN-029} 100
  • FailureRate {Machine=DNCENGWIN-135} 100
  • FailureRate {Machine=DNCENGWIN-136} 100

Go to rule

@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=DNCENGWIN-026} 100
  • FailureRate {Machine=DNCENGWIN-029} 100
  • FailureRate {Machine=DNCENGWIN-135} 100
  • FailureRate {Machine=DNCENGWIN-136} 100

Go to rule

@meghnave meghnave self-assigned this Jan 30, 2025
@meghnave
Copy link

The machines have been turned offline and ICM created

@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=DNCENGWIN-012} 100
  • FailureRate {Machine=DNCENGWIN-026} 100
  • FailureRate {Machine=DNCENGWIN-029} 100
  • FailureRate {Machine=DNCENGWIN-135} 100
  • FailureRate {Machine=DNCENGWIN-136} 100

Go to rule

2 similar comments
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=DNCENGWIN-012} 100
  • FailureRate {Machine=DNCENGWIN-026} 100
  • FailureRate {Machine=DNCENGWIN-029} 100
  • FailureRate {Machine=DNCENGWIN-135} 100
  • FailureRate {Machine=DNCENGWIN-136} 100

Go to rule

@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=DNCENGWIN-012} 100
  • FailureRate {Machine=DNCENGWIN-026} 100
  • FailureRate {Machine=DNCENGWIN-029} 100
  • FailureRate {Machine=DNCENGWIN-135} 100
  • FailureRate {Machine=DNCENGWIN-136} 100

Go to rule

@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=DNCENGWIN-012} 100

Go to rule

@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=DNCENGWIN-012} 100
  • FailureRate {Machine=DNCENGWIN-014} 100

Go to rule

3 similar comments
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=DNCENGWIN-012} 100
  • FailureRate {Machine=DNCENGWIN-014} 100

Go to rule

@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=DNCENGWIN-012} 100
  • FailureRate {Machine=DNCENGWIN-014} 100

Go to rule

@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=DNCENGWIN-012} 100
  • FailureRate {Machine=DNCENGWIN-014} 100

Go to rule

@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=DNCENGWIN-012} 100
  • FailureRate {Machine=DNCENGWIN-014} 100
  • FailureRate {Machine=DNCENGWIN-036} 100

Go to rule

1 similar comment
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=DNCENGWIN-012} 100
  • FailureRate {Machine=DNCENGWIN-014} 100
  • FailureRate {Machine=DNCENGWIN-036} 100

Go to rule

@meghnave
Copy link

meghnave commented Feb 4, 2025

Created https://portal.microsofticm.com/imp/v5/incidents/details/593172565/summary as well

@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💔 Metric state changed to alerting

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

  • FailureRate {Machine=DNCENGWIN-012} 100
  • FailureRate {Machine=DNCENGWIN-014} 100
  • FailureRate {Machine=DNCENGWIN-036} 87

Go to rule

@dotnet-eng-status dotnet-eng-status bot added Inactive Alert Issues from Grafana alerts that are now "OK" and removed Active Alert Issues from Grafana alerts that are now active labels Feb 5, 2025
@dotnet-eng-status Dotnet Eng Status
Copy link
Author

💚 Metric state changed to ok

Description and instructions for this alert

Please note that this alert will fire every 12 hours as the list of machines can change while the alert is alive. So please keep an eye on the list of machines in the comment.

Go to rule

@meghnave
Copy link

meghnave commented Feb 5, 2025

https://portal.microsofticm.com/imp/v5/incidents/details/593172565/summary resolved and alert cleared, closing.

@meghnave meghnave closed this as completed Feb 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@meghnave meghnave

Labels
Critical Grafana Alert Issues opened by Grafana Inactive Alert Issues from Grafana alerts that are now "OK" Ops - First Responder Production Tied to the Production environment (as opposed to Staging)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@meghnave