Bug 2223959: osd: make osd pod to sleep when osds are flapping #521
Conversation
|
@sp98: No Bugzilla bug is referenced in the title of this pull request. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
sp98
changed the title
|
@sp98: No Bugzilla bug is referenced in the title of this pull request. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
sp98
changed the title
|
/bugzila refresh |
|
/bugzilla refresh |
|
@agarwal-mudit: Bugzilla bug 2223959 is in a bug group that is not in the allowed groups for this repo.
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/bugzilla refresh |
|
/retest |
openshift-ci
bot
added
bugzilla/severity-high
|
@agarwal-mudit: This pull request references Bugzilla bug 2223959, which is valid. 3 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@openshift-ci[bot]: GitHub didn't allow me to request PR reviews from the following users: nehaberry. Note that only red-hat-storage members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
When OSDs flap, ceph stops the OSD daemon if its marked down greater than 5 times in 600 seconds. But OSD pod restarts and marks the OSD `up` again. This causes the PGs mapped to these OSDs to peer. While the PGs are peering, IO to these PGs are blocked. So we need to ensure that if ceph is marking OSD `down` due to flapping, OSD pod should not restart to mark the OSDs `up` again. This PR adds a sleep to the OSD pod if the container returned with a 0 exit code Default behavior is to sleep for 6 hrs. But user can configure it from the ceph cluster spec. Signed-off-by: sp98 <sapillai@redhat.com> (cherry picked from commit 4eb9f62) Signed-off-by: sp98 <sapillai@redhat.com>
|
/approve |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: sp98, travisn The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@sp98: All pull requests linked via external trackers have merged: Bugzilla bug 2223959 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
When OSDs flap, ceph stops the OSD daemon if its marked down greater than 5 times in 600 seconds. But OSD pod restarts and marks the OSD
upagain. This causes the PGs mapped to these OSDs to peer. While the PGs are peering, IO to these PGs are blocked.So we need to ensure that if ceph is marking OSD
downdue to flapping, OSD pod should not restart to mark the OSDsupagain.This PR adds a sleep to the OSD pod if the container returned with a 0 exit code Default behavior is to sleep for 6 hrs. But user can configure it from the ceph cluster spec.
Signed-off-by: sp98 sapillai@redhat.com
(cherry picked from commit 4eb9f62)
Description of your changes:
Which issue is resolved by this Pull Request:
Resolves #
Checklist:
skip-cion the PR.