Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OCPBUGS-43027: [release-4.17] config-daemon: Restart all instances of device-plugin #1015

Merged
merged 1 commit into from
Oct 21, 2024
Merged

OCPBUGS-43027: [release-4.17] config-daemon: Restart all instances of device-plugin #1015

merged 1 commit into from
Oct 21, 2024

Conversation

zeeke
Copy link
Contributor

@zeeke zeeke commented Oct 10, 2024

When the operator changes the device-plugin Spec (e.g. .Spec.NodeSelector), it may happen that there are two device plugin pods for a given node, one that is terminating, the other that is initializing.
If the config-daemon executes restartDevicePluginPod() at the same time, it may kill the terminating pod, while the initializing one will run with the old dp configuration. This may cause one or more resources to not being advertised, until a manual device plugin restart occurs.

Make the config-daemon restart all the device-plugin instances it founds for its own node.

4.17 backport of

@zeeke
config-daemon: Restart all instances of device-plugin
af40ada 
When the operator changes the device-plugin Spec (e.g. .Spec.NodeSelector), it may happen
that there are two device plugin pods for a given node, one that is terminating, the other that is
initializing.
If the config-daemon executes `restartDevicePluginPod()` at the same time, it may kill the terminating
pod, while the initializing one will run with the old dp configuration. This may cause one or more resources
to not being advertised, until a manual device plugin restart occurs.

Make the config-daemon restart all the device-plugin instances it founds for its own node.

Signed-off-by: Andrea Panattoni <apanatto@redhat.com>
@zeeke
Copy link
Contributor Author

zeeke commented Oct 10, 2024

/jira cherrypick OCPBUGS-41356

@openshift-ci-robot
Copy link
Contributor

@zeeke: Jira Issue OCPBUGS-41356 has been cloned as Jira Issue OCPBUGS-43027. Will retitle bug to link to clone.
/retitle OCPBUGS-43027: config-daemon: Restart all instances of device-plugin

In response to this:

/jira cherrypick OCPBUGS-41356

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot changed the title config-daemon: Restart all instances of device-plugin OCPBUGS-43027: config-daemon: Restart all instances of device-plugin Oct 10, 2024
@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Oct 10, 2024
@openshift-ci-robot
Copy link
Contributor

@zeeke: This pull request references Jira Issue OCPBUGS-43027, which is invalid:

  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected dependent Jira Issue OCPBUGS-41356 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is MODIFIED instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

When the operator changes the device-plugin Spec (e.g. .Spec.NodeSelector), it may happen that there are two device plugin pods for a given node, one that is terminating, the other that is initializing.
If the config-daemon executes restartDevicePluginPod() at the same time, it may kill the terminating pod, while the initializing one will run with the old dp configuration. This may cause one or more resources to not being advertised, until a manual device plugin restart occurs.

Make the config-daemon restart all the device-plugin instances it founds for its own node.

4.17 backport of

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 10, 2024
@zeeke
Copy link
Contributor Author

zeeke commented Oct 10, 2024

/jira backport release-4.16,release-4.15,release-4.14

@zeeke zeeke changed the title OCPBUGS-43027: config-daemon: Restart all instances of device-plugin OCPBUGS-43027: [release-4.17] config-daemon: Restart all instances of device-plugin Oct 10, 2024
@openshift-ci-robot
Copy link
Contributor

@zeeke: The following backport issues have been created:

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.16
/cherrypick release-4.15
/cherrypick release-4.14

In response to this:

/jira backport release-4.16,release-4.15,release-4.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-cherrypick-robot
Copy link

@openshift-ci-robot: once the present PR merges, I will cherry-pick it on top of release-4.16 in a new PR and assign it to you.

In response to this:

@zeeke: The following backport issues have been created:

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.16
/cherrypick release-4.15
/cherrypick release-4.14

In response to this:

/jira backport release-4.16,release-4.15,release-4.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@zeeke
Copy link
Contributor Author

zeeke commented Oct 10, 2024

/test e2e-telco5g-sriov

@zeeke
Copy link
Contributor Author

zeeke commented Oct 10, 2024

/retest

@zeeke
Copy link
Contributor Author

zeeke commented Oct 17, 2024

/jira refresh

@evgenLevin @bn222 @wizhaoredhat please take a look at this backport

@openshift-ci-robot
Copy link
Contributor

@zeeke: This pull request references Jira Issue OCPBUGS-43027, which is invalid:

  • expected dependent Jira Issue OCPBUGS-41356 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

/jira refresh

@evgenLevin @bn222 @wizhaoredhat please take a look at this backport

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@wizhaoredhat
Copy link
Contributor

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 17, 2024
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Oct 17, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wizhaoredhat, zeeke

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@zeeke
Copy link
Contributor Author

zeeke commented Oct 21, 2024

/jira refresh
/label backport-risk-assessed
@evgenLevin @ajaggapa when you have time, take a look at this backport. I need cherry-pick-approved label

@openshift-ci openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Oct 21, 2024
@openshift-ci-robot openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Oct 21, 2024
@openshift-ci-robot
Copy link
Contributor

@zeeke: This pull request references Jira Issue OCPBUGS-43027, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.17.z) matches configured target version for branch (4.17.z)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-41356 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-41356 targets the "4.18.0" version, which is one of the valid target versions: 4.18.0
  • bug has dependents

Requesting review from QA contact:
/cc @evgenLevin

In response to this:

/jira refresh
/label backport-risk-assessed
@evgenLevin @ajaggapa when you have time, take a look at this backport. I need cherry-pick-approved label

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot removed the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Oct 21, 2024
@evgenLevin
Copy link
Contributor

/label cherry-pick-approved

@openshift-ci openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Oct 21, 2024
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Oct 21, 2024

@zeeke: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot openshift-merge-bot bot merged commit 921d478 into openshift:release-4.17 Oct 21, 2024
12 checks passed
@openshift-ci-robot
Copy link
Contributor

@zeeke: Jira Issue OCPBUGS-43027: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-43027 has been moved to the MODIFIED state.

In response to this:

When the operator changes the device-plugin Spec (e.g. .Spec.NodeSelector), it may happen that there are two device plugin pods for a given node, one that is terminating, the other that is initializing.
If the config-daemon executes restartDevicePluginPod() at the same time, it may kill the terminating pod, while the initializing one will run with the old dp configuration. This may cause one or more resources to not being advertised, until a manual device plugin restart occurs.

Make the config-daemon restart all the device-plugin instances it founds for its own node.

4.17 backport of

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-cherrypick-robot
Copy link

@openshift-ci-robot: new pull request created: #1023

In response to this:

@zeeke: The following backport issues have been created:

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.16
/cherrypick release-4.15
/cherrypick release-4.14

In response to this:

/jira backport release-4.16,release-4.15,release-4.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-cherrypick-robot openshift-cherrypick-robot mentioned this pull request Oct 21, 2024
Open
@openshift-cherrypick-robot
Copy link

@openshift-ci-robot: new pull request created: #1024

In response to this:

@zeeke: The following backport issues have been created:

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.16
/cherrypick release-4.15
/cherrypick release-4.14

In response to this:

/jira backport release-4.16,release-4.15,release-4.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-cherrypick-robot openshift-cherrypick-robot mentioned this pull request Oct 21, 2024
Open
@openshift-cherrypick-robot
Copy link

@openshift-ci-robot: new pull request created: #1025

In response to this:

@zeeke: The following backport issues have been created:

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.16
/cherrypick release-4.15
/cherrypick release-4.14

In response to this:

/jira backport release-4.16,release-4.15,release-4.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-cherrypick-robot openshift-cherrypick-robot mentioned this pull request Oct 21, 2024
Open
@openshift-bot
Copy link
Contributor

[ART PR BUILD NOTIFIER]

Distgit: sriov-network-config-daemon
This PR has been included in build sriov-network-config-daemon-container-v4.17.0-202410211805.p0.g921d478.assembly.stream.el9.
All builds following this will include this PR.

@openshift-bot
Copy link
Contributor

[ART PR BUILD NOTIFIER]

Distgit: sriov-network-operator
This PR has been included in build sriov-network-operator-container-v4.17.0-202410211805.p0.g921d478.assembly.stream.el9.
All builds following this will include this PR.

@openshift-bot
Copy link
Contributor

[ART PR BUILD NOTIFIER]

Distgit: sriov-network-webhook
This PR has been included in build sriov-network-webhook-container-v4.17.0-202410211805.p0.g921d478.assembly.stream.el9.
All builds following this will include this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pliurh pliurh Awaiting requested review from pliurh

@s1061123 s1061123 Awaiting requested review from s1061123

@evgenLevin evgenLevin Awaiting requested review from evgenLevin

Assignees

@mffiedler mffiedler

@rbbratta rbbratta

@anuragthehatter anuragthehatter

@huiran0826 huiran0826

@asood-rh asood-rh

@jechen0648 jechen0648

@evgenLevin evgenLevin

@wizhaoredhat wizhaoredhat

@qiowang721 qiowang721

@ajaggapa ajaggapa

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.