[BUG] Support bundle kit should respect node selector & taint toleration

[innobead]

Describe the bug (🐛 if you encounter this issue)

It seems there are no node selectors or taint toleration passed to the support bundle kit, so it will cause deploying a support bundle agent in any node in the cluster. Currently, the image only supports arm64/amd64/s390x, so any non-supported arch nodes will have a pending pod to cause the support bundle kit agent daemonset unable to start successfully.

Need to respect the same node selector & taint toleration for system management components.

To Reproduce

#5613 (comment)

Expected behavior

A clear and concise description of what you expected to happen.

Log or Support bundle

If applicable, add the Longhorn managers' log or support bundle when the issue happens.
You can generate a Support Bundle using the link at the footer of the Longhorn UI.

Environment

• Longhorn version: 1.4.0/1.4.1
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl):
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version:
  • Number of management node in the cluster:
  • Number of worker node in the cluster:
• Node config
  • OS type and version:
  • CPU per node:
  • Memory per node:
  • Disk type(e.g. SSD/NVMe):
  • Network bandwidth between the nodes:
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
• Number of Longhorn volumes in the cluster:

Additional context

cc @longhorn/qa for any system management components (instance manager, share manager, backing image, support bundle kit manager), need to verify pod deployment use cases.

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

  • [BUG] Support bundle kit should respect node selector & taint toleration #5614 (comment)
  • test(support-bundle): agent with node-selector and taint-toleration longhorn-tests#1286

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at:

  • chore(support-bundle): version bump #5639
    The PR for the chart change is at:
  • chore(support-bundle): version bump #5639

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at:

  • fix(support-bundle): agent doesn't respect node-selector and taint-toleration longhorn-manager#1798

• Which areas/issues this PR might have potential impacts on?
  Area manager, support-bundle
  Issues

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at:

  • test(support-bundle): agent with node-selector and taint-toleration longhorn-tests#1286
    The automation test case PR is at
  • test(support-bundle): agent with node-selector and taint-toleration longhorn-tests#1286
    The issue of automation test case implementation is at (please create by the template):
  • [TEST][BUG] Support bundle kit should respect node selector & taint toleration #5631

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

  • [BUG] Support bundle kit should respect node selector & taint toleration #5614 (comment)
  • test(support-bundle): agent with node-selector and taint-toleration longhorn-tests#1286

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at:

  • chore(support-bundle): version bump #5747
    The PR for the chart change is at:
  • chore(support-bundle): version bump #5747

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at:

  • fix(agent): tainted nodes should not be counted rancher/support-bundle-kit#70

• Which areas/issues this PR might have potential impacts on?
  Area manager, support-bundle
  Issues

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at:

  • fix(support-bundle): revert master node taint tolerations longhorn-tests#1309
    The automation test case PR is at
  • fix(support-bundle): revert master node taint tolerations longhorn-tests#1309
    The issue of automation test case implementation is at (please create by the template):
  • [TEST][BUG] Support bundle kit should respect node selector & taint toleration #5631

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at

[c3y1huang]

• Fixing test cases to filter LH nodes instead of K8s nodes:
  • fix(support-bundle): fix support-bundle agent test cases longhorn-tests#1311
• Found race conditions when running tests in the pipeline:
  • fix(agent): completed without expected nodes rancher/support-bundle-kit#71
  • chore(support-bundle): version bump #5769

[yangchiu]

Verified passed on master-head (support-bundle-kit v0.0.24). The support bundle kit test cases for node selector and taint toleration can pass, and recently there's no support bundle downloading issue in daily regression.