Skip to content

Commit

Permalink
Merge pull request #615 from red-hat-storage/sync_us--master
Browse files Browse the repository at this point in the history
Syncing latest changes from upstream master for rook
  • Loading branch information
subhamkrai authored Apr 8, 2024
2 parents 498ba23 + 8c8844e commit fe01063
Show file tree
Hide file tree
Showing 8 changed files with 19 additions and 105 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/golangci-lint.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ jobs:
steps:
- uses: actions/setup-go@v5
with:
go-version: "1.21"
go-version: "1.22.2"
check-latest: true
- name: govulncheck
uses: golang/govulncheck-action@v1
86 changes: 10 additions & 76 deletions Documentation/CRDs/Cluster/network-providers.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,27 +79,6 @@ to or from host networking after you update this setting, you will need to
[failover the mons](../../Storage-Configuration/Advanced/ceph-mon-health.md#failing-over-a-monitor)
in order to have mons on the desired network configuration.

## CSI Host Networking

Host networking for CSI pods is controlled independently from CephCluster networking. CSI can be
deployed with host networking or pod networking. CSI uses host networking by default, which is the
recommended configuration. CSI can be forced to use pod networking by setting the operator config
`CSI_ENABLE_HOST_NETWORK: "false"`.
When CSI uses pod networking (`"false"` value), it is critical that `csi-rbdplugin`,
`csi-cephfsplugin`, and `csi-nfsplugin` pods are not deleted or updated without following a special
process outlined below. If one of these pods is deleted, it will cause all existing PVCs on the
pod's node to hang permanently until all application pods are restarted.

The process for updating CSI plugin pods is to follow the following steps on each Kubernetes node
sequentially:
1. `cordon` and `drain` the node
2. When the node is drained, delete the plugin pod on the node (optionally, the node can be rebooted)
3. `uncordon` the node
4. Proceed to the next node when pods on the node rerun and stabilize

For modifications, see [Modifying CSI Networking](#modifying-csi-networking).

## Multus
`network.provider: multus`

Expand All @@ -114,13 +93,6 @@ isolation.

### Multus Prerequisites

These prerequisites apply when:
- CephCluster `network.selector['public']` is specified, AND
- Operator config `CSI_ENABLE_HOST_NETWORK` is `"true"` (or unspecified), AND
- Operator config `CSI_DISABLE_HOLDER_PODS` is `"true"`

If any of the above do not apply, these prerequisites can be skipped.

In order for host network-enabled Ceph-CSI to communicate with a Multus-enabled CephCluster, some
setup is required for Kubernetes hosts.

Expand Down Expand Up @@ -155,10 +127,6 @@ understand and implement these requirements.
need to be an order of magnitude larger (or more) than the host address space to allow the
storage cluster to grow in the future.

If these prerequisites are not achievable, plan to set the Rook operator config
`CSI_ENABLE_HOST_NETWORK: "false"` as documented in the [CSI Host Networking](#csi-host-networking)
section.
### Multus Configuration

Refer to [Multus documentation](https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md)
Expand Down Expand Up @@ -425,28 +393,23 @@ present in the Rook operator namespace must plan to set `CSI_DISABLE_HOLDER_PODS
Rook v1.14 is installed and before v1.16 is installed by following the migration sections below.
CephClusters with no holder pods do not need to follow migration steps.

Helm users will set `csi.disableHolderPods: true` in values.yaml instead of `CSI_DISABLE_HOLDER_PODS`.

CephClusters that do not use `network.provider: multus` can follow the
[Disabling Holder Pods](#disabling-holder-pods) section.

CephClusters that use `network.provider: multus` will need to plan the migration more carefully.
Read the [Disabling Holder Pods with Multus and CSI Host Networking](#disabling-holder-pods-with-multus-and-csi-host-networking)
section. Decide whether to use CSI host networking or not following the sections below.
Read the [Disabling Holder Pods with Multus](#disabling-holder-pods-with-multus) section in full
before beginning.

!!! hint
To determine if holder pods are deployed, use
`kubectl --namespace $ROOK_OPERATOR get pods | grep plugin-holder`

## Modifying CSI networking

### Disabling Holder Pods with Multus and CSI Host Networking
### Disabling Holder Pods with Multus

This migration section applies in the following scenarios:
- CephCluster `network.provider` is `"multus"`, AND
- Operator config `CSI_DISABLE_HOLDER_PODS` is changed to `"true"`, AND
- Operator config `CSI_ENABLE_HOST_NETWORK` is (or is modified to be) `"true"`

If the scenario does not apply, skip ahead to the
[Disabling Holder Pods](#disabling-holder-pods) section below.
This migration section applies when any CephCluster `network.provider` is `"multus"`. If the
scenario does not apply, skip ahead to the [Disabling Holder Pods](#disabling-holder-pods) section.

**Step 1**
Before setting `CSI_ENABLE_HOST_NETWORK: "true"` and `CSI_DISABLE_HOLDER_PODS: "true"`, thoroughly
Expand Down Expand Up @@ -525,16 +488,14 @@ If the above check succeeds for all nodes, proceed with the

### Disabling Holder Pods

This migration section applies when `CSI_DISABLE_HOLDER_PODS` is changed to `"true"`.

**Step 1**
If any CephClusters have Multus enabled (`network.provider: "multus"`), follow the
[Disabling Holder Pods with Multus and CSI Host Networking](#disabling-holder-pods-with-multus-and-csi-host-networking)
[Disabling Holder Pods with Multus](#disabling-holder-pods-with-multus)
steps above before continuing.
**Step 2**
Begin by setting `CSI_DISABLE_HOLDER_PODS: "true"` -- also set the desired value of
`CSI_ENABLE_HOST_NETWORK` if needed.
Begin by setting `CSI_DISABLE_HOLDER_PODS: "true"`. If `CSI_ENABLE_HOST_NETWORK` is set to
`"false"`, also set this value to `"true"` at the same time.

After this, `csi-*plugin-*` pods will restart, and `csi-*plugin-holder-*` pods will remain running.

Expand Down Expand Up @@ -581,30 +542,3 @@ daemonset.apps "csi-rbdplugin-holder-my-cluster" deleted

**Step 6**
The migration is now complete! Congratulations!

### Applying CSI Networking

This migration section applies in the following scenario:
- `CSI_ENABLE_HOST_NETWORK` is modified, AND
- `CSI_DISABLE_HOLDER_PODS` is `"true"`

**Step 1**
If `CSI_DISABLE_HOLDER_PODS` is unspecified or is `"false"`, follow the
[Disabling Holder Pods](#disabling-holder-pods) section first.

**Step 2**
Begin by setting the desired `CSI_ENABLE_HOST_NETWORK` value.

**Step 3**
At this stage, PVCs for running applications are still using the the old network. These PVCs must be
migrated to the new network. Follow the below process to do so.

For each node in the Kubernetes cluster:
1. `cordon` and `drain` the node
2. Wait for all pods to drain
3. `uncordon` and `undrain` the node
4. Wait for the node to be rehydrated and stable
5. Proceed to the next node

**Step 4**
The migration is now complete! Congratulations!
2 changes: 1 addition & 1 deletion Documentation/Upgrade/rook-upgrade.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ The upgrade steps in this guide will clarify what Helm handles automatically.

!!! important
If there are pods named `csi-*plugin-holder-*` in the Rook operator namespace, set the new
config `disableHolderPods: false` in the values.yaml before upgrading to v1.14.
config `csi.disableHolderPods: false` in the values.yaml before upgrading to v1.14.

The `rook-ceph` helm chart upgrade performs the Rook upgrade.
The `rook-ceph-cluster` helm chart upgrade performs a [Ceph upgrade](#ceph-version-upgrades) if the Ceph image is updated.
Expand Down
22 changes: 1 addition & 21 deletions PendingReleaseNotes.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,6 @@
# v1.14 Pending Release Notes
# v1.15 Pending Release Notes

## Breaking Changes

- The minimum supported version of Kubernetes is v1.25.
Upgrade to Kubernetes v1.25 or higher before upgrading Rook.
- The Rook operator config `CSI_ENABLE_READ_AFFINITY` was removed. v1.13 clusters that have modified
this value to be `"true"` must set the option as desired in each CephCluster as documented
[here](https://rook.github.io/docs/rook/v1.14/CRDs/Cluster/ceph-cluster-crd/#csi-driver-options)
before upgrading to v1.14.
- Rook is beginning the process of deprecating CSI network "holder" pods.
If there are pods named `csi-*plugin-holder-*` in the Rook operator namespace, see the
[detailed documentation](./Documentation/CRDs/Cluster/network-providers.md#holder-pod-deprecation)
to disable them. This is optional for v1.14, but will be required in a future release.

## Features

- Kubernetes versions **v1.25** through **v1.29** are supported.
- Ceph daemon pods using the `default` service account now use a new `rook-ceph-default` service account.
- Allow setting the Ceph `application` on a pool
- Create object stores with shared metadata and data pools. Isolation between object stores is enabled via RADOS namespaces.
- The feature support for VolumeSnapshotGroup has been added to the RBD and CephFS CSI driver.
- Support for virtual style hosting for s3 buckets in the CephObjectStore.
- Add option to specify prefix for the OBC provisioner.
- Support Azure Key Vault for storing OSD encryption keys.
- Separate image repository and tag values in the helm chart for the CSI images.
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ require (
go.starlark.net v0.0.0-20231121155337-90ade8b19d09 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/crypto v0.21.0 // indirect
golang.org/x/net v0.22.0 // indirect
golang.org/x/net v0.23.0 // indirect
golang.org/x/oauth2 v0.18.0 // indirect
golang.org/x/sys v0.18.0 // indirect
golang.org/x/term v0.18.0 // indirect
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -1069,8 +1069,8 @@ golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE=
golang.org/x/net v0.19.0/go.mod h1:CfAk/cbD4CthTvqiEl8NpboMuiuOYsAr/7NOjZJtv1U=
golang.org/x/net v0.22.0 h1:9sGLhx7iRIHEiX0oAJ3MRZMUCElJgy7Br1nO+AMN3Tc=
golang.org/x/net v0.22.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg=
golang.org/x/net v0.23.0 h1:7EYJ93RZ9vYSZAIb2x3lnuvqO5zneoD6IvWjuhfxjTs=
golang.org/x/net v0.23.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
Expand Down
2 changes: 1 addition & 1 deletion pkg/apis/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ require (
github.com/ryanuber/go-glob v1.0.0 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
golang.org/x/crypto v0.21.0 // indirect
golang.org/x/net v0.22.0 // indirect
golang.org/x/net v0.23.0 // indirect
golang.org/x/oauth2 v0.18.0 // indirect
golang.org/x/sys v0.18.0 // indirect
golang.org/x/term v0.18.0 // indirect
Expand Down
4 changes: 2 additions & 2 deletions pkg/apis/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -910,8 +910,8 @@ golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE=
golang.org/x/net v0.19.0/go.mod h1:CfAk/cbD4CthTvqiEl8NpboMuiuOYsAr/7NOjZJtv1U=
golang.org/x/net v0.22.0 h1:9sGLhx7iRIHEiX0oAJ3MRZMUCElJgy7Br1nO+AMN3Tc=
golang.org/x/net v0.22.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg=
golang.org/x/net v0.23.0 h1:7EYJ93RZ9vYSZAIb2x3lnuvqO5zneoD6IvWjuhfxjTs=
golang.org/x/net v0.23.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
Expand Down

0 comments on commit fe01063

Please sign in to comment.