Merge pull request #694 from BlaineEXE/resync-1.15-to-4.17

Resync rook/rook:release-1.15 to release-4.17

.github/workflows/auto-assign.yaml

@@ -10,7 +10,7 @@ jobs:
   assign:
     permissions:
       # write permissions are needed to assign the issue.
-      contents: write
+    issues: write
     name: Run self assign job
     runs-on: ubuntu-latest
     steps:

Documentation/CRDs/Object-Storage/ceph-object-store-crd.md

@@ -65,8 +65,11 @@ spec:
   #zone:
     #name: zone-a
   #hosting:
+  #  advertiseEndpoint:
+  #    dnsName: "mystore.example.com"
+  #    port: 80
+  #    useTls: false
   #  dnsNames:
-  #    - "mystore.example.com"
   #    - "mystore.example.org"
 ```
 
@@ -101,8 +104,7 @@ The gateway settings correspond to the RGW daemon settings.
 * `sslCertificateRef`: If specified, this is the name of the Kubernetes secret(`opaque` or `tls`
     type) that contains the TLS certificate to be used for secure connections to the object store.
     If it is an opaque Kubernetes Secret, Rook will look in the secret provided at the `cert` key name. The value of the `cert` key must be
-    in the format expected by the [RGW
-    service](https://docs.ceph.com/docs/master/install/ceph-deploy/install-ceph-gateway/#using-ssl-with-civetweb):
+    in the format expected by the [RGW service](https://docs.ceph.com/docs/master/install/ceph-deploy/install-ceph-gateway/#using-ssl-with-civetweb):
     "The server key, server certificate, and any other CA or intermediate certificates be supplied in
     one file. Each of these items must be in PEM form." They are scenarios where the certificate DNS is set for a particular domain
     that does not include the local Kubernetes DNS, namely the object store DNS service endpoint. If
@@ -115,7 +117,10 @@ The gateway settings correspond to the RGW daemon settings.
     cluster. Rook will look in the secret provided at the `cabundle` key name.
 * `hostNetwork`: Whether host networking is enabled for the rgw daemon. If not set, the network settings from the cluster CR will be applied.
 * `port`: The port on which the Object service will be reachable. If host networking is enabled, the RGW daemons will also listen on that port. If running on SDN, the RGW daemon listening port will be 8080 internally.
-* `securePort`: The secure port on which RGW pods will be listening. A TLS certificate must be specified either via `sslCerticateRef` or `service.annotations`
+* `securePort`: The secure port on which RGW pods will be listening. A TLS certificate must be
+    specified either via `sslCerticateRef` or `service.annotations`. Refer to
+    [enabling TLS](../../Storage-Configuration/Object-Storage-RGW/object-storage.md#enabling-tls)
+    documentation for more details.
 * `instances`: The number of pods that will be started to load balance this object store.
 * `externalRgwEndpoints`: A list of IP addresses to connect to external existing Rados Gateways
     (works with external mode). This setting will be ignored if the `CephCluster` does not have
@@ -155,9 +160,30 @@ The [zone](../../Storage-Configuration/Object-Storage-RGW/ceph-object-multisite.
 
 ## Hosting Settings
 
-The hosting settings allow you to host buckets in the object store on a custom DNS name, enabling virtual-hosted-style access to buckets similar to AWS S3 (https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html).
-
-* `dnsNames`: a list of DNS names to host buckets on. These names need to valid according RFC-1123. Otherwise it will fail. Each endpoint requires wildcard support like [ingress loadbalancer](https://kubernetes.io/docs/concepts/services-networking/ingress/#hostname-wildcards). Do not include the wildcard itself in the list of hostnames (e.g., use "mystore.example.com" instead of "*.mystore.example.com"). Add all the hostnames like openshift routes otherwise access will be denied, but if the hostname does not support wild card then virtual host style won't work those hostname. By default cephobjectstore service endpoint and custom endpoints from cephobjectzone is included. The feature is supported only for Ceph v18 and later versions.
+`hosting` settings allow specifying object store endpoint configurations. These settings are only
+supported for Ceph v18 and higher.
+
+A common use case that requires configuring hosting is allowing
+[virtual host-style](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html)
+bucket access. This use case is discussed in more detail in
+[Rook object storage docs](../../Storage-Configuration/Object-Storage-RGW/object-storage.md#virtual-host-style-bucket-access).
+
+* `advertiseEndpoint`: By default, Rook advertises the most direct connection to RGWs to dependent
+    resources like CephObjectStoreUsers and ObjectBucketClaims. To advertise a different address
+    (e.g., a wildcard-enabled ingress), define the preferred endpoint here. Default behavior is
+    documented in more detail [here](../../Storage-Configuration/Object-Storage-RGW/object-storage.md#object-store-endpoint)
+    * `dnsName`: The valid RFC-1123 (sub)domain name of the endpoint.
+    * `port`: The nonzero port of the endpoint.
+    * `useTls`: Set to true if the endpoint is HTTPS. False if HTTP.
+* `dnsNames`: When this or `advertiseEndpoint` is set, Ceph RGW will reject S3 client connections
+    who attempt to reach the object store via any unspecified DNS name. Add all DNS names that the
+    object store should accept here. These must be valid RFC-1123 (sub)domain names.
+    Rook automatically adds the known CephObjectStore service DNS name to this list, as well as
+    corresponding CephObjectZone `customEndpoints` (if applicable).
+
+!!! Note
+    For DNS names that support wildcards, do not include wildcards.
+    E.g., use `mystore.example.com` instead of `*.mystore.example.com`.
 
 ## Runtime settings
 

Documentation/CRDs/specification.md

@@ -1915,7 +1915,9 @@ ObjectStoreHostingSpec
 </td>
 <td>
 <em>(Optional)</em>
-<p>Hosting settings for the object store</p>
+<p>Hosting settings for the object store.
+A common use case for hosting configuration is to inform Rook of endpoints that support DNS
+wildcards, which in turn allows virtual host-style bucket addressing.</p>
 </td>
 </tr>
 </table>
@@ -8977,6 +8979,60 @@ and prepares same OSD on that disk</p>
 </tr>
 </tbody>
 </table>
+<h3 id="ceph.rook.io/v1.ObjectEndpointSpec">ObjectEndpointSpec
+</h3>
+<p>
+(<em>Appears on:</em><a href="#ceph.rook.io/v1.ObjectStoreHostingSpec">ObjectStoreHostingSpec</a>)
+</p>
+<div>
+<p>ObjectEndpointSpec represents an object store endpoint</p>
+</div>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>dnsName</code><br/>
+<em>
+string
+</em>
+</td>
+<td>
+<p>DnsName is the DNS name (in RFC-1123 format) of the endpoint.
+If the DNS name corresponds to an endpoint with DNS wildcard support, do not include the
+wildcard itself in the list of hostnames.
+E.g., use &ldquo;mystore.example.com&rdquo; instead of &ldquo;*.mystore.example.com&rdquo;.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>port</code><br/>
+<em>
+int32
+</em>
+</td>
+<td>
+<p>Port is the port on which S3 connections can be made for this endpoint.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>useTls</code><br/>
+<em>
+bool
+</em>
+</td>
+<td>
+<p>UseTls defines whether the endpoint uses TLS (HTTPS) or not (HTTP).</p>
+</td>
+</tr>
+</tbody>
+</table>
 <h3 id="ceph.rook.io/v1.ObjectEndpoints">ObjectEndpoints
 </h3>
 <p>
@@ -9160,18 +9216,40 @@ bool
 <tbody>
 <tr>
 <td>
+<code>advertiseEndpoint</code><br/>
+<em>
+<a href="#ceph.rook.io/v1.ObjectEndpointSpec">
+ObjectEndpointSpec
+</a>
+</em>
+</td>
+<td>
+<em>(Optional)</em>
+<p>AdvertiseEndpoint is the default endpoint Rook will return for resources dependent on this
+object store. This endpoint will be returned to CephObjectStoreUsers, Object Bucket Claims,
+and COSI Buckets/Accesses.
+By default, Rook returns the endpoint for the object store&rsquo;s Kubernetes service using HTTPS
+with <code>gateway.securePort</code> if it is defined (otherwise, HTTP with <code>gateway.port</code>).</p>
+</td>
+</tr>
+<tr>
+<td>
 <code>dnsNames</code><br/>
 <em>
 []string
 </em>
 </td>
 <td>
 <em>(Optional)</em>
-<p>A list of DNS names in which bucket can be accessed via virtual host path. These names need to valid according RFC-1123.
-Each domain requires wildcard support like ingress loadbalancer.
-Do not include the wildcard itself in the list of hostnames (e.g. use &ldquo;mystore.example.com&rdquo; instead of &ldquo;*.mystore.example.com&rdquo;).
-Add all hostnames including user-created Kubernetes Service endpoints to the list.
-CephObjectStore Service Endpoints and CephObjectZone customEndpoints are automatically added to the list.
+<p>A list of DNS host names on which object store gateways will accept client S3 connections.
+When specified, object store gateways will reject client S3 connections to hostnames that are
+not present in this list, so include all endpoints.
+The object store&rsquo;s advertiseEndpoint and Kubernetes service endpoint, plus CephObjectZone
+<code>customEndpoints</code> are automatically added to the list but may be set here again if desired.
+Each DNS name must be valid according RFC-1123.
+If the DNS name corresponds to an endpoint with DNS wildcard support, do not include the
+wildcard itself in the list of hostnames.
+E.g., use &ldquo;mystore.example.com&rdquo; instead of &ldquo;*.mystore.example.com&rdquo;.
 The feature is supported only for Ceph v18 and later versions.</p>
 </td>
 </tr>
@@ -9376,7 +9454,9 @@ ObjectStoreHostingSpec
 </td>
 <td>
 <em>(Optional)</em>
-<p>Hosting settings for the object store</p>
+<p>Hosting settings for the object store.
+A common use case for hosting configuration is to inform Rook of endpoints that support DNS
+wildcards, which in turn allows virtual host-style bucket addressing.</p>
 </td>
 </tr>
 </tbody>

Documentation/Getting-Started/quickstart.md

@@ -36,7 +36,7 @@ To configure the Ceph storage cluster, at least one of these local storage optio
 A simple Rook cluster is created for Kubernetes with the following `kubectl` commands and [example manifests](https://github.com/rook/rook/blob/master/deploy/examples).
 
 ```console
-$ git clone --single-branch --branch master https://github.com/rook/rook.git
+$ git clone --single-branch --branch v1.15.0-beta.0 https://github.com/rook/rook.git
 cd rook/deploy/examples
 kubectl create -f crds.yaml -f common.yaml -f operator.yaml
 kubectl create -f cluster.yaml

Documentation/Storage-Configuration/Monitoring/ceph-monitoring.md

@@ -48,7 +48,7 @@ There are two sources for metrics collection:
 From the root of your locally cloned Rook repo, go the monitoring directory:
 
 ```console
-$ git clone --single-branch --branch master https://github.com/rook/rook.git
+$ git clone --single-branch --branch v1.15.0-beta.0 https://github.com/rook/rook.git
 cd rook/deploy/examples/monitoring
 ```
 

Documentation/Storage-Configuration/Object-Storage-RGW/object-storage.md

@@ -50,7 +50,7 @@ spec:
       codingChunks: 1
   preservePoolsOnDelete: true
   gateway:
-    sslCertificateRef:
+    # sslCertificateRef:
     port: 80
     # securePort: 443
     instances: 1
@@ -159,7 +159,7 @@ spec:
     dataPoolName: rgw-data-pool
     preserveRadosNamespaceDataOnDelete: true
   gateway:
-    sslCertificateRef:
+    # sslCertificateRef:
     port: 80
     instances: 1
 ```
@@ -200,7 +200,7 @@ Then create a secret with the user credentials:
 kubectl -n rook-ceph create secret generic --type="kubernetes.io/rook" rgw-admin-ops-user --from-literal=accessKey=<access key of the user> --from-literal=secretKey=<secret key of the user>
 ```
 
-If you have an external `CephCluster` CR, you can instruct Rook to consume external gateways with the following:
+For an external CephCluster, configure Rook to consume external RGW servers with the following:
 
 ```yaml
 apiVersion: ceph.rook.io/v1
@@ -216,24 +216,35 @@ spec:
         # hostname: example.com
 ```
 
-Use the existing `object-external.yaml` file. Even though multiple endpoints can be specified, it is recommend to use only one endpoint. This endpoint is randomly added to `configmap` of OBC and secret of the `cephobjectstoreuser`. Rook never guarantees the randomly picked endpoint is a working one or not.
-If there are multiple endpoints, please add load balancer in front of them and use the load balancer endpoint in the `externalRgwEndpoints` list.
+See `object-external.yaml` for a more detailed example.
 
-When ready, the message in the `cephobjectstore` status similar to this one:
+Even though multiple `externalRgwEndpoints` can be specified, it is best to use a single endpoint.
+Only the first endpoint in the list will be advertised to any consuming resources like
+CephObjectStoreUsers, ObjectBucketClaims, or COSI resources. If there are multiple external RGW
+endpoints, add load balancer in front of them, then use the single load balancer endpoint in the
+`externalRgwEndpoints` list.
 
-```console
-kubectl -n rook-ceph get cephobjectstore external-store
-NAME                                 PHASE
-external-store                       Ready
+## Object store endpoint
 
-```
+The CephObjectStore resource `status.info` contains `endpoint` (and `secureEndpoint`) fields, which
+report the endpoint that can be used to access the object store as a client. This endpoint is also
+advertised as the default endpoint for CephObjectStoreUsers, ObjectBucketClaims, and
+Container Object Store Interface (COSI) resources.
 
-Any pod from your cluster can now access this endpoint:
+Each object store also creates a Kubernetes service that can be used as a client endpoint from
+within the Kubernetes cluster. The DNS name of the service is
+`rook-ceph-rgw-<objectStoreName>.<objectStoreNamespace>.svc`. This service DNS name is the default
+`endpoint` (and `secureEndpoint`).
 
-```console
-$ curl 10.100.28.138:8080
-<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
-```
+For [external clusters](#connect-to-an-external-object-store), the default endpoint is the first
+`spec.gateway.externalRgwEndpoint` instead of the service DNS name.
+
+The advertised endpoint can be overridden using `advertiseEndpoint` in the
+[`spec.hosting` config](../../CRDs/Object-Storage/ceph-object-store-crd.md#hosting-settings).
+
+Rook always uses the advertised endpoint to perform management operations against the object store.
+When [TLS is enabled](#enable-tls), the TLS certificate must always specify the endpoint DNS name to
+allow secure management operations.
 
 ## Create a Bucket
 
@@ -490,6 +501,82 @@ kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o jsonpa
 kubectl -n rook-ceph get secret rook-ceph-object-user-my-store-my-user -o jsonpath='{.data.SecretKey}' | base64 --decode
 ```
 
+## Enable TLS
+
+TLS is critical for securing object storage data access, and it is assumed as a default by many S3
+clients. TLS is enabled for CephObjectStores by configuring
+[`gateway` options](../../CRDs/Object-Storage/ceph-object-store-crd.md#gateway-settings).
+Set `securePort`, and give Rook access to a TLS certificate using `sslCertificateRef`.
+`caBundleRef` may be necessary as well to give the deployed gateway (RGW) access to the TLS
+certificate's CA signing bundle.
+
+Ceph RGW only supports a **single** TLS certificate. If the given TLS certificate is a concatenation
+of multiple certificates, only the first certificate will be used by the RGW as the server
+certificate. Therefore, the TLS certificate given must include all endpoints that clients will use
+for access as subject alternate names (SANs).
+
+The [CephObjectStore service endpoint](#object-store-endpoint) must be added as a SAN on the TLS
+certificate. If it is not possible to add the service DNS name as a SAN on the TLS certificate,
+set `hosting.advertiseEndpoint` to a TLS-approved endpoint to help ensure Rook and clients use
+secure data access.
+
+!!! note
+    OpenShift users can use add `service.beta.openshift.io/serving-cert-secret-name` as a service
+    annotation instead of using `sslCertificateRef`.
+
+## Virtual host-style Bucket Access
+
+The Ceph Object Gateway supports accessing buckets using
+[virtual host-style](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html)
+addressing, which allows addressing buckets using the bucket name as a subdomain in the endpoint.
+
+AWS has deprecated the the alternative path-style addressing mode which is Rook and Ceph's default.
+As a result, many end-user applications have begun to remove path-style support entirely. Many
+production clusters will have to enable virtual host-style address.
+
+Virtual host-style addressing requires 2 things:
+
+1. An endpoint that supports [wildcard addressing](https://en.wikipedia.org/wiki/Wildcard_DNS_record)
+2. CephObjectStore [hosting](../../CRDs/Object-Storage/ceph-object-store-crd.md#hosting-settings) configuration.
+
+Wildcard addressing can be configured in myriad ways. Some options:
+
+- Kubernetes [ingress loadbalancer](https://kubernetes.io/docs/concepts/services-networking/ingress/#hostname-wildcards)
+- Openshift [DNS operator](https://docs.openshift.com/container-platform/latest/networking/dns-operator.html)
+
+The minimum recommended `hosting` configuration is exemplified below. It is important to ensure that
+Rook advertises the wildcard-addressable endpoint as a priority over the default. TLS is also
+recommended for security, and the configured TLS certificate should specify the advertise endpoint.
+
+```yaml
+spec:
+  ...
+  hosting:
+    advertiseEndpoint:
+      dnsName: my.wildcard.addressable.endpoint.com
+      port: 443
+      useTls: true
+```
+
+A more complex `hosting` configuration is exemplified below. In this example, two
+wildcard-addressable endpoints are available. One is a wildcard-addressable ingress service that is
+accessible to clients outside of the Kubernetes cluster (`s3.ingress.domain.com`). The other is a
+wildcard-addressable Kubernetes cluster service (`s3.rook-ceph.svc`). The cluster service is the
+preferred advertise endpoint because the internal service avoids the possibility of the ingress
+service's router being a bottleneck for S3 client operations.
+
+```yaml
+spec:
+  ...
+  hosting:
+    advertiseEndpoint:
+      dnsName: s3.rook-ceph.svc
+      port: 443
+      useTls: true
+  dnsNames:
+    - s3.ingress.domain.com
+```
+
 ## Object Multisite
 
 Multisite is a feature of Ceph that allows object stores to replicate its data over multiple Ceph clusters.