feat: Solution for multi-glance setup in block only deploy

[timolow]

This PR is focused around adding a means to allow multiple glance servers to operate in an environment that does not provide a backing NFS/CephFS/RBD backend.

To do multi "master" glance we must solution for the following:

• No single point of failure
• Avoid split brain storage
• Allow for scaleable growth

The method for this edge case is rook-ceph running on PVC. As we already need a SC to provide backing storage for SQL/Rabbit and memcache we would already have a SC storage provider setup, a SC does not always need to provide NFS or sometimes we may not want the extra complexity of providing NFS for a single service.

Set up is much the same, in this PR I will demonstrate the setup and operation of rook-ceph with PVC backend.

Provide a backing SC with general SC name, in this example the SC is provided by ISCSI

(genestack) root@genestack-controller1:/opt/genestack/submodules/openstack-helm# kubectl get sc
NAME                   PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
general (default)      org.democratic-csi.iscsi        Delete          Immediate           true                   3h15m

I have setup the policy placement for OSDs to run only on nodes tagged with 'openstack-control-plane'

Deploy the rook-ceph operator as normal

kubectl apply -k /opt/genestack/kustomize/rook-operator/

After operator gets deployed, deploy rook-cluster-external-pvc

kubectl apply -k /opt/genestack/kustomize/rook-cluster-external-pvc/

You will see that OSDs are scheduled on nodes tagged as infrastructure nodes

(genestack) root@genestack-controller1:/opt/genestack/submodules/openstack-helm# kubectl --namespace rook-ceph get pods -o wide  | grep osd
rook-ceph-osd-0-5d67dc9854-8lw6l                                  2/2     Running     0              108m   10.233.64.64    genestack-controller1.lab.underworld.local   <none>           <none>
rook-ceph-osd-1-6cd4dc6db-gv7mk                                   2/2     Running     0              106m   10.233.64.67    genestack-controller2.lab.underworld.local   <none>           <none>
rook-ceph-osd-2-664cd9867b-c2l5k                                  2/2     Running     0              106m   10.233.64.70    genestack-controller3.lab.underworld.local   <none>           <none>

Once fully setup, you can create a modified cephfs (myfs) filesystem for multi-attach glance

kubectl apply -k /opt/genestack/kustomize/rook-defaults-external-pvc/

You will be left with two SCs, general and general-multi-attach

(genestack) root@genestack-controller1:/opt/genestack/submodules/openstack-helm# kubectl get sc -A
NAME                   PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
general (default)      org.democratic-csi.iscsi        Delete          Immediate           true                   3h25m
general-multi-attach   rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   85m

General Glance smoke test of storage system:

(genestack) root@genestack-controller1:/opt/genestack/submodules/openstack-helm# kubectl get pvc -A
NAMESPACE     NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
kube-system   ovndb-backup                      Bound    pvc-f5c0e38e-dafe-4e7a-a625-6fd82399d623   1Gi        RWO            general                88m
openstack     data-memcached-0                  Bound    pvc-ef6a20b6-f928-484b-9281-2c3b8ac96159   10Gi       RWO            general                89m
openstack     data-memcached-1                  Bound    pvc-8e600f51-5221-474d-8dd9-61b89766ae1f   10Gi       RWO            general                89m
openstack     data-memcached-2                  Bound    pvc-c58cad1f-5ba3-487d-b733-5157b31e901b   10Gi       RWO            general                89m
openstack     galera-mariadb-galera-0           Bound    pvc-cd32d6d0-303a-459e-a4f0-707800e33f31   10Gi       RWO            general                99m
openstack     galera-mariadb-galera-1           Bound    pvc-3d36d699-0094-492c-a41c-91f45e6659f4   10Gi       RWO            general                99m
openstack     galera-mariadb-galera-2           Bound    pvc-8fa73f8d-acf7-46d7-9dbb-174acae333bc   10Gi       RWO            general                99m
openstack     glance-images                     Bound    pvc-26295959-d4ec-4181-a94a-8c7e7f0d01b8   10Gi       RWX            general-multi-attach   79m
openstack     mariadb-backup                    Bound    pvc-b2c6b406-c1f1-4156-a202-3d254f34a696   1Gi        RWO            general                97m
openstack     persistence-rabbitmq-server-0     Bound    pvc-6df8022d-c288-405b-a450-2f19c9d6fc9b   5Gi        RWO            general                91m
openstack     persistence-rabbitmq-server-1     Bound    pvc-4d0b21fd-ae98-444a-8ecf-24cd9a16e109   5Gi        RWO            general                91m
openstack     persistence-rabbitmq-server-2     Bound    pvc-0f5c3e87-934a-4c87-8274-315d6e3aef36   5Gi        RWO            general                91m
openstack     postgresql-archive-postgresql-0   Bound    pvc-ed78f2bc-60f1-43a3-932e-621e80e62e88   5Gi        RWO            general                93m
openstack     postgresql-data-postgresql-0      Bound    pvc-fb26c7f1-0c6c-4554-ba83-9ebf1d74a9a8   5Gi        RWO            general                93m
openstack     storage-mariadb-galera-0          Bound    pvc-3b9a7062-ee03-4a2a-80a3-159b56ef2078   10Gi       RWO            general                99m
openstack     storage-mariadb-galera-1          Bound    pvc-20e70a5f-130d-4dde-a0d6-9f2aa2c71e17   10Gi       RWO            general                99m
openstack     storage-mariadb-galera-2          Bound    pvc-1c8ccdc6-f47b-4275-acaf-f0d8d06666b5   10Gi       RWO            general                99m
rook-ceph     rook-ceph-mon-a                   Bound    pvc-0e2a2ce0-ca53-4510-b1cb-062bc151425c   10Gi       RWO            general                116m
rook-ceph     rook-ceph-mon-b                   Bound    pvc-f563eca9-e843-4ca2-822f-d65246cfe853   10Gi       RWO            general                116m
rook-ceph     rook-ceph-mon-c                   Bound    pvc-2d0c5893-2b68-43b1-8fb7-c85af45cd239   10Gi       RWO            general                116m
rook-ceph     set1-data-0cqvz9                  Bound    pvc-8c744d3e-af91-4a4f-b273-bb101acb3dc3   100Gi      RWO            general                111m
rook-ceph     set1-data-1lvmx5                  Bound    pvc-df43af88-2ee1-4f73-b225-6ce3e216242d   100Gi      RWO            general                111m
rook-ceph     set1-data-2whn72                  Bound    pvc-dfe97c71-42cd-4d82-81c9-abc2c68354cc   100Gi      RWO            general                111m
(genestack) root@genestack-controller1:/opt/genestack/submodules/openstack-helm# kubectl --namespace openstack exec -ti openstack-admin-client -- openstack image list
+--------------------------------------+---------------------+--------+
| ID                                   | Name                | Status |
+--------------------------------------+---------------------+--------+
| 273aa51d-a4b7-46a0-aeff-1a9bd4e1d43a | Cirros 0.6.2 64-bit | active |
+--------------------------------------+---------------------+--------+
(genestack) root@genestack-controller1:/opt/genestack/submodules/openstack-helm# kubectl --namespace rook-ceph get cephclusters.ceph.rook.io
NAME        DATADIRHOSTPATH   MONCOUNT   AGE    PHASE   MESSAGE                        HEALTH      EXTERNAL   FSID
rook-ceph   /var/lib/rook     3          129m   Ready   Cluster created successfully   HEALTH_OK              9a6657cd-f3ab-4d70-b276-a05e2ca03e1b

[cloudnull]

👍🏼 lgtm, setup seems fairly straight forward. Can you add a doc page to our storage backends that covers these options?

[timolow]

@cloudnull First pass on documentation done and attached to this PR.

[sulochan]

👍🏽 lgtm