Merge pull request #833 from tesshuflower/volume_populator_docs

Volume populator docs

CHANGELOG.md

@@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Restic - New option to run a restic unlock before the backup in the next sync.
 - Restic - Allow passing through of RCLONE_ env vars from the restic secret to
   the mover job.
+- Volume Populator added for ReplicationDestinations.
 
 ### Changed
 

docs/usage/index.rst

@@ -14,6 +14,7 @@ Usage
    rsync-tls/index
    syncthing/index
    cli/index
+   volume-populator/index
 
 There are four different replication methods built into VolSync. Choose the method that best fits your use-case:
 
@@ -54,3 +55,8 @@ Metrics
 VolSync :doc:`exposes a number of metrics <metrics/index>` that permit monitoring
 the status of replication relationships via Prometheus.
 
+Volume Populator
+================
+
+VolSync provides a :doc:`Volume Populator <volume-populator/index>` to allow creation of PVCs that reference a
+ReplicationDestination as a dataSourceRef.

docs/usage/rclone/database_example.rst

@@ -9,6 +9,19 @@ First, create the source namespace and deploy the source MySQL database.
 .. code:: console
 
    $ kubectl create ns source
+   $ kubectl annotate namespace source volsync.backube/privileged-movers="true"
+
+.. note::
+    The second command to annotate the namespace is used to enable the rclone data mover to run in privileged mode.
+    This is because this simple example runs MySQL as root. For your own applications, you can run unprivileged by
+    setting the ``moverSecurityContext`` in your ReplicationSource/ReplicationDestination to match that of your
+    application in which case the namespace annotation will not be required. See the
+    :doc:`permission model documentation </usage/permissionmodel>` for more details.
+
+Deploy the source MySQL database.
+
+.. code:: console
+
    $ kubectl create -f examples/source-database/ -n source
 
 Verify the database is running.
@@ -41,6 +54,30 @@ Add a new database.
    > exit
    $ exit
 
+
+Now edit ``examples/rclone/rclone.conf`` with your rclone configuration, or you can deploy minio as object-storage to use
+with the examples.
+
+To start minio in your cluster, run:
+
+.. code:: console
+
+   $ hack/run-minio.sh
+
+If using minio then you can edit ``examples/rclone/rclone.conf`` to match the following:
+
+.. code-block:: none
+    :caption: rclone.conf for use with local minio
+
+    [rclone-bucket]
+    type = s3
+    provider = Minio
+    env_auth = false
+    access_key_id = access
+    secret_access_key = password
+    region = us-east-1
+    endpoint = http://minio.minio.svc.cluster.local:9000
+
 Now, deploy the ``rclone-secret`` followed by ``ReplicationSource`` configuration.
 
 .. code:: console
@@ -78,6 +115,7 @@ on the destination.
 .. code:: console
 
    $ kubectl create ns dest
+   $ kubectl annotate namespace dest volsync.backube/privileged-movers="true"
    $ kubectl create secret generic rclone-secret --from-file=rclone.conf=./examples/rclone/rclone.conf -n dest
    $ kubectl create -f examples/rclone/volsync_v1alpha1_replicationdestination.yaml -n dest
 
@@ -88,20 +126,14 @@ destination side. At the end of the each successful iteration, the ``Replication
 updated with the latest snapshot image.
 
 Now deploy the MySQL database to the ``dest`` namespace which will use the data that has been replicated.
-First we need to identify the latest snapshot from the ``ReplicationDestination`` object. Record the values of
-the latest snapshot as it will be used to create a pvc. Then create the Deployment, Service, PVC,
-and Secret.
 
-Ensure that the next synchronization cycle does not start while the following
-steps are being completed or VolSync may replace the existing snapshot with a
-new one before the database starts.
+The PVC uses the VolSync volume populator feature and sets the ReplicationDestination
+as its dataSourceRef. This will populate the PVC with the latest snapshot contents from the ReplicationDestination.
+
+Create the Deployment, Service, PVC, and Secret.
 
 .. code:: console
 
-   # Get the latest snapshot name
-   $ kubectl get replicationdestination database-destination -n dest --template={{.status.latestImage.name}}
-   # Substitute that name into the database PVC template
-   $ sed -i 's/snapshotToReplace/volsync-dest-database-destination-20201203174504/g' examples/destination-database/mysql-pvc.yaml
    # Start the database
    $ kubectl create -n dest -f examples/destination-database/
 

docs/usage/restic/database_example.rst

@@ -13,11 +13,24 @@ A MySQL database will be used as the example application.
 Creating source PVC to be backed up
 -----------------------------------
 
-Create a namespace called ``source``, and deploy the source MySQL database.
+Create a namespace called ``source``
 
 .. code-block:: console
 
    $ kubectl create ns source
+   $ kubectl annotate namespace source volsync.backube/privileged-movers="true"
+
+.. note::
+    The second command to annotate the namespace is used to enable the restic data mover to run in privileged mode.
+    This is because this simple example runs MySQL as root. For your own applications, you can run unprivileged by
+    setting the ``moverSecurityContext`` in your ReplicationSource/ReplicationDestination to match that of your
+    application in which case the namespace annotation will not be required. See the
+    :doc:`permission model documentation </usage/permissionmodel>` for more details.
+
+Deploy the source MySQL database.
+
+.. code:: console
+
    $ kubectl -n source create -f examples/source-database/
 
 Verify the database is running:
@@ -205,6 +218,7 @@ To restore from the backup, create a destination, deploy ``restic-config`` and
 .. code-block:: console
 
    $ kubectl create ns dest
+   $ kubectl annotate namespace dest volsync.backube/privileged-movers="true"
    $ kubectl -n dest create -f examples/restic/source-restic/
 
 To start the restore, create a empty PVC for the data:
@@ -242,9 +256,13 @@ Once the restore is complete, the ``.status.lastManualSync`` field will match
 To verify restore, deploy the MySQL database to the ``dest`` namespace which will use the data that has
 been restored from sourcePVC backup.
 
+Create the Deployment, Service, and Secret.
+
 .. code-block:: console
 
-   $ kubectl create -n dest -f examples/destination-database/
+   $ kubectl create -n dest -f examples/destination-database/mysql-secret.yaml
+   $ kubectl create -n dest -f examples/destination-database/mysql-deployment.yaml
+   $ kubectl create -n dest -f examples/destination-database/mysql-service.yaml
 
 Validate that the mysql pod is running within the environment.
 

docs/usage/rsync/database_example.rst

@@ -113,14 +113,14 @@ lines:
 
    Status:
      Conditions:
-       Last Transition Time:  2020-12-03T16:07:35Z
+       Last Transition Time:  2023-08-03T16:07:35Z
        Message:               Reconcile complete
        Reason:                ReconcileComplete
        Status:                True
        Type:                  Reconciled
      Last Sync Duration:      4.511334577s
-     Last Sync Time:          2020-12-03T16:09:04Z
-     Next Sync Time:          2020-12-03T16:10:00Z
+     Last Sync Time:          2023-08-03T16:09:04Z
+     Next Sync Time:          2023-08-03T16:10:00Z
 
 We will modify the source database by creating an additional database in the
 mysql pod running in the source namespace.
@@ -151,18 +151,22 @@ the destination.
 Now the mysql database will be deployed to the destination namespace which will use the
 data that has been replicated.
 
-First we need to identify the latest snapshot from the ReplicationDestination
-object. Record the values of the latest snapshot as it will be used to create a
-pvc. Then create the Deployment, Service, PVC, and Secret. Ensure that the above
-steps are completed before a new replication cycle starts or the latest snapshot
-may be replaced before it can be used.
+First we need to wait for the next synchronization iteration to complete so the
+changes made above to add a new database will be replicated to the destination.
 
 .. code:: console
 
-   $ kubectl get replicationdestination database-destination -n dest --template={{.status.latestImage.name}}
-   volsync-dest-database-destination-20201203174504
+   $ kubectl get replicationdestination database-destination -n dest --template={{.status.lastSyncTime}}
+   2023-08-03T16:29:01Z
+
+When the above has been updated to a newer time as in the example above, then we can proceed to create the
+Deployment, Service, PVC, and Secret.
+
+The PVC uses the VolSync volume populator feature and sets the ReplicationDestination
+as its dataSourceRef. This will populate the PVC with the latest snapshot contents from the ReplicationDestination.
+
+.. code:: console
 
-   $ sed -i 's/snapshotToReplace/volsync-dest-database-destination-20201203174504/g' examples/destination-database/mysql-pvc.yaml
    $ kubectl create -n dest -f examples/destination-database/
 
 Validate that the mysql pod is running within the environment.

docs/usage/volume-populator/index.rst

@@ -0,0 +1,160 @@
+=======================================
+ReplicationDestination Volume Populator
+=======================================
+
+.. toctree::
+   :hidden:
+
+.. sidebar:: Contents
+
+   .. contents:: ReplicationDestination Volume Populator
+      :local:
+
+When a PVC is created that directly references a ReplicationDestination object, VolSync's Volume Populator controller will automatically fill the PVC with the most recent replicated data, alleviating the need to manually specify the name of the Snapshot.
+
+.. note::
+    The VolumePopulator feature of VolSync is available with kubernetes v1.22 and
+    above with the `AnyVolumeDataSource` feature gate enabled. The `AnyVolumeDataSource` feature gate is
+    enabled by default as of v1.24.
+
+When replicating or restoring a PVC via a ReplicationDestination using a VolumeSnapshot,
+the end result is a VolumeSnapshot that contains the latestImage from the last successful synchronization.
+Previously to create a PVC with the synchronized data, you needed to create a PVC with a dataSourceRef that points to
+the VolumeSnapshot created by the ReplicationDestination.
+The VolSync volume populator means you can instead point the dataSourceRef of a PVC to the VolSync
+ReplicationDestination resource. VolSync will take care of finding the latestImage available in the
+ReplicationDestination (or waiting for it to appear after replication completes) and then populating the PVC with
+the contents of the VolumeSnapshot.
+
+
+Configuring a PVC with ReplicationDestination Volume Populator
+==============================================================
+
+This example will assume you have a working ReplicationDestination setup that is replicating/synchronizing data.
+The example used here uses the Rclone mover, but other movers could be used. For more information about setting up the
+replication itself, refer to the docs for the mover you are interested in.
+
+.. note::
+    The ReplicationDestination used with the volume populator must use a copyMethod of `Snapshot`.
+
+Here's an example of a ReplicationDestination.
+
+.. code-block:: yaml
+    :caption: ReplicationDestination object
+
+    ---
+    apiVersion: volsync.backube/v1alpha1
+    kind: ReplicationDestination
+    metadata:
+      name: rclone-replicationdestination
+      namespace: dest
+    spec:
+      trigger:
+        # Every 6 minutes, offset by 3 minutes
+        schedule: "3,9,15,21,27,33,39,45,51,57 * * * *"
+      rclone:
+        rcloneConfigSection: "aws-s3-bucket"
+        rcloneDestPath: "volsync-test-bucket/mysql-pvc-claim"
+        rcloneConfig: "rclone-secret"
+        copyMethod: Snapshot
+        accessModes: [ReadWriteOnce]
+        capacity: 10Gi
+        storageClassName: my-sc
+        volumeSnapshotClassName: my-vsc
+    status:
+      lastSyncDuration: 30.038338887s
+      lastSyncTime: "2023-08-13T07:29:36Z"
+      latestImage:
+        apiGroup: snapshot.storage.k8s.io
+        kind: VolumeSnapshot
+        name: volsync-rclone-replicationdestination-dest-20230813072935
+
+Now a PVC can be created that uses this ReplicationDestination. This PVC will be populated with the contents of the
+VolumeSnapshot indicated in the ``status.latestImage`` of the ReplicationDestination.
+
+.. code-block:: yaml
+    :caption: PVC object
+
+    ---
+    apiVersion: v1
+    kind: PersistentVolumeClaim
+    metadata:
+      name: restored-pvc
+      namespace: dest
+    spec:
+      accessModes: [ReadWriteOnce]
+      dataSourceRef:
+        kind: ReplicationDestination
+        apiGroup: volsync.backube
+        name: rclone-replicationdestination
+      resources:
+        requests:
+          storage: 10Gi
+      storageClassName: my-vsc
+
+The ``.spec.dataSourceRef.kind`` must be set to ``ReplicationDestination`` and the ``.spec.dataSourceRef.apiGroup``
+must be set to ``volsync.backube``.
+
+``.spec.dataSourceRef.name`` should be the name of your ReplicationDestination.
+
+The VolSync volume populator controller will start to populate the volume with the latest available snapshot
+indicated in the ReplicationDestination at ``.status.latestImage``.  In the above example ReplicationDestination this
+would be: ``volsync-rclone-replicationdestination-dest-20230813072935``.
+
+.. note::
+  If no latestImage exists yet, then the PVC will remain in pending state until the ReplicationDestination completes
+  and a snapshot is available.  In this way you could create a ReplicationDestination and a PVC that uses the
+  ReplicationDestination at the same time. The PVC will only start the volume population process after the
+  ReplicationDestination has completed a replication and a snapshot is available (as seen in ``.status.latestImage``).
+
+  Additionally, if the storage class used (``my-vsc`` in the example) has a ``volumeBindingMode`` of
+  ``WaitForFirstConsumer``, the volume populator will need to wait until there is a consumer of the PVC before it gets
+  populated. When a consumer does come along (for example a pod that wants to mount the PVC), then the volume will be
+  populated at that time. At this point the VolSync volume populator controller will take the latestImage from the
+  ReplicationDestination, which may have been updated if additional replications have occurred since the PVC was
+  created.
+
+Once the PVC has been populated, the status should be updated and it can be used. Here is an example of the PVC
+after it has been populated.
+
+.. code-block:: yaml
+    :caption: PVC object after volume population complete
+
+    ---
+    apiVersion: v1
+    kind: PersistentVolumeClaim
+    metadata:
+      annotations:
+        pv.kubernetes.io/bind-completed: "yes"
+        pv.kubernetes.io/bound-by-controller: "yes"
+        volume.beta.kubernetes.io/storage-provisioner: hostpath.csi.k8s.io
+        volume.kubernetes.io/storage-provisioner: hostpath.csi.k8s.io
+      creationTimestamp: "2023-08-13T07:24:23Z"
+      finalizers:
+      - kubernetes.io/pvc-protection
+      name: restored-pvc
+      namespace: dest
+      resourceVersion: "4520"
+      uid: 55f748d8-538c-4457-b36f-ad1f956290d2
+    spec:
+      accessModes: [ReadWriteOnce]
+      dataSource:
+        kind: ReplicationDestination
+        apiGroup: volsync.backube
+        name: rclone-replicationdestination
+      dataSourceRef:
+        kind: ReplicationDestination
+        apiGroup: volsync.backube
+        name: rclone-replicationdestination
+      resources:
+        requests:
+          storage: 10Gi
+      storageClassName: my-vsc
+      volumeMode: Filesystem
+      volumeName: pvc-dec29e3a-6f21-4eb0-84b2-b9d9d875f486
+    status:
+      accessModes:
+      - ReadWriteOnce
+      capacity:
+        storage: 10Gi
+      phase: Bound

examples/destination-database/mysql-pvc.yaml

@@ -6,10 +6,10 @@ metadata:
 spec:
   accessModes:
     - ReadWriteOnce
-  dataSource:
-    kind: VolumeSnapshot
-    apiGroup: snapshot.storage.k8s.io
-    name: snapshotToReplace
+  dataSourceRef:
+    kind: ReplicationDestination
+    apiGroup: volsync.backube
+    name: database-destination  # Name of the ReplicationDestination
   resources:
     requests:
       storage: 2Gi

examples/rclone/rclone.conf

@@ -1,4 +1,4 @@
-[aws-s3-bucket]
+[rclone-bucket]
 type = s3
 provider = AWS
 env_auth = false

examples/rclone/volsync_v1alpha1_replicationdestination.yaml

@@ -8,7 +8,7 @@ spec:
   trigger:
     schedule: "*/5 * * * *"
   rclone:
-    rcloneConfigSection: "aws-s3-bucket"
+    rcloneConfigSection: "rclone-bucket"
     rcloneDestPath: "volsync-test-bucket/mysql-pv-claim"
     rcloneConfig: "rclone-secret"
     copyMethod: Snapshot