E2E_test_flow.md

E2E Backup/Restore with CSI Flowchart - Mongo

Steps below explain E2E backup/restore Test suite of App mongo-persistent using CSI in AWS cluster.

1. Pre-Backup:

1.1 Create VolumeSnapshotClass with Driver which is similar to Provider of available csi-storage class.

Note: Storage Class’s Provider and VolumeSnapshotClass’s Driver should match.

VolumeSnapshotClass:

apiVersion: snapshot.storage.k8s.io/v1
deletionPolicy: Retain
driver: ebs.csi.aws.com
kind: VolumeSnapshotClass
metadata:
 annotations:
   snapshot.storage.kubernetes.io/is-default-class: 'true'
 labels:
   velero.io/csi-volumesnapshot-class: 'true' 
 name: oadp-example-snapclass

1.2 Installing Application and checking if application is ready for the backup by Running preBackup Step

2. Backup:

2.1 For Backup, create volumeSnapshot CR that points to mongo-pvc.

Path to volume after creating volumeSnapshot CR:

VolumeSnapshot → mongo-pvc → pv → volume 

volumeSnapshot is created with readyToUse: nil

As described in the image, source of the volumeSnapshot is mongo-pvc. Here is the example of VolumeSnapshot:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
 generateName: velero-mongo-
 name: velero-mongo-6sg7g
 namespace: mongo-persistent
spec:
 source:
   persistentVolumeClaimName: mongo-pvc
 volumeSnapshotClassName: oadp-example
status:
 boundVolumeSnapshotContentName: snapcontent
 creationTime: '2022-07-13T15:28:28Z'
 readyToUse: nil
 restoreSize: 1Gi

2.2 CSI snapshotter makes gRPC call to provider’s Driver to create snapShot, and update volumeSnapshot’s readyToUse nil to false

2.3 CSI-snapshotter create volumeSnapshotContent with snapshotHandle returned by Driver

Source of volumeSnapshotContent is volumeHandle, a unique string that identifies volume. Similarly, snapshotHandle is the unique identifier of the volume snapshot created on the storage backend. Here is the example of volumeSnapshotContent:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
 annotations:
   snapshot.storage.kubernetes.io/volumesnapshot-being-deleted: 'yes'
 name: snapcontent
spec:
 deletionPolicy: Retain
 driver: ebs.csi.aws.com
 source:
   volumeHandle: vol-081e4301172d6fa52
 volumeSnapshotClassName: oadp-example-snapclass
 volumeSnapshotRef:
   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshot
   name: velero-mongo-6sg7g
   namespace: mongo-persistent
status:
 readyToUse: true
 restoreSize: 1073741824
 snapshotHandle: snap-083de977db057e37c
 

2.4 Update volumeSnapshot and volumeSnapshotContent readyToUse: true

2.5 End of Backup

3. Uninstall the application

Delete VolumeSnapshot created during the backup. Otherwise deleting namespace in the cluster during uninstallation will trigger the VolumeSnapshot deletion, which will cause snapshot deletion on the cloud providers, then backup cannot restore the PV.

4. Restore:

4.1 Backup resources from S3 bucket

resources such as namespace, services, routes, secrets e.t.c.

4.2 Create PVC

Velero creates PVC with DataSource: VolumeSnapshot that leads to snapshot. (PVC, VS, and, VSC configuration files are available at the end)

VolumeSnapshotContent→VolumeSnapshot→ PVC’s dataSource → PV → Volume 

Note: Backup will re-create VolumeSnapshotContent,because some parameter in the VolumeSnapshotContent Spec is immutable, e.g. VolumeSnapshotRef and Source.

4.3 Create PV, CSI-provisioner observes PVC and creates PV as per claim

4.4 Create volume, CSI-provisioner asks provider’s driver to create volume and return volume-ID

4.5 PV→ Volume, CSI-provisioner provides volume-ID to PV and bounds PV to volume

4.6 End of the restore