File System Backup does not work without including pod in backup/restore

[klllmxx]

For I try to restore the backup item,all pvs,pvcs can be find,but data not found
something like this

command:
deploy:
velero install --provider aws --use-node-agent --privileged-node-agent --plugins velero-plugin-for-aws:v1.4.1,velero-plugin-for-csi:v0.4.2 --bucket prod-backup --secret-file /root/middleware/velero/velero-auth.txt --namespace velero --image velero:v1.13.1 --snapshot-location-config region=minio --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.127.131:9000 --uploader-type restic

[klllmxx]

backup:
velero backup create cluster-backup888 --include-namespaces prod --include-resources persistentvolumeclaims,persistentvolumes --item-operation-timeout 600s --default-volumes-to-fs-backup
restore:
velero restore create redis-restore --from-backup cluster-backup888

[klllmxx]

How can I do... ....

[kaovilai]

Velero does not restic restore data to volumes when backups do not include pods mounting those volumes.

[klllmxx]

Velero does not restic restore data to volumes when backups do not include pods mounting those volumes.

So....command like this?
velero backup create cluster-857 --include-namespaces prod --include-resources persistentvolumeclaims,persistentvolumes.pods,serviceaccounts,secrets --item-operation-timeout 600s --default-volumes-to-fs-backup

[klllmxx]

Velero does not restic restore data to volumes when backups do not include pods mounting those volumes.

About when I try to backup namespaces by prod,the restore schedule not completed
command:
velero backup create cluster-859 --include-namespaces prod
velero restore create redis-result --from-backup cluster-859

[root@k8s-master-prod data]# cd prod-redis-data-redis-cluster-0-pvc-3e40d814-af8b-4c75-b602-1bb28c355d63/
[root@k8s-master-prod prod-redis-data-redis-cluster-0-pvc-3e40d814-af8b-4c75-b602-1bb28c355d63]# ls
[root@k8s-master-prod prod-redis-data-redis-cluster-0-pvc-3e40d814-af8b-4c75-b602-1bb28c355d63]#

[kaovilai]

Secondly velero does not override existing volume.

You have to delete PVC/pv for volume to be restored with data.

[klllmxx]

Secondly velero does not override existing volume.

You have to delete PVC/pv for volume to be restored with data.
Delete PVC/pv already, and delete pv folders from storage backend also, my kubernetes storage is nfs-client-provisioner.

And try use v1.15.0 to backup resources like this:
Name: cluster1
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: velero.io/resource-timeout=10m0s
velero.io/source-cluster-k8s-gitversion=v1.28.1
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=28

Phase: Failed (run velero backup logs cluster1 for more information)

Namespaces:
Included: prod
Excluded:

Resources:
Included: persistentvolumeclaims, persistentvolumes.pods, serviceaccounts, secrets
Excluded:
Cluster-scoped: auto

Label selector:

Or label selector:

Storage Location: default

Velero-Native Snapshot PVs: true
Snapshot Move Data: false
Data Mover: velero

TTL: 720h0m0s

CSISnapshotTimeout: 10m0s
ItemOperationTimeout: 10m0s

Hooks:

Backup Format Version: 1.1.0

Started: 2024-12-05 14:15:25 +0800 CST
Completed: <n/a>

Expiration: 2025-01-04 14:15:25 +0800 CST

Backup Volumes:
<error getting backup volume info: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline>

command:
velero install --provider aws --use-node-agent --privileged-node-agent --plugins velero-plugin-for-aws:v1.4.1 --bucket prod-backup --secret-file /root/middleware/velero/velero-auth.txt --namespace velero --image velero:v1.15.0 --use-volume-snapshots --snapshot-location-config region=minio --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://192.168.127.131:9000 --uploader-type restic

velero backup create cluster1 --include-namespaces prod --include-resources persistentvolumeclaims,persistentvolumes.pods,serviceaccounts,secrets --item-operation-timeout 600s --snapshot-volumes=true --default-volumes-to-fs-backup=true

time="2024-12-05T06:40:53Z" level=error msg="Error getting backup store for this location" backupLocation=velero/default controller=backup-sync error="rpc error: code = Unknown desc = Invalid s3 url http://192.168.127.131:9000\u00a0--uploader-type, URL must be valid according to https://golang.org/pkg/net/url/#Parse and start with http:// or https://" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:255" error.function=main.newAWSConfig logSource="pkg/controller/backup_sync_controller.go:103"
time="2024-12-05T06:40:53Z" level=info msg="plugin process exited" backupLocation=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-sync id=93 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws
time="2024-12-05T06:40:53Z" level=error msg="Error getting a backup store" backup-storage-location=velero/default controller=backup-storage-location error="rpc error: code = Unknown desc = Invalid s3 url http://192.168.127.131:9000\u00a0--uploader-type, URL must be valid according to https://golang.org/pkg/net/url/#Parse and start with http:// or https://" error.file="/go/src/velero-plugin-for-aws/velero-plugin-for-aws/object_store.go:255" error.function=main.newAWSConfig logSource="pkg/controller/backup_storage_location_controller.go:138"
time="2024-12-05T06:40:53Z" level=info msg="BackupStorageLocation is invalid, marking as unavailable" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121"
time="2024-12-05T06:40:53Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation "default" is unavailable: rpc error: code = Unknown desc = Invalid s3 url http://192.168.127.131:9000\u00a0--uploader-type, URL must be valid according to https://golang.org/pkg/net/url/#Parse and start with http:// or https://)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:179"
time="2024-12-05T06:40:53Z" level=info msg="plugin process exited" backup-storage-location=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-storage-location id=101 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws

[klllmxx]

Secondly velero does not override existing volume.

You have to delete PVC/pv for volume to be restored with data.

By the way,one more ask about the how to change this images for restore steps:

[kaovilai]

@klllmxx
create this configmap

apiVersion: v1
kind: ConfigMap
metadata:
  # any name can be used; Velero uses the labels (below)
  # to identify it rather than the name
  name: fs-restore-action-config
  # must be in the velero namespace
  namespace: velero
  # the below labels should be used verbatim in your
  # ConfigMap.
  labels:
    # this value-less label identifies the ConfigMap as
    # config for a plugin (i.e. the built-in restore
    # item action plugin)
    velero.io/plugin-config: ""
    # this label identifies the name and kind of plugin
    # that this ConfigMap is for.
    velero.io/pod-volume-restore: RestoreItemAction
data:
  # The value for "image" can either include a tag or not;
  # if the tag is *not* included, the tag from the main Velero
  # image will automatically be used.
  image: myregistry.io/my-custom-helper-image[:OPTIONAL_TAG]

[klllmxx]

@klllmxx create this configmap

apiVersion: v1
kind: ConfigMap
metadata:
  # any name can be used; Velero uses the labels (below)
  # to identify it rather than the name
  name: fs-restore-action-config
  # must be in the velero namespace
  namespace: velero
  # the below labels should be used verbatim in your
  # ConfigMap.
  labels:
    # this value-less label identifies the ConfigMap as
    # config for a plugin (i.e. the built-in restore
    # item action plugin)
    velero.io/plugin-config: ""
    # this label identifies the name and kind of plugin
    # that this ConfigMap is for.
    velero.io/pod-volume-restore: RestoreItemAction
data:
  # The value for "image" can either include a tag or not;
  # if the tag is *not* included, the tag from the main Velero
  # image will automatically be used.
  image: myregistry.io/my-custom-helper-image[:OPTIONAL_TAG]

After deploy velero then deploy this one is best or having other way try？and need before deploy velero set this one before install step nessessary?

[kaovilai]

imo create the config map then create/install velero deployment

[klllmxx]

imo create the config map then create/install velero deployment

ammm...my deployment using for crds with command...no more using helm...

[klllmxx]

also about restore step must be having pods informations?

[klllmxx]

imo create the config map then create/install velero deployment
Just want to restore data inside the pv
https://techdocs.broadcom.com/us/en/vmware-tanzu/application-catalog/tanzu-application-catalog/services/tac-doc/apps-tutorials-backup-restore-data-redis-cluster-kubernetes-index.html

[kaovilai]

also about restore step must be having pods informations?

if your backup is specific enough to your workload such as only your intended namespaces, you can just restore from that backup without specifying anything else. Otherwise you can filter what gets restored by namespaces, resource types.

[klllmxx]

Phase: Completed

Looked good right? What's the issue now?

No more pv persistence data but otherthing can be back man
Can you see link:https://techdocs.broadcom.com/us/en/vmware-tanzu/application-catalog/tanzu-application-catalog/services/tac-doc/apps-tutorials-backup-restore-data-redis-cluster-kubernetes-index.html?
and my full steps ?

[kaovilai]

I can see, but I don't use helm or tanzu so I don't have specifics whether it should work or not.

[klllmxx]

I can see, but I don't use helm or tanzu so I don't have specifics whether it should work or not.

Watch point sir!I give this link actually talking about using velero to restore redis-cluster man!And my kubernetes csi storages is nfs! What is the relationship with Helm or Tanzu?

[klllmxx]

I can see, but I don't use helm or tanzu so I don't have specifics whether it should work or not.

Watch point sir!I give this link actually talking about using velero to restore redis-cluster man!And my kubernetes csi storages is nfs! What is the relationship with Helm or Tanzu?

By the way,this link only backup pv and pvc also success restore pv persistence data.

[kaovilai]

What is the relationship with Helm or Tanzu?

The article you linked
mentions

You have configured Helm to use the Tanzu Application Catalog chart repository following the instructions for Tanzu Application Catalog.
You have previously deployed the Redis Cluster Helm chart on the source cluster and added some data to it. Example command

I am not familiar with Redis, and I do not have access to Tanzu Platform cloud services account per the prerequisites of the document. I will not be able to reproduce your issue unfortunately. Perhaps other maintainers more familiar with the document can help.

[klllmxx]

What is the relationship with Helm or Tanzu?

The article you linked mentions

You have configured Helm to use the Tanzu Application Catalog chart repository following the instructions for Tanzu Application Catalog.
You have previously deployed the Redis Cluster Helm chart on the source cluster and added some data to it. Example command

I am not familiar with Redis, and I do not have access to Tanzu Platform cloud services account per the prerequisites of the document. I will not be able to reproduce your issue unfortunately. Perhaps other maintainers more familiar with the document can help.

That's is simple question man,it has nothing to do with helm and cloud，basically velero backup pv and pvc resources only and use this backup to restore to the new migrated cluster and the data can be back,do you understanding what i say？by the way,this link article was not written by me，thank you sir~

[kaovilai]

do you understanding what i say

Honestly, I was trying my best to understand what you mean and may have misunderstood the question.

If I understand your question, you mean to ask if you could backup just the PV/PVC and nothing else (ie. without pods) like the article and restore this PV/PVC data in the new cluster.

Yes but you need to

1. Not use file system backup.
   1. Use VolumeSnapshotLocation provided by cloud provider plugin, in the article you can see in the velero install screenshot they had created volumesnapshotlocation as well. This method will not move snapshots to a different cloud provider and/or region. For aws, I would try read the entire plugin readme. I think the article you linked is being purposefully vague. You need to follow cloud plugin docs.
   2. Use Velero Built-In Data Mover

[klllmxx]

do you understanding what i say

Honestly, I was trying my best to understand what you mean and may have misunderstood the question.

If I understand your question, you mean to ask if you could backup just the PV/PVC and nothing else (ie. without pods) like the article and restore this PV/PVC data in the new cluster.

Yes but you need to

1. Not use file system backup.

   1. Use VolumeSnapshotLocation provided by cloud provider plugin, in the article you can see in the screenshot they had created volumesnapshotlocation as well. This method will not move snapshots to a different cloud provider and/or region. For aws, I would try read the entire plugin readme. I think the article you linked is being purposefully vague. You need to follow cloud plugin docs.velero install
   2. Use Velero Built-In Data Mover

Same like backup and restore steps shown with the link I sent you have been
Also I using backend storage is nfs with nfs-client-provisioner sir

[kaovilai]

If your nfs client provisioner does not have a CSI driver with snapshot capability then without adding pods to use file system backup you won't be able to migrate PV/PVC data.

[klllmxx]

If your nfs client provisioner does not have a CSI driver with snapshot capability then without adding pods to use file system backup you won't be able to migrate PV/PVC data.
Yeah Yeah~ It's okay to carry pod resources for backup, but is it necessary to carry pod resources for recovery? Can't recover PVC and persistent data in PV separately?

[kaovilai]

is it necessary to carry pod resources for recovery?

Yes it is necessary with file system backup. Until such time that velero create a pod for you, your backup will need to contain a pod mounting PV for file system backup to work on these volumes.

[klllmxx]

is it necessary to carry pod resources for recovery?

Yes it is necessary with file system backup. Until such time that velero create a pod for you, your backup will need to contain a pod mounting PV for file system backup to work on these volumes.

Recovery steps also necessary carry pod resources right you mean ?

[kaovilai]

yes

[klllmxx]

yes

ok now i see,good,but offical document information not cleary to explain this one so hope update the recovery documentation is best i think sir

[kaovilai]

PodVolumeRestore - represents a FSB restore of a pod volume. The main Velero restore process creates one or more of these when it encounters a pod that has associated FSB backups. Each node in the cluster runs a controller for this resource (in the same daemonset as above) that handles the PodVolumeRestores for pods on that node. PodVolumeRestore is backed by restic or kopia, the controller invokes restic or kopia internally, refer to restic integration and kopia integration for details.

Our docs already highlight this.

If more improvement is needed leave a suggestion or PR.

[klllmxx]

PodVolumeRestore - represents a FSB restore of a pod volume. The main Velero restore process creates one or more of these when it encounters a pod that has associated FSB backups. Each node in the cluster runs a controller for this resource (in the same daemonset as above) that handles the PodVolumeRestores for pods on that node. PodVolumeRestore is backed by restic or kopia, the controller invokes restic or kopia internally, refer to restic integration and kopia integration for details.

Our docs already highlight this.

If more improvement is needed leave a suggestion or PR.

Thank you sir you are best man!!~~hahahaha