Unable to backup pv with restic since last kubernetes upgrade

[fschuyts]

What steps did you take and what happened:
Running the velero backup command on a namespace running an basic app with a pvc mounted returns the following error 👍

Errors:
Velero: name: /hello-kubernetes-custom-6b44877dfc-xtntp message: /Error backing up item error: /pod volume backup failed: error exposing host path for pod volume: error identifying unique volume path on host for volume hello in pod hello-kubernetes-custom-6b44877dfc-xtntp: expected one matching path: /host_pods/c1bfc678-a5f1-48e5-9266-7c980f237000/volumes/*/pvc-017566fa-2a7d-44e3-b6b3-5c9091c44f76/mount, got 0

What did you expect to happen:

Was expecting the backup to be successful especially that it was working fine before upgrading kubernetes from 1.24 to 1.25

Anything else you would like to add:
A little of history.
We were running TKGI 1.15.8 with kubernetes 1.24. We were using VCP for volumes provisionning.
As recommended by VMWare and deprecation of VCP in future relase we migrate to CNS ( CSI driver ).
Migration to CSI driver was successful and restic backup was still working fine.
After upgrading TKGI to 1.16.8 and so kubernetes to 1.25.16, the restic backup is not working anymore.

Having a look deeper, restic is looking on the following path /var/vcap/data/kubelet/pods/78de8e79-cbc4-41cd-a890-a153b411d66b/volumes/kubernetes.iovsphere-volume/pvc-51e2458a-7aea-490a-8505-65e686e6bb2b/mount but for some reason now it looks like the datas are in the parent folder /var/vcap/data/kubelet/pods/78de8e79-cbc4-41cd-a890-a153b411d66b/volumes/kubernetes.iovsphere-volume/pvc-51e2458a-7aea-490a-8505-65e686e6bb2b

According to the documentation from VMWare ( Broadcom ), the backup configuration remains the same. Nothing has been changed.

Is there a way to configure restic to check in the parent folder instead of the mount folder ?

I have already tried with several versions of velero ( 1.11, 1.12, 1.13) but still no luck.

Environment:

• helm version (use helm version): 3.16.2
• helm chart version and app version (use helm list -n <YOUR NAMESPACE>):
  already tried :
  • helm chart 5.0.2 with app version 1.11.1
  • helm chart 5.2.2 with app version 1.12.3
  • helm chart 6.7.0 with app version 1.13.2
• Kubernetes version (use kubectl version): 1.25.16
• Cloud provider or hardware configuration: TKGI 1.16.8 on vSphere/vSaN
• OS (e.g. from /etc/os-release): worker nodes are running Ubuntu 22.04.4 LTS

[reasonerjt]

Hi this is a well-known issue in TKGi, and AFAIK there is an internal KB, please reach out to official Broadcom support.

[fschuyts]

Hi this is a well-known issue in TKGi, and AFAIK there is an internal KB, please reach out to official Broadcom support.

Hello,

Thanks for the reply @reasonerjt
Unfortunately, from support perspective, TKGI 1.16.8 is out of support. Any other way to find out the KB you are talking about especially if it is an internal one ?

[reasonerjt]

Unfortunately, I don't think I have access to the TKGi KB, but it seems you need to modify the volume "host-pods" of the daemonset to make sure restic can access the data of the pod in the hostpath.

[fschuyts]

Unfortunately, I don't think I have access to the TKGi KB, but it seems you need to modify the volume "host-pods" of the daemonset to make sure restic can access the data of the pod in the hostpath.

This is already done through the helm chart or are you talking about something else ?

Here is the current volume and volumemount configuration on the daemonset :

volumeMounts:
- mountPath: /host_pods
  mountPropagation: HostToContainer
  name: host-pods
volumes:
- hostPath:
    path: /var/vcap/data/kubelet/pods
    type: ""
  name: host-pods