CSI driver-smb not mounting despite no errors #557

sj10v2 · 2022-11-28T14:15:59Z

sj10v2
Nov 28, 2022

Hello,

I'm unable to get the csi smb driver to work properly on my microk8s 1,25 cluster running on ubuntu 22.04. I'm hoping someone on this forum is able to assist me in resolving this issue. I've deployed a PV and PVC with the following charts

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-smb
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - vers=3.0
    - uid=1001
    - gid=1001
    - noperm
  csi:
    driver: smb.csi.k8s.io
    readOnly: false
    volumeHandle: d1190baaasdf # make sure it's a unique id in the cluster
    volumeAttributes:
      source: "//192.168.40.30/test-smb"
    nodeStageSecretRef:
      name: smbcreds

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-smb
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: ""
  volumeMode: Filesystem
  volumeName: pv-smb

Mounted pvc to deployment with following


        securityContext:
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: false
          runAsGroup: 1001
          runAsNonRoot: false
          runAsUser: 0

        volumeMounts:
        - mountPath: /data
          name: data

      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: pvc-smb

smb driver node reports successful mount as shown in logs below

I1128 13:25:58.463582       1 utils.go:77] GRPC request: {"staging_target_path":"/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/c7f3c9017db890efa1eb7819671214aee1e812078559bef28ab5ba894791d36e/globalmount","target_path":"/var/snap/microk8s/common/var/lib/kubelet/pods/c644ec16-5f35-41f9-b27f-cfa878c4c82d/volumes/kubernetes.io~csi/pv-smb/mount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","vers=3.0","uid=1001","gid=1001","noperm"],"volume_mount_group":"1001"}},"access_mode":{"mode":5}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"smb-test-c497fbcb6-xl767","csi.storage.k8s.io/pod.namespace":"default","csi.storage.k8s.io/pod.uid":"c644ec16-5f35-41f9-b27f-cfa878c4c82d","csi.storage.k8s.io/serviceAccount.name":"smb-test","source":"//192.168.40.30/test-smb"},"volume_id":"d1190baaasdf"}
I1128 13:25:58.464106       1 nodeserver.go:78] NodePublishVolume: mounting /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/c7f3c9017db890efa1eb7819671214aee1e812078559bef28ab5ba894791d36e/globalmount at /var/snap/microk8s/common/var/lib/kubelet/pods/c644ec16-5f35-41f9-b27f-cfa878c4c82d/volumes/kubernetes.io~csi/pv-smb/mount with mountOptions: [bind] volumeID(d1190baaasdf)
I1128 13:25:58.464127       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/c7f3c9017db890efa1eb7819671214aee1e812078559bef28ab5ba894791d36e/globalmount /var/snap/microk8s/common/var/lib/kubelet/pods/c644ec16-5f35-41f9-b27f-cfa878c4c82d/volumes/kubernetes.io~csi/pv-smb/mount)
I1128 13:25:58.465417       1 mount_linux.go:183] Mounting cmd (mount) with arguments ( -o bind,remount /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/c7f3c9017db890efa1eb7819671214aee1e812078559bef28ab5ba894791d36e/globalmount /var/snap/microk8s/common/var/lib/kubelet/pods/c644ec16-5f35-41f9-b27f-cfa878c4c82d/volumes/kubernetes.io~csi/pv-smb/mount)
I1128 13:25:58.466751       1 nodeserver.go:85] NodePublishVolume: mount /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/c7f3c9017db890efa1eb7819671214aee1e812078559bef28ab5ba894791d36e/globalmount at /var/snap/microk8s/common/var/lib/kubelet/pods/c644ec16-5f35-41f9-b27f-cfa878c4c82d/volumes/kubernetes.io~csi/pv-smb/mount volumeID(d1190baaasdf) successfully

but folder comes up empty

sudo ls -la /var/snap/microk8s/common/var/lib/kubelet/pods/c644ec16-5f35-41f9-b27f-cfa878c4c82d/volumes/kubernetes.io~csi/pv-smb/mount
total 8
drwxr-xr-x 2 root root 4096 Nov 28 08:25 .
drwxr-x--- 3 root root 4096 Nov 28 08:25 ..

and df -h command reveals nothing is being mounted.

Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.6G  2.4M  1.6G   1% /run
/dev/nvme0n1p2  457G   21G  414G   5% /
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           7.8G  4.0K  7.8G   1% /tmp
/dev/nvme0n1p1  511M  5.3M  506M   2% /boot/efi
tmpfs           1.6G   80K  1.6G   1% /run/user/126
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/52949f8c217eb868b2f6b458c95e6cc6cef4ab2621c1568a72bf9388b563f9dd/shm
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/4b096ac4cf30065bb51101399f6b3a66a01619074f64e38cd6072e117e3e4d94/shm
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/e184d723eec8b2e83ffc7d718c0790d9273af8d75656b6bfa9c6c25591fd381f/shm
tmpfs           1.6G   68K  1.6G   1% /run/user/1000
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/b1898e4dc88a68eeba541c6e0dd1685ef68a5f5a316aea5547d893f733d42b3b/shm
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/dd7c2b822966d0a75db4f1e286097a9b4d0e151d65b963acc12d42f08de7de41/shm
shm              64M     0   64M   0% /var/snap/microk8s/common/run/containerd/io.containerd.grpc.v1.cri/sandboxes/41989b21e2bb8d8c5d5238d2d25002970c61e6a00b5188d047aeb9d8491cfd44/shm

Any help will be greatly appreciated... Thanks

Answered by andyzhangx

Nov 28, 2022

not sure whether you have set this:

Microk8s based kubernetes recommended settings:
- --set linux.dnsPolicy=ClusterFirstWithHostNet with --set controller.dnsPolicy=ClusterFirstWithHostNet -
  external smb server cannot be found based on Default dns.
- --set linux.kubelet="/var/snap/microk8s/common/var/lib/kubelet" - sets correct path to microk8s kubelet even
  though a user has a folder link to it.

View full answer

andyzhangx · 2022-11-28T14:42:15Z

andyzhangx
Nov 28, 2022
Maintainer

could you provide NodeStageVolume logs in the driver logs?

1 reply

sj10v2 Nov 28, 2022
Author

nodeStageVolume logentry below

I1128 12:41:52.734375 1 nodeserver.go:201] NodeStageVolume: targetPath(/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/c7f3c9017db890efa1eb7819671214aee1e812078559bef28ab5ba894791d36e/globalmount) volumeID(d1190baaasdf) context(map[source://192.168.40.30/test-smb]) mountflags([dir_mode=0777 file_mode=0777 vers=3.0 uid=1001 gid=1001 noperm]) mountOptions([dir_mode=0777 file_mode=0777 vers=3.0 uid=1001 gid=1001 noperm])
I1128 12:43:22.720960 1 utils.go:76] GRPC call: /csi.v1.Node/NodeStageVolume
I1128 12:43:22.721116 1 nodeserver.go:201] NodeStageVolume: targetPath(/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/c7f3c9017db890efa1eb7819671214aee1e812078559bef28ab5ba894791d36e/globalmount) volumeID(d1190baaasdf) context(map[source://192.168.40.30/test-smb]) mountflags([dir_mode=0777 file_mode=0777 vers=3.0 uid=1001 gid=1001 noperm]) mountOptions([dir_mode=0777 file_mode=0777 vers=3.0 uid=1001 gid=1001 noperm])

andyzhangx · 2022-11-28T15:06:20Z

andyzhangx
Nov 28, 2022
Maintainer

not sure whether you have set this:

Microk8s based kubernetes recommended settings:
- --set linux.dnsPolicy=ClusterFirstWithHostNet with --set controller.dnsPolicy=ClusterFirstWithHostNet -
  external smb server cannot be found based on Default dns.
- --set linux.kubelet="/var/snap/microk8s/common/var/lib/kubelet" - sets correct path to microk8s kubelet even
  though a user has a folder link to it.

4 replies

sj10v2 Nov 28, 2022
Author

regarding DNS, i'm mounting PV using ip address not domain name
volumeAttributes:
source: "//192.168.40.30/test-smb'

Do i still need to change the dnspolicy configurations?

regarding linux.kubelet see nodestagevolume logs
nodeserver.go:201] NodeStageVolume: targetPath(/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/c7f3c9017db890efa1eb7819671214aee1e812078559bef28ab5ba894791d36e/globalmount) volumeID(d1190baaasdf) context(map[source://192.168.40.30/test-smb]) mountflags([dir_mode=0777 file_mode=0777 vers=3.0 uid=1001 gid=1001 noperm]) mountOptions([dir_mode=0777 file_mode=0777 vers=3.0 uid=1001 gid=1001 noperm])

andyzhangx Nov 29, 2022
Maintainer

that's ok, but I don't see mounting successfully in NodeStageVolume, could you check whether you have set this correctly per #443 (comment)

sj10v2 Nov 29, 2022
Author

Setting linux.kubelet to /var/snap/microk8s/common/var/lib/kubelet fixed the issue. Many thanks for the help

andyzhangx Nov 29, 2022
Maintainer

should be addressed by doc PR: #552

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CSI driver-smb not mounting despite no errors #557

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

CSI driver-smb not mounting despite no errors #557

sj10v2 Nov 28, 2022

Replies: 2 comments · 5 replies

andyzhangx Nov 28, 2022 Maintainer

sj10v2 Nov 28, 2022 Author

andyzhangx Nov 28, 2022 Maintainer

sj10v2 Nov 28, 2022 Author

andyzhangx Nov 29, 2022 Maintainer

sj10v2 Nov 29, 2022 Author

andyzhangx Nov 29, 2022 Maintainer

sj10v2
Nov 28, 2022

Replies: 2 comments 5 replies

andyzhangx
Nov 28, 2022
Maintainer

sj10v2 Nov 28, 2022
Author

andyzhangx
Nov 28, 2022
Maintainer

sj10v2 Nov 28, 2022
Author

andyzhangx Nov 29, 2022
Maintainer

sj10v2 Nov 29, 2022
Author

andyzhangx Nov 29, 2022
Maintainer