Shared storage for 3 pod in deployment configuration

[sharpec]

Hello everybody.
I'am newlly on Kubernetes.
Now i stay work on ha cluster (3master + 3 agent) with deployment's pod with 3 replicas.
I have test the cluster with Locust and this work very well.

Now i need to mount a shared disk for all pod in RWX.
The latest k3s image dont have a nfs package.
There is an alternative to nfs?

I have find many alternative, but i'm not sure:

• Portworx (isnt fully free)
• goofys
• minio
• s3fs
• longhorn (has experimental RWX mode)
• glusterfs (removed from k3s)

I would like the storage to be geographically distributed as well, with the 3 pods accessing indiscriminately on a single file system.

Any idea?
Thank you

[mysticaltech]

Hello @sharpec, we have out-of-the-box support for Longhorn; you can try it! And if you need Geo distribution, just choose one location per nodepool.

[sharpec]

Thanks for answer @mysticaltech
i have set in terraform.tfvars

enable_longhorn = true
disable_hetzner_csi = true

I have tried with this config but this is the result

pod.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: fastapi
  name: fastapi
spec:
  replicas: 3
  selector:
    matchLabels:
      run: fastapi
  #strategy:
  #    type: Recreate
  template:
    metadata:
      labels:
        run: fastapi
    spec:
      containers:
      - image: mycustomrepo/repo1:tag1
        name: fastapi
        #volumeMounts:
        #- name: volv
        #  mountPath: /data
        ports:
        - containerPort: 80
        volumeMounts:
          - mountPath: /data
            name: rwx-test
        mount:
      imagePullSecrets:
      - name: regcred
      volumes:
        - name: rwx-test
          persistentVolumeClaim:
            claimName: rwx-test

rwx.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rwx-test
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: longhorn
  resources:
    requests:
      storage: 1Gi

❯ kubectl --kubeconfig kubeconfig.yaml get events --all-namespaces --sort-by='.metadata.creationTimestamp'

NAMESPACE   LAST SEEN   TYPE      REASON                   OBJECT                                                        MESSAGE
default     34m         Normal    ExternalProvisioning     persistentvolumeclaim/rwx-test                                waiting for a volume to be created, either by external provisioner "driver.longhorn.io" or manually created by system administrator
default     34m         Normal    Provisioning             persistentvolumeclaim/rwx-test                                External provisioner is provisioning volume for claim "default/rwx-test"
longhorn    34m         Normal    Detached                 volume/pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e               volume pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e has been detached
longhorn    34m         Normal    Start                    volume/pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e               Persistent Volume pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e started to use/reuse Longhorn volume pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e
default     34m         Normal    ProvisioningSucceeded    persistentvolumeclaim/rwx-test                                Successfully provisioned volume pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e
longhorn    33m         Normal    Start                    replica/pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e-r-d7db1fbe   Starts pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e-r-d7db1fbe
longhorn    33m         Normal    Start                    replica/pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e-r-e7fcc9bc   Starts pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e-r-e7fcc9bc
default     33m         Normal    SuccessfulCreate         replicaset/fastapi-56d9d7f9b5                                 Created pod: fastapi-56d9d7f9b5-cwfwz
default     33m         Normal    ScalingReplicaSet        deployment/fastapi                                            Scaled up replica set fastapi-56d9d7f9b5 to 1
default     33m         Normal    Scheduled                pod/fastapi-56d9d7f9b5-cwfwz                                  Successfully assigned default/fastapi-56d9d7f9b5-cwfwz to k3s-agent-small-fsn1-rxp
longhorn    33m         Normal    Pulling                  pod/share-manager-pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e    Pulling image "longhornio/longhorn-share-manager:v1_20211020"
longhorn    33m         Normal    Start                    engine/pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e-e-16bf616e    Starts pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e-e-16bf616e
longhorn    33m         Normal    Attached                 volume/pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e               volume pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e has been attached to k3s-agent-small-fsn1-rxp
longhorn    33m         Normal    Created                  pod/share-manager-pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e    Created container share-manager
longhorn    33m         Normal    Started                  pod/share-manager-pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e    Started container share-manager
longhorn    33m         Normal    Pulled                   pod/share-manager-pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e    Successfully pulled image "longhornio/longhorn-share-manager:v1_20211020" in 5.528554564s
longhorn    33m         Warning   Unhealthy                pod/share-manager-pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e    Readiness probe failed: cat: /var/run/ganesha.pid: No such file or directory
default     33m         Normal    SuccessfulAttachVolume   pod/fastapi-56d9d7f9b5-cwfwz                                  AttachVolume.Attach succeeded for volume "pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e"
default     2m28s       Warning   FailedMount              pod/fastapi-56d9d7f9b5-cwfwz                                  MountVolume.MountDevice failed for volume "pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e" : rpc error: code = Internal desc = mount failed: exit status 32...
default     10m         Warning   FailedMount              pod/fastapi-56d9d7f9b5-cwfwz                                  Unable to attach or mount volumes: unmounted volumes=[rwx-test], unattached volumes=[rwx-test kube-api-access-p8v29]: timed out waiting for the condition
default     28m         Warning   FailedMount              pod/fastapi-56d9d7f9b5-cwfwz                                  Unable to attach or mount volumes: unmounted volumes=[rwx-test], unattached volumes=[kube-api-access-p8v29 rwx-test]: timed out waiting for the condition

❯ kubectl --kubeconfig kubeconfig.yaml logs share-manager-pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e -n longhorn

time="2022-05-16T12:34:40Z" level=info msg="starting RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576"
time="2022-05-16T12:34:40Z" level=info msg="ending RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576"
time="2022-05-16T12:34:41Z" level=debug msg="volume pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e device /dev/longhorn/pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e contains filesystem of format " encrypted=false volume=pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e
I0516 12:34:42.569952       1 mount_linux.go:425] Disk "/dev/longhorn/pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/longhorn/pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e]
I0516 12:34:43.609562       1 mount_linux.go:435] Disk successfully formatted (mkfs): ext4 - /dev/longhorn/pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e /export/pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e
time="2022-05-16T12:34:44Z" level=info msg="starting nfs server, volume is ready for export" encrypted=false volume=pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e
time="2022-05-16T12:34:44Z" level=info msg="Running NFS server!"
time="2022-05-16T12:34:44Z" level=info msg="starting health check for volume" encrypted=false volume=pvc-b4af4cae-5923-4ab2-adfd-860ba002ff0e

[mysticaltech]

@karbowiak Any ideas on how to fix this?

[karbowiak]

@mysticaltech sadly not off the top of my head - i'll have a whack at it in the weekend, see if i can replicate the issue and find the cause - i gotta use RWX for a few volumes myself as well at some point 😅

[mysticaltech]

Ok, thanks @karbowiak!

[mysticaltech]

@sharpec Please, if you want to install other CSI solutions, you can disable Hetzner CSI (there is a variable for that) and install the one you wish to with helm.

If somehow the installation fails because you need a package installed in the OS, let me know which one and for which helm chart.

As a tool, we aim to be as flexible as possible!

[tijnhendrikson]

I ran into the same issue and added the nfs-client to the agents/control plane if Longhorn is enabled. I briefly tested it and was able to deploy a PVC with ReadWriteMany successfully. I created a PR (#199) with the changes. I'm new to Terraform, so not sure if this is the way to go.

[sharpec]

Awesome! I'll try tomorrow. Thank you guys

[mysticaltech]

Thanks, @tijnhendrikson! Merged and released with v1.2.2.