QNAP-CSI-PlugIn unable to run dd on iscsi block device

[brunnels]

I'm trying to use QNAP-CSI-PlugIn to dynamically create iscsi backed PVC's on my qnap. During the last part of the configuration the node pod tries to use dd if=/dev/sdc bs=4096 count=512 status=none to determine the block device filesystem type but it's not permitted.

time="2024-06-17T21:20:30Z" level=debug msg="Device found." device=/dev/sdc logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=debug msg="<<<< devices.waitForDevice" device=/dev/sdc logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=debug msg=">>>> osutils.execCommandWithTimeout." args="[/dev/sdc]" command=blkid logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI timeout=5s workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=debug msg=">>>> osutils.execCommandWithTimeoutAndInput." args="[/dev/sdc]" command=blkid logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI timeoutSeconds=5s workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=debug msg="<<<< osutils.execCommandWithTimeoutAndInput." command=blkid error="exit status 2" logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=debug msg="<<<< osutils.execCommandWithTimeout." logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=info msg="Could not get FSType for device; err: exit status 2." device=/dev/sdc logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=debug msg="<<<< devices.getDeviceFSType" logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=debug msg=">>>> devices.isDeviceUnformatted" device=/dev/sdc logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=debug msg=">>>> osutils.execCommandWithTimeout." args="[if=/dev/sdc bs=4096 count=512 status=none]" command=dd logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI timeout=5s workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=debug msg=">>>> osutils.execCommandWithTimeoutAndInput." args="[if=/dev/sdc bs=4096 count=512 status=none]" command=dd logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI timeoutSeconds=5s workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=debug msg="<<<< osutils.execCommandWithTimeoutAndInput." command=dd error="exit status 2" logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=debug msg="<<<< osutils.execCommandWithTimeout." logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=error msg="failed to read the device" device=/dev/sdc error="exit status 2" logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=debug msg="<<<< devices.isDeviceUnformatted" logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=error msg="Unable to identify if the device is unformatted; err: exit status 2" device=/dev/sdc logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=debug msg="<<<< iscsi.AttachISCSIVolume" logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI workflow="node_server=stage"
time="2024-06-17T21:20:30Z" level=debug msg="Attach iSCSI volume is not complete, waiting." error="exit status 2" increment=5.294453817s logLayer=csi_frontend requestID=a8cdb52d-55fc-4b7a-bd80-d76317794b59 requestSource=CSI workflow="node_server=stage"

/dev is mounted in the container and talosctl disks on the node shows that the iscsi block device resides at /dev/sdc

I have pod-security.kubernetes.io/enforce: privileged label set on the qnap-csi operator namespace.

Is there anything else that's needed to allow the dd command to work?

[rothgar]

Are you using a talos installation with the iscsi extension? dd isn't provided by talos so I assume it's a binary in your container.

I would try doing a talos install with iscsi and linux-utils from the image factory
https://factory.talos.dev/?arch=amd64&cmdline-set=true&extensions=-&extensions=siderolabs%2Fiscsi-tools&extensions=siderolabs%2Futil-linux-tools&platform=metal&target=metal&version=1.7.5

You can also update an existing talos node with those extensions via (this is the amd64 bare metal installer)

talos upgrade --image factory.talos.dev/installer/613e1592b2da41ae5e265e8789429f22e121aab91cb4deb6bc3c0b6262961245:v1.7.5

[brunnels]

@rothgar I do have iscsi extension on all nodes and I can see the iscsi volume get created in my qnap so all that is working. I think it's an issue with the security policy on the pod that the QNAP operator creates. I am able to use the talos debug daemonset to add a pod in the same namespace as the qnap operator and when I ssh into it I can run the same dd commands on the iscsi device that the qnap operator created.

I opened an issue with the QNAP project and detailed my findings there. qnap-dev/QNAP-CSI-PlugIn#13