Error: Failed adding new disk: Failed to open: Failed to luksOpen: /dev/disk/by-path/..., exit status 2, Failed to load key in kernel keyring. No key available with this passphrase.

[nobuto-m]

When I follow the FDE document:
https://canonical-microceph.readthedocs-hosted.com/en/latest/explanation/fde-osd/

It fails with the following output.

ubuntu@mc-1:~$ sudo microceph disk add /dev/vdc --wipe --encrypt
Error: Failed adding new disk: Failed to open: Failed to luksOpen: /dev/disk/by-path/virtio-pci-0000:08:00.0, exit status 2, Failed to load key in kernel keyring.
No key available with this passphrase.


NOTE: OSD Encryption requires a snapd >= 2.59.1
Verify your version of snapd by running "snap version"

ubuntu@mc-1:~$ snap version
snap    2.60.1+git1186.g661258f
snapd   2.60.1+git1186.g661258f
series  16
ubuntu  22.04
kernel  5.15.0-79-generic

$ snap list microceph
Name       Version        Rev  Tracking     Publisher   Notes
microceph  0+git.c07ce73  585  latest/edge  canonical✓  -

[sabaini]

Hi @nobuto-m ,

Apologies for the late reply.

I couldn't immediately repro this issue. Would you be able to share some more details:

• Any apparmor or seccomp messages in dmesg? It would be ideal to get a sosreport --all-logs report
• I presume you have run sudo snap connect microceph:dm-crypt, is that right?
• Can you add this disk without encryption, i.e. run sudo microceph disk add /dev/vdc --wipe?

Thanks in advance.

[nobuto-m]

Sure thing. I have the following list of snaps installed on 3 machines and the microceph is behind the microcloud cluster.

$ snap list
Name        Version                  Rev    Tracking       Publisher   Notes
core20      20230622                 1974   latest/stable  canonical✓  base
core22      20230801                 864    latest/stable  canonical✓  base
lxd         5.17-e5ead86             25505  latest/stable  canonical✓  -
microceph   0+git.a77ce2c            627    latest/edge    canonical✓  -
microcloud  0+git.091ee78            540    latest/edge    canonical✓  -
microovn    22.03.2+snap4a6fea39cd   244    22.03/stable   canonical✓  -
snapd       2.60.3+git1281.g75a2072  20184  latest/edge    canonical✓  snapd

And this microcloud cluster was set up by the following snippet and the content of it should be straightforward, I think.
https://github.com/nobuto-m/quick-microcloud/blob/master/redeploy-microcloud.sh

mc-1:~$ sudo microcloud init 
Select an address for MicroCloud's internal traffic:

 Using address "192.168.123.172" for MicroCloud

Limit search for other MicroCloud servers to 192.168.123.172/24? (yes/no) [default=yes]: 
Scanning for eligible servers ...

 Selected "mc-2" at "192.168.123.160"
 Selected "mc-3" at "192.168.123.155"

Would you like to set up local storage? (yes/no) [default=yes]: 
Select exactly one disk from each cluster member:

Select which disks to wipe:

 Using "/dev/disk/by-id/scsi-SATA_QEMU_HARDDISK_QM00001" on "mc-1" for local storage pool
 Using "/dev/disk/by-id/scsi-SATA_QEMU_HARDDISK_QM00001" on "mc-2" for local storage pool
 Using "/dev/disk/by-id/scsi-SATA_QEMU_HARDDISK_QM00001" on "mc-3" for local storage pool

Would you like to set up distributed storage? (yes/no) [default=yes]: 
Select from the available unpartitioned disks:

Select which disks to wipe:

 Using 1 disk(s) on "mc-2" for remote storage pool
 Using 1 disk(s) on "mc-3" for remote storage pool
 Using 1 disk(s) on "mc-1" for remote storage pool

Configure distributed networking? (yes/no) [default=yes]: no
Initializing a new cluster
 Local MicroCloud is ready
 Local LXD is ready
 Local MicroOVN is ready
 Local MicroCeph is ready
Awaiting cluster formation ...
 Peer "mc-3" has joined the cluster
 Peer "mc-2" has joined the cluster
Cluster initialization is complete
MicroCloud is ready

To answer your questions,

I presume you have run sudo snap connect microceph:dm-crypt, is that right?

Yes, it errors out as expected in the initial run, then I ran it explicitly.

mc-1:~$ sudo microceph disk add /dev/vdd --wipe --encrypt
Error: Failed adding new disk: Encryption unsupported on this machine: dm-crypt interface connection missing: 
Use "sudo snap connect microceph:dm-crypt" to enable encryption.

$ sudo snap connect microceph:dm-crypt

$ sudo microceph disk add /dev/vdd --wipe --encrypt
Error: Failed adding new disk: Failed to open: Failed to luksOpen: /dev/disk/by-path/virtio-pci-0000:0c:00.0, exit status 2, Failed to load key in kernel keyring.
No key available with this passphrase.


NOTE: OSD Encryption requires a snapd >= 2.59.1
Verify your version of snapd by running "snap version"

$ snap version
snap    2.60.3+git1281.g75a2072
snapd   2.60.3+git1281.g75a2072
series  16
ubuntu  22.04
kernel  5.15.0-79-generic

Can you add this disk without encryption, i.e. run sudo microceph disk add /dev/vdc --wipe?

Yes, it works without encryption. As you can see below, the device was added as osd.3 (4th drive in the cluster).

ubuntu@mc-1:~$ sudo microceph disk add /dev/vdd --wipe 
ubuntu@mc-1:~$ echo $?
0
ubuntu@mc-1:~$ sudo microceph.ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME      STATUS  REWEIGHT  PRI-AFF
-1         0.12476  root default                            
-2         0.06238      host mc-1                           
 0         0.03119          osd.0      up   1.00000  1.00000
 3         0.03119          osd.3      up   1.00000  1.00000
-4         0.03119      host mc-2                           
 2         0.03119          osd.2      up   1.00000  1.00000
-3         0.03119      host mc-3                           
 1         0.03119          osd.1      up   1.00000  1.00000

Any apparmor or seccomp messages in dmesg? It would be ideal to get a sosreport --all-logs report

sosreport-mc-1-2023-09-06-nuygxvw.tar.gz

[sabaini]

Many thanks, I believe I could reproduce your results.

It appears that the seccomp assertions are not applied to running snap processes. Restarting the microceph.daemon process after connecting dm-crypt works around this issue. I have filed a case for snapd in bug #2034585 for this and will document the workaround in the MicroCeph docs as well, possibly will have to add a daemon restart in an interface hook

[nobuto-m]

I think the following part needs the same change.

microceph/microceph/ceph/osd.go

Line 206 in cecb46e

helper := fmt.Sprint("Use \"sudo snap connect microceph:dm-crypt\" to enable encryption.")

Originally posted by @nobuto-m in #211 (comment)

[sabaini]

@nobuto-m +1 , added in #217