virtme: enable ssh support #210

arighi · 2024-12-23T09:54:05Z

Add support for specifying --server ssh, to automatically start and configure sshd in the virtme-ng guest, and --client ssh to connect to a virtme-ng guest via SSH.

The server's port can be customized with the --port NUM option (default is 2222).

Example usage (start a vng instance with sshd enabled):

$ vng --server ssh --port 2222

In another shell session on the host, connect to the running instance as a client:

$ vng --client ssh --port 2222

Note that it's also possible to use ssh or scp directly, the --client option is provided for convenience and to be consistent with the vsock option.

Edit: additionally, introduce the following shortcuts for convenience, to quickly enable remote connection (either vsock or ssh):

$ vng --console PORT
$ vng --ssh PORT

Example:

 $ vng --ssh

 # In another terminal
 $ ssh -p 2222 localhost

matttbe

Thank you for looking at this! It looks good, just a few questions, mainly around the new script and options.

virtme/guest/virtme-sshd-script

virtme/commands/run.py

virtme_ng/run.py

virtme/commands/run.py

matttbe

Thank you for the new version!

It looks good to me, apart a few missing mkdir -p. I also added a few questions and suggestions, but they might not require changes.

README.md

virtme_ng/run.py

virtme/guest/virtme-sshd-script

virtme/commands/run.py

matttbe

Sorry, a few more comments :)

But only one thing to change I think to properly fill authorized_keys file.

virtme/guest/virtme-sshd-script

virtme_ng/run.py

virtme/commands/run.py

virtme/guest/virtme-sshd-script

arighi · 2024-12-29T17:26:41Z

About the host keys, thanks for clarifying. I think most VM managers automatically generate new keys, but they also give the option to pass some pre-generated keys (using cloud-init for example) with special config options. We can probably follow the same approach, auto-generate them for now and in the future add an option to pass some pre-generate host keys if it's really needed.

matttbe

Thank you for the last update!

This last version looks good to me, and I tested it.

I have just two minors comments on the sshd script, but you can ignore them and merge directly if you prefer.

virtme/guest/virtme-sshd-script

Add support for specifying `--server ssh`, to automatically start and configure sshd in the virtme-ng guest, and `--client ssh` to connect to a virtme-ng guest via SSH. The server's port can be customized with the `--port NUM` option (default is 2222). Example usage (start a vng instance with sshd enabled): $ vng --server ssh --port 2222 In another shell session on the host, connect to the running instance as a client: $ vng --client ssh --port 2222 Note that it's also possible to use ssh or scp directly, the --client option is provided for convenience and to be consistent with the vsock option. Signed-off-by: Andrea Righi <arighi@nvidia.com>

Simplify the remote console and ssh options for the vng front-end: - to start a console or ssh server: $ vng --console [PORT] $ vng --ssh [PORT] - to start a console or ssh client session: $ vng --console-client [PORT] $ vng --ssh-client [PORT] Signed-off-by: Andrea Righi <arighi@nvidia.com>

arighi · 2024-12-30T16:02:42Z

@matttbe I've applied your patch to virtme-sshd-script and added more details to the comment about the check for the read-only rootfs.

I think this PR is decent enough for a merge at this point and we can improve things in-tree later. Moreover, after this PR is merged I was considering to cut a new release, because we have some really cool features. :)

Thank you so much for all the help on this!

matttbe · 2024-12-30T16:56:44Z

@matttbe I've applied your patch to virtme-sshd-script and added more details to the comment about the check for the read-only rootfs.

Thank you!

I think this PR is decent enough for a merge at this point and we can improve things in-tree later.

Sounds good to me!

Moreover, after this PR is merged I was considering to cut a new release, because we have some really cool features. :)

Good idea!

Once released, I will use it in my script to use the console instead of the serial!

Thank you so much for all the help on this!

You are welcome! Thank you for the modifications and the different updates!

marcosps · 2024-12-30T18:14:01Z

Thanks a lot for working on this feature! I just tried to test it, and the vng -r --ssh works, but when I try running a new instance of vng as client with vng --ssh-client, it returns this error message:

kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 2222

The same happens when trying to connect with plain ssh:

$ ssh -p 2222 localhost
kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 2222

Is it something wrong with my machine? I mean, I never really playing with portfwd or anything like that, so I'm not sure what can be wrong in this case: if it's related to vng or something on my machine's ssh.

Also, I installed virtme-ng using pipx:

$ pipx install --force .

Inside the virtme-ng clonned dir.

arighi · 2024-12-30T18:19:07Z

Thanks a lot for working on this feature! I just tried to test it, and the vng -r --ssh works, but when I try running a new instance of vng as client with vng --ssh-client, it returns this error message:
kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 2222

Hi @marcosps, to make sure I understand, when you say it works, you mean that you're able to ssh into the first instance, but other (new) instances are not working? Or even the first one is not working?

matttbe · 2024-12-30T18:19:12Z

@marcosps thank you for testing it!

Can you ssh localhost from inside the vm?

marcosps · 2024-12-30T18:24:11Z

Thanks a lot for working on this feature! I just tried to test it, and the vng -r --ssh works, but when I try running a new instance of vng as client with vng --ssh-client, it returns this error message:
kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 2222
Hi @marcosps, to make sure I understand, when you say it works, you mean that you're able to ssh into the first instance, but other (new) instances are not working? Or even the first one is not working?

@arighi it "works" because I see the VM started (the server), and QEMU is running.

IIUC, by using --ssh on the "server" I would still see a bash session on the "server" as the same as before, and only the --ssh-client on a different session would allow me to "join" in the same VM, right?

@marcosps thank you for testing it!

Can you ssh localhost from inside the vm?

@matttbe

Nope: ssh: connect to host localhost port 22: Connection refused

matttbe · 2024-12-30T18:28:22Z

Can you ssh localhost from inside the vm?

@matttbe: Nope: ssh: connect to host localhost port 22: Connection refused

This feature has a few assumptions: the user has a "normal" key at the traditional place (~/.ssh/id_*.pub), overlayfs is used in the home dir, only the current user executing vng can log in, sshd is installed on the host (or chroot), etc.

Maybe best to check inside the VM what's there: sshd running? .authorized_keys OK? etc.? Maybe run ssh with -vvv.

arighi · 2024-12-30T20:51:48Z

@marcosps you can also try to run virtme-sshd-script (from the guest tools dir) manually inside the guest to see if you get an explicit error, it looks like sshd failed to start in your case.

marcosps · 2024-12-30T21:02:45Z

Can you ssh localhost from inside the vm?

@matttbe: Nope: ssh: connect to host localhost port 22: Connection refused

This feature has a few assumptions: the user has a "normal" key at the traditional place (~/.ssh/id_*.pub), overlayfs is used in the home dir, only the current user executing vng can log in, sshd is installed on the host (or chroot), etc.

Yep, sshd is installed, I'm trying to ssh into the VM with the same user, by "overlayfs in the home dir", you mean, having overlayfs support on the VM? if yes, than I'm covered. I also have the "normal" keys in my ~/.ssh/

Maybe best to check inside the VM what's there: sshd running? .authorized_keys OK? etc.? Maybe run ssh with -vvv.

@marcosps you can also try to run virtme-sshd-script (from the guest tools dir) manually inside the guest to see if you get an explicit error, it looks like sshd failed to start in your case.

Thanks @matttbe and @arighi it seems that my sshd is failing to start:

mpdesouza@virtme-ng:~/git/linux> /home/mpdesouza/git/virtme-ng/virtme/guest/virtme-sshd-script
rm: cannot remove '/var/run/nologin': Operation not permitted
/usr/etc/ssh/sshd_config: Permission denied
mpdesouza@virtme-ng:~/git/linux> ls -l /usr/etc/ssh/sshd_config
-rw-r----- 1 root root 3727 Oct 28 08:18 /usr/etc/ssh/sshd_config

I'm using openSUSE Tumbleweed, which already uses the usrmerge (either way, I'm running as a normal user, which doesn't allow me to start sshd either way)....

arighi · 2024-12-30T21:07:48Z

...

Thanks @matttbe and @arighi it seems that my sshd is failing to start:
mpdesouza@virtme-ng:~/git/linux> /home/mpdesouza/git/virtme-ng/virtme/guest/virtme-sshd-script
rm: cannot remove '/var/run/nologin': Operation not permitted
/usr/etc/ssh/sshd_config: Permission denied
mpdesouza@virtme-ng:~/git/linux> ls -l /usr/etc/ssh/sshd_config
-rw-r----- 1 root root 3727 Oct 28 08:18 /usr/etc/ssh/sshd_config
I'm using openSUSE Tumbleweed, which already uses the usrmerge (either way, I'm running as a normal user, which doesn't allow me to start sshd either way)....

You need to run the sshd script as root (in the guest), and you probably need to export virtme_ssh_user=YOUR_USER, sorry I forgot to mention that.

marcosps · 2024-12-30T21:26:37Z

...
Thanks @matttbe and @arighi it seems that my sshd is failing to start:
mpdesouza@virtme-ng:~/git/linux> /home/mpdesouza/git/virtme-ng/virtme/guest/virtme-sshd-script
rm: cannot remove '/var/run/nologin': Operation not permitted
/usr/etc/ssh/sshd_config: Permission denied
mpdesouza@virtme-ng:~/git/linux> ls -l /usr/etc/ssh/sshd_config
-rw-r----- 1 root root 3727 Oct 28 08:18 /usr/etc/ssh/sshd_config
I'm using openSUSE Tumbleweed, which already uses the usrmerge (either way, I'm running as a normal user, which doesn't allow me to start sshd either way)....
You need to run the sshd script as root (in the guest), and you probably need to export virtme_ssh_user=YOUR_USER, sorry I forgot to mention that.

Ah, ok ok, makes sense. Sorry about it. I'm now running as root, but I'm still seeing the EACCESS error:

# /home/mpdesouza/git/virtme-ng/virtme/guest/virtme-sshd-script 
/usr/etc/ssh/sshd_config: Permission denied

# id
'uid=0(root) gid=0(root) groups=0(root)

# /home/mpdesouza/git/virtme-ng/virtme/guest/virtme-sshd-script 
/usr/etc/ssh/sshd_config: Permission denied

# echo $virtme_ssh_user 
mpdesouza

I tried to strace it, and it seems that's still triggering the EACCES, even tough I'm root:

# strace --follow-forks /home/mpdesouza/git/virtme-ng/virtme/guest/virtme-sshd-scr
ipt

.... <lots and lots of output>
[pid   415] newfstatat(AT_FDCWD, "/usr/etc/ssh/sshd_config", {st_mode=S_IFREG|0640, st_size=3727, ...}, 0) = 0
[pid   415] newfstatat(AT_FDCWD, "/etc/ssh/sshd_config", 0x7fffe0f8ac00, 0) = -1 ENOENT (No such file or directory)
[pid   415] openat(AT_FDCWD, "/usr/etc/ssh/sshd_config", O_RDONLY) = -1 EACCES (Permission denied)
[pid   415] dup(2)                      = 7
[pid   415] fcntl(7, F_GETFL)           = 0x2 (flags O_RDWR)
[pid   415] fstat(7, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0
[pid   415] write(7, "/usr/etc/ssh/sshd_config: Permis"..., 44/usr/etc/ssh/sshd_config: Permission denied
) = 44
[pid   415] close(7)                    = 0
[pid   415] exit_group(1)               = ?
[pid   415] +++ exited with 1 +++

So I'm not sure what's going on... starting sshd on my host system as root works, so I'm not sure why this doesn't work...

arighi · 2024-12-30T22:16:30Z

I tried to strace it, and it seems that's still triggering the EACCES, even tough I'm root:

# strace --follow-forks /home/mpdesouza/git/virtme-ng/virtme/guest/virtme-sshd-scr
ipt

.... <lots and lots of output>
[pid   415] newfstatat(AT_FDCWD, "/usr/etc/ssh/sshd_config", {st_mode=S_IFREG|0640, st_size=3727, ...}, 0) = 0
[pid   415] newfstatat(AT_FDCWD, "/etc/ssh/sshd_config", 0x7fffe0f8ac00, 0) = -1 ENOENT (No such file or directory)
[pid   415] openat(AT_FDCWD, "/usr/etc/ssh/sshd_config", O_RDONLY) = -1 EACCES (Permission denied)
[pid   415] dup(2)                      = 7
[pid   415] fcntl(7, F_GETFL)           = 0x2 (flags O_RDWR)
[pid   415] fstat(7, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0
[pid   415] write(7, "/usr/etc/ssh/sshd_config: Permis"..., 44/usr/etc/ssh/sshd_config: Permission denied
) = 44
[pid   415] close(7)                    = 0
[pid   415] exit_group(1)               = ?
[pid   415] +++ exited with 1 +++

So I'm not sure what's going on... starting sshd on my host system as root works, so I'm not sure why this doesn't work...

Ah ok, makes sense, that's because sshd is trying to access /usr/etc/ssh/sshd_config on the host via virtiofsd, but virtiofsd (that runs as your regular user) probably doesn't have access to that file, hence EACCESS.

In my case I have the config in /etc/ssh/sshd_config, that is world readable. Maybe we can fix this by generating a custom sshd_config in tmp and bind mount to the real sshd_config if we don't have read access.

matttbe · 2024-12-30T22:17:07Z

I'm using openSUSE Tumbleweed, which already uses the usrmerge (either way, I'm running as a normal user, which doesn't allow me to start sshd either way)....

You need to run the sshd script as root (in the guest), and you probably need to export virtme_ssh_user=YOUR_USER, sorry I forgot to mention that.

Ah, ok ok, makes sense. Sorry about it. I'm now running as root, but I'm still seeing the EACCESS error:

From the host, can you access /usr/etc/ssh/sshd_config?
If not, and if vng is started by your user, I guess it is normal not to have access to this file from the guest.

matttbe · 2024-12-30T22:20:14Z

Maybe we can fix this by generating a custom sshd_config in tmp and bind mount to the real sshd_config if we don't have read access.

Or could it be enough to write something in the troubleshooting section in the README file for the moment?
Or maybe it is easy to maintain a small config file? Can we easily find where this file should be located?

arighi · 2024-12-30T22:23:46Z

Maybe we can fix this by generating a custom sshd_config in tmp and bind mount to the real sshd_config if we don't have read access.

Or could it be enough to write something in the troubleshooting section in the README file for the moment? Or maybe it is easy to maintain a small config file? Can we easily find where this file should be located?

Yeah, finding the proper location for the confiig file can be tricky, I'd hope that the first path that sshd checks is /etc/ssh//sshd_config, if that's the case we may just place a custom config there and be happy...

matttbe · 2024-12-30T22:44:19Z

Maybe we can fix this by generating a custom sshd_config in tmp and bind mount to the real sshd_config if we don't have read access.

Or could it be enough to write something in the troubleshooting section in the README file for the moment? Or maybe it is easy to maintain a small config file? Can we easily find where this file should be located?

Yeah, finding the proper location for the confiig file can be tricky, I'd hope that the first path that sshd checks is /etc/ssh//sshd_config

I guess it is hardcoded at build time, using info passed to the configure script.

if that's the case we may just place a custom config there and be happy...

Maybe it is easier to maintain and always use a minimal custom config. At least we would be sure the port is always 22 for example.

arighi · 2024-12-30T22:53:41Z

I think all distro are configuring sshd to use the config from m /etc/ssh/sshd_config, if we maintain a small file there we can probably solve 90% of the issues (or worst case we can check a couple of known paths depending on the distro)

arighi · 2024-12-31T07:34:35Z

@marcosps @matttbe #212 should fix the EACESS issue.

Moreover, I was thinking, why don't we just allow empty password for all users (see arighi/virtme-ng-init#16) and set PermitEmptyPasswords yes in the custom sshd_config? In this way we can completely get rid of the authorized_keys logic, simplify the sshd setup and make everything more robust.

At the end we don't care about security inside the guest, because it's for testing purposes only, and from a security standpoint it's also safe, because all the filesystem operations are still performed by virtiofsd, that runs as the host user, independently on the user in the guest.

matttbe · 2024-12-31T11:00:52Z

virtme/commands/run.py

+
+    # Setup a port forward network interface for the guest.
+    qemuargs.extend(["-device", "%s,netdev=ssh" % (arch.virtio_dev_type("net"))])
+    qemuargs.extend(["-netdev", "user,id=ssh,hostfwd=tcp::%d-:22" % args.port])


As mentioned on arighi/virtme-ng-init#16 (review), this should be modified:

"hostfwd=tcp:127.0.0.1:%d-:22" % args.port

marcosps · 2024-12-31T12:51:47Z

I tried to strace it, and it seems that's still triggering the EACCES, even tough I'm root:
# strace --follow-forks /home/mpdesouza/git/virtme-ng/virtme/guest/virtme-sshd-scr
ipt

.... <lots and lots of output>
[pid   415] newfstatat(AT_FDCWD, "/usr/etc/ssh/sshd_config", {st_mode=S_IFREG|0640, st_size=3727, ...}, 0) = 0
[pid   415] newfstatat(AT_FDCWD, "/etc/ssh/sshd_config", 0x7fffe0f8ac00, 0) = -1 ENOENT (No such file or directory)
[pid   415] openat(AT_FDCWD, "/usr/etc/ssh/sshd_config", O_RDONLY) = -1 EACCES (Permission denied)
[pid   415] dup(2)                      = 7
[pid   415] fcntl(7, F_GETFL)           = 0x2 (flags O_RDWR)
[pid   415] fstat(7, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0
[pid   415] write(7, "/usr/etc/ssh/sshd_config: Permis"..., 44/usr/etc/ssh/sshd_config: Permission denied
) = 44
[pid   415] close(7)                    = 0
[pid   415] exit_group(1)               = ?
[pid   415] +++ exited with 1 +++
So I'm not sure what's going on... starting sshd on my host system as root works, so I'm not sure why this doesn't work...
Ah ok, makes sense, that's because sshd is trying to access /usr/etc/ssh/sshd_config on the host via virtiofsd, but virtiofsd (that runs as your regular user) probably doesn't have access to that file, hence EACCESS.

In my case I have the config in /etc/ssh/sshd_config, that is world readable. Maybe we can fix this by generating a custom sshd_config in tmp and bind mount to the real sshd_config if we don't have read access.

ahhh, ok, yes, it makes sense that virtiofsd runs as a normal user... thanks a lot for spotting this problem! I'll test your PR to check if that fixes my issue!

arighi requested a review from matttbe December 23, 2024 09:54

arighi force-pushed the ssh-support branch 4 times, most recently from 37d6cdf to 67cb602 Compare December 23, 2024 14:16

matttbe requested changes Dec 28, 2024

View reviewed changes

matttbe reviewed Dec 28, 2024

View reviewed changes

virtme/commands/run.py Outdated Show resolved Hide resolved

arighi force-pushed the ssh-support branch 4 times, most recently from 7b53b12 to 14a9d94 Compare December 28, 2024 18:17

matttbe requested changes Dec 29, 2024

View reviewed changes

arighi force-pushed the ssh-support branch from 14a9d94 to bf096ec Compare December 29, 2024 13:54

matttbe requested changes Dec 29, 2024

View reviewed changes

arighi force-pushed the ssh-support branch from bf096ec to 18ce464 Compare December 29, 2024 17:57

matttbe approved these changes Dec 30, 2024

View reviewed changes

virtme/guest/virtme-sshd-script Show resolved Hide resolved

virtme/guest/virtme-sshd-script Outdated Show resolved Hide resolved

arighi added 2 commits December 30, 2024 16:58

arighi force-pushed the ssh-support branch from 18ce464 to 0d6ca43 Compare December 30, 2024 15:58

arighi merged commit 6e7dc3e into main Dec 30, 2024
6 checks passed

matttbe reviewed Dec 31, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

virtme: enable ssh support #210

virtme: enable ssh support #210

arighi commented Dec 23, 2024 •

edited

Loading

matttbe left a comment

matttbe left a comment

matttbe left a comment

arighi commented Dec 29, 2024

matttbe left a comment

arighi commented Dec 30, 2024

matttbe commented Dec 30, 2024

marcosps commented Dec 30, 2024

arighi commented Dec 30, 2024

matttbe commented Dec 30, 2024

marcosps commented Dec 30, 2024

matttbe commented Dec 30, 2024

arighi commented Dec 30, 2024

marcosps commented Dec 30, 2024

arighi commented Dec 30, 2024

marcosps commented Dec 30, 2024

arighi commented Dec 30, 2024

matttbe commented Dec 30, 2024

matttbe commented Dec 30, 2024

arighi commented Dec 30, 2024

matttbe commented Dec 30, 2024

arighi commented Dec 30, 2024

arighi commented Dec 31, 2024

matttbe Dec 31, 2024

marcosps commented Dec 31, 2024

virtme: enable ssh support #210

virtme: enable ssh support #210

Conversation

arighi commented Dec 23, 2024 • edited Loading

matttbe left a comment

Choose a reason for hiding this comment

matttbe left a comment

Choose a reason for hiding this comment

matttbe left a comment

Choose a reason for hiding this comment

arighi commented Dec 29, 2024

matttbe left a comment

Choose a reason for hiding this comment

arighi commented Dec 30, 2024

matttbe commented Dec 30, 2024

marcosps commented Dec 30, 2024

arighi commented Dec 30, 2024

matttbe commented Dec 30, 2024

marcosps commented Dec 30, 2024

matttbe commented Dec 30, 2024

arighi commented Dec 30, 2024

marcosps commented Dec 30, 2024

arighi commented Dec 30, 2024

marcosps commented Dec 30, 2024

arighi commented Dec 30, 2024

matttbe commented Dec 30, 2024

matttbe commented Dec 30, 2024

arighi commented Dec 30, 2024

matttbe commented Dec 30, 2024

arighi commented Dec 30, 2024

arighi commented Dec 31, 2024

matttbe Dec 31, 2024

Choose a reason for hiding this comment

marcosps commented Dec 31, 2024

arighi commented Dec 23, 2024 •

edited

Loading