console not working

[jimc]

Ive lost my console.

I just checked out v6.9 (unaltered),
created ./builds/x0 subdir for output,
copied Makefile from ./builds/bar; its a 1-liner:
[jimc@frodo x0]$ cat Makefile

Automatically generated by /home/jimc/projects/lx/linux.git/Makefile: don't edit

include /home/jimc/projects/lx/linux.git/Makefile

and ran vng -bv there.

the kernel builds cleanly, and boots fine, but console is un-responsive

except that ^ax will terminate the QEMU session.

all builds/foos in all my worktrees now have this problem.

Any idea what might be causing this ?
Ive done nothing to alter QEMU setup / install
all qemu-* parts are from fedora fc40, unaltered.
(rpm -vvV qemu-kvm to verify it)

[arighi]

Hi, which version of virtme-ng are you using? We had this console issue in fedora (and other distro using a recent version of qemu), that should be fixed now by 4aab16e, included in v1.23.

[jimc]

Im on 1.25

[jimc@frodo x0]$ qemu-x86_64 --version
qemu-x86_64 version 8.1.92 (v8.2.0-rc2)
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers

I rolled back to v1.22 to test that, same result.

but rolling back to v1.20 did work - I have console again.

that said, inside the vm, vng still shows v1.22

[ 2.665666] virtme-ng-init: Starting systemd-udevd version 255.6-1.fc40
[ 2.666396] virtme-ng-init: triggering udev coldplug
[ 2.735372] virtme-ng-init: waiting for udev to settle
[ 3.219672] virtme-ng-init: udev is done
[ 3.220212] virtme-ng-init: initialization done
_ _
__ ()_ | | _ __ ___ ___ _ __ __ _
\ \ / / | | | _ _ \ / _ ____| _ \ / _ |
\ V /| | | | || | | | | | /| | | | (| |
_/ ||| _|| || |_|_| || ||__ |
|___/
kernel version: 6.9.0-virtme x86_64

[root@v6 linux.git]#
[root@v6 linux.git]#
[root@v6 linux.git]#
[root@v6 linux.git]# vng --version
virtme-ng 1.22
[root@v6 linux.git]#

virtme v1.21 breaks console

[jimc@frodo x0]$ vrun_
virtme-ng 1.21
doing: vng --verbose --name v6.9-36-gd4140b527ddc --user root --cwd ../.. -a dynamic_debug.verbose=2 -p 4
virtme: waiting for virtiofsd to start
virtme: use 'microvm' QEMU architecture
No EFI environment detected.
early console in extract_kernel
input_data: 0x00000000029dc2b9
input_len: 0x0000000000b08706
output: 0x0000000001000000
output_len: 0x000000000249f160
kernel_total_size: 0x0000000002230000
needed_size: 0x0000000002600000
trampoline_32bit: 0x0000000000000000
Physical KASLR using RDRAND RDTSC...
Virtual KASLR using RDRAND RDTSC...

Decompressing Linux... Parsing ELF... Performing relocations... done.
Booting the kernel (entry_offset: 0x0000000000000000).
[ 0.000000] Linux version 6.9.0-virtme (jimc@frodo) (gcc (GCC) 14.1.1 20240507 (Red Hat 14.1.1-1), GNU ld version 2.41-37.fc40) #1 SMP PREEMPT_DYNAMIC Fri May 24 09:30:11 MDT 2024
[ 0.000000] Command line: virtme_hostname=v6.9-36-gd4140b527ddc virtme_link_mods=/home/jimc/projects/lx/linux.git/builds/x0/.virtme_mods/lib/modules/0.0.0 virtme_rw_overlay0=/etc virtme_rw_overlay1=/lib virtme_rw_overlay2=/home virtme_rw_overlay3=/opt virtme_rw_overlay4=/srv virtme_rw_overlay5=/usr virtme_rw_overlay6=/var console=hvc0 earlyprintk=serial,ttyS0,115200 virtme_console=ttyS0 psmouse.proto=exps "virtme_stty_con=rows 40 cols 148 iutf8" TERM=xterm-256color virtme_chdir=home/jimc/projects/lx/linux.git rootfstype=virtiofs root=ROOTFS raid=noautodetect ro dynamic_debug.verbose=2 init=/home/jimc/.local/lib/python3.12/site-packages/virtme/guest/bin/virtme-ng-init
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable

and the working 1.20 run

[jimc@frodo x0]$ vrun_
virtme-ng 1.20
doing: vng --verbose --name v6.9-36-gd4140b527ddc --user root --cwd ../.. -a dynamic_debug.verbose=2 -p 4
virtme: waiting for virtiofsd to start
virtme: use 'microvm' QEMU architecture
No EFI environment detected.
early console in extract_kernel
input_data: 0x00000000029dc2b9
input_len: 0x0000000000b08706
output: 0x0000000001000000
output_len: 0x000000000249f160
kernel_total_size: 0x0000000002230000
needed_size: 0x0000000002600000
trampoline_32bit: 0x0000000000000000
Physical KASLR using RDRAND RDTSC...
Virtual KASLR using RDRAND RDTSC...

Decompressing Linux... Parsing ELF... Performing relocations... done.
Booting the kernel (entry_offset: 0x0000000000000000).
[ 0.000000] Linux version 6.9.0-virtme (jimc@frodo) (gcc (GCC) 14.1.1 20240507 (Red Hat 14.1.1-1), GNU ld version 2.41-37.fc40) #1 SMP PREEMPT_DYNAMIC Fri May 24 09:30:11 MDT 2024
[ 0.000000] Command line: virtme_hostname=v6.9-36-gd4140b527ddc virtme_link_mods=/home/jimc/projects/lx/linux.git/builds/x0/.virtme_mods/lib/modules/0.0.0 virtme_rw_overlay0=/etc virtme_rw_overlay1=/lib virtme_rw_overlay2=/home virtme_rw_overlay3=/opt virtme_rw_overlay4=/srv virtme_rw_overlay5=/usr virtme_rw_overlay6=/var earlyprintk=serial,ttyS0,115200 console=ttyS0 psmouse.proto=exps "virtme_stty_con=rows 40 cols 148 iutf8" TERM=xterm-256color virtme_chdir=home/jimc/projects/lx/linux.git rootfstype=virtiofs root=ROOTFS raid=noautodetect ro dynamic_debug.verbose=2 init=/home/jimc/.local/lib/python3.12/site-packages/virtme/guest/bin/virtme-ng-init
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved

[jimc@frodo virtme-ng.git]$ git bisect bad
f3749da is the first bad commit
commit f3749da (HEAD, origin/kernel-log-stderr)
Author: Andrea Righi andrea.righi@canonical.com
Date: Sun Feb 4 10:34:39 2024 +0100

virtme-ng: redirect kernel log to stderr in interactive mode

Redirect kernel messages to stderr when running in interactive mode,
similar to what we do in command/script mode.

In this way it is possible, for example, to save kernel logs to a file
even when running in interactive mode, such as:

 $ vng -vr 2>/tmp/kernel.log

NOTE: unfortunately we can't redirect early boot kernel messages,
earlyprintk doesn't seem to work with virconsole devices. So, early
messages will be still sent to stdout, in this way we don't break the
old behavior.

This addresses discussion #60.

Signed-off-by: Andrea Righi <andrea.righi@canonical.com>

virtme/architectures.py | 8 ++++----
virtme/commands/run.py | 31 +++++++++++++++++++++++--------
virtme/guest/virtme-init | 10 +++++++++-
virtme_ng_init | 2 +-
4 files changed, 37 insertions(+), 14 deletions(-)

[jimc@frodo virtme-ng.git]$ git bisect log
git bisect start

status: waiting for both good and bad commits

bad: [47c7b57] virtme-ng v1.21

git bisect bad 47c7b57

status: waiting for good commit(s), bad commit known

good: [00caf60] virtme-ng v1.20

git bisect good 00caf60

bad: [eacf6bb] virtme-ng: fix sending kernel log to stderr in script mode

git bisect bad eacf6bb

bad: [ef77b37] doc: replace the screenshot with a video in README.md

git bisect bad ef77b37

bad: [a28ac8b] Merge pull request #66 from arighi/kernel-log-stderr

git bisect bad a28ac8b

bad: [f3749da] virtme-ng: redirect kernel log to stderr in interactive mode

git bisect bad f3749da

first bad commit: [f3749da] virtme-ng: redirect kernel log to stderr in interactive mode

[jimc@frodo virtme-ng.git]$

I did just this between each step

[jimc@frodo virtme-ng.git]$ BUILD_VIRTME_NG_INIT=1 pip3 install --verbose -r requirements.txt .

I perused the patch, contemplated hacking out the virtme-console,
but figured I could wait for a more informed approach :-)

[arighi]

Thanks for tracking this down and sorry for the late response, I've been traveling to conferences recently.

I can't reproduce this on Ubuntu, Debian, or Arch, so there's probably something in the fc40 environment that is preventing to create the console device.

Few questions.

Is the console working if you run vng without -v (in non-verbose mode it doesn't create the virtio console used for the dmesg / stderr redirection, so it should fallback to the old behavior of sending all the output to a regular stdio console).

Have you tried to boot the stock fc40 kernel? (I'm wondering if your kernel may be missing a specific config required by the virtioconsole)

Is this a fairly standard fc40 installation? Do you see any potential security-related restrictions that may prevent qemu/vng from redirecting stdin/stdout/stderr in the guest (i.e., SELinux, AppArmor rules, ...)

[jimc]

running stock 6.8.11-300.fc40.x86_64
on both host and in vng -vr or vng -r
fails as before with origin/main

I branched and did rebase -i
to reorder the problem patch and 2 dependents to the end:

3bc27ae (tag: fedora-console-a1, fedora-console) qemu: remove unnecessary '-serial none' arg
0650445 virtme-ng: fix sending kernel log to stderr in script mode
04a4f88 virtme-ng: redirect kernel log to stderr in interactive mode

• testing here passes, so 3 above are the cause of my trouble
  1023f4e (HEAD) init: move all temp files to /run/tmp

So at this point Ive got a working setup.

I'll spend some time soon to figure out which part breaks on fedora

[arighi]

Hm... I'm confused... I think you already found that the problem was introduced in virtme-ng: redirect kernel log to stderr in interactive mode. And we had a problem in fedora introduced by this commit, but that was fixed with qemu: remove unnecessary '-serial none' arg. However, this looks like a different issue right?

Unfortunately I don't have a fedora box at the moment, I'll try to install one in the next days and try to reproduce the problem.

[jimc]

the problem is that patch.

to be sure, I "floated" the commit (and 2 which needed it)
backwards by successive rebases.

the problem moves with it.

so I can use HEAD~3 on the reordered fork,
and figure out what the last 3 (mainly the 1) are doing later.

[jimc@frodo virtme-ng.git]$ git log --oneline -n4 fedora-console
3bc27ae (tag: fedora-console-a1, fedora-console) qemu: remove unnecessary '-serial none' arg
0650445 virtme-ng: fix sending kernel log to stderr in script mode
04a4f88 virtme-ng: redirect kernel log to stderr in interactive mode
1023f4e (HEAD) init: move all temp files to /run/tmp

This way I get the latest features, (except the 1)
and dont have to worry about the changing meaning
of several options in the meantime.

so not front burner afaic
sorry for the crappy explanation

separately -

we have --custom # load conf=val s from a file

could we also have --custom-item "CONFIG_FOO=m"

it means needing fewer custom-fragment files,
by allowing single modifiers,
and it directly expresses the meaningful config-combos
rather than hiding all the combinatorics in N different files

it also makes it super clear when scripting variations on a baseline --custom file

--conf-item=CONFIG_KMEMLEAK=y

or maybe even drop the /^CONFIG_/
--config-item=KMEMLEAK=y
--config-item=DYNAMIC_DEBUG_CORE=Y

[arighi]

ok I think I understand now, you rebased re-adding the patches that are introducing the console issue on top, so that you can follow the development and quickly revert the problematic commits.

The problem is for sure introduced by the stderr redirection, but I don't want to revert that commit, because this feature is super useful, also it works on all the other distro, so we just need to understand what fedora is doing differently. I'm planning to allocate some time to prepare a fc40 box in the next days and try to repeat this issue.

About the config change feel free to send a PR if you want and I'll be happy to review / apply it, it seems useful.

[jimc]

ok, for some reason, I cant push my config-item branch to my own fork of your repo.
so Ive got nothing to PR.
I used git send-email instead.

[arighi]

oh... that is odd, if it's in your namespace you should have all the permissions... anyway, an email is also fine :)

[arighi]

One thing I haven't asked (I think)... have you tried to run vng with --no-virtme-ng-init? Does it make any difference? Thanks!

[jimc]

I havent tried that
for many months, I more or less forgot it exists.

related to that, when I switched just now from my config-item branch
I got
[jimc@frodo virtme-ng.git]$ git checkout main
M virtme_ng_init
Switched to branch 'main'

Previously when that happened, I just did git reset --hard,
presuming that should restore main's version of the submodule.
Could this cause or cover up a problem ?

anyway, VIOLA ! IT WORKS

[jimc@virtme-ng virtme-ng.git]$ QEMU: Terminated
[jimc@frodo virtme-ng.git]$ ./vng --no-virtme-ng-init -r
_ _
__ ()_ | | _ __ ___ ___ _ __ __ _
\ \ / / | | | _ _ \ / _ ____| _ \ / _ |
\ V /| | | | || | | | | | /| | | | (| |
_/ ||| _|| || |_|_| || ||__ |
|___/
kernel version: 6.8.11-300.fc40.x86_64 x86_64
(CTRL+d to exit)

[jimc@virtme-ng virtme-ng.git]$
[jimc@virtme-ng virtme-ng.git]$

(the 2nd prompt means the console saw my )
It also works with -vr

so this is good.
What do you want me to try next ? :-D

might it be useful to bisect on the submodule version ?
how to do that ??

[arighi]

ok, that's what I was suspecting, you updated virtme-ng, but not the virtme-ng-init module. Usually I run git submodule update --init --recursive when I update the virtme-ng repo to make sure I get all the required changes also in virtme-ng-init.

In this case after switching to the new stderr redirection you were still using the old "init", that's why the console didn't work.

I guess it's all solved now, just remember to update virtme-ng-init every time you update virtme-ng from git. And try --no-virtme-ng-init if something odd is happening. :)

We can probably close this now (if you see other issues feel free to re-open it), thanks for testing and reporting this!

[jimc]

not quite that simple.

I do have this in my .git/config, I think that means I did the recursive init
[submodule]
active = .

but after doing git reset --hard HEAD,
I still have:

[jimc@frodo virtme-ng.git]$ git diff
diff --git a/virtme_ng_init b/virtme_ng_init
index eeaf7c5..f20306d 160000
--- a/virtme_ng_init
+++ b/virtme_ng_init
@@ -1 +1 @@
-Subproject commit eeaf7c5
+Subproject commit f20306d

I must be doing this wrong :-}

[jimc]

apparently, Im _detached_/ :-)

[jimc@frodo virtme_ng_init]$ git fetch
[jimc@frodo virtme_ng_init]$ git log --oneline
f20306d (HEAD) virtme-ng: honor virtme_user when running user script
2f0d249 virtme-ng-init: make virtme-script a constant
af4cfe1 virtme-ng-init: support long commands in graphic mode

I fixed this, with cd into submod
eeaf7c5 (HEAD, origin/main, origin/HEAD) virtme-ng-init: move all temp files to /run/tmp
8a051da Merge pull request #9 from ThibF/thibf/allow_tmp_overlay
9f75983 Merge pull request #10 from ThibF/thibf/build_install_doc

then
git checkout eeaf7c

this fixes git diff in parent.
but it still leaves me with the bug

35 ./vng -vr
36 ./vng -vr --no-virtme-ng-init

35 breaks, 36 works.

its obviously low priority here, so I'll take a few hints,
and investigate myself

[arighi]

Are you recompiling virtme-ng-init after updating the submodule?

[jimc]

On Sat, Jun 15, 2024 at 4:45 PM Andrea Righi ***@***.***> wrote: Are you recompiling virtme-ng-init after updating the submodule?

oh, you mean, like: ***@***.*** virtme-ng.git]$ make cd virtme_ng_init && cargo install --path . --root ../virtme/guest Installing virtme-ng-init v0.1.0 (/home/jimc/projects/virtme-ng.git/virtme_ng_init) Updating crates.io index Compiling virtme-ng-init v0.1.0 (/home/jimc/projects/virtme-ng.git/virtme_ng_init) Finished `release` profile [optimized] target(s) in 1.63s Replacing ../virtme/guest/bin/virtme-ng-init Replaced package `virtme-ng-init v0.1.0 (/home/jimc/projects/virtme-ng.git/virtme_ng_init)` with `virtme-ng-init v0.1.0 (/home/jimc/projects/virtme-ng.git/virtme_ng_init)` (executable `virtme-ng-init`) warning: be sure to add `../virtme/guest/bin` to your PATH to be able to run the installed binaries and with that, this now works: ***@***.*** virtme-ng.git]$ ./vng -vr so, thanks for all your help isolating this, sorry for the noise.

[arighi]

Yep, exactly, virtme-ng-init is a Rust project, so it requires a make every time it is updated. And no problem at all, glad we figure this out. :)