virtme-ng: use pre-compiled mainline kernels

[arighi]

This is a PoC of an idea discussed with @marcosps at LPC 2023 in Richmond (VA): use pre-compiled kernels to get rid of the build time when we need to recompile specific kernel versions.

For the pre-compiled kernels we can rely on the Ubuntu mainline repository: Ubuntu periodically provides deb packages for the main upstream kernel versions (for debugging purposes), that are essentially vanilla kernels with the generic Ubuntu .config, so they are general-purpose kernels suitable for any kind of test.

This can help to speed up time (and energy 💚) when we need to test specific upstream kernel versions, by cutting completely the kernel rebuild time.

See commit description for implementation details and usage.

Feedbacks are welcome!

[SPYFF]

Great feature! My only concern here is linux-tools package AFAIK does not compiled for these mainline kernel builds. We have modules, image and headers, but IMO tools like bpftool, perf, etc. are missing.

I brought this up about a year ago (https://lists.ubuntu.com/archives/kernel-team/2022-July/131852.html).
But this feature still has value in its own right.

[arighi]

Great feature! My only concern here is linux-tools package AFAIK does not compiled for these mainline kernel builds. We have modules, image and headers, but IMO tools like bpftool, perf, etc. are missing.

I brought this up about a year ago (https://lists.ubuntu.com/archives/kernel-team/2022-July/131852.html). But this feature still has value in its own right.

Good point! I'll try to ping someone and see if we can start publishing linux-tools as well.

However, about perf one of the dev at LPC told me that perf is backward and forward compatible, so potentially we can use any version of perf with any kernel. We are shipping it in linux-tools (that has a strict dependency with the kernel version), but theoretically we could just relax this dependency. It is just convenient to release a new perf when the new kernel is released, but for the pre-compiled mainline kernels I think we can use any perf that is installed on the host (inside the virtme-ng session).

I'm pretty sure this applies to bpftrace (to a large degree) and with BTF enabled we should be able to run pretty much any bpftrace script.

Excluding perf and bpftrace what's missing from linux-tools is acpidbg (maybe not very useful inside virtme-ng), turbostat/cpupower (definitely not useful inside virtme-ng), usbip/usbipd (could be useful, but I'm pretty sure this is also not strictly dependent on the kernel version).

Are you aware of any other tool that may be strictly dependent on the specific kernel version that we're running?

[SPYFF]

Good point! I'll try to ping someone and see if we can start publishing linux-tools as well.

Would be great, thanks!

However, about perf one of the dev at LPC told me that perf is backward and forward compatible, so potentially we can use any version of perf with any kernel. We are shipping it in linux-tools (that has a strict dependency with the kernel version), but theoretically we could just relax this dependency. It is just convenient to release a new perf when the new kernel is released, but for the pre-compiled mainline kernels I think we can use any perf that is installed on the host (inside the virtme-ng session).

Thanks for the info, I missed that, good to know perf is forward compatible.

I'm pretty sure this applies to bpftrace (to a large degree) and with BTF enabled we should be able to run pretty much any bpftrace script.

Excluding perf and bpftrace what's missing from linux-tools is acpidbg (maybe not very useful inside virtme-ng), turbostat/cpupower (definitely not useful inside virtme-ng), usbip/usbipd (could be useful, but I'm pretty sure this is also not strictly dependent on the kernel version).

Are you aware of any other tool that may be strictly dependent on the specific kernel version that we're running?

I mostly use bpftrace/libbpf-tools for basic tracing and debugging stuff, and as you pointed out, it relies on BTF, so not much external dependency required. I agree, linux-tools are mostly useful for bare-metal testing, and less valuable for virtme-ng.

The problem is that the host may be running a very different configuration from Ubuntu, possibly with a very old kernel and tooling, and since there is no easily available and installable linux-tools package, you have to go back to source to install things.

perf and bpftool are evolving very fast, keeping up with the kernel. When running virtme-ng on an LTS host, the tooling of the host kernel may be more than a year behind the mainline. Very usecase-dependent, but experimentation is exactly why this can easily be a problem: I suspect no one wants to recompile fresh tooling from source for an otherwise precompiled, fresh kernel (even if it is of course much less of a problem than recompiling the kernel itself from scratch).

As a next step, I think it would be good to have linux-tools package for the mainline kernel. For example, Debian does this, but I don't have enough experience to recommend it as an alternative.

However, I think this feature is very good in this form, because in most cases you can experiment without linux-tools.

[arighi]

...

The problem is that the host may be running a very different configuration from Ubuntu, possibly with a very old kernel and tooling, and since there is no easily available and installable linux-tools package, you have to go back to source to install things.

perf and bpftool are evolving very fast, keeping up with the kernel. When running virtme-ng on an LTS host, the tooling of the host kernel may be more than a year behind the mainline. Very usecase-dependent, but experimentation is exactly why this can easily be a problem: I suspect no one wants to recompile fresh tooling from source for an otherwise precompiled, fresh kernel (even if it is of course much less of a problem than recompiling the kernel itself from scratch).

As a next step, I think it would be good to have linux-tools package for the mainline kernel. For example, Debian does this, but I don't have enough experience to recommend it as an alternative.

However, I think this feature is very good in this form, because in most cases you can experiment without linux-tools.

Yeah, this is a valid point, it shouldn't be too hard to provide linux-tools as well, I'll talk directly to the maintainer of the mainline builds in Ubuntu, it would definitely provide an additional value to this feature, especially for those that are not running the very latest kernel (as you correctly pointed out).

In the meantime, I think the lack of linux-tools shouldn't be a blocker to add this feature in virtme-ng, at the end even if you recompile your own kernel from source and your user-space tools on the host are not recent enough you may experience exactly the same issues. Basically the pre-compiled kernels can simply help you to save build time if you want to do a quick test exactly with a specific upstream version, but they don't provide anything more than what a regular kernel built from source would provide.

Thank you for your feedback @SPYFF (useful as always!)

[SPYFF]

Agree, its a non-blocker.