Enable offloading for 4-5x speedup

[nirs]

You can get 4-5x times better performance with offloading:
https://github.com/nirs/vmnet-helper/blob/main/docs/performance.md#offloading

In libkrun set features to COMPAT_NET_FEATURES:

anylinuxfs/anylinuxfs/src/main.rs

Line 986 in 71045cb

0, // COMPAT_NET_FEATURES,

Enable offloading in vment-helper:

• --enable-checksum-offload
• --enable-tso
  

  anylinuxfs/anylinuxfs/src/vm_network.rs

  Line 139 in 71045cb

  vmnet_helper_cmd

Note: you need libkrun with this fix:
containers/libkrun#556

[nohajc]

I remember reading that offloading speeds up data transfers in one direction while crippling the performance in the other. Is that no longer the case?

I suppose it's related to the libkrun issue you fixed?

[nirs]

I remember reading that offloading speeds up data transfers in one direction while crippling the performance in the other. Is that no longer the case?

It is still the case on macOS < 26.2:
https://github.com/nirs/vmnet-helper/blob/main/docs/performance.md#macos-156

But on macOS >= 26.2 this issue is gone:
https://github.com/nirs/vmnet-helper/blob/main/docs/performance.md#macos-263

Since you already check macOS version for rootless usage, you can tweak the options when running on macOS >= 26.2. You can remove the special code when macOS 28 is released and macOS 15 becomes unsupported.

I suppose it's related to the libkrun issue you fixed?

The issue was revealed when using podman libkrun machine as registry cache, and can be easily reproduced with iperf3 when offloading is enabled. The fix makes this configuration usable.

[nohajc]

Thank you for the tip! I'll definitely enable this when your libkrun fix is released. I use the homebrew version dynamically linked.

Any specific vmnet-helper version needed? I still need to upgrade from 0.9.

[nirs]

Any specific vmnet-helper version needed? I still need to upgrade from 0.9.

It should work with any version but using latest version is best.

[Golui]

Wanted to try this out before the libkrun patch is available in a release, with ZFS on FreeBSD. However, anylinuxfs forces the network helper to be gvproxy.

anylinuxfs/common-utils/src/lib.rs

Lines 156 to 158 in bdb142a

	match os {
	OSType::FreeBSD => NetHelper::GvProxy,
	OSType::Linux => self,

Is there any reason for this?

[nohajc]

Yes, I need to document this... In my tests, vmnet-helper with FreeBSD leads to networking issues such as failing NFS mounts and very high I/O latency. Haven't tried to debug this yet.

[nohajc]

Should be reproducible by simply mounting a zpool with multiple datasets.

Anyway, I can't tell if this is a vmnet or FreeBSD virtio-net issue.

[nirs]

@nohajc Any tips for using freebsd image with krunkit? I would like to test it.

[nohajc]

I never used krunkit. If you run anylinuxfs image install freebsd-15.0, it will boostrap an image for you which is usable with libkrun (and has a script that starts an NFS server). See ~/.anylinuxfs/freebsd-15.0.

There are some quirks though. FreeBSD cannot boot with virtio-console so in my setup, it uses a legacy serial console instead.

Not sure if you can make it work out-of-the-box with krunkit but anylinuxfs shell -i freebsd might be usable for testing too... If you patch it to use vmnet-helper. I can help with that if needed.

[Golui]

krunkit does not support direct kernel booting afaik, which your VM boostrapping relies on.

I have started working on a patch today that should add legacy console capabilities to krunkit based on what @nohajc did for anylinuxfs, but I wasn't able to actually get any output for even stock alpine linux images (even after adjusting the kernel boot parameters to include console=hvc0). I'll drop it here in case you find it useful.

diff --git a/src/virtio.rs b/src/virtio.rs
index 9a6e337..cbbd3df 100644
--- a/src/virtio.rs
+++ b/src/virtio.rs
@@ -53,6 +53,8 @@ extern "C" {
     fn krun_add_vsock_port2(ctx_id: u32, port: u32, c_filepath: *const c_char, listen: bool)
         -> i32;
     fn krun_add_virtiofs(ctx_id: u32, c_tag: *const c_char, c_path: *const c_char) -> i32;
+    fn krun_disable_implicit_console(ctx_id: u32) -> i32;
+    fn krun_add_serial_console_default(ctx_id: u32, input_fd: c_int, output_fd: c_int) -> i32;
     fn krun_set_console_output(ctx_id: u32, c_filepath: *const c_char) -> i32;
     fn krun_add_net_unixgram(
         ctx_id: u32,
@@ -221,40 +223,58 @@ impl KrunContextSet for BlkConfig {
 }
 
 /// Configuration of a virtio-serial device.
-#[derive(Clone, Debug, PartialEq)]
+#[derive(Clone, Debug, Default, PartialEq)]
 pub struct SerialConfig {
     /// Path of a file to use as the device's log.
-    pub log_file_path: PathBuf,
+    pub log_file_path: Option<PathBuf>,
+    pub legacy: bool,
 }
 
 impl FromStr for SerialConfig {
     type Err = anyhow::Error;
 
     fn from_str(s: &str) -> Result<Self, Self::Err> {
+        let mut serial_cfg = Self::default();
         let mut args = parse_args(s.to_string())?;
-        check_required_args(&args, "virtio-serial", &["logFilePath"])?;
 
-        let log_file_path = args.remove("logFilePath").unwrap();
+        if let Some(log_file_path) = args.remove("logFilePath") {
+            serial_cfg.log_file_path = PathBuf::from_str(log_file_path.as_str())
+                .context("logFilePath argument not a valid path")
+                .ok();
+        }
+
+        if let Some(arg_legacy) = args.remove("legacy") {
+            serial_cfg.legacy = parse_boolean(&arg_legacy)?;
+        }
         check_unknown_args(args, "virtio-serial")?;
 
-        Ok(Self {
-            log_file_path: PathBuf::from_str(log_file_path.as_str())
-                .context("logFilePath argument not a valid path")?,
-        })
+        Ok(serial_cfg)
     }
 }
 
 /// Set the krun console output to be written to the virtio-serial's log file.
 impl KrunContextSet for SerialConfig {
     unsafe fn krun_ctx_set(&self, id: u32) -> Result<(), anyhow::Error> {
-        let path_cstr = path_to_cstring(&self.log_file_path)?;
+        if let Some(path_ok) = &self.log_file_path {
+            if self.legacy {
+                return Err(anyhow!("unable to use legacy console with logFilePath"));
+            }
+            let path_cstr = path_to_cstring(&path_ok)?;
+            if krun_set_console_output(id, path_cstr.as_ptr()) < 0 {
+                return Err(anyhow!(
+                    "unable to set krun console output redirection to virtio-serial log file"
+                ));
+            }
 
-        if krun_set_console_output(id, path_cstr.as_ptr()) < 0 {
-            return Err(anyhow!(
-                "unable to set krun console output redirection to virtio-serial log file"
-            ));
+            return Ok(());
+        } else if self.legacy {
+            if krun_disable_implicit_console(id) < 0 {
+                return Err(anyhow!("unable to disable implicit console"));
+            }
+            if krun_add_serial_console_default(id, 0, 1) < 0 {
+                return Err(anyhow!("unable to add serial default console"));
+            }
         }
-
         Ok(())
     }
 }

[nohajc]

@Golui Make sure you're running the latest libkrun with krunkit. There were some fixes related to FreeBSD booting and the console output. But that was even before 1.17 I think.

[nohajc]

If the console stuff in krunkit is solved, it should probably be able to boot the official FreeBSD VM image as well. There's raw UFS or ZFS:

https://download.freebsd.org/releases/VM-IMAGES/15.0-RELEASE/aarch64/Latest/

[Golui]

I was actually unable to mount anything with a bleeding edge libkrun. Figured I would track the issue down myself instead of filing an issue for an unreleased version of libkrun.

I hit it a few hours ago and I'll be bisecting it shortly to see what broke. Can't build 1.17.4 on my system due to some init cross-compile issues and can't simply cherry-pick commits that fix it.

macOS: root_disk: /Users/golui/.anylinuxfs/freebsd-15.0/freebsd-microvm-disk.img
macOS: num_vcpus: 4
macOS: ram_size_mib: 8192
macOS: vmproxy args: ["/vmproxy-bsd", "/dev/vtbd1", "tank", "-b", "127.0.0.1", "-t", "zfs_member", "-m", "-v"]
  BSD: add net default: gateway 192.168.127.1
  BSD: Error: Failed to initialize network: Failed to expose port: PortDef { local: ":111", remote: "192.168.127.2:111" }: error sending request for url (http://192.168.127.1/services/forwarder/expose): client error (Connect): tcp connect error: Network is down (os error 50)
macOS: NFS server not ready
macOS: libkrun VM exited with status: 0
macOS: Failed to receive VM report: channel is empty and sending half is closed
macOS: gvproxy exited with status: 0

I haven't actually updated my FreeBSD image for some time, I'll give latest a shot with krunkit linked agains my patched libkrun.

It's been a while since I looked at it, but you don't do anything regarding console in the FreeBSD init, right?

[nohajc]

It's been a while since I looked at it, but you don't do anything regarding console in the FreeBSD init, right?

Not really, no. I do use my own init fork to make it even build on BSD (that PR is opened upstream, not merged yet).

It has been the same since day one though. I'm gonna have to try the latest libkrun too if you say it's broken...

Need to stay ahead before 1.18 is released. Otherwise I'd probably have to start bundling my own libkrun copy.