virtual router - resources / system load

[wverleger]

Hi,

running ACS 4.22.0 with KVM hypervisors (ubuntu24 and alma linux 9.6).

Before upgrading to ACS 4.22.0 from 4.19.3.0 we had a VPC with redundant routers (4 vCPU / 1G memory each).
This VPC produces a lot of traffic and to secure the sessions between the routers we scaled them up to 4 vCPU / 1G memory.

After upgrading to ACS 4.22.0 we encountered perfomance problems within the VPC. After disabling redundant routers and scaling up the remaining router to: 8 vCPU / 4G memory the system stabilised.

On the hypervisor (ubuntu 24.04) we see a high system load (I guess I/O interrupts, scheduler, etc) for this qemu process:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   7479 root      20   0 9920.3m 857580  23040 R  96.7   0.1     25,56 CPU 3/KVM
   7483 root      20   0 9920.3m 857580  23040 R  94.3   0.1     25,06 CPU 7/KVM
   7481 root      20   0 9920.3m 857580  23040 R  89.0   0.1     23,07 CPU 5/KVM
   7477 root      20   0 9920.3m 857580  23040 R  88.7   0.1     23,13 CPU 1/KVM
   7476 root      20   0 9920.3m 857580  23040 R  81.3   0.1     20,43 CPU 0/KVM
   7480 root      20   0 9920.3m 857580  23040 R  78.7   0.1     20,54 CPU 4/KVM
   7478 root      20   0 9920.3m 857580  23040 R  65.0   0.1     16,52 CPU 2/KVM
   7482 root      20   0 9920.3m 857580  23040 S  58.7   0.1     15,12 CPU 6/KVM
   7474 root      20   0 9920.3m 857580  23040 R  58.0   0.1     15,18 vhost-7464
   7473 root      20   0 9920.3m 857580  23040 R  44.0   0.1     10,21 vhost-7464
   7470 root      20   0 9920.3m 857580  23040 R  32.0   0.1      8,46 vhost-7464
   7472 root      20   0 9920.3m 857580  23040 S   7.7   0.1 134:28.09 vhost-7464
   7471 root      20   0 9920.3m 857580  23040 S   5.3   0.1  85:29.00 vhost-7464
   7464 root      20   0 9920.3m 857580  23040 S   0.3   0.1   0:30.19 qemu-system-x86
   7468 root      20   0 9920.3m 857580  23040 S   0.0   0.1   0:00.01 qemu-system-x86
   7469 root      20   0 9920.3m 857580  23040 S   0.0   0.1   0:02.24 vhost-7464
   7475 root      20   0 9920.3m 857580  23040 S   0.0   0.1   0:04.83 IO mon_iothread
   7486 root      20   0 9920.3m 857580  23040 S   0.0   0.1   0:00.00 vnc_worker
   7488 root      20   0 9920.3m 857580  23040 S   0.0   0.1   0:00.00 kvm-nx-lpage-re
 340663 root      20   0 9920.3m 857580  23040 S   0.0   0.1   0:00.00 worker
 340701 root      20   0 9920.3m 857580  23040 S   0.0   0.1   0:00.00 worker
 340737 root      20   0 9920.3m 857580  23040 S   0.0   0.1   0:00.00 worker
 340738 root      20   0 9920.3m 857580  23040 S   0.0   0.1   0:00.00 worker
 340739 root      20   0 9920.3m 857580  23040 S   0.0   0.1   0:00.00 worker
 340740 root      20   0 9920.3m 857580  23040 S   0.0   0.1   0:00.00 worker
 340741 root      20   0 9920.3m 857580  23040 S   0.0   0.1   0:00.00 worker
 340742 root      20   0 9920.3m 857580  23040 S   0.0   0.1   0:00.00 worker

While the router vm itself is mostly idling within the vm:

Tasks: 150 total,   1 running, 149 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us,  0.0 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.8 si,  0.0 st
%Cpu1  :  0.0 us,  0.0 sy,  0.0 ni, 98.5 id,  0.0 wa,  0.0 hi,  1.5 si,  0.0 st
%Cpu2  :  0.0 us,  0.0 sy,  0.0 ni, 98.8 id,  0.0 wa,  0.0 hi,  1.2 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni, 98.1 id,  0.0 wa,  0.0 hi,  1.9 si,  0.0 st
%Cpu4  :  0.8 us,  0.4 sy,  0.0 ni, 97.4 id,  0.0 wa,  0.0 hi,  1.5 si,  0.0 st
%Cpu5  :  0.0 us,  0.0 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
%Cpu6  :  0.0 us,  0.0 sy,  0.0 ni, 98.5 id,  0.0 wa,  0.0 hi,  1.5 si,  0.0 st
%Cpu7  :  0.0 us,  0.0 sy,  0.0 ni, 97.9 id,  0.0 wa,  0.0 hi,  2.1 si,  0.0 st
MiB Mem :   3911.8 total,   3436.8 free,    448.7 used,    122.2 buff/cache
MiB Swap:    487.0 total,    487.0 free,      0.0 used.   3463.0 avail Mem

On other hypervisors (alma linux 9.6) we see routers idling both on hypervisor and within the router.

1. Is there any known change regarding virtual routers from 4.19.3.0 to 4.22.0 we are overlooking?
2. Could this be an expected behaviour (high hypervisor load, low system load in router)?

Thank you!