After upgrading from 4.20 to 4.22 , System VM isnt getting internet connection on production server.

[carbon-arts]

problem

After upgrading from 4.20 to 4.22 , System VM isnt connecting internet , ping to 8.8.8.8 is failing .

ran ssvm-check.sh on the secondarystoragevm :

root@s-634-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
================================================
First DNS server is  192.168.10.1
PING 192.168.10.1 (192.168.10.1): 56 data bytes
64 bytes from 192.168.10.1: icmp_seq=0 ttl=64 time=0.514 ms
64 bytes from 192.168.10.1: icmp_seq=1 ttl=64 time=0.444 ms
--- 192.168.10.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.444/0.479/0.514/0.035 ms
Good: Can ping DNS server
================================================
Good: DNS resolves cloudstack.apache.org
================================================
nfs is currently mounted
Mount point is /mnt/SecStorage/77fbce28-3eaa-38ea-87ac-9bee91c393ff
Good: Can write to mount point
Mount point is /mnt/SecStorage/3a7f442d-940a-3bba-97ba-a88ffac04f49
Good: Can write to mount point
================================================
Management server is 192.168.10.30. Checking connectivity.
Good: Can connect to management server 192.168.10.30 port 8250
================================================
Good: Java process is running
================================================
Tests Complete. Look for ERROR or WARNING above.
root@s-634-VM:~# ping google.com
PING google.com (142.250.65.78): 56 data bytes
92 bytes from s-634-VM (<pub ip gateway>): Destination Host Unreachable
92 bytes from s-634-VM (<pub ip gateway>): Destination Host Unreachable
92 bytes from s-634-VM (<pub ip gateway>): Destination Host Unreachable
92 bytes from s-634-VM (<pub ip gateway>): Destination Host Unreachable
^C--- google.com ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

root@s-634-VM:~# 

kindly help...

versions

Cloudstack mgmt : 4.22 , ubuntu 22.04
kvm host agent : 4.22 , ubuntu 22.04

The steps to reproduce the bug

Upgrade from 4.20 to 4.22.

What to do about it?

Normally upgradation to an lts version should be painless.
Right now upgradation is an nerve wracking activity as our production ACS is down.

[weizhouapache]

how about the CPVM ?

[carbon-arts]

root@v-633-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
================================================
First DNS server is  192.168.10.1
PING 192.168.10.1 (192.168.10.1): 56 data bytes
64 bytes from 192.168.10.1: icmp_seq=0 ttl=64 time=0.472 ms
64 bytes from 192.168.10.1: icmp_seq=1 ttl=64 time=0.406 ms
--- 192.168.10.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.406/0.439/0.472/0.033 ms
Good: Can ping DNS server
================================================
Good: DNS resolves cloudstack.apache.org
================================================
ERROR: Storage nfs is not currently mounted
Verifying if we can at least ping the storage
Storage address list is empty, trying to ping storage network gateway instead ()
ping: missing host operand
Try 'ping --help' or 'ping --usage' for more information.
WARNING: Cannot ping nfs storage network gateway
routing table follows
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         <pub ip>  0.0.0.0         UG    0      0        0 eth2
10.0.0.0        192.168.10.1    255.0.0.0       UG    0      0        0 eth1
134.195.41.192  0.0.0.0         255.255.255.192 U     0      0        0 eth2
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth0
172.16.0.0      192.168.10.1    255.240.0.0     UG    0      0        0 eth1
192.168.0.0     192.168.10.1    255.255.0.0     UG    0      0        0 eth1
192.168.10.0    0.0.0.0         255.255.255.0   U     0      0        0 eth1
================================================
Management server is 192.168.10.30. Checking connectivity.
Good: Can connect to management server 192.168.10.30 port 8250
================================================
Good: Java process is running
================================================
Tests Complete. Look for ERROR or WARNING above.
root@v-633-VM:~# 

same no ping to google.com

[weizhouapache]

@carbon-arts
please check if system vms can reach the public gateway

[carbon-arts]

root@s-634-VM:~# ping <pub gateway>
PING <pub gateway> (<pub gateway>): 56 data bytes
92 bytes from s-634-VM (<pub ip>): Destination Host Unreachable
92 bytes from s-634-VM (<pub ip>): Destination Host Unreachable
92 bytes from s-634-VM (<pub ip>): Destination Host Unreachable
92 bytes from s-634-VM (<pub ip>): Destination Host Unreachable
^C--- <pub gateway> ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss
root@s-634-VM:~# 

it cant.