T6641: Add vyos-network-event-logger Service #4216

HollyGurza · 2024-12-03T09:25:24Z

The service parses and logs network events for improved monitoring and diagnostics. Supported event types include:

RTM_NEWROUTE, RTM_DELROUTE
RTM_NEWLINK, RTM_DELLINK
RTM_NEWADDR, RTM_DELADDR
RTM_NEWNEIGH, RTM_DELNEIGH, RTM_GETNEIGH
RTM_NEWRULE, RTM_DELRULE

Added operational mode commands for filtered log retrieval:

show log network-event <event-type> <interface>: Retrieve logs filtered by event type and interface.
show interfaces <type> <name> event-log <event-type>: Display interface-specific logs filtered by event type.

Change Summary

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Code style update (formatting, renaming)
Refactoring (no functional changes)
Migration from an old Vyatta component to vyos-1x, please link to related PR inside obsoleted component
Other (please describe):

Related Task(s)

https://vyos.dev/T6641

Related PR(s)

Component(s) name

Proposed changes

log examples:

run show log network-event interface eth1

Dec 03 09:23:34 vyos-network-event-logger[21158]: [LINK]   3: eth1:  <BROADCAST,MULTICAST>  mtu 1500  qdisc fq_codel  state DOWN  group 0  link/ether 0c:89:0a:2e:00:01  brd ff:ff:ff:ff:ff:ff  [altname enp0s4] [altname ens4]
Dec 03 09:23:34 vyos-network-event-logger[21158]: [NEIGH]  Deleted  ff02::16  dev eth1 lladdr 33:33:00:00:00:16    NOARP
Dec 03 09:23:40 vyos-network-event-logger[21158]: [LINK]   3: eth1:  <BROADCAST,MULTICAST,UP,LOWER_UP>  mtu 1500  qdisc fq_codel  state UP  group 0  link/ether 0c:89:0a:2e:00:01  brd ff:ff:ff:ff:ff:ff  [altname enp0s4] [altname ens4]
Dec 03 09:23:40 vyos-network-event-logger[21158]: [ROUTE]  broadcast 10.0.0.255 tos 0  dev eth1 table local proto kernel scope link src 10.0.0.1
Dec 03 09:23:40 vyos-network-event-logger[21158]: [ROUTE]  unicast 10.0.0.0/24 tos 0  dev eth1 table main proto kernel scope link src 10.0.0.1
Dec 03 09:23:40 vyos-network-event-logger[21158]: [ROUTE]  multicast ff00::/8 tos 0  dev eth1 table local proto kernel metric 256  pref medium

How to test

To check the correct parsing you can use logs from ip monitor label
configure network-event logger to listen events:

conf
set service monitoring network-event event addr
set service monitoring network-event event link
set service monitoring network-event event neigh
set service monitoring network-event event route
set service monitoring network-event event rule
commit

you can use IP utils to manipulate the interface and check logs or use some set interface * commands e.g.

generate pki ca install test_ca

conf
set interfaces bonding bond1 address 11.11.11.11/32
commit
run show interfaces bonding bond1 event-log
del interfaces bonding bond1
commit

set interfaces bridge br1 address 11.11.11.11/32
commit
run show interfaces bridge br1 event-log
del interfaces bridge br1
commit

set interfaces dummy dum1 address 11.11.11.11/32
commit
run show interfaces dummy dum1  event-log
del interfaces dummy dum1
commit

sudo ip link set down dev eth1
sudo ip link set up dev eth1
run show interfaces ethernet eth1 event-log

set interfaces geneve gnv1  address 11.11.11.11/32
set interfaces geneve gnv1 remote 127.0.0.1
set interfaces geneve gnv1 vni 10
commit
run show interfaces geneve gnv1 event-log
del interfaces geneve gnv1
commit

set interfaces input ifb1 redirect eth1
commit
run show interfaces input ifb1  event-log
del interfaces input ifb1
commit

set interfaces loopback lo address 11.11.11.11/32
commit
run show interfaces loopback lo event-log
del interfaces loopback lo address
commit

set interfaces macsec macsec1 address '11.11.11.11/32'
set interfaces macsec macsec1 security cipher 'gcm-aes-128'
set interfaces macsec macsec1 source-interface 'eth1'
commit
run show interfaces macsec macsec1 event-log
del interfaces macsec macsec1
commit

set interfaces pseudo-ethernet peth1 address '11.11.11.11/32'
set interfaces pseudo-ethernet peth1 source-interface 'eth1'
commit
run show interfaces pseudo-ethernet peth1 event-log
del interfaces pseudo-ethernet peth1
commit

set interfaces sstpc sstpc1 server '127.0.0.1'
set interfaces sstpc sstpc1 ssl ca-certificate 'test_ca'
commit
run show interfaces sstpc sstpc1 event-log
del interfaces sstpc sstpc1
commit

set interfaces tunnel tun1 address '11.11.11.11/32'
set interfaces tunnel tun1 encapsulation 'gre'
set interfaces tunnel tun1 source-interface 'eth1'
commit
run show interfaces tunnel tun1 event-log
del interfaces tunnel tun1
commit

set interfaces virtual-ethernet veth1 address '11.11.11.11/32'
set interfaces virtual-ethernet veth1 peer-name 'veth2'
set interfaces virtual-ethernet veth2 peer-name 'veth1'
commit
run show interfaces virtual-ethernet veth1 event-log
del interfaces virtual-ethernet veth1
del interfaces virtual-ethernet veth2
commit

set interfaces vti vti1 address 11.11.11.11/32
commit
run show interfaces vti vti1 event-log
del interfaces vti vti1
commit

set interfaces vxlan vxlan1 address '11.11.11.11/32'
set interfaces vxlan vxlan1 mtu '1400'
set interfaces vxlan vxlan1 source-interface 'eth1'
set interfaces vxlan vxlan1 vni '12'
commit
run show interfaces vxlan vxlan1 event-log
del interfaces vxlan
commit

Smoketest result

vyos@vyos:~$ python3 /usr/libexec/vyos/tests/smoke/cli/test_service_monitoring_network_event.py 
test_network_event_log (__main__.TestMonitoringNetworkEvent.test_network_event_log) ... ok

----------------------------------------------------------------------
Ran 1 test in 6.043s

OK

Checklist:

I have read the CONTRIBUTING document
I have linked this PR to one or more Phabricator Task(s)
I have run the components SMOKETESTS if applicable
My commit headlines contain a valid Task id
My change requires a change to the documentation
I have updated the documentation accordingly

github-actions · 2024-12-03T09:25:41Z

👍
No issues in PR Title / Commit Title

jestabro · 2024-12-10T14:34:04Z

As mentioned on Slack, this should be rebased over current and pushed to sync with the changes in
#4226
vyos/vyos-build#824

A local build of this PR confirms a successful build and smoketests.

The service parses and logs network events for improved monitoring and diagnostics. Supported event types include: - `RTM_NEWROUTE`, `RTM_DELROUTE` - `RTM_NEWLINK`, `RTM_DELLINK` - `RTM_NEWADDR`, `RTM_DELADDR` - `RTM_NEWNEIGH`, `RTM_DELNEIGH`, `RTM_GETNEIGH` - `RTM_NEWRULE`, `RTM_DELRULE` Added operational mode commands for filtered log retrieval: - `show log network-event <event-type> <interface>`: Retrieve logs filtered by event type and interface. - `show interfaces <type> <name> event-log <event-type>`: Display interface-specific logs filtered by event type.

github-actions · 2024-12-10T16:25:17Z

CI integration ❌ failed!

Details

CI logs

CLI Smoketests (no interfaces) 👍 passed
CLI Smoketests (interfaces only) ❌ failed
Config tests 👍 passed
RAID1 tests 👍 passed
TPM tests 👍 passed

c-po · 2024-12-19T19:22:28Z

Is there any downside not enabling this service by default?

HollyGurza · 2024-12-20T07:33:13Z

Is there any downside not enabling this service by default?

at least, you must remember to turn it on to analyze some situations and reproduce these? otherwise logs will not stored.
I'm not sure if this should be enabled by default

dmbaturin

I left a bunch of suggestions for improvement.

dmbaturin · 2025-01-06T11:33:38Z

op-mode-definitions/include/show-interface-type-event-log.xml.i

+  <properties>
+    <help>Show interface network event log</help>
+  </properties>
+  <command>journalctl --no-hostname --boot --unit vyos-network-event-logger.service | grep "\b$4\b"</command>


journalctl has a built-in --grep= option that supports PCRE, so I think there is no need to use an external filter here. The built-in will be faster and will preserve coloring, etc.

dmbaturin · 2025-01-06T11:36:12Z

op-mode-definitions/include/show-interface-type-event-log.xml.i

+      <properties>
+        <help>Show log for route network events</help>
+      </properties>
+      <command>journalctl --no-hostname --boot --unit vyos-network-event-logger.service | grep "$(echo "\[$6\]" | tr '[:lower:]' '[:upper:]')" | grep "\b$4\b"</command>


The journalctl command is duplicated a lot here. Should we make it a script?

I'm also not convinced about the value of the tr call here.

dmbaturin · 2025-01-06T11:38:21Z

op-mode-definitions/include/show-interface-type-event-log.xml.i

+<!-- included start from show-nht.xml.i -->
+<node name="event-log">
+  <properties>
+    <help>Show interface network event log</help>


Suggested change

<help>Show interface network event log</help>

<help>Show network interface change event log</help>

dmbaturin · 2025-01-06T11:43:19Z

interface-definitions/service_monitoring_network_event.xml.in

+<interfaceDefinition>
+  <node name="service">
+    <children>
+      <node name="monitoring">


We should probably make a skeleton node for monitoring and include everything else into it, although it's a topic for a different PR.

dmbaturin · 2025-01-06T11:56:24Z

src/services/vyos-network-event-logger

+        logger.setLevel(logging.INFO)
+
+
+# https://github.com/torvalds/linux/blob/adc218676eef25575469234709c2d87185ca223a/include/uapi/linux/neighbour.h#L46


Should we move the constants to a separate module? There may be more places in the future where they might be useful for handling netlink events.

dmbaturin · 2025-01-06T14:06:11Z

op-mode-definitions/include/show-interface-type-event-log.xml.i

+    </leafNode>
+    <leafNode name="rule">
+      <properties>
+        <help>Show log for rule network events</help>


Suggested change

<help>Show log for rule network events</help>

<help>Show log for PBR rule change events</help>

dmbaturin · 2025-01-06T14:06:44Z

op-mode-definitions/show-log.xml.in

+              </tagNode>
+              <node name="route">
+                <properties>
+                  <help>Show log for route network events</help>


Suggested change

<help>Show log for route network events</help>

<help>Show log for route events</help>

dmbaturin · 2025-01-06T14:06:57Z

op-mode-definitions/show-log.xml.in

+              </node>
+              <node name="link">
+                <properties>
+                  <help>Show log for link network events</help>


Suggested change

<help>Show log for link network events</help>

<help>Show log for network link events</help>

dmbaturin · 2025-01-06T14:07:10Z

op-mode-definitions/show-log.xml.in

+              </node>
+              <node name="addr">
+                <properties>
+                  <help>Show log for addr network events</help>


Suggested change

<help>Show log for addr network events</help>

<help>Show log for network address events</help>

dmbaturin · 2025-01-06T14:07:28Z

op-mode-definitions/show-log.xml.in

+              </node>
+              <node name="neigh">
+                <properties>
+                  <help>Show log for neigh network events</help>


Suggested change

<help>Show log for neigh network events</help>

<help>Show log for neighbor table events</help>

c-po · 2025-01-08T19:44:38Z

Another reason for having this running always in the background:

From FRR show interface eth0

Interface eth0 is up, line protocol is up
  Link ups:       1    last: 2025/01/08 08:03:49.20
  Link downs:     0    last: (never)

HollyGurza requested a review from a team as a code owner December 3, 2024 09:25

HollyGurza requested review from dmbaturin, sarthurdev, zdc, jestabro, c-po and nicolas-fort December 3, 2024 09:25

github-actions bot added the current label Dec 3, 2024

github-actions bot assigned HollyGurza Dec 3, 2024

HollyGurza force-pushed the T6641 branch from 02a4e02 to 3ad5985 Compare December 4, 2024 02:43

HollyGurza force-pushed the T6641 branch from 3ad5985 to b9e3eb8 Compare December 10, 2024 15:07

HollyGurza requested review from sever-sever and fett0 as code owners December 10, 2024 15:07

dmbaturin requested changes Jan 6, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

T6641: Add vyos-network-event-logger Service #4216

T6641: Add vyos-network-event-logger Service #4216

HollyGurza commented Dec 3, 2024

github-actions bot commented Dec 3, 2024 •

edited

Loading

jestabro commented Dec 10, 2024

github-actions bot commented Dec 10, 2024

c-po commented Dec 19, 2024

HollyGurza commented Dec 20, 2024

dmbaturin left a comment

dmbaturin Jan 6, 2025

dmbaturin Jan 6, 2025

dmbaturin Jan 6, 2025

dmbaturin Jan 6, 2025

dmbaturin Jan 6, 2025

dmbaturin Jan 6, 2025

dmbaturin Jan 6, 2025

dmbaturin Jan 6, 2025

dmbaturin Jan 6, 2025

dmbaturin Jan 6, 2025

c-po commented Jan 8, 2025

	<help>Show interface network event log</help>
	<help>Show network interface change event log</help>

		logger.setLevel(logging.INFO)


		# https://github.com/torvalds/linux/blob/adc218676eef25575469234709c2d87185ca223a/include/uapi/linux/neighbour.h#L46

	<help>Show log for rule network events</help>
	<help>Show log for PBR rule change events</help>

	<help>Show log for route network events</help>
	<help>Show log for route events</help>

	<help>Show log for link network events</help>
	<help>Show log for network link events</help>

	<help>Show log for addr network events</help>
	<help>Show log for network address events</help>

	<help>Show log for neigh network events</help>
	<help>Show log for neighbor table events</help>

T6641: Add vyos-network-event-logger Service #4216

Are you sure you want to change the base?

T6641: Add vyos-network-event-logger Service #4216

Conversation

HollyGurza commented Dec 3, 2024

Change Summary

Types of changes

Related Task(s)

Related PR(s)

Component(s) name

Proposed changes

How to test

Smoketest result

Checklist:

github-actions bot commented Dec 3, 2024 • edited Loading

jestabro commented Dec 10, 2024

github-actions bot commented Dec 10, 2024

Details

c-po commented Dec 19, 2024

HollyGurza commented Dec 20, 2024

dmbaturin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

c-po commented Jan 8, 2025

github-actions bot commented Dec 3, 2024 •

edited

Loading