[sanity_check][bgp] Add default route check in sanity for single asic #16235

yaqiangz · 2024-12-26T05:33:29Z

Description of PR

Summary:
Fixes # (issue)

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

Approach

What is the motivation for this PR?

BGP routes would be setup during add-topo https://github.com/sonic-net/sonic-mgmt/blob/master/ansible/roles/vm_set/tasks/add_topo.yml#L276.
But there are some scenarios that route in DUT has been messed up, but bgp sessions are all up, sanity would treat it as healthy and wouldn't take action to recover it.

Loopbackv4 address has been replaced, it would cause all kernel routes from bgp miss
In some test cases announce or withdraw routes from ptf but fail to recover (i.e. test_stress_routes)

Healthy status:

admin@sonic:~$ ip route show default
default nhid 282 proto bgp src 10.1.0.32 metric 20 
        nexthop via 10.0.0.57 dev PortChannel101 weight 1 
        nexthop via 10.0.0.59 dev PortChannel103 weight 1 
        nexthop via 10.0.0.61 dev PortChannel105 weight 1 
        nexthop via 10.0.0.63 dev PortChannel106 weight 1 
admin@sonic:~$ show ip bgp sum

IPv4 Unicast Summary:
BGP router identifier 10.1.0.32, local AS number 65100 vrf-id 0
BGP table version 2890
RIB entries 2893, using 648032 bytes of memory
Peers 6, using 4451856 KiB of memory
Peer groups 4, using 256 bytes of memory


Neighbhor      V     AS    MsgRcvd    MsgSent    TblVer    InQ    OutQ  Up/Down      State/PfxRcd  NeighborName
-----------  ---  -----  ---------  ---------  --------  -----  ------  ---------  --------------  --------------
10.0.0.57      4  65200        763        764         0      0       0  11:46:17             1439  ARISTA01M1
10.0.0.59      4  65200        763        765         0      0       0  11:46:17             1439  ARISTA02M1
10.0.0.61      4  65200        763        765         0      0       0  11:46:17             1439  ARISTA03M1
10.0.0.63      4  65200        763        765         0      0       0  11:46:17             1439  ARISTA04M1
10.0.0.65      4  64001        712        761         0      0       0  11:46:15                2  ARISTA01MX
10.0.0.67      4  64002        712        761         0      0       0  11:46:15                2  ARISTA02MX

Total number of neighbors 6

Issue status, no default route, but show ip bgp sum looks good

admin@sonic:~$ ip route show default
admin@sonic:~$ show ip bgp sum

IPv4 Unicast Summary:
BGP router identifier 10.1.0.32, local AS number 65100 vrf-id 0
BGP table version 2892
RIB entries 2893, using 648032 bytes of memory
Peers 6, using 4451856 KiB of memory
Peer groups 4, using 256 bytes of memory


Neighbhor      V     AS    MsgRcvd    MsgSent    TblVer    InQ    OutQ  Up/Down      State/PfxRcd  NeighborName
-----------  ---  -----  ---------  ---------  --------  -----  ------  ---------  --------------  --------------
10.0.0.57      4  65200        764        767         0      0       0  11:47:14             1439  ARISTA01M1
10.0.0.59      4  65200        764        768         0      0       0  11:47:14             1439  ARISTA02M1
10.0.0.61      4  65200        764        768         0      0       0  11:47:14             1439  ARISTA03M1
10.0.0.63      4  65200        764        768         0      0       0  11:47:14             1439  ARISTA04M1
10.0.0.65      4  64001        713        764         0      0       0  11:47:12                2  ARISTA01MX
10.0.0.67      4  64002        713        764         0      0       0  11:47:12                2  ARISTA02MX

Total number of neighbors 6

How did you do it?

Add default routes check in sanity check, and re-announce routes if issue happen

How did you verify/test it?

Run sanity check

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

mssonicbld · 2024-12-26T05:33:31Z

/azp run

azure-pipelines · 2024-12-26T05:33:42Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2024-12-26T05:39:24Z

/azp run

azure-pipelines · 2024-12-26T05:39:34Z

Azure Pipelines successfully started running 1 pipeline(s).

yaqiangz · 2024-12-26T05:51:18Z

Hi @yejianquan could you please help to confirm whether this change is expected in multi-asic?

yejianquan · 2024-12-26T09:03:40Z

Hi @cyw233 , could you please help to verify if this new check/recover works well on chassis devices?

cyw233 · 2024-12-26T09:14:48Z

Hi @cyw233 , could you please help to verify if this new check/recover works well on chassis devices?

Sure, will verify cc. @yaqiangz

mssonicbld · 2024-12-26T09:16:36Z

/azp run

azure-pipelines · 2024-12-26T09:16:48Z

Azure Pipelines successfully started running 1 pipeline(s).

yaqiangz · 2024-12-26T09:28:28Z

Confirmed that multi-asic has different behavior, this PR is only for single asic

mssonicbld · 2024-12-26T09:37:53Z

/azp run

azure-pipelines · 2024-12-26T09:38:04Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2024-12-26T13:08:43Z

/azp run

azure-pipelines · 2024-12-26T13:08:55Z

Azure Pipelines successfully started running 1 pipeline(s).

StormLiangMS

LGTM

…sonic-net#16235) What is the motivation for this PR? BGP routes would be setup during add-topo https://github.com/sonic-net/sonic-mgmt/blob/master/ansible/roles/vm_set/tasks/add_topo.yml#L276. But there are some scenarios that route in DUT has been messed up, but bgp sessions are all up, sanity would treat it as healthy and wouldn't take action to recover it. Loopbackv4 address has been replaced, it would cause all kernel routes from bgp miss In some test cases announce or withdraw routes from ptf but fail to recover (i.e. test_stress_routes) Healthy status: admin@sonic:~$ ip route show default default nhid 282 proto bgp src 10.1.0.32 metric 20 nexthop via 10.0.0.57 dev PortChannel101 weight 1 nexthop via 10.0.0.59 dev PortChannel103 weight 1 nexthop via 10.0.0.61 dev PortChannel105 weight 1 nexthop via 10.0.0.63 dev PortChannel106 weight 1 admin@sonic:~$ show ip bgp sum IPv4 Unicast Summary: BGP router identifier 10.1.0.32, local AS number 65100 vrf-id 0 BGP table version 2890 RIB entries 2893, using 648032 bytes of memory Peers 6, using 4451856 KiB of memory Peer groups 4, using 256 bytes of memory Neighbhor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd NeighborName ----------- --- ----- --------- --------- -------- ----- ------ --------- -------------- -------------- 10.0.0.57 4 65200 763 764 0 0 0 11:46:17 1439 ARISTA01M1 10.0.0.59 4 65200 763 765 0 0 0 11:46:17 1439 ARISTA02M1 10.0.0.61 4 65200 763 765 0 0 0 11:46:17 1439 ARISTA03M1 10.0.0.63 4 65200 763 765 0 0 0 11:46:17 1439 ARISTA04M1 10.0.0.65 4 64001 712 761 0 0 0 11:46:15 2 ARISTA01MX 10.0.0.67 4 64002 712 761 0 0 0 11:46:15 2 ARISTA02MX Total number of neighbors 6 Issue status, no default route, but show ip bgp sum looks good admin@sonic:~$ ip route show default admin@sonic:~$ show ip bgp sum IPv4 Unicast Summary: BGP router identifier 10.1.0.32, local AS number 65100 vrf-id 0 BGP table version 2892 RIB entries 2893, using 648032 bytes of memory Peers 6, using 4451856 KiB of memory Peer groups 4, using 256 bytes of memory Neighbhor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd NeighborName ----------- --- ----- --------- --------- -------- ----- ------ --------- -------------- -------------- 10.0.0.57 4 65200 764 767 0 0 0 11:47:14 1439 ARISTA01M1 10.0.0.59 4 65200 764 768 0 0 0 11:47:14 1439 ARISTA02M1 10.0.0.61 4 65200 764 768 0 0 0 11:47:14 1439 ARISTA03M1 10.0.0.63 4 65200 764 768 0 0 0 11:47:14 1439 ARISTA04M1 10.0.0.65 4 64001 713 764 0 0 0 11:47:12 2 ARISTA01MX 10.0.0.67 4 64002 713 764 0 0 0 11:47:12 2 ARISTA02MX Total number of neighbors 6 How did you do it? Add default routes check in sanity check, and re-announce routes if issue happen How did you verify/test it? Run sanity check

mssonicbld · 2024-12-27T06:01:35Z

Cherry-pick PR to 202311: #16246

…sonic-net#16235) What is the motivation for this PR? BGP routes would be setup during add-topo https://github.com/sonic-net/sonic-mgmt/blob/master/ansible/roles/vm_set/tasks/add_topo.yml#L276. But there are some scenarios that route in DUT has been messed up, but bgp sessions are all up, sanity would treat it as healthy and wouldn't take action to recover it. Loopbackv4 address has been replaced, it would cause all kernel routes from bgp miss In some test cases announce or withdraw routes from ptf but fail to recover (i.e. test_stress_routes) Healthy status: admin@sonic:~$ ip route show default default nhid 282 proto bgp src 10.1.0.32 metric 20 nexthop via 10.0.0.57 dev PortChannel101 weight 1 nexthop via 10.0.0.59 dev PortChannel103 weight 1 nexthop via 10.0.0.61 dev PortChannel105 weight 1 nexthop via 10.0.0.63 dev PortChannel106 weight 1 admin@sonic:~$ show ip bgp sum IPv4 Unicast Summary: BGP router identifier 10.1.0.32, local AS number 65100 vrf-id 0 BGP table version 2890 RIB entries 2893, using 648032 bytes of memory Peers 6, using 4451856 KiB of memory Peer groups 4, using 256 bytes of memory Neighbhor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd NeighborName ----------- --- ----- --------- --------- -------- ----- ------ --------- -------------- -------------- 10.0.0.57 4 65200 763 764 0 0 0 11:46:17 1439 ARISTA01M1 10.0.0.59 4 65200 763 765 0 0 0 11:46:17 1439 ARISTA02M1 10.0.0.61 4 65200 763 765 0 0 0 11:46:17 1439 ARISTA03M1 10.0.0.63 4 65200 763 765 0 0 0 11:46:17 1439 ARISTA04M1 10.0.0.65 4 64001 712 761 0 0 0 11:46:15 2 ARISTA01MX 10.0.0.67 4 64002 712 761 0 0 0 11:46:15 2 ARISTA02MX Total number of neighbors 6 Issue status, no default route, but show ip bgp sum looks good admin@sonic:~$ ip route show default admin@sonic:~$ show ip bgp sum IPv4 Unicast Summary: BGP router identifier 10.1.0.32, local AS number 65100 vrf-id 0 BGP table version 2892 RIB entries 2893, using 648032 bytes of memory Peers 6, using 4451856 KiB of memory Peer groups 4, using 256 bytes of memory Neighbhor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd NeighborName ----------- --- ----- --------- --------- -------- ----- ------ --------- -------------- -------------- 10.0.0.57 4 65200 764 767 0 0 0 11:47:14 1439 ARISTA01M1 10.0.0.59 4 65200 764 768 0 0 0 11:47:14 1439 ARISTA02M1 10.0.0.61 4 65200 764 768 0 0 0 11:47:14 1439 ARISTA03M1 10.0.0.63 4 65200 764 768 0 0 0 11:47:14 1439 ARISTA04M1 10.0.0.65 4 64001 713 764 0 0 0 11:47:12 2 ARISTA01MX 10.0.0.67 4 64002 713 764 0 0 0 11:47:12 2 ARISTA02MX Total number of neighbors 6 How did you do it? Add default routes check in sanity check, and re-announce routes if issue happen How did you verify/test it? Run sanity check

mssonicbld · 2024-12-27T06:01:36Z

Cherry-pick PR to 202411: #16247

mssonicbld · 2024-12-27T06:01:38Z

Cherry-pick PR to 202405: #16248

…#16235) What is the motivation for this PR? BGP routes would be setup during add-topo https://github.com/sonic-net/sonic-mgmt/blob/master/ansible/roles/vm_set/tasks/add_topo.yml#L276. But there are some scenarios that route in DUT has been messed up, but bgp sessions are all up, sanity would treat it as healthy and wouldn't take action to recover it. Loopbackv4 address has been replaced, it would cause all kernel routes from bgp miss In some test cases announce or withdraw routes from ptf but fail to recover (i.e. test_stress_routes) Healthy status: admin@sonic:~$ ip route show default default nhid 282 proto bgp src 10.1.0.32 metric 20 nexthop via 10.0.0.57 dev PortChannel101 weight 1 nexthop via 10.0.0.59 dev PortChannel103 weight 1 nexthop via 10.0.0.61 dev PortChannel105 weight 1 nexthop via 10.0.0.63 dev PortChannel106 weight 1 admin@sonic:~$ show ip bgp sum IPv4 Unicast Summary: BGP router identifier 10.1.0.32, local AS number 65100 vrf-id 0 BGP table version 2890 RIB entries 2893, using 648032 bytes of memory Peers 6, using 4451856 KiB of memory Peer groups 4, using 256 bytes of memory Neighbhor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd NeighborName ----------- --- ----- --------- --------- -------- ----- ------ --------- -------------- -------------- 10.0.0.57 4 65200 763 764 0 0 0 11:46:17 1439 ARISTA01M1 10.0.0.59 4 65200 763 765 0 0 0 11:46:17 1439 ARISTA02M1 10.0.0.61 4 65200 763 765 0 0 0 11:46:17 1439 ARISTA03M1 10.0.0.63 4 65200 763 765 0 0 0 11:46:17 1439 ARISTA04M1 10.0.0.65 4 64001 712 761 0 0 0 11:46:15 2 ARISTA01MX 10.0.0.67 4 64002 712 761 0 0 0 11:46:15 2 ARISTA02MX Total number of neighbors 6 Issue status, no default route, but show ip bgp sum looks good admin@sonic:~$ ip route show default admin@sonic:~$ show ip bgp sum IPv4 Unicast Summary: BGP router identifier 10.1.0.32, local AS number 65100 vrf-id 0 BGP table version 2892 RIB entries 2893, using 648032 bytes of memory Peers 6, using 4451856 KiB of memory Peer groups 4, using 256 bytes of memory Neighbhor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd NeighborName ----------- --- ----- --------- --------- -------- ----- ------ --------- -------------- -------------- 10.0.0.57 4 65200 764 767 0 0 0 11:47:14 1439 ARISTA01M1 10.0.0.59 4 65200 764 768 0 0 0 11:47:14 1439 ARISTA02M1 10.0.0.61 4 65200 764 768 0 0 0 11:47:14 1439 ARISTA03M1 10.0.0.63 4 65200 764 768 0 0 0 11:47:14 1439 ARISTA04M1 10.0.0.65 4 64001 713 764 0 0 0 11:47:12 2 ARISTA01MX 10.0.0.67 4 64002 713 764 0 0 0 11:47:12 2 ARISTA02MX Total number of neighbors 6 How did you do it? Add default routes check in sanity check, and re-announce routes if issue happen How did you verify/test it? Run sanity check

[sanity_check][bgp] Add default route check in sanity

6a166f8

yaqiangz force-pushed the azure-master_sanity_bgp branch from 4cfafc9 to 6a166f8 Compare December 26, 2024 05:39

yaqiangz added Request for 202311 branch Request for 202405 branch Request for 202411 branch labels Dec 26, 2024

Skip multi-asic

33da168

yaqiangz changed the title ~~[sanity_check][bgp] Add default route check in sanity~~ [sanity_check][bgp] Add default route check in sanity for single asic Dec 26, 2024

Fix modular_chassis

b4c35a5

Update chassis

d1023cc

yaqiangz requested a review from StormLiangMS December 26, 2024 23:35

StormLiangMS approved these changes Dec 27, 2024

View reviewed changes

StormLiangMS merged commit ca01ce6 into sonic-net:master Dec 27, 2024
17 checks passed

StormLiangMS added Approved for 202311 branch Approved for 202405 branch Approved for 202411 branch labels Dec 27, 2024

mssonicbld added the Created PR to 202311 branch label Dec 27, 2024

mssonicbld added the Created PR to 202411 branch label Dec 27, 2024

mssonicbld added the Created PR to 202405 branch label Dec 27, 2024

mssonicbld added Included in 202311 branch and removed Created PR to 202311 branch labels Dec 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[sanity_check][bgp] Add default route check in sanity for single asic #16235

[sanity_check][bgp] Add default route check in sanity for single asic #16235

yaqiangz commented Dec 26, 2024

mssonicbld commented Dec 26, 2024

azure-pipelines bot commented Dec 26, 2024

mssonicbld commented Dec 26, 2024

azure-pipelines bot commented Dec 26, 2024

yaqiangz commented Dec 26, 2024

yejianquan commented Dec 26, 2024

cyw233 commented Dec 26, 2024

mssonicbld commented Dec 26, 2024

azure-pipelines bot commented Dec 26, 2024

yaqiangz commented Dec 26, 2024 •

edited

Loading

mssonicbld commented Dec 26, 2024

azure-pipelines bot commented Dec 26, 2024

mssonicbld commented Dec 26, 2024

azure-pipelines bot commented Dec 26, 2024

StormLiangMS left a comment

mssonicbld commented Dec 27, 2024

mssonicbld commented Dec 27, 2024

mssonicbld commented Dec 27, 2024

[sanity_check][bgp] Add default route check in sanity for single asic #16235

[sanity_check][bgp] Add default route check in sanity for single asic #16235

Conversation

yaqiangz commented Dec 26, 2024

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

mssonicbld commented Dec 26, 2024

azure-pipelines bot commented Dec 26, 2024

mssonicbld commented Dec 26, 2024

azure-pipelines bot commented Dec 26, 2024

yaqiangz commented Dec 26, 2024

yejianquan commented Dec 26, 2024

cyw233 commented Dec 26, 2024

mssonicbld commented Dec 26, 2024

azure-pipelines bot commented Dec 26, 2024

yaqiangz commented Dec 26, 2024 • edited Loading

mssonicbld commented Dec 26, 2024

azure-pipelines bot commented Dec 26, 2024

mssonicbld commented Dec 26, 2024

azure-pipelines bot commented Dec 26, 2024

StormLiangMS left a comment

Choose a reason for hiding this comment

mssonicbld commented Dec 27, 2024

mssonicbld commented Dec 27, 2024

mssonicbld commented Dec 27, 2024

yaqiangz commented Dec 26, 2024 •

edited

Loading