From 7a8d0b17bd67958e150455dbb2fa55514afc0933 Mon Sep 17 00:00:00 2001 From: Lawrence Lee Date: Fri, 30 Aug 2024 11:16:24 -1000 Subject: [PATCH] Update DASH HLD to Rev 2.2 (#1791) Restrict allowed values for the routing_type field in DASH_ROUTE_TABLE Specify behavior of route groups once all ENI bindings are removed Deprecate/rename some action_type fields to be consistent with the latest sonic-dash-api changes. --- doc/dash/dash-sonic-hld.md | 35 ++++++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/doc/dash/dash-sonic-hld.md b/doc/dash/dash-sonic-hld.md index 4e8336998b1..cc8208dde15 100644 --- a/doc/dash/dash-sonic-hld.md +++ b/doc/dash/dash-sonic-hld.md @@ -1,6 +1,6 @@ # SONiC-DASH HLD ## High Level Design Document -### Rev 2.0 +### Rev 2.2 # Table of Contents @@ -50,6 +50,7 @@ | 1.7 | 02/20/2024 | Prince Sunny | Introduce Route Group Table | | 2.0 | 04/08/2024 | Prince Sunny | Schema updates for PL, PL-NSG, metering | | 2.1 | 08/22/2024 | Mukesh M Velayudhan | Add local Region ID field in appliance | +| 2.2 | 08/28/2024 | Lawrence Lee | Route table `routing_type` restrictions, delete op behavior | # About this Manual This document provides more detailed design of DASH APIs, DASH orchestration agent, Config and APP DB Schemas and other SONiC buildimage changes required to bring up SONiC image on an appliance card. General DASH HLD can be found at [dash_hld](https://github.com/sonic-net/DASH/tree/main/documentation/general/dash-high-level-design.md). @@ -178,12 +179,11 @@ DASH Sonic implementation is targeted for appliance scenarios and must handles m 8. In normal operation, mappings churn often followed by routes and least for ACLs. 9. ENIs shall have an admin-state that enables normal connections and forwarding only *after* all configurations for an ENI is applied during initial creation. When the ENI is admin-state down, the packets destined to this ENI shall be dropped. Order of operation/configuration shall be enforced by the controller. Sonic implementation shall honor the state set by controller and ENI shall accept and forward traffic only if the admin-state is set to 'up'. 10. ENI must support 1M active bi-directional TCP connections or UDP flows however the connection pool can be oversubscribed. An oversubscription of 2:1 would be expected, so the connection pool can be more optimal if executed as one large table where ENI can be a part of the key. -11. During VNET or ENI delete, implementation must support ability to delete all *mappings* or *routes* in a single API call. -12. Add and Delete APIs are idempotent. As an example, deleting an object that doesn't exists shall not return an error. -13. During a delete operation, if there is a dependency (E.g. mappings still present when a VNET is deleted), implementation shall return *error* and shall not perform any force-deletions or delete dependencies implicitly. -14. During a bulk operation, if any part/subset of API fails, implementation shall return *error* for the entire API. Sonic implementation shall validate the entire API as pre-checks before applying and return accordingly. -15. Implementation must have flexible memory allocation for ENI and not reserve max scale during initial create (e.g 100k routes). This is to allow oversubscription. -16. Implementation must not have silent failures for APIs. E.g accepting an API from controller, returning success and failing in the backend. This is orthogonal to the idempotency of APIs described above for ADD and Delete operations. Intent is to ensure SDN controller and Sonic implementation is in-sync +11. Add and Delete APIs are idempotent. As an example, deleting an object that doesn't exists shall not return an error. +12. With some exceptions, during a delete operation, if there is a dependency (E.g. ), implementation shall return *error* and shall not perform any force-deletions or delete dependencies implicitly. See **Implicit deletion of SAI objects** below for exceptions. +13. During a bulk operation, if any part/subset of API fails, implementation shall return *error* for the entire API. Sonic implementation shall validate the entire API as pre-checks before applying and return accordingly. +14. Implementation must have flexible memory allocation for ENI and not reserve max scale during initial create (e.g 100k routes). This is to allow oversubscription. +15. Implementation must not have silent failures for APIs. E.g accepting an API from controller, returning success and failing in the backend. This is orthogonal to the idempotency of APIs described above for ADD and Delete operations. Intent is to ensure SDN controller and Sonic implementation is in-sync ## 1.7 ACL requirements @@ -211,6 +211,17 @@ ACL is essential for NSGs and have different stages. In the current model, there - ACL rules are not expected to have both tags and prefixes of same type configured in the same rule. For e.g, same rule shall not have both src tag and src prefix configured, but it is possible to have src tag and dst prefix or vice-versa - Counters can be attached to ACL rules optionally for retrieving the number of connections/flows. It is not required to get the packet/byte counter as in the traditional model. A new SAI counter type shall be required for this. +## 1.8 Implicit deletion of SAI objects +Due to memory constraints, certain high-volume table entries will not be cached in orchagent/SAI. Once programmed to hardware, SONiC is unable to perform operations on specific entries without being explicitly configured. This has the following implications: +- For a given 'parent' which is depended on by uncached 'child' entries, deleting the parent must also delete all child entries in the same SAI API call. This is to avoid needing a separate delete op for each child. For example, `DASH_ROUTE_GROUP_TABLE` is a parent for `DASH_ROUTE_TABLE`, which corresponds to `sai_outbound_routing_entry_t`. A command to delete a route group must, in the same SAI API call, delete all `sai_outbound_routing_entry_t` objects in that group. See the table below for all such dependencies between tables. +- If a previously bound route group is unbound from all ENIs, it must be implicitly deleted (in both orchagent cache and SAI). Since route information is not cached, it is not possible to reprogram routes to hardware, so once a route group is completely unbound it cannot be rebound to another ENI and must be deleted. Adding routes for a route group which has been deleted in this manner will result in an error. (Note that the previous bullet also applies here, so all routes in the unbound route group must also be deleted) + +| Parent Table | Child Table | Child SAI Type(s) | +| ------------ | ----------- | ---------------- | +| `DASH_ROUTE_GROUP_TABLE` | `DASH_ROUTE_TABLE` | `sai_outbound_routing_entry_t` | +| `DASH_ACL_GROUP_TABLE` | `DASH_ACL_RULE_TABLE` | ACL rule ID (`sai_object_id_t`) | +| `DASH_VNET_TABLE` | `DASH_VNET_MAPPING_TABLE` | `sai_outbound_ca_to_pa_entry_t`,
`sai_pa_validation_entry_t`,
`sai_outbound_routing_entry_t` (only if `SAI_OUTBOUND_ROUTING_ENTRY_ATTR_DST_VNET_ID` matches the deleted VNET) | + # 2 Packet Flows The following section captures at a high-level on the VNET packet flow. Detailed lookup and pipeline behavior can be referenced *here*. @@ -519,7 +530,8 @@ DASH_ROUTE_GROUP_TABLE:{{group_id}} ``` DASH_ROUTE_TABLE:{{group_id}}:{{prefix}} - "action_type": {{routing_type}} + "action_type": {{routing_type}} (DEPRECATED) + "routing_type": {{routing_type}} "vnet":{{vnet_name}} (OPTIONAL) "appliance":{{appliance_id}} (OPTIONAL) "overlay_ip":{{ip_address}} (OPTIONAL) @@ -535,13 +547,14 @@ DASH_ROUTE_TABLE:{{group_id}}:{{prefix}} ``` key = DASH_ROUTE_TABLE:group_id:prefix ; Route route table with CA prefix for packet Outbound ; field = value -action_type = routing_type ; reference to routing type +action_type = routing_type ; reference to routing type (DEPRECATED) +routing_type = routing_type ; replacement for the deprecated `action_type` field. Must be one of {vnet, vnet_direct, direct, servicetunnel, drop}. vnet = vnet name ; destination vnet name if routing_type is {vnet, vnet_direct}, a vnet other than eni's vnet means vnet peering appliance = appliance id ; appliance id if routing_type is {appliance} overlay_ip = ip_address ; overly_ip to lookup if routing_type is {vnet_direct}, use dst ip from packet if not specified overlay_sip_prefix = ip_prefix ; overlay ipv6 src ip if routing_type is {servicetunnel}, transform last 32 bits from packet (src ip) overlay_dip_prefix = ip_prefix ; overlay ipv6 dst ip if routing_type is {servicetunnel}, transform last 32 bits from packet (dst ip) -underlay_sip = ip_address ; underlay ipv4 src ip if routing_type is {servicetunnel,privatelink}; this is the ST GW VIP (for ST traffic) or custom VIP. If specified, overrides pl_underlay_sip from DASH_ENI_TABLE +underlay_sip = ip_address ; underlay ipv4 src ip if routing_type is {servicetunnel} or for DASH_VNET_MAPPING_TABLE entries in this prefix where routing_type is {privatelink}; this is the ST GW VIP (for ST traffic) or custom VIP. If specified, overrides pl_underlay_sip from DASH_ENI_TABLE underlay_dip = ip_address ; underlay ipv4 dst ip to override if routing_type is {servicetunnel}, use dst ip from packet if not specified metering_policy_en = bool ; Metering policy lookup enable (optional), default = false (OBSOLETED). If aggregated or/and bits is 0, metering policy is applied metering_class_or = uint32 ; Metering class-id 'or' bits @@ -593,7 +606,7 @@ DASH_VNET_MAPPING_TABLE:{{vnet}}:{{ip_address}} ``` key = DASH_VNET_MAPPING_TABLE:vnet:ip_address ; CA-PA mapping table for Vnet ; field = value -action_type = routing_type ; reference to routing type +routing_type = routing_type ; reference to routing type underlay_ip = ip_address ; PA address for the CA mac_address = MAC address as string ; Inner dst mac metering_class_or = uint32 ; metering class 'or' bits