IPv6 as SCION dataplane

[matzf]

IPv6 as SCION dataplane

A short while ago, the idea to implement the SCION dataplane as an IPv6 routing header was floated in the chat and was received rather positively.
This discussion intendeds to explore this idea in a bit more detail. The expected outcome is to collectively either reject the idea or come up with a more detailed proposal on how this could work.

Background

SCION is an inter-domain routing architecture.
In the traditional IPv4/6 internet, BGP is the "control plane" for inter-domain routing. BGP controls the routing tables, and the dataplane is IP, i.e. the packets sent from source to destination are IP packets.

SCION has it's own dataplane. The packets sent from source to destination host are SCION packets. Still, IP is the default (and currently only) "underlay" for AS-local packet delivery.

SCION also has it's own address type, consisting of a part identifying the AS (ISD-AS, 16 + 48 == 64 bits) and a part identifying a host in this AS (typically IPv4 or IPv6 or special SCION service identifiers).
Furthermore SCION has it's own control message protocol (SCMP), which is effectively an extended version of ICMPv6.

The SCION packet format is "syntactically" relatively similar to IPv6, after which it was modeled. The two main differences between SCION and IPv6 packets are:

• SCION packets contain a SCION path, a list of cryptographically authenticated local forwarding instructions for each AS
• the addresses (SCION address vs IPv6 address)

Idea

Instead of having separate SCION packet format, use IPv6 as the SCION dataplane: Include the SCION Path information in an IPv6 routing extension header [RFC8200, section 4.4, IANA IPv6 Routing types].

If we adopt the current SCION dataplane path format, this SCION routing header could look like this:

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|  Next Header  |  Hdr Ext Len  |  RT_SCION     | Segments Left |   // RT_SCION := IANA assignment for Routing Type: SCION (e.g. 7)
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                      Destination Address                      |
|                              ...                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|CI |  CurrHF   |    RSV    |  Seg0Len  |  Seg1Len  |  Seg2Len  |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           InfoField                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                              ...                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           InfoField                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           HopField                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           HopField                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                              ...                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

When the SCION routing header is present, the Destination Address (DA) of the IPv6 packet is replaced with the address of the next SCION router. Any intermediate IP router must not inspect the routing header but forward the packet toward this DA.
This is analogous to the Segment Routing Header (SRH) RFC8754.

Note that there are some redundancies now that indicate that some tweaking of the detailed format could be beneficial, e.g. the Segments Left is somewhat redundant with CurrHF.
The SRH also has the "hops" in reversed order, presumably to simplify the check for the last segment. If this is really useful, we could apply that same thing as well.

Border routers

In this proposals, SCION border routers are IPv6 routers that process the SCION routing header. The processing rules for SCION path remain the same as with the current SCION dataplane; routers checks the validity of the current HopField, increment the Current Hop and update the Segment Identifier field.
The router determines the IP address of the next hop; for ingress routers to an AS, this is the IP address determined by the egress interface ID. For egress routers, this is simply the IP address of the configured neighbor router. If this is the last hop, the next hop is the destination address contained in the SCION routing extension.
The router then replaces the DA of the IP packet with this next hop address and sends it out towards this address.

Addressing

This is perhaps the biggest challenge for this idea.

• One option is to keep the bespoke SCION address types with their explicit separation of AS and end host identifier.
  Only the "host identifier" part would be included in the source/destination address of the IPv6 part.
  The AS identifier is only strictly needed for the path lookup, but not for the forwarding.
  It might still make sense to include at least the source AS identifier information somewhere in the packet, e.g. as part of the routing option or as a separate option somewhere.

However, this seems somewhat unappealing as this requires new entry types for DNS and any other naming systems. It also feels excessive to add another 64bit AS identifier on top of a 128 bit host address.
Ideally, we just use IPv6 addresses. Either:

• Allocate some prefix of IPv6 addresses for SCION, and encode the ISD-AS in the "routing prefix" part of the address.
  Entities can easily parse these addresses, e.g. to use the ISD-AS identifier in a path lookup.

  This would probably be very hard to fit into the IP allocation schemes in use with IANA and the RIRs, which seems to be entirely detached from AS number allocation.

• Encode the ISD-AS somehow in the "interface identifier" part of the IPv6 address. Nobody needs this anyway. (?)

• Add a lookup system to determine the AS given an IPv6 host address (a bit like in LISP).
  This could be based on DNS, or based on the SCION control plane, or based on RPKI.

Service Addresses

On top of the addressing in general, there is also the question how to represent the anycast addresses for the SCION control service(s). This seems easy in comparison...

SCION-IP-gateway

The SCION-IP-gateway becomes an entirely different thing.
Instead of tunneling IP packets over SCION, like it currently does, it is now just a middlebox that adds the SCION routing header to IPv6 packets where this is missing.
For this it needs to determine the destination AS of the IPv6 packet. Either this can done based on local routing rules ("these prefixes should be delivered to that AS"), or by determining the AS that the destination address belongs to. This can either require a lookup , or simply parsing the IPv6 address (see discussion on addresses above).

There is no need for an exit gateway, as the routing option does not necessarily need to be stripped.
The IPv6 packet can just be forwarded "normally" to its destination host once it reaches the end of the SCION path.
This can also be used to use SCION routing for only parts of the inter-domain routing path (~analogous to the SBAS system).
As a nice side benefit, this gives direct interoperability between hosts/applications that support SCION natively and those that use a gateway.

One challenge might be that without an explicit tunnel connection to a remote SIG, it becomes harder to keep track of path health.

Benefits

• Keep all the nice SCION properties (routing and fault isolation, explicit path failover, multi path, stateless routers,...)
• Reuse a lot of stuff from IPv6; "only" add SCION header processing to network devices and hosts, instead of building a copy of the entire stack and tooling.
• Would likely appear much more appealing than a separate dataplane for the wider internet community and IETF.

Challenges

• We lose IPv4 support. We lose the ability to seamlessly communicate between ASes internally using IPv4 vs. IPv6 (and any other potential local address type).
• A lot of effort invested in current system with the SCION packet types. Not all of this would still be useful.
• Implementation requires changes in IP stacks which may be harder to pull off than building a system on top.
• The "clean-slate design" claim becomes (even more) questionable, even though it still applies to the control plane and PKI parts of SCION.
  Overall, the "branding" for SCION might become less clear^[dubious].
• The obvious name for this idea would be "SCION Routing over IPv6", but the abbreviation SRv6 is already taken.

---

I wrote down specific options to get the discussion started. This is in no way meant to restrict the "proposal space"; as any reader will no doubt have noticed, my knowledge of IPv6 details is on the level of reading Wikipedia articles.

[matzf]

@sgmonroy and @oncilla explained to me in an off-line discussion how we can use the same approach as used in Segment Routing, to get AS local forwarding without having to support the SCION routing extension in all local IP routers;
The idea is to put the next hop address as the destination address of the IP packet and simply replacing this at every border router (instead of, as I had previously proposed, keeping the destination address as the actual destination and inspecting the SCION routing header even for local forwarding).
The only cost of this approach is having to keep the actual destination address in the SCION routing header (i.e. a 16 byte overhead). This is still far better than ending up having to encapsulate the packets locally.
I've already updated the description above.

[nicorusti]

Thanks for putting this together @matzf , I really like the proposal and I see a lot of value in pushing it forward! From an interoperability point of view, the fact that SCION currently has its own data plane is definitely something that raises some eyebrows, especially around the IETF. Moving to IPv6 EH would definitely help in that direction.

On some specific topics:

About addressing & SIGs:
I think you summarized the options well, however each one is achieving something slightly different. The key issue to me seems where and how to do IP to ISD-AS mapping:

• No default mapping (as it is today): the SCION endpoint must look up, out of band, the destination ISD-AS (currently with the DNS TXT record, or based on routes if it is a SIG). This leads to the side property that an host address can be reused in multiple ISD-ASes. This keeps the mechanism roughly as they are now.
• Implicit mapping (ISD-AS encoded somewhere in the IP header): this would create somehow a static mapping IP-> ISD-AS. I’m not sure how feasible it is (neither in the routing part, neither in the interface identifier). I think all bits in an IPv6 could be in principle in use
• Explicit lookup system: The challenge here is how to distribute prefixes to ISD mappings. How about SIAM? I’m not too familiar with it, could it be used?
  Regarding existing methods, they all rely on BGP in some way:
  • You mentioned RPKI. The actual prefix → AS mapping is relayed by BGP.. ROAs then can only help to validate prefix origin once you receive the prefix via BGP
  • LISP relies on regular internet and BGP for the initial EID to ETR lookup. One method is to use RFC6836 (LISP+ALT). This approach uses a GRE overlay connecting ALT-routers that speak BGP to each other to exchange EID prefixes to ETR mappings. There is also a map server (RFC6833). In a nutshell, this approach relies on BGP

I think choosing between the approaches above depends on what interoperability & transition mechanism we envision being widely used in the long term. Based on those, then the SIG would look a bit different. What do we actually want to achieve here?

Additional challenges to keep in mind:

• Some IPv6 EH are widely dropped. draft-vyncke-v6ops-james contains measurements showing that routing EH are dropped in up to 10-20% of cases, depending on routing type and upper level protocol. The segment routing header (routing type 4) gets dropped more frequently as it is expected to be used on intra-domain networks only.
• Getting a IANA routing type (via IETF Review or IESG) will take some IETF effort and time.

[matzf]

Regarding dropping packets with routing headers. I read the results of this study a bit different. Generally the routing headers work as expected:

Other routing header types (2 == mobile IPv6 [RFC6275], 3 == RPL [RFC6554], and even 5 == CRH-16 and 6 == CRH-32[I-D.draft-bonica-6man-comp-rtg-hdr]) can be transmitted over the global Internet without being dropped (assuming that the 2.5% of dropped packets are within the measurement error). At least, this is true for UDP. We still need to investigate the differences for TCP equivalent transmissions.

Note that forwarding packets with the SCION routing header would only be required in ASes that explicitly support SCION (at least as long as we ignore some SBAS-style mechanisms), which seems significantly easier to achieve than transmitting these packets over the global Internet. This makes me somewhat optimistic that the routers generally treat these routing headers as expected and that it should be rather easy for an AS to ensure that these packets are forwarded properly when it explicitly opts to deploy SCION.

[matzf]

Tunneling

Coming from #3961/#4280, i.e. our struggle to find a way to get rid of the "dispatcher" for receiving SCION packets on end hosts, this IPv6 approach would have the great advantage that most end hosts support IPv6/UDP and we wouldn't have to do anything to explicitly dispatch incoming UDP packets. Even ICMP error messages might be dispatched properly. That is, as long as the IPv6 stack ignores an unknown routing header like it's supposed to.

However, sending packets with a SCION routing header would still require either kernel support or raw IP sockets (and thus elevated permissions, which is not ideal).
One possible way to address this would be to support a UDP-based tunneling mode for outgoing packets in the routers. Either Foo-over-UDP (FOU) or Generic-UDP-Encapsulation (GUE) or would seem appropriate.
For transmitting packets, applications would "manually" create the full IPv6 packet header including the SCION routing header and encapsulate this in a UDP dataframe sent to the router.
The router does not need to encapsulate incoming packets; traffic thus looks a bit asymmetric, packets are only encapsulated on the very first hop between the end host and the egress router.
The router can support both encapsulated and non-encapsulated traffic e.g. distinguished by different local IP addresses. Applications on hosts with native SCION support (or using raw IP sockets) can avoid the MTU penalty incurred by the encapsulation.

The same tunneling mechanism can also give us (back) support for IPv4-only networks. Outgoing SCION-IPv6 packets are encapsulated as described above. We can use IPv4-mapped IPv6 addresses for the host addresses in the SCION-IPv6 packets, and routers can encapsulate incoming packets for destination addresses in this range. Hosts in IPv4-only networks would then need configuration to locally strip the encapsulation for receiving packets (either using the OS built-in support for these encapsulation protocols, or a by a custom background service a bit like the dispatcher).

Note: this is all a bit speculative, I haven't tested anything of this, but it promises to solve a bunch of problems at the same time. Unless someone already sees a hole that invalidates this idea, I will try build a proof-of-concept prototype (or try to find someone who can build it).

[jiceatscion]

There's an implementation of RFC8754 in the Linux Kernel. So, we could probably take advantage of that if we want to stop being limited by user-space-only code. V4 support would still need some tunneling, of course.

[jiceatscion]

I've been giving some thoughts to the addressing issue as a concern in itself. Whether or not we use IPV6 as our native dataplane, there would be considerable benefits if applications could use real V6 addresses to reference SCION hosts: we get to use unmodified DNS, unmodified URLs and, in contexts where it is ok for the application to not choose a route itself, unmodified socket APIs. This would mean that my vanilla web browser could connect to a vanilla web server over scion without even knowing it; as long a scion router can be inserted into the networking stack (which can be done with either a kernel module or by repurposing some existing interception mechanisms such as used for virtual networks).

Even if we can't inject the router transparently we could still make unmodified applications work either by way of extensions or by way of dynamic linking trickeries. The important part is v6-compatible addresses, therefore standard URLS, address configs, and DNS.

Now what would it take for scion addresses to be V6 addresses? Just fewer bits. They would probably need to be shorter yet.

Here are the constraints:

• RFC4291 insists on the 64 LSbits to be available to contain a NIC number.
• RFC6890 Instructs IANA to reserve 2001::/23 for experimental protocols, which they allocate in blocks of /29. Many of those are available an dprobably easy to get. (https://www.iana.org/assignments/iana-ipv6-special-registry/iana-ipv6-special-registry.xhtml)
  RFC3587 reminds that IANA does not allocate addresses outside the 2000::/3 range (https://www.iana.org/assignments/ipv6-unicast-address-assignments/ipv6-unicast-address-assignments.xhtml). However it is not forbidden from doing so and there's still plenty of space (e.g. 2d00:0000::/8 is not assigned and nothing is after that).

So, we could:

• Easily get a /28 prefix. leaving us 36 + 64 bits where to fit the scion address bits.
• May be get a large slice from the normal pool. For example a whole /8, would leave us 56 + 64 bits.

Here's why I think we can fit an entire scion address, ISD (16 bits), AS (48), and HOST (a whole V6 address) into less than 128 bits?

• HOST: if we can make scion addresses be a special case of V6 address, HOST does not need to be a full V6 address. The scion address can uniquely identify the host just by including the same 64 LSbits as the regular v6 address would.
• AS: we do not need 48bits to represent AS numbers; not even 32. The existing AS numbers fit on 19 bits as of today. Since AS numbers are assigned sequentially, with no blocks or vanity prefixes, we'd need to allocate 1 per second for the next 136 years before we run out. So, how about we stick with 32 and revisit in 136 years? If we really want to have a special-purpose range on top of that, we can throw in one or two extra bits, but not 16!
  ISD: It is not necessary to uniquely identify an AS since AS numbers are supposed to be unique in themselves. So, does it really need to hog our most critical commodity, address bits?

With that, we can easily fit in a /8 address block:

• HOST: 64 bits
• AS: 40 bits (who's counting?)
• ISD: 16 bits

If that is easier to get, we could fit in a /14 block and grow later:

• HOST: 64bits
• AS: 34 bits
• ISD: 16 bits (we use only 14, though, so two are fair game)

For a meager /28 it's more work. Either:

• HOST: 64 bits
• AS: 32 bits (no special range)
• ISD: 4 bits (is it even useful at that point?)

Or:

• HOST: 64 bits
• AS: 36 bits
• ISD: gone - find some other way to get AS -> ISD mapping.

Or (near term work-around):

• HOST: 64 bits
• AS: 22 bits (enough to contain current ASNs plus pure scion ones)
• ISD: 14 bits

[jiceatscion]

FWIW, I wrote the above a bit more formally in a design proposal:
https://raw.githubusercontent.com/jiceatscion/scion/v6_address_design/doc/dev/design/V6AddressSupport.rst

[vstambolic]

I would like to provide an update on the "IPv6 as SCION Dataplane" project that stemmed from this discussion. I did it as a part of my Semester Project within the Network Security Group at ETH during my third semester of master's studies last autumn under the supervision of @marcfrei and @FR4NK-W.

Our research primarily focused on evaluating the survivability of IPv6 packets with (Routing) Extension Headers. To assess the survivability rate of packets, we set up several experiments in various network environments and tried to identify the factors influencing packet drops. Unfortunately, the results turned out to be quite discouraging. Here, I wanted to share a brief overview of the methodology we used, the evaluation results, the identified constraints, and some of the conclusions we made.

Methodology

To measure the survivability rate of IPv6 packets with Extension Headers, we employed several probes positioned at various locations as vantage points. The vantage points included an instance hosted by Digital Ocean, and two instances in the AWS ecosystem.

Each probe deployed at these locations sends standardized IPv6 packets containing a Routing Extension Header. For the layer 4 protocol, we test both UDP and TCP headers, following the example from previous research. We experimented with varying EH sizes, ranging from 0 bytes to 256 bytes of EH optional data. Probes deployed at vantage points send packets in batches. We determine the survivability rate by counting the responses from the target nodes. More specifically, we monitor the number of successfully received responses against the total number of packets sent within each batch.

Results - DigitalOcean Vantage Point: Diverse Destinations

In one of the experiments, we selected diverse types of environments as targets to assess the survivability rate in different network settings. The chosen destinations featured Tier 1 ISPs PCCW in Hong Kong and Verizon in the United States, the national research and education network ZAMREN in Zambia, and the AWS Ireland instance from a well-known cloud service provider network. As a vantage point, we employed an instance from the DigitalOcean environment.

With the exception of PCCW, all UDP packets with an EH data size of 112 bytes survive, along with a large portion of UDP packets with 128 bytes of EH data. After further investigation, we discovered that most of the drops of large packets once occur within the first AS. More precisely, the border routers of the DigitalOcean AS set a restriction on the EH data size to 128 bytes. For packets directed towards PCCW Global, the limit lies within the range of 64 to 96 bytes, this time due to constraints imposed by border routers from other intermediate ASes.

Our findings proved that the survival rate does not directly depend on the geolocation or the service provider. Instead, it is influenced by the technical capabilities and operational behavior of the intermediate routers along the path. The critical factor proves to be the AS with the smallest maximum allowed packet size. In the case of PCCW Global, this corresponds to an intermediary AS with a capped limit of 96 bytes, whereas in other scenarios, this limit is set to 128 bytes by the initial AS, DigitalOcean.

Identified Constraints

We attribute the observed network behavior primarily to operational and technical constraints associated with intermediate routers.

Regarding the technical constraints, modern high-end routers are known to parse up to 192 or 384 bytes of header rfc9098, which generally proves to be sufficient for handling most Extension Header use cases effectively. However, as seen from the plots, this threshold falls significantly lower in realistic network environments. In our other experiments, we determined that the limit falls to as low as 48 bytes in some environments. This is clearly far from what is necessary to accommodate particularly large SCION packets.

Furthermore, large extension headers in IPv6 may also result in disruption to load-balancing, packet filtering and security mechanisms. For example, before accessing the layer 4 header, a load-balancing system must first process the entire IPv6 packet, along with its extension headers. If the system is not able to access the required information, it will most likely drop the packet. We attribute some of the detected packet drops to these types of constraints as well.

We also detected that some firewalls are by default configured to reject packets with unknown Routing Header types, both for inbound and outbound traffic. This means that introducing new Routing Header types like SCION, would require some reconfiguration of firewall rules in some networks.

Summary

To summarize, our measurements showed that today’s networks, including those of major service providers, are generally not well-prepared to support the integration of SCION into IPv6. However, we note that deploying SCION within an AS nonetheless requires some infrastructural changes and router reconfigurations. In this context, integrating SCION into the IPv6 framework and ensuring the smooth forwarding of packets with SCION Routing Headers becomes another integral step in deploying SCION within an AS. Furthermore, IPv6 is still a relatively young protocol, and as it matures, we can anticipate that ASes will provide better support for EHs. That said, even though SCION-IPv6 integration doesn't seem feasible at this moment, the idea definitely remains plausible for future consideration.

Related Work

https://datatracker.ietf.org/doc/html/draft-vyncke-v6ops-james-03
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4556007