2019/09/06/dx-gateway-deep-dive/

[utterances-bot]

AWS Direct Connect Gateway Deep Dive - Edge Cloud

A deep dive look into the AWS Direct Connect Gateway

https://www.edge-cloud.net/2019/09/06/dx-gateway-deep-dive/

[HammadAlam]

Fantastic write up.

[2stacks]

Great post! I was hoping you could clarify one thing. It may be my lack of understanding but this part seems to be contradicting.

In the case of a Hosted Connection - which only provides a single Virtual Interface - that VIF can either be a private, public or transit Virtual Interface. AWS Direct Connect Hosted Virtual Interfaces do not support Transit VIFs at all.

Perhaps I don't understand the difference between a "Hosted Connection" and a "Hosted Virtual Interface."

[chriselsen]

Have a look at the "How AWS Direct Connect Partners Enable Access to AWS Direct Connect" section on https://aws.amazon.com/directconnect/partners/ for a detailed explanation. But in a nutshell: Hosted connection provide reserved bandwidth and the ability to create Transit VIFs if the booked speed is 1 Gbps or higher. Hosted Virtual Interfaces are oversubscribed by the partner, offer no reserved bandwidth and don't exist as Transit VIF. Especially because of the lack of Transit VIF support, I see partners moving from offering Hosted Virtual Interface model to offering Hosted Connections.

[2stacks]

Thank you for the follow up. This makes perfect sense now. I'm working with Verizon to set up direct connect so this information is very helpful.

…

On Sat, Mar 14, 2020 at 7:58 PM Christian Elsen ***@***.***> wrote: Have a look at the "How AWS Direct Connect Partners Enable Access to AWS Direct Connect" section on https://aws.amazon.com/directconnect/partners/ for a detailed explanation. But in a nutshell: Hosted connection provide reserved bandwidth and the ability to create Transit VIFs if the booked speed is 1 Gbps or higher. Hosted Virtual Interfaces are oversubscribed by the partner, offer no reserved bandwidth and don't exist as Transit VIF. Especially because of the lack of Transit VIF support, I see partners moving from offering Hosted Virtual Interface model to offering Hosted Connections. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#32 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADUNEHC7IRAAWLPYMQRJ5Q3RHQK35ANCNFSM4J2WBDYA> .

[crash5723]

Great right ups. Major kudos for the effort. One thing I can't seem to get my head around with regard to the DxGW / TGW relationship. Hopefully, you can sort me out. I have a situation where we will have two separate Direct Connect Links going to AWS. The idea is for these links to be HA or at the very least Active/Standby. Will I have each dxcon use the same DxGW and then that DxGW attaches to the TGW? Or am I looking at a scenario where I have two DxGW that attach to the TGW. I read your write up with regard to the dxcon / vpn connectivity, but I wasn't sure that this was the same. Am I affected by the "1 transit VIF per dxcon" ? Right now we have just the 1 dxcon connected to the DxGW with a transit VIF (and public VIF), but we are adding the second dxcon. Any help would be greatly appreciated.

[chriselsen]

You can connect both DX to the same DX-GW, which then connects to the TGW. Have a look at the "Local preference BGP communities" section of https://docs.aws.amazon.com/directconnect/latest/UserGuide/routing-and-bgp.html. If you want to use the two DX in Active/Active you should set the same local pref community - e.g 7224:7200 - on prefixes over both Transit VIFs.

[clgeisler]

Thanks for your write-ups. I'm new to AWS networking, and am uncertain of some things. We have a GovCloud West account with a transit gateway setup and a single vpn attachment and a single vpc attachment with static routes. No BGP. We are now trying to establish a hosted 1G Direct Connect from Colo in the east. We provisioned the direct connect to the East region and associated to our public account. We want to establish a connection to the same govcloud west transit gateway. My thought was we will need a direct connect gateway and transit VIF for this. This will dictate that we need to setup BGP. Which is fine.

However. when setting this up, we wanted to specify the VLAN, ASN, and IP that is used to our on-prem (Colo cage) router (customer gateway) for the direct connect gateway. It didn't seem to alow us to choose the VLAN or ASN for the AWS side . We could only specify the ASN associated with the on-prem router (customer gateway). Do we need to do something differnt or setup BGP on the transit gateway first?

Secondly, is there anyway to encrypt this traffic over the direct connect? It seems the one AWS option is to setup a public VIF for the direct connect and setup a VPN attachment that way. The other option seems to be deploying and EC2 into the VPC, but that doesn't seem to scale well (will need EC2 VPN gateway for every VPC.

Thanks for your input

[chriselsen]

I know that the wording in the console and documentation is a bit confusing. When you setup a DX Virtual Interface (VIF) - incl. Transit VIF - you actually specify your side of the connection within that mask. Therefore you have to enter your ASN and VLAN there. In the case of a Transit VIF, the AWS-side ASN is the one that you gave to the DX Gateway. Because that's what your router will for forming the BGP session with.
You are right about using DX Gateway with DX-Connection and VIF in the commercial account to connect on-premises to GovCloud. That's how it's done. And you're also right about the two options of getting encryption between AWS and on-premises: 1) AWS Site-to-Site (IPSec) VPN over Public VIF 2) EC2 based software router that terminates any kind of VPN tunnel. For the public VIF you will want to filter what IP prefixes you receive over it, otherwise you get all prefixes from AS16509. Reach out to your AWS SA and request a session with a Network Specialist. That should get you sorted out on any open questions pretty quickly.

[tkvenu]

Christain - help ( any pointers would help)
client env : on-prem DC is connected to AWS account1 using DX/direct connect ....

new request: can we create a separate/brand new AWS account2 (should not be interconnected to account1) and make use of a common DX connection to account2 in parallel with account1?

you can consider account1 (control tower1) /owner1 and account2 ( control tower2)/ owner2

[chriselsen]

Have a look at the Multi-Account support section above, which explains how DX connections, DX-GW and VGW can be distributed across multiple accounts.

[wongwiyan]

Excellent write up. I learnt a lot from it. Can you elaborate about how flow between VPCs using hairpin thru the on-premises network is prevented.
(note https://docs.aws.amazon.com/directconnect/latest/UserGuide/virtualgateways.html
suggests the same thing by saying
The following traffic flows are not supported:
Direct communication between the VPCs that are associated with a single Direct Connect gateway. This includes traffic from one VPC to another by using a hairpin through an on-premises network through a single Direct Connect gateway.)
However this contradicts with https://www.megaport.com/blog/aws-vgw-vs-dgw-vs-tgw/ where it says CIDR addresses can’t overlap, and traffic will not route from VPC-A to the Direct Connect Gateway and to VPC-B. It will instead route as follows: VPC-A > Direct Connect > Data Center Router > Direct Connect > VPC-B.)

It is important to point out that the red depicted data flow not only includes >BGP routing traffic, but also routed traffic. Looking at the VIFs facing on->premises, you will not receive BGP route announcements from the Direct >Connect Gateway that were originated by one of the other VIFs. But even >attempting to place a static route towards the Direct Connect for traffic from >one VIF to another will fail.

[chriselsen]

Excellent write up. I learnt a lot from it. Can you elaborate about how flow between VPCs using hairpin thru the on-premises network is prevented.

I updated the corresponding section with additional clarification. The Megaport statement is actually not fully correct. Let's have a look at their setup and assume that you are announcing a 10.0.0.0/8 route into the DX-GW from on-premises. In that case traffic from one of the VPCs in AWS account 1 towards e.g. 10.20.0.0/16 would be attracted towards the DX-GW. Once traffic hits the DX-GW, there is a route present towards 10.20.0.0/16, which points towards the upper VPC in account 2. Therefore traffic would flow via the DX-GW between these two VPCs - which is the red path depicted in my article above.

[evandengdbw]

Great post~ I am wondering where is the DX gateway deployed at? At the direct connect location or other place? By the way, I found that some documents describe DX gateways as bgp route reflectors which not process data traffic. But some other documents tell that DX gateway consists of logical devices which contain route layer and dataplane. What is the AWS logical device? I am so confused. Could you please help me out?

[chriselsen]

Yes, the Direct Connect Gateway is a distributed route reflector. I added a section above to clarify this. Hope that helps.

[evandengdbw]

Thanks a lot for your excellent write-ups.
Since data traffic flows from direct connect location to transit gateway/VPC via the AWS backbone, does the DX data traffic flows in the backbone via overlay network or underlay network ? does it use VXLAN?
Could you please give some details about the DX location and the edge device AWS uses? Does AWS use dedicated router at the edge to build EBGP sessions with customer router and build IBGP connection with DX gateway to exchange the routes（L3 access) or just build EBGP between DX gateway and customer（L2 access in DX location）?