2019/08/16/aws-dxgw-with-ipsec-vpn-backup/

[utterances-bot]

AWS Transit Gateway with Direct Connect Gateway and Site-to-Site (IPSec) VPN Backup - Edge Cloud

This article shows you how to setup a primary active Direct Connect connection between an AWS Transit Gateway and on-premises networks via Direct Connect Gateway, while leveraging a Site-to-Site (IPSec) VPN as backup.

https://www.edge-cloud.net/2019/08/16/aws-dxgw-with-ipsec-vpn-backup/

[HammadAlam]

Great article. Couple of questions.

a. in "DX Gateway announced prefixes" you mentioned "number of prefixes per AWS Transit Gateway from AWS to on-premises on a transit virtual interface (via Direct Connect Connect Gateway) is limited to 20"

This means that from a DX Gateway perspective, it can announce max of 60 prefixes to on-prem (with 3 Transit Gateways attached)?

I have read in some places where it makes it feel like the limit of 20 prefixes is per DXGW and not per Transit Gateway.

b. How many Direct Connect can be attached to a single DX Gateway?

[chriselsen]

Have a look at https://www.edge-cloud.net/2019/09/06/dx-gateway-deep-dive/#bgp-prefixes which should answer your questions in more detail:

a. in "DX Gateway announced prefixes" you mentioned "number of prefixes per AWS Transit Gateway from AWS to on-premises on a transit virtual interface (via Direct Connect Connect Gateway) is limited to 20"
This means that from a DX Gateway perspective, it can announce max of 60 prefixes to on-prem (with 3 Transit Gateways attached)?

Yes, that's correct.

b. How many Direct Connect can be attached to a single DX Gateway?

You can have 30 Virtual interfaces per AWS Direct Connect gateway. In case of Transit Gateway these Virtual interfaces are Transit VIFs. As there can only be 1 Transit VIF per DX connection (dxcon), you need 30 dxcon to achieve this. In the case of VGW, which uses Private VIFs, all these 30 Private VIFs could be on the same dxcon.

[HammadAlam]

Fantastic and thanks for quick resply. One more thing re DX/DXGW/Transit GW if you dont mind:

Ref: https://www.amazonaws.cn/en/directconnect/faqs/

Q. I have two Direct Connect connections, both 1G, I want all incoming traffic into my network load balanced across these two connections, can I use community based routing to achieve such load balancing across the locations?

Yes, you can use community based routing to enable load balancing across Direct Connect locations; To do so, any prefixes requiring load-balancing must be marked with the same communities.

The way I understand the question is to load balance across 2 DX paths from AWS to on-prem.

So if 10.10.0.0/16 was advertised by 2 DX connected to same or different DXGW, the DXGW(s) install this route in the Transit Gateway route table attached to VPCs saying to get to 10.10.0.0/16 the next hop is . However, if both DX attachments are advertising same routes and Transit Gateway does not support ECMP on DX, how would the load balancing work as mentioned in the FAQ.

Any insight is much appreciated!

[chriselsen]

In this case the standard BGP Best Path Selection Algorithm applies. If ECMP is disabled on the TGW, then step 9 in the reference above becomes false and the algorithm continues to step 10, which says "out of several external paths, this criterion selects the first received path". So whatever route the TGW saw first will be selected.
This step minimizes route-flap because a newer path does not displace an older one, even if the newer path would be the preferred route based on the next decision criteria (steps 11, 12, and 13).

[HammadAlam]

Thanks again Christian. Your input is very valuable and appreciated.

I have been unable to find any AWS documentation that said ECMP for BGP at TGW is supported for DXGW attachment/association. In fact, several articles clearly said that the Transit Gateway ECMP checkbox is only for IPSec VPN connections. For ex, https://docs.aws.amazon.com/vpc/latest/tgw/tgw-transit-gateways.html

So what I understand from your last comment is that if same routes with same pref etc are being advertised to the Transit Gateway from two different DX Gateways, Transit Gateway will ECMP/Load Balance traffic over both paths.

[chriselsen]

TGW can ECMP across multiple DX-GW - obviously if ECMP is enabled on the TGW.
You can even announce the same prefix over 1+ DX-GW and 1+ VPN tunnels. In that case traffic would flow over the DX-GW(s) only, but still via ECMP. If you withdraw the prefixes from all DX-GW, the VPN(s) will be used.

[HammadAlam]

Thanks a lot. ECMP over DXGW is a key distinction that I wanted to validate. Much appreciated!!
Following you on Twitter now :)

[fabbio1980]

And the simple conf : DX and VPN as a backup without TGW, is going to be same?

[chriselsen]

And the simple conf : DX and VPN as a backup without TGW, is going to be same?

If you are not using TGW for the DX and VPN, you'll be using a VGW within a single VPC. In that case things are much easier as the VGW propagate the same VPC routes to on-premises. For prefixes announced from on-premises to VGW, BGP routes over DX are preferred.

[SheanLV]

Great write-up, thank you for the in depth info. I did have a question on the AWS side of a BGP peering, both VPN and DX. Will the AWS side of the BGP peering take action based upon the MED set by the customer gateway? If I have multiple private VIFs for example, can influence AWS routing preference by setting a MED of 50 on connection and 100 on the other?

[chriselsen]

Multi Exit Discriminator is not supported over Direct Connect. Instead you can use Local Preference BGP Communities (https://docs.aws.amazon.com/directconnect/latest/UserGuide/routing-and-bgp.html#bgp-communities-private-transit) to control which of your private VIFs should be preferred by AWS.
Over AWS IPSec VPN, MEDs are supported.