Home

Understanding the AWS Transit Gateway

During AWS re-Invent 2018 event, Amazon announced the Transit Gateway virtual distributed routeing device. What exactly is it? What does it do? What are the limitations? There was a lot of buzz around the announcement and for good reason. Traditionally (I know right “traditional” keyword in cloud speak) one had to deploy third-party virtual-machines such as the Cisco CSRv or a firewall device such as Palo Alto Networks or Fortinet Fortigate. The Transit Gateway takes direct aim at these solutions.

Transit VPCs

By now, we’re all pretty familiar with the lack of one aws vpc to speak with another aws vpc by transiting through a central aws vpc. If you didn’t know this already, now you do. Confused?

VPC-B <------> VPC-A <------> VPC-C

Given that both VPC-B and VPC-C are peered with VPC-A, this means B cannot talk to C by going through A and like wise, though A can communicate with both. To avoid this “limitation” (in quotes, because there are good reasons for it) a third-party solution is necessary. For example, AWS has a document using the Cisco CSRv router, here’s the Cisco link. There are a few others as well, probably too many to list.

What these solutions do is create an overlay on-top of the VPC Networking, some would traverse the VPC Peerings and create tunnels between, others such as the Cisco would utilize VPC Endpoints.

Traditional Limitations

Besides being complex and not cloud-native some limitations such as handling high-availability set in. There’s no layer-2 in the cloud. Handling keep-alive for these solutions turned out to be quite a hassle, utilizing CloudWatch Alarms to signal when something went wrong. How about sizing? The virtual-machines and license needed to be sized correctly from the get-go. Upgrading this turned into a hot-potato upgrade procedure, where AMI images would be created from running systems, new bigger ones spun up, then other ones torn down.

Running things in the cloud as you do on-premises is a mistake, a costly one so many make.

What if the virtual-machines were oversized? Then you’re in the same boat. Do you pay more for something not being used and wait until business traffic catches up? Do you downgrade and then upgrade later? There’s a cost to all this, with virtual-machines you’re paying hourly for the compute, storage, networking i/o and inter-vpc traffic. The bigger the VM the more it costs.

There are personnel costs, maintenance fees (you know, smartnet), licensing fees (don’t get me started), but I won’t get into that today.

Enter the Transit Gateway

The AWS Transit Gateway was introduced to address most if not all of the limitations face with traditional approaches. With the Transit Gateway there’s automatic scaling of throughput with pretty decent defaults. Here’s a list of defaults from the FAQS Page

Type Limit Default
Number of AWS Transit Gateway attachments 5,000
Maximum bandwidth per VPN connection* 1.25 Gbps
Maximum bandwidth (burst) per VPC connection 50 Gbps
Number of AWS Transit Gateways per account 5
Number of AWS Transit Gateway attachments per VPC 5
Number of routes 10,000
Number of Direct Connect gateways per AWS Transit Gateway 20

You also get aws cloud-native API integration to use tools such as Terraform and Cloudformation amongst others. Yes, you can use cloudformaiton to deploy other solutions, by instantiating EC2 Instances and passing user data. But really? That something you want to be doing?

I’m sure there are edge cases to use third-party solutions, this is meant to encompass most use-cases.

Simplified pricing (not necessarily cheaper if you’re looking at just $ numbers) by charging per-connection per-hour and the amount of traffic that flows through those connections. Keep in mind that one VPC to VPC attachment is taking up two (2) connections, one for each end.

Transit Gateway Use-Cases

Besides what we’ve already covered and simplifying your cloud network. There are other use cases that would be similar two third-party solutions. Having a security-doamin where all traffic in and out of your cloud network goes through a security-hardened VPC. A central-services where things like custom-dns, authentication, authorization, identity and management are enforced at this VPC layer.

Topology Diagram

Here’s the diagram we’ll be using for this example, Where we’ll have a central-core VPC that routes internet-bound traffic and any VPC-to-VPC flows can go direct to each other.

Click on the image for a full-size version. Transit Gateway VPC Diagram

Toplogy breakdown:

  • Core VPC - This is where any VPC-to-outbound traffic will flow through (our transit VPC), the name Core was shorter for pictures and spacing.

    • VPC CIDR is 10.244.0.0/16 has one private and one public subnet of 10.244.0.0/24 and 10.244.128.0/24 respectively, each with a single EC2 Instance residing.
    • Has a single NAT Gateway and distributed Internet Gateway for outbound external traffic to flow.
  • Stub-1 VPC - Contains a single EC2 Instance to test communication, in later posts, it will expand to hosts certain applications.

    • VPC CIDR is 10.245.0.0/16 with a single private subnet of 10.245.128.0/24
  • Stub-2 VPC - Same as Sub-1 for now

    • VPC CIDR is 10.246.0.0/16 with a single private subnet of 10.246.128.0/24
  • VPC-to-VPC traffic can flow directly through the transit gateway.

I kept this simple and single availability zone to keep focus on the Transit Gateway functionality. In real-world applications, please have multi-az and multi-region if possible. For future post, I will upgrade the infrastructure to also have VPN functionality to the Transit Gateway, for this post it would be waaaaayyyyy too long 👻.

Transit Gateway Componenets

Like all things AWS the Transit Gateway service is broken down into multiple components to consume.

  • The Transit Gateway - itself is the hosted service (“serverless”, ok just kidding..maybe) by AWS.
  • The Transit Gateway VPC Attachments - These point to the VPCs and Subnets within those VPCs to attach to the transit gateway.
  • The Transit Gateway Route Tables - As the name implies, these are logical route tables to create routing domain segmentation (VRFs?).

    • If you wanted to have a subset of VPCs communicate with some and not others, ths is how you would separate that.
  • The Transit Gateway Routes - Like any other routes, these are CIDR based routes that point to a Transit Gateway Route Table and the attached VPC.

In the AWS Console under the Transit VPC section only the Transit Gateway, Transit Gateway Attachments and Transit Gateway Route Tables are visible. The routes are created inside the Route Table section. The only time we’ll be touching the console for this is to take screenshots of what’s created.

The entire infrastructure in the diagram above was deployed with terraform and the code is available here. If you’re curious, this entire thing takes just under three (3) minutes to deploy, your mileage may vary.

terraform apply plan.tfplan 5.75s user 1.46s system 4% cpu 2:55.28 total

The Transit (CORE) VPC

This is where all outbound traffic will flow to and from, whether that’s headed to the Internet or later on to On-Premises network.

Lets take a quick look at what this VPC looks like.

aws ec2 describe-vpcs --vpc-ids vpc-0614b23941147065a --region us-east-1
{
    "Vpcs": [
        {
            "CidrBlock": "10.244.0.0/16",
            "VpcId": "vpc-0614b23941147065a",
            "InstanceTenancy": "default",
            "CidrBlockAssociationSet": [
                {
                    "CidrBlock": "10.244.0.0/16"
                }
            ],
            "Tags": [
                {
                    "Key": "Name",
                    "Value": "Core-VPC"
                }
            ]
        }
    ]
}

Some of the output was removed and will be from most CLI output for brevity and to make things concise. The CIDR for this VPC as you can see is the 10.244.0.0/16 network.

The Other VPCs

Here’s a screenshot of the other VPCs so we don’t spend all day on JSON output formatting.

Other VPCs Image

You can see the corresponding CIDRs assigned and the state of each VPC.

The VPC Routing Tables

Lets take a look at the routing tables for the VPCs to make sure things are “pointing” in the right direction. Let’s see the view from the Core VPC.

The Core VPC

Public Route Table Core VPC Public Route Table

The public route table has three routes as per the diagram

  • 10.244.0.0/16 pointing to “local” per AWS speak, means VPC wide
  • 0.0.0.0/0 towards the internet gateway
  • 10.244.0.0/14 towards the Transit Gateway

Private Route Table Core VPC Private Route Table

The private route table has three routes as well, what you’ll see in common is the local route so I’ll stop including it.

  • 0.0.0.0/0 towards the NAT Gateway
  • 10.244.0.0/14 towards the Transit Gateway

The diagram below depicts this in a more visual manner, the public route table in this demo is strictly for a bastion EC2 instance to SSH and do things inside the cloud network.

Diagram Core VPC Diagram

The Stub-1 VPC

For the sake of time, readability and sanity I will not include details for the Stub-2 VPC as they’re mirrors of each other with exceptions of CIDR Blocks and subnets, I will include the image diagram.

The Stub-1 and 2 VPC only have private route tables and no public ones as all traffic will flow through the Core VPC.

Private Route Table Stub-1 VPC Private Route Table

Aside from the usual local route, you can see there’s two additional routes.

  • 10.244.0.0/14 towards the Transit Gateway
  • 0.0.0.0/0 also twards the Transit Gateway

Why /14? Because it covers all the VPC CIDR networks I’m using quite nicely and has room for another. Why 10.244 you ask? Because I like that number, don’t ask anything else again, and I think it’s the default /16 for Weaveworks Weavenet Kubernetes CNI? Or one of those, and I was playing with it for a bit, stuck with me.

Here are the diagrams as visual aids in what we’re working with, also a nice distraction from all the text, I know, I know.

Stub-1 VPC Diagram Stub-1 VPC Diagram

Stub-2 VPC Diagram Stub-2 VPC Diagram

The Transit Gateway

{
    "TransitGateways": [
        {
            "TransitGatewayId": "tgw-0c045a4c92c630a20",
            "State": "available",
            "Options": {
                "AmazonSideAsn": 65100,
                "AutoAcceptSharedAttachments": "enable",
                "DefaultRouteTableAssociation": "enable",
                "AssociationDefaultRouteTableId": "tgw-rtb-0f72208f89e2bc18a",
                "DefaultRouteTablePropagation": "enable",
                "PropagationDefaultRouteTableId": "tgw-rtb-0f72208f89e2bc18a",
                "VpnEcmpSupport": "enable",
                "DnsSupport": "enable"
            }
        }
    ]
}

The gateway itself has a few things to take note of. Besides the typical ID and ARN fields for all AWS resources.

  • The AmazonSideAsn field is set to 65100 per the terraform code, if left blank this defaults to an Amazon default ASN.
  • The DefaultRouteTableAssociation field controls whether any attachments will automatically be assigned to the default route table of the gateway. By default this is disabled, I enabled it in the code for simplicity, in real world you will probably have a few route tables and this should be left alone.
  • The DefaultRouteTablePropagation controls whether the default route table propagates it’s route with any VPN or Direct Connect connections, by default this is enabled.

Transit Gateway VPC Attachment

What is a VPC attachment for a Transit Gateway? Exactly what it sounds like, except that aside from being attached to the VPC itself, the Transit Gateway also needs the subnets, lets take a look…

Only one subnet per availability zone is allowed, does not matter if it’s private or public though multiple subnets in different availability zones are suggested (you know, for obvious reasons).

aws ec2 describe-transit-gateway-vpc-attachments --transit-gateway-attachment-ids tgw-attach-00031ea6b5a4d1c86 --region us-east-1
{
    "TransitGatewayVpcAttachments": [
        {
            "TransitGatewayAttachmentId": "tgw-attach-00031ea6b5a4d1c86",
            "TransitGatewayId": "tgw-0c045a4c92c630a20",
            "VpcId": "vpc-0614b23941147065a",
            "State": "available",
            "SubnetIds": ["subnet-0e3d19b79d09f89e1"],
            "Options": {
                "DnsSupport": "enable",
                "Ipv6Support": "disable"
            }
        }
    ]
}

Taking a look at the VPC attachment two thing should become familiar when looking at the output.

  • The first is the VpcId of the attachment, this Transit Gateway is now attached to the given VPC.
  • Next is the list of SubnetIds, there’s only one here because both the public and private subnets are in the same availability zone. This specific output was for the Core VPC, the output for any of the stubs looks similar with different IDs.

See a picture of a stub VPC from the console view. Stub Transit Gateway VPC Attachment

The Transit Gateway Routing Tables

The last thing to look at and for the sake of this post, it’s only the default transit gateway route table created. Later on we’ll dive deeper into traffic segmentation with separate route tables, security zones and more.

Look a the default route table routes Default Route Table Routes

There are four routes for our purpose in this transit gateway, see pictured below…

Transit Gateway Routes

The non-default routes, 10.244.0.0/16, 10.245.0.0/16, and 10.246.0.0/16 are automatically added when the VPC attachment is created. The next route of 0.0.0.0/0 or default-route is was manually added by the terraform code. Because the Stub VPCs point their own default route to the Transit Gateway and the transit gateway’s default points towards the Core VPC all outbound traffic from the stubs transits the Core VPC.

Here’s a reminder (because it’s a long post) of what the infrastructure looks like. Follow the red lines for internet traffic flows and the blue lines for internal traffic flows.

Transit Gateway VPC Diagram

In this example, because there are no separate routing tables and no domains are being enforced, only routes. Any VPC can communicate directly with another VPC without going through the Core VPC, only internet bound traffic flows through the Core as it’s the only exit point.

In the next post on the subject, we’ll cover how to force even internal traffic through the core vpc.

Bonus (lets see some traffic)

If you stuck with it for now, lets see this actually work. Any reference to ssh keys have already been copied to the specified hosts (if you’re reading this, should be easy enough to do, if not, leave me a note). This is only to show that any VPC can talk to any other VPC directly, and to the internet through the core.

VPC to VPC Traffic

Lets login to bastion host and ssh into stub-1 instance ec2-2.

[I] ssh -i ~/.aws-keys/aws-dev-key -o StrictHostKeyChecking=no ec2-user@54.234.36.142
Last login: Fri May 17 22:04:16 2019 from d-137-103-55-25.fl.cpe.atlanticbb.net

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
[ec2-user@ec2-1a-ssh-bastion ~]$

From here we’ll ping and ssh into stub 2 ec2-3 host.

  • ssh from bastion to ec2-2
[ec2-user@ec2-1a-ssh-bastion ~]$ ssh -i .ssh/ssh-key -o StrictHostKeyChecking=no ec2-user@10.245.128.160

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
  • ping from ec2-2 in stub-1 vpc to ec2-3 in stub-2 vpc
[ec2-user@ec2-2 ~]$ ping 10.246.128.231 -c 3
PING 10.246.128.231 (10.246.128.231) 56(84) bytes of data.
64 bytes from 10.246.128.231: icmp_seq=1 ttl=254 time=1.49 ms
64 bytes from 10.246.128.231: icmp_seq=2 ttl=254 time=0.896 ms
64 bytes from 10.246.128.231: icmp_seq=3 ttl=254 time=0.942 ms

--- 10.246.128.231 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.896/1.110/1.494/0.274 ms

look at the IP addresses and reference the diagram to make sure they match to the VPC subnets

  • SSH from ec2-2 in stub-1 to ec2-3 in stub-2
[ec2-user@ec2-2 ~]$ ssh -i .ssh/ssh-key -o StrictHostKeyChecking=no ec2-user@10.246.128.231
Warning: Permanently added '10.246.128.231' (ECDSA) to the list of known hosts.
Last login: Fri May 17 20:00:22 2019 from ip-10-244-0-70.ec2.internal

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
[ec2-user@ec2-3 ~]$

I did ssh to make sure there’s bi-directional communication, as ping doesn’t necessarily test that, unless you ping from the other side. SSH also tests state and that nothing’s breaking in the middle.

VPC to Internet

Now lets test internet connectivity on a stub VPC with no default internet or nat gateways, only a default route to the Core VPC.

As a reminder, here’s what one of the stub VPCs route tables looks like. Stub VPC Route Table

  • Ping 4.2.2.2 because that’s just how we do it…
[ec2-user@ec2-2 ~]$ ping -c 3 4.2.2.2
PING 4.2.2.2 (4.2.2.2) 56(84) bytes of data.
64 bytes from 4.2.2.2: icmp_seq=1 ttl=49 time=3.79 ms
64 bytes from 4.2.2.2: icmp_seq=2 ttl=49 time=2.41 ms
64 bytes from 4.2.2.2: icmp_seq=3 ttl=49 time=2.29 ms

--- 4.2.2.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 2.292/2.836/3.798/0.683 ms
  • Ping google.com and test DNS resolution
[ec2-user@ec2-2 ~]$ ping -c 3 google.com
PING google.com (172.217.15.110) 56(84) bytes of data.
64 bytes from iad30s21-in-f14.1e100.net (172.217.15.110): icmp_seq=1 ttl=46 time=3.08 ms
64 bytes from iad30s21-in-f14.1e100.net (172.217.15.110): icmp_seq=2 ttl=46 time=1.36 ms
64 bytes from iad30s21-in-f14.1e100.net (172.217.15.110): icmp_seq=3 ttl=46 time=1.32 ms

--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 1.323/1.924/3.085/0.821 ms
  • CURL this post and test bi-directional TCP communication
[ec2-user@ec2-2 ~]$ curl -I -X GET https://yandy.io/2019/05/understanding-the-aws-transit-gateway/
HTTP/2 200
cache-control: public, max-age=0, must-revalidate
content-type: text/html; charset=UTF-8
date: Sat, 18 May 2019 01:25:16 GMT
etag: "181e5fdc7911e0caa1e71ed7a1dc4034-ssl"
strict-transport-security: max-age=31536000
age: 0
server: Netlify
x-nf-request-id: 4e332a70-7253-4515-b609-a6180d1c2085-60215040

Check that out, status code of 200 means all is good.

Summary

With a well placed terraform modules or cloudformation stacks and CI/CD pipelines, one can turn projects and business solutions into repeatable and well documented infrastructure. Routes can be added with a few lines of code, if VPCs get deleted things automatically clean up. It’s almost like re-inventing routing in a cloud-native way.

Networking in the cloud it’s at an infancy, it can be simple and confusing at the same time. Troubleshooting it can a nightmare due to lack of visibility. Make the APIs work for your benefit, create alarms, create logging, transfer those to monitoring tools. The AWS Transit Gateway is just another tool, it’s not a hammer, so don’t treat every problem like it’s a nail for it, but it’s an extremely useful and powerful construct.

It’s going to be a hybrid world, there’s no one tool for every job. Just educate yourself, research, find the right solution and concentrate on solving business problems not technical problems though sometimes they go hand in hand.