Deploy BIG-IP in AWS with HA across AZ’s - without using EIP’s
Background: The CloudFormation templates that are provided and supported by F5 are an excellent resource for customers to deploy BIG-IP VE in AWS. Along with these templates, documentation guiding your F5 deployment in AWS is an excellent resource. And of course, DevCentral articles are helpful. I recommend reading about HA topologies in AWS to start. I hope my article today can shed more light on an architecture that will suit a specific set of requirements: No Elastic IP's (EIP’s), High Availability (HA)across AZ’s, and multiple VPC’s. Requirements behind this architecture choice: I recently had a requirement to deploy BIG-IP appliances in AWS across AZ’s. I had read the official deployment guide, but I wasn’t clear on how to achieve failover without EIP’s. I was given 3 requirements: HA across AZ’s. In this architecture, we required a pair of BIG-IP devices in Active/Standby, where each device was in a different AZ. I needed to be able to fail over between devices. No EIP’s. This requirement existed because a 3 rd party firewall was already exposed to the Internet with a public IP address. That firewall would forward inbound requests to the BIG-IP VE in AWS, which in turn proxied traffic to a pair of web servers. Therefore, there was no reason to associate an EIP (a public IP address) with the BIG-IP interface. In my demo below I have not exposed a public website through a 3 rd party firewall, but to do so is a simple addition to this demo environment. Multiple VPC’s. This architecture had to allow for multiple VPC’s in AWS. There was already a “Security VPC” which contained firewalls, BIG-IP devices, and other devices, and I had to be able to use these devices for protecting applications that were across 1 or more disparate VPC’s. Meeting the requirements: HA across AZ’s This is the easiest of the requirements to meet because F5 has already provided templates to deploy these in AWS. I personally used a 2-nic device, with a BYOL license, deployed to an existing VPC, so that meant my template was this one. After this deployment is complete, you should have a pair of devices that will sync their configuration. At time of failover The supported F5 templates will deploy with the Advanced HA iApp. It is important that you configure this iApp after you have completed your AWS deployments. The iApp uses IAM permissions deployed with the template to make API calls to AWS at the time of failover. The API calls will update the route tables that you specify within the iApp wizard. Because this iApp is installed on both devices, either device can update the route in your route tables to point to its own interface. Update as of Dec 2019 This article was first written Feb 2019, and in Dec 2019 F5 released the Cloud Failover Extension (CFE), which is a cloud-agnostic, declarative way to configure failover in multiple public clouds. You can use the CFE instead of the Advanced HA iApp to achieve high availability between BIG-IP devices in cloud. Update as of Apr 2020 Your API calls will typically be sent to the public Internet endpoints for AWS EC2 API calls. Optionally, you can use AWS VPC endpoints to keep your API calls out to AWS EC2 from traversing the public Internet. My colleague Arnulfo Hernandez has written an article explaining how to do this. No EIP’s Configure an “alien IP range” I’m recycling another DevCentral solution here. You will need to choose an IP range for your VIP network that does not fall within the CIDR that is assigned to your VPC. Let’s call it an “alien range” because it “doesn’t belong” in your VPC and you couldn’t assign IP addresses from this range to your AWS ENI’s. Despite that, now create a route table within AWS that points this “alien range” to your Active BIG-IP device’s ENI (if you’re using a 2+ nic device, point it to the correct data plane NIC, not the Mgmt. interface). Don’t forget to associate the route table with specific subnets, per your design. Alternatively, you could add this route to the default VPC route table. Create a VIP on your active device Now create a VIP on your active device and configure the IP address as an IP within your alien range. Ensure the config replicates to your standby device. Ensure that source/destination checking is disabled on the ENI’s that your AWS routes are pointing to (on both Standby and Active devices). You should now have a VIP that you can target from other hosts in your VPC, provided that the route you created above is applied to the traffic destined to the VIP. Multiple VPC’s For extra credit, we’ll set up a Transit Gateway. This will allow other VPCs to route traffic to this “alien range” also, provided that the appropriate routes exist in the remote VPC’s, and also are applied to the appropriate Transit Gateway Route Table. Again, I’m recycling ideas that have already been laid out in other DevCentral posts. I won’t re-hash how to set up a transit gateway in AWS, because you can follow the linked post above. Sufficed to say, this is what you will need to set up if you want to route between multiple VPC’s using a transit gateway: 2 or more VPC’s A transit gateway in AWS Multiple transit gateway attachments that attach the transit gateway and each VPC you would like to route between. You will need one attachment per VPC. A transit gateway route table that is associated with each attachment. I will point out that you need to add a route for your “alien range” in your transit gateway route table, and in your remote VPC’s. That way, hosts in your remote VPC’s will forward traffic destined to your alien range (VIP network) to the transit gateway, and the transit gateway will forward it to your VPC, and the route you created in Step A will forward that traffic to your active BIG-IP device. Completed design: After the above configuration, you should have an environment that looks like the diagram below: Tips Internet access for deployments: When you deploy your BIG-IP devices, they will need Internet access to pull down some resources, including the iApp. So if you are deploying devices into your existing VPC, make sure you have a reachable Internet Gateway in AWS so that the devices have Internet access through both their management interface, and their data plane interface(s). Internet access for failover: Remember that an API call to AWS will still use an outbound request to the Internet. Make sure you allow the BIG-IP devices to make outbound connections over HTTPS to the Internet. If this is not available, you will find that your route tables are not updated at time of failover. (If you have a hard requirement that your devices should not have outbound access to internet, you can follow Arnulfo's guide linked to above and use VPC endpoints to keep this traffic on your local VPC) iApp logs: you can enable this in the iApp settings. I have used the logs (in /var/ltm/log) to troubleshoot issues that I have created myself. That’s how I learned not to accidentally cut off Internet access for your devices! Don’t forget about return routes if SNAT is disabled: Just like on-prem environments, if you disable SNAT, the pool member will need to route return traffic back to the BIG-IP device. You will commonly set up a default route (0.0.0.0/0) in AWS, point it at the ENI of the active BIG-IP device, and associate this route table with the subnet containing the pool members. If the pool members are in a remote VPC, you will need to create this route on the transit gw route table also. Don’t accidentally cut off internet access: When you configure the default route of 0.0.0.0/0 to point to eth1 of the BIG-IP device, don’t apply this route to every subnet in your Security VPC. It may be easy to do so accidentally, but remember that it could cause the API calls to update route tables to fail when the Standby device becomes Active. Don’t forget to disable source/dest check on your ENI’s. This is configured by the template, but if you have other devices that require it, remember to check this setting.A Primer On F5 AWS CloudFormation Templates
If there was a magic button for application deployment we'd all use it. For those of us in the real world we use templates. AWS CloudFormation Templates (CFT) serve as a guide for creating and deploying resources into your ecosystem. Please key in on the word template. As with any other guide, you provide the additional details to graduate a template into a functional and living configuration. F5 provides several supported CFTs to accommodate BIG-IP and BIG-IQ deployment scenarios. Multiple interfaces, disaster recovery options, and VPC design flexibilities provide a lot of options to digest. And with options comes questions and potential complexity. We'll discuss the CFTs offered and how they can add benefit without extending complexity to you and your team. On Choosing Templates It seems initially easier to take an on-premises design and apply it to your cloud migration strategy. Don't. This is the time to reevaluate architectural decisions made over the years and it's also the time to evaluate how each cloud provider offers infrastructure. AWS provides a familiar network-centric model compartmentalized within virtual private networks (VPC). Taking advantage of these cloud methodologies may sway architectural decisions a little. You may even find starting over is quicker than shoe-horning in legacy designs. Regardless of your application and architecture needs, we probably have a templateto help along your deployments. F5 AWS CloudFormation Templates are available at F5's Github page and are grouped by availability model, followed by NIC count, and lastly licensing model. The master readme file within the AWS CloudFormation Template repository provides ample documentation to help you chose. We have two preliminary CFT branches for the adventurous; supported and experimental. Supported - If you're new to AWS CloudFormation Templates, start here. With a bit of tweaking you can get BIG-IP up and running in no time. You'll supply your environment variables or you can use the templates to walk through the EC2 options during BIG-IP deployment. It's your call. They're easy to integrate into your architecture designs and you can contact support with questions. Win win. Experimental - There's no support and they're work's in progress but for the hardened CloudFormation Alchemist, you can turn our works-in-progress into your gold. These are for people curious what were working on and ok experimenting themselves with different deployment options. On Availability Models Understanding how each cloud provider offers redundant infrastructure should be reflected in your cloud design. Amazon AWS offers availability zones within a specific regions to provide redundancy. In an AWS Virtual Private Cloud (VPC), a subnet may only resides within a single availability zone so if you need hardware and site redundancy you must to configure multiple subnets deployed against different availability zones. F5 provides several availability models depending on what your needs. Our AWS CFTs provide guidance for: Standalone - In relation to BIG-IP in a disaster recovery environment, Standalone denotes running BIG-IP with no secondary device to receive traffic in the event of a system, EC2 virtual machine, or availability zone failure. System outages or network issues would create a potential service disruption. But you play fast and loose so you're not worried right...? Failover Models Layer 2 access isn't available in AWS (or any major cloud provider technically) so we cannot rely on our good ol' gratuitous ARP for failover communication. Instead we rely on AWS API calls to migrate the public elastic IP across private IPs;between active and standby BIG-IP instances. Using API services instead of L2does create multi-second delays for traffic failover but most cloud applications are much more session resilient than their on-premises counterparts. There are two primary types of active-passive failover models. Same-Net - BIG-IP is deployed within the same subnet. This offers redundancy for the BIG-IP systems from any software issues or AMI instance failures. Because both BIG-IPs are deployed on the same subnet, you're in the same availability zone and still at risk if the zone experiences a failure. Across-Net - BIG-IP is deployed across subnets and thusly can be deployed across multiple availability zones. This provides BIG-IP redundancy in the event of an availability zone failure. Providing failover redundancy for BIG-IP devices requires two or more interfaces. Single interface config sync across availability zones provides configuration changes across devices but does not facilitate the API-based traffic failover in the event of a failure. For more information on this read Amazon Web Services: Single NIC config sync across Availability Zones. Autoscale BIG-IP can autoscale within AWS to mirror application traffic fluctuations. F5 autoscales viaBIG-IQ (using Per-App VE), via BIG-IP DNS, or by using AWS Elastic Load Balancer (ELB). Since we're talking about templates, we'll focus on the CFTs offering Autoscale via BIG-IP DNS and via ELB. Via DNS - Autoscaling BIG-IP using DNS is done with BIG-IP DNS (formerly BIG-IP GTM). BIG-IP DNS can talk natively to globally deployed BIG-IP instances through an XML protocol called iQuery to provide multiple traffic failover methodologies going beyond basic connection thresholds or DNS round robin load balancing. For this to work, the BIG-IP management interfaces must be able to communicate over publicly routable networks. This connection can be secured with firewall/VPC ACLs and VPC security groups. Via ELB - Using Amazon's native Elastic Load Balancer configuration, BIG-IP's are deployed into a AWS Auto Scaling Group. As BIG-IP's are dynamically deployed, ELB services discover them and add them to the pool. You'll need to use hourly licensing for BIG-IP in this scenario as you'll not be able to manually license automaticallyspun up and torn down BIG-IPs. Each BIG-IP is deployed as a standalone system responsible only for it's own traffic and does not communicate with other BIG-IP systems. If an instance is taken offline, AWS ELB services will redistribute the connections to other online BIG-IP defined within your ELB configuration. I know it sounds weird to load balance "load balancers" but if you're only using BIG-IP for basic L3 load balancing in AWS you're missing out on most of BIG-IP'sL3 through L7 traffic and security services. When deployed with ELB services, you're complimenting BIG-IP feature sets, not taking away. This goes back to being creative with your cloud architecture. On Interfaces And AMIs AWS CloudFormation Templates help reduce the need to "future proof" your architecture. It simplifies managing multiple designs because one size does not fit all. That'll becomes very apparent when determining what Amazon Machine Images (AMI) to use when deploying BIG-IP. The maximum number of interfaces a single BIG-IP can use is determined by the AMI selected during deployment. This is where using templates and later discussed cloud-init configs can help reduce the stress if you run into an AMI limitation. A great independent web site to help you evaluate Amazon's current AMI offerings is EC2Instances.info. Amazon's always adding new (and sometimes deprecating old) instances to meet the ever growing needs of it's end users so it's good to stay current. Example - The m4.large is a good 2vCPU 8GB RAM AMI. And while it allows for up to 20 IPs to be assigned, it only allows 2 interfaces. This is fine for many systems but you may have a design where BIG-IP needs 3 or more interfaces. That means you're going to need the m5.large which has 3 available interfaces, better networking performance, and a slightly higher hourly cost. This example illustrates the need to pay attention when chosing instances to define in your templates but then again, templates make it easy to update your designs. On Cloud-Init CloudFormation templates complete the tasks needed to deploy EC2 resources BIG-IP will use but there's still more configuration to do in order to successfully pass traffic. In Alex Applebaum's F5 in AWS Part 5 - Cloud-init, Single-NIC, and Auto Scale Out in BIG-IP he discusses how Cloud-Init can provision configurations within anoperating system once the CFT templates have completed their job. Introduced in BIG-IP virtual edition v13, Cloud-Init provides youa way you to build and complete the bootstrapping process to get BIG-IP up and running without user interaction. Check out Alex's article! On What's Next This article is an primer. It's here to get you familiar with the templates we offer for Amazon Web Services. Coming soon we'll dive into one of the more complicated templates and dissect it down so you can understand not only how they work but maybe give you an idea of how you can use them better. A lot of people still manually configure their AWS VPC's and while that's perfectly fine.There are other ways, and maybe better ways, to complete the same tasks. Lastly, what do you want to learn about? If there are any topics that may interest you, let us know and we'll see if we can either write an article, create a lightboard lesson, or start a discussion in Q&A. Thanks for reading and happy admining. Additional Resources: Get Started with BIG-IP VE In AWS F5 Github CloudFormation Template Home F5 In AWS Part 1 : AWS Networking Basics F5 In AWS Part 2: Running BIG-IP in an EC2 Virtual Private Cloud F5 In AWS Part 3: Advanced Topologies and More on Highly-Available Services F5 In AWS Part 4: Orchestrating BIG-IP Application Services with Open-Source Tools F5 In AWS Part 5: Cloud-init, Single-NIC, and Auto Scale Out of BIG-IP in v12
