Solving for true-source IP with global load balancers in Google Cloud

Background

Recently a customer approached us with requirements that may seem contradictory: true source IP persistence, global distribution of load balancers (LB's), TCP-only proxying, and alias IP ranges in Google Cloud. With the help of PROXY protocol support, we offered a straightforward solution that is worth documenting for others.

Customer requirements

• We have NGINX WAF running on VM instances in Google Cloud Platform (GCP)
• We want to expose these to the Internet with a global load balancer
• We must know the true source IP of clients when traffic reaches our WAF
• We do not want so use an application (HTTP/S) load balancer in Google.
  • i.e., we do not want to perform TLS decryption with GCP or use HTTP/HTTPS load balancing
  • therefore, we cannot use X-Forwarded-For headers to preserve true source IP
• Additionally, we'd like to use Cloud Armor. How can we add on a CDN/DDoS/etc provider if needed?

Let's solve for these requirements by finding the load balancer to use, and then how to preserve and use true source IP.

Which load balancer type fits best?

This guide outlines our options for Google LB’s. Because our requirements include global, TCP-only load balancing, we will choose the highlighted LB type of “Global external proxy Network Load Balancer”.

Requirements of global load balancer, and TCP-only, determined this load balancer type.

Proxy vs Passthrough

Notice that global LB’s proxy traffic. They do not preserve source IP address as a passthrough LB does. Global IP addresses are advertised from multiple, globally-distributed front end locations, using Anycast IP routing.

Proxying from these locations allows traffic symmetry, but Source NAT causes loss of the original client IP address. I've added some comments into a Google diagram below to show our core problem here:

PROXY protocol support with Google load balancers

Google’s TCP LB documentation outlines our challenge and solution:

"By default, the target proxy does not preserve the original client IP address and port information. You can preserve this information by enabling the PROXY protocol on the target proxy."

Without PROXY protocol support, we could only meet 2 out of 3 core requirements with any given load balancer type. PROXY protocol allows us to meet all 3 simultaneously.

You can meet these 3 requirements simultaneously if you use PROXY protocol

Setting up our environment in Google

The script below configures a global TCP proxy network load balancer and associated objects. It is assumed that a VPC network, subnet, and VM instances exist already.

This script assumes the VM’s are F5 BIG-IP devices, although our demo will use Ubuntu VM’s with NGINX installed. Both BIG-IP and NGINX can easily receive and parse PROXY protocol.

# GCP Environment Setup Guide for Global TCP Proxy LB with Proxy Protocol. Credit to Tony Marfil, @tmarfil

# Step 1: Prerequisites
# Before creating the network endpoint group, ensure the following GCP resources are already configured:
# 
# -A VPC network named my-vpc.
# -A subnet within this network named outside.
# -Instances ubuntu1 and ubuntu2 should have alias IP addresses configured: 10.1.2.16 and 10.1.2.17, respectively, both using port 80 and 443.
# 
# Now, create a network endpoint group f5-neg1 in the us-east4-c zone with the default port 443.

gcloud compute network-endpoint-groups create f5-neg1 \
--zone=us-east4-c \
--network=my-vpc \
--subnet=outside \
--default-port=443

# Step 2: Update the Network Endpoint Group
# 
# Add two instances with specified IPs to the f5-neg1 group.

gcloud compute network-endpoint-groups update f5-neg1 \
--zone=us-east4-c \
--add-endpoint 'instance=ubuntu1,ip=10.1.2.16,port=443' \
--add-endpoint 'instance=ubuntu2,ip=10.1.2.17,port=443'

# Step 3: Create a Health Check
# 
# Set up an HTTP health check f5-healthcheck1 that uses the serving port.

gcloud compute health-checks create http f5-healthcheck1 \
--use-serving-port

# Step 4: Create a Backend Service
# 
# Configure a global backend service f5-backendservice1 with TCP protocol and attach the earlier health check.

gcloud compute backend-services create f5-backendservice1 \
--global \
--health-checks=f5-healthcheck1 \
--protocol=TCP 

# Step 5: Add Backend to the Backend Service
# 
# Link the network endpoint group f5-neg1 to the backend service.

gcloud compute backend-services add-backend f5-backendservice1 \
    --global \
    --network-endpoint-group=f5-neg1 \
    --network-endpoint-group-zone=us-east4-c \
    --balancing-mode=CONNECTION \
    --max-connections=1000

# Step 6: Create a Target TCP Proxy
# 
# Create a global target TCP proxy f5-tcpproxy1 to handle routing to f5-backendservice1.

gcloud compute target-tcp-proxies create f5-tcpproxy1 \
--backend-service=f5-backendservice1 \
--proxy-header=PROXY_V1 \
--global

# Step 7: Create a Forwarding Rule
# 
# Establish global forwarding rules for TCP traffic on port 80 & 443.

gcloud compute forwarding-rules create f5-tcp-forwardingrule1 \
--ip-protocol TCP \
--ports=80 \
--global \
--target-tcp-proxy=f5-tcpproxy1

gcloud compute forwarding-rules create f5-tcp-forwardingrule2 \
--ip-protocol TCP \
--ports=443 \
--global \
--target-tcp-proxy=f5-tcpproxy1

# Step 8: Create a Firewall Rule
# 
# Allow ingress traffic on specific ports for health checks with the rule allow-lb-health-checks.

gcloud compute firewall-rules create allow-lb-health-checks \
    --direction=INGRESS \
    --priority=1000 \
    --network=my-vpc \
    --action=ALLOW \
    --rules=tcp:80,tcp:443,tcp:8080,icmp \
    --source-ranges=35.191.0.0/16,130.211.0.0/22 \
    --target-tags=allow-health-checks

# Step 9: Add Tags to Instances
# 
# Tag instances ubuntu1 and ubuntu2 to include them in health checks.

gcloud compute instances add-tags ubuntu1 --tags=allow-health-checks --zone=us-east4-c
gcloud compute instances add-tags ubuntu2 --tags=allow-health-checks --zone=us-east4-c


## TO PULL THIS ALL DOWN: (uncomment the lines below)
# gcloud compute firewall-rules delete allow-lb-health-checks --quiet
# gcloud compute forwarding-rules delete f5-tcp-forwardingrule1 --global --quiet
# gcloud compute forwarding-rules delete f5-tcp-forwardingrule2 --global --quiet
# gcloud compute target-tcp-proxies delete f5-tcpproxy1 --global --quiet
# gcloud compute backend-services delete f5-backendservice1 --global --quiet
# gcloud compute health-checks delete f5-healthcheck1 --quiet
# gcloud compute network-endpoint-groups delete f5-neg1 --zone=us-east4-c --quiet
#
# Then delete your VM's and VPC network if desired.

Receiving PROXY protocol using NGINX

We now have 2x Ubuntu VM's running in GCP that will receive traffic when we target our global TCP proxy LB's IP address.

Let’s use NGINX to receive and parse the PROXY protocol traffic. When proxying and "stripping" the PROXY protocol headers from traffic, NGINX can append an additional header containing the value of the source IP obtained from PROXY protocol:

server {
  listen 80 proxy_protocol; # tell NGINX to expect traffic with PROXY protocol
  server_name customer1.my-f5.com;

  location / {
    proxy_pass http://localhost:3000;
    proxy_http_version 1.1;
    proxy_set_header x-nginx-ip $server_addr; # append a header to pass the IP address of the NGINX server
    proxy_set_header x-proxy-protocol-source-ip $proxy_protocol_addr; # append a header to pass the src IP address obtained from PROXY protocol
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr; # append a header to pass the src IP of the connection between Google's front end LB and NGINX
    proxy_cache_bypass $http_upgrade;
  }
}

Displaying true source IP in our web app

You might notice above that NGINX is proxying to http://localhost:3000. I have a simple NodeJS app to display a page with HTTP headers:

const express = require('express');
const app = express();
const port = 3000;

// set the view engine to ejs
app.set('view engine', 'ejs');

app.get('/', (req, res) => {
        const proxy_protocol_addr = req.headers['x-proxy-protocol-source-ip'];
        const source_ip_addr = req.headers['x-real-ip'];
        const array_headers = JSON.stringify(req.headers, null, 2);
        const nginx_ip_addr = req.headers['x-nginx-ip'];
    res.render('index', {
        proxy_protocol_addr: proxy_protocol_addr,
        source_ip_addr: source_ip_addr,
        array_headers: array_headers,
        nginx_ip_addr: nginx_ip_addr
    });
})
app.listen(port, () => {
  console.log('Server is listenting on port 3000');
})

For completeness, NodeJS is using the EJS template engine to build our page. The file views/index.ejs is here:

<!DOCTYPE html>
<html lang="en">
<head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width, initial-scale-1">
        <title>Demo App</title>
</head>
<body class="container">
<main>
        <h2>Hello World!</h2>
        <p>True source IP (the value of <code>$proxy_protocol_addr</code>) is <b><%= typeof proxy_protocol_addr != 'undefined' ? proxy_protocol_addr : '' %></b></p>
        <p>IP address that NGINX recieved the connection from (the value of <code>$remote_addr</code>) is <b><%= typeof source_ip_addr != 'undefined' ? source_ip_addr : '' %> </b></p>
        <p>IP address that NGINX is running on (the value of <code>$server_addr</code>) is <b><%= typeof nginx_ip_addr != 'undefined' ? nginx_ip_addr : '' %></b><p>
        <h3>Request Headers at the app:</h3>
        <pre><%= typeof array_headers != 'undefined' ? array_headers : '' %></pre>
</main>
</body>
</html>

 

Cloud Armor

Cloud Armor is an easy add-on when using Google load balancers. If required, an admin can:

1. Create a Cloud Armor security policy
2. Add rules (for example, rate limiting) to this policy
3. Attach the policy to a TCP load balancer
4. In this way “edge protection” is applied to your Google workloads with little effort.

Our end result

This small demo app shows that true source IP can be known to an application running on Google VM’s when using the Global TCP Network Load Balancer. We’ve achieved this using PROXY protocol and NGINX. We’ve used NodeJS to display a web page with proxied header values.

Simple demo app showing source IP obtained from parsing PROXY protocol

Thanks for reading. Please reach out with any questions!