Protecting APIs with Access Policy Manager and custom iRules
The problem: Unprotected API - Vulnerable to Overload Without Rate-Limiting Enforcement Our customer in the B2B sector is encountering a challenge with their public API. Despite having implemented a custom method for generating long-lived API keys, they find themselves unable to enforce rate-limiting effectively. This absence of rate-limiting mechanisms poses significant challenges, potentially resulting in the overloading of their system due to excessive requests or the exploitation of their API by unauthorized users. Without proper rate-limiting controls in place, the customer faces risks to both the performance and security of their API infrastructure, necessitating a solution to mitigate these concerns and ensure the smooth operation of their services for their clients. Our customers wants to offer two tiers of service level agreements (SLAs) - gold and standard. Complicating matters further, the API key, integral to authentication, is transmitted via a custom HTTP header. The solution: BIG-IP APM and Custom iRules for Effective Rate-Limiting My solution involves leveraging the API Protection feature of BIG-IP APM in conjunction with a custom iRule. By utilizing this combination, our customer can effectively extract the API Keys from HTTP requests and enforce rate limiting on specific API endpoints. As for now they only want to enforce rate limiting on the POST endpoints. This approach empowers the customer to secure their API while efficiently managing and controlling access to critical endpoints, ensuring optimal performance and safeguarding against abuse or overload. With this iRule we can to extract the API key from the HTTP Requests and store it in a variable, that can later be used by the API Protection feature of the APM. API Keys and the associated SLA level are stored in a Data Groupof the typestring. # Enable (1) or disable (0) logging globally when RULE_INIT { set static::debug 1 } # Access and analyze the HTTP header data for SLA value when HTTP_REQUEST { set sla [class lookup [HTTP::header value apikey] dg_apikeys] if { $static::debug } {log local0. "Made it to HTTP_REQUEST event with SLA value $sla."} } # Evaluate SLA value during per-request access policy execution when ACCESS_PER_REQUEST_AGENT_EVENT { set id [ACCESS::perflow get perflow.irule_agent_id] if { $id eq "read-sla" } { if { $static::debug } {log local0. "Made it to iRule agent in perrequest policy with SLA value $sla."} ACCESS::perflow set perflow.custom "$sla" } } And this is how the Per Request Policy in the API Protection profile looks. It uses the value of the API Key (extracted with the help of the the iRule) and the Source IP of the client to enforce Rate Limiting on the POST endpoints, using two different SLAs. In the APM log you should see the following message, once the client exceeds his quota defined in the SLA. Apr 28 20:12:42 ltm-apm-16.mylab.local notice tmm[11094]: 01870075:5: (null):/Common: API request with weight (1) violated the quota in rate limiting config(/Common/demo_api_ratelimiting_auto_rate_limiting_standard). Apr 28 20:12:42 ltm-apm-16.mylab.local notice tmm[11094]: 0187008d:5: /Common/demo_api_ratelimiting_ap:Common:6600283561834577940: Execution of per request access policy (/Common/demo_api_ratelimiting_prp) done with ending type (Reject) Further reading: You can find a more detailed write-up on my GitHub page: https://github.com/webserverdude/f5_APM_API_Protection There you can find the Per Request Policy explained in all details. The Data Group with for the iRule. A demo API for testing my solution. A Postman Collection for working with my demo API.
Rate Limiting based on ACCESS TOKEN (OAuth 2.0)
Problem this snippet solves: When publishing web services, you need to implement some rate limiting functions to avoid abuses. There are plenty of ways to setup Rate limiting How to use this snippet: The code below setup a rate limiting based on the ACCESS TOKEN. The client will receive a response "429 Too much requests" after 1000 requests in a window of 300 seconds. The client can request its current status by doing a request to /rate_limit_status. He will then receive the following JSON message : { "x-rate-limit-limit": 1000, "x-rate-limit-remaining": 800, "x-rate-limit-reset": 100 } Code : when RULE_INIT { ### # rate limit options ### set static::request_limit 1000 set static::window_size 300 ### # define URI endpoints ### set static::status_uri "/rate_limit_status" } when HTTP_REQUEST { ### # initialize vars ### set access_token "" set client_ip "" ### # retrieve the access_token. It will be used as a mandatory key to evaluate rate limiting ### if { [HTTP::header exists Authorization] and [HTTP::header Authorization] contains "Bearer" } { set access_token [getfield [HTTP::header Authorization] " " 2] set client_ip [IP::client_addr] } if { !($access_token eq "") } { ### # provide client with rate limit status ### set key [sha1 $access_token] set count [table lookup -notouch $key] set time [table timeout -remaining $key] ### # Provide a status page to the client ### if { [HTTP::path] eq $static::status_uri and [HTTP::method] eq "GET" } { if { $count > 0 } { set x_rate_limit_limit "$static::request_limit" set x_rate_limit_remaining "[expr {$static::request_limit-$count}]" set x_rate_limit_reset "$time" } else { set x_rate_limit_limit "$static::request_limit" set x_rate_limit_remaining "$static::request_limit" set x_rate_limit_reset "$static::window_size" } HTTP::respond 200 content "{\"x-rate-limit-limit\": $x_rate_limit_limit,\"x-rate-limit-remaining\": $x_rate_limit_remaining,\"x-rate-limit-reset\": $x_rate_limit_reset}" noserver Content-Type "application/json" Connection Close event disable all } else { ### # Handle the case where a client reach the rate limit ### if { $count >= $static::request_limit } { set x_rate_limit_limit "$static::request_limit" set x_rate_limit_remaining "0" set x_rate_limit_reset "$time" HTTP::respond 429 content "{\"x-rate-limit-limit\": $x_rate_limit_limit,\"x-rate-limit-remaining\": $x_rate_limit_remaining,\"x-rate-limit-reset\": $x_rate_limit_reset}" noserver Content-Type "application/json" Connection Close event disable all } else { if { $count == 0 } { table add $key 1 $static::window_size $static::window_size } else { table incr -notouch $key } } } } } Tested this on version: 11.5
Bot and Request Limiting iRule
Problem this snippet solves: This iRule limits robots and what they can do. Furthermore, it restricts requests per second and blacklists a client that goes above the limit Note: Not CMP Compatible. Code : when RULE_INIT { #Define blacklist timeout set ::bl_timeout 30 #Define request per minute threshold set ::req_limit 5 #Expiration for tracking IPs set ::expiration_time 300 #Sets iRule Runlevel 0-log only 1 - Logging and Blocking set ::runlevel 1 } when HTTP_REQUEST { #Captures User-Agent header to check for known robots set ua [string tolower [HTTP::header User-Agent]] log local0. "User Agent: $ua" #Checks to see if the connection is a known robot or requests the robot.txt file if { ([matchclass $ua contains $::RUA]) or ([string tolower [HTTP::uri]] contains "robot.txt") } { set robot 1 log local0. "Robot Detected" } else { set robot 0 } #Defines client_ip variable with the address of the client set client_ip [IP::client_addr] log local0. "Client IP: $client_ip" #Robot logic if { $robot > 0 }{ set bl_check [session lookup uie blacklist_$client_ip] log local0. "Value of bl_check variable: $bl_check" set req_uri [string tolower [HTTP::uri]] log local0. "Request URI: $req_uri" #Checks to see if IP address is on blacklist if { $bl_check ne ""}{ log local0.warn "Request Blocked: $client_ipClient on Blacklist[HTTP::request]" if { $::runlevel > 0 }{ HTTP::respond 403 } } #Checks to see if Robot is allowed and sets restrictions. Default is no access switch -glob $ua { "*slurp*" - "*yahooseeker*" - "*googlebot*" - "*msnbot*" - "*teoma*" - "*voyager*" { if { [matchclass $req_uri starts_with $::robot_block] }{ log local0.warn "Request Blocked: $client_ipRequest Blocked. Robot not following Robot.txt[HTTP::request]" if { $::runlevel > 0 }{ HTTP::respond 403 } } else { pool dave_pool } } default { log local0.warn "Request Blocked: $client_ipRequest Blocked, Unauthoried Robot[HTTP::request]" if { $::runlevel > 0 }{ HTTP::respond 403 } } } } #Logic for non-robots. Checks to see if blacklisted set bl_check [session lookup uie blacklist_$client_ip] log local0. "Non-Robot bl_check: $bl_check" if { $bl_check ne "" }{ log local0.warn "Request Blocked: $client_ipClient on Blacklist[HTTP::request]" log local0.warn "Session Record: $bl_check" if { $::runlevel > 0 }{ HTTP::respond 403 } } set curr_time [clock seconds] set timekey starttime_$client_ip set reqkey reqcount_$client_ip set request_count [session lookup uie $reqkey] log local0. "Request Count: $request_count" #If user uses search their request count is reset if { [HTTP::uri] starts_with "/search" }{ session delete uie $reqkey } #Sets up new count for first time connections. If not a new connection, connection count is incremented and the iRule checks to #see if over the threshold if { $request_count eq "" } { log local0. "Request Count is 0" set request_count 1 session add uie $reqkey $request_count $::expiration_time log local0. "Current Time: $curr_time" log local0. "Timekey Value: $timekey" log local0. "Reqkey value: $reqkey" session add uie $timekey [expr {$curr_time - 2}] [expr {$::expiration_time + 2}] log local0. "Request Count is now: $request_count" } else { set start_time [session lookup uie $timekey] log local0. "Start Time: $start_time" log local0. "Request Count (beyond first request): $request_count" incr request_count session add uie $reqkey $request_count $::expiration_time set elapsed_time [expr {$curr_time - $start_time}] log local0. "Elapsed Time: $elapsed_time" if {$elapsed_time < 60} { set elapsed_time 60 } set curr_rate [expr {$request_count / ($elapsed_time/60)}] log local0. "Current Rate of Request for $client_ip: $curr_rate" if {$curr_rate > $::req_limit}{ log local0.warn "Request Blocked: $client_ipClient over Threshold. Added to Blacklist[HTTP::request]" if { $::runlevel > 0 }{ session add uie blacklist_$client_ip $::bl_timeout HTTP::respond 403 } } } } Tested this on version: 9.0
HTTP Rate Limiting - CMP Compatible
Problem this snippet solves: This iRule is a CMP compatible Request per Second Throttling iRule. Code : when HTTP_REQUEST { set debug 1 set expiration_time 300 set client_ip [IP::client_addr] set req_limit 200 set curr_time [clock seconds] set timekey starttime set reqkey reqcount set request_count [session lookup uie $reqkey] if {$debug}{log local0. "Request Count: $request_count"} #Sets up new count for first time connections. If not a new connection, connection count is incremented and the iRule checks to #see if over the threshold if { $request_count eq "" } { if {$debug}{log local0. "Request Count is 0"} set request_count 1 session add uie $reqkey $request_count $expiration_time if {$debug}{log local0. "Current Time: $curr_time"} if {$debug}{log local0. "Timekey Value: $timekey"} if {$debug}{log local0. "Reqkey value: $reqkey"} session add uie $timekey [expr {$curr_time - 2}] [expr {$expiration_time + 2}] if {$debug}{log local0. "Request Count is now: $request_count"} } else { set start_time [session lookup uie $timekey] if {$debug}{log local0. "Start Time: $start_time"} if {$debug}{log local0. "Request Count (beyond first request): $request_count"} incr request_count session add uie $reqkey $request_count $expiration_time set elapsed_time [expr {$curr_time - $start_time}] if {$debug}{log local0. "Elapsed Time: $elapsed_time"} if {$elapsed_time < 60} { set elapsed_time 60 } set curr_rate [expr {$request_count / ($elapsed_time/60)}] if {$debug}{log local0. "Current Rate of Request: $curr_rate"} if {$curr_rate > $req_limit}{ if {$debug}{log local0.warn "Request Blocked: $client_ipClient over Threshold.[HTTP::request]"} HTTP::redirect http://www.company.com/sorry.html } } }
