HTTP Brute Force Mitigation Playbook: Bad Actor Behavior and Gathering Statistics using BIG-IP LTM Policies, iRules and BIG-IP AVR - Chapter 2
Gathering Statistics and Bad Actor Behavior In this Chapter of the HTTP Brute Force Mitigation Playbook series, we will review BIG-IP Local Traffic Manager (LTM) and Application Visibility and Reporting (AVR) modules' features to show how it can used to gather statistics and how we can use these statistics as a base line of what an organization may consider as normal traffic and malicious traffic and how it relates to HTTP Brute Force Attacks. Normal Traffic and Malicious Traffic Definition Normally, we consider normal traffic as client activities associated to an application that is expected and does not affect availability and performance. Malicious traffic, on the other hand, are traffic bound to hit any public facing application or device with the goal of impacting availability, performance and its security. We will focus on malicious traffic related to HTTP Brute Force Attacks. Bad Actor Behavior In the case of HTTP Brute Force Attacks, it is common to observe these considered Bad Actor Behavior from malicious traffic: A spoofed HTTP User-Agent header Traffic from unexpected Geolocation(s) Excessive traffic from a certain source IP Multiple Login attempts on a web application from the same or distributed source IP using different credentials HTTP requests to an application that iterates thru a HTTP parameter's value either from a single or distributed traffic source The malicious traffic could be sent slowly and in low volume in an attempt to be more stealthy and applying on or more of the Bad Actor Behavior combinations or also be sent thru high volume traffic and will have noticeable impact such as high CPU usage on the application server Prerequisites Managing the BIGIP configuration requires Administrator access. Ensure access to Configuration Utility (Web GUI) and SSH is available. These management interfaces will be helpful in configuring, verifying and troubleshooting on the BIGIP.Having access to serial console output of the BIGIP is also helpful. When working with F5 Support, having an F5 Support/AskF5 account will be helpful.To access your AskF5 account, access the following link: https://support.f5.com/csp/my-support/home Another helpful site is F5 ihealth, where qkviews extracted from the BIGIP device can be uploaded and analysed. Logs, statistics, heuristics, graphs and BIGIP configuration extracted thru a qkview are available in ihealth when uploaded and will aid in diagnosing and troubleshooting efforts. https://ihealth.f5.com/ Register an account to these F5 Sites as these will helpful in working with F5 Support.When troubleshooting, generating a qkview or extracting bigip logs or viewing configuration in the GUI require access to the BIGIP and having the administrative access to it and referenced F5 Support sites will be helpful. Local Traffic Manager (LTM) and Application Visibility and Reporting (AVR) license are required to use the related features. LTM and AVR modules should be provisioned to access related configuration discussed in this chapter. Using Local Traffic Manager (LTM) features to gather HTTP request data Local Traffic Manager (LTM)features such as iRules and LTM Policy can be used to log messages generated from a certain stage of the HTTP request. In the case of a HTTP brute force attack, inspecting the source IP, HTTP headers, the HTTP request payload can provide information on what malicious traffic may look like. Required License and Provisioning BIG-IP LTM module should licensed and provisioned for LTM features to be available. A "harmless" HTTP request The following "curl" command will receive a proper response from a web site it is requesting content from. curl --silent --output /dev/null -lvk https://172.16.8.84 -H "User-Agent: im-a-web-browser" The web server does not validate whether this curl request is coming from a legitimate web browser as a normal user would use.the curl "-H" option allows a HTTP header to be included in the curl HTTP request, in this case "User-Agent: im-a-web-browser" will be the HTTP User-Agentand value that the web server will receive. A legitimate HTTP User-Agent value can also be used for this option and the web server will simply accept and process the request without validating whether the request is from a real web browser or an automated client . Using Local Traffic Manager (LTM) Policy to log HTTP request data A Local Traffic Manager (LTM) Policy can be applied to a Virtual Server and match a traffic pattern and execute defined actions. LTM Policy configuration is straightforward and is able to log essential data, such as in the case of a HTTP Brute Force attack, gather source IP, HTTP headers and HTTP payload. It can log locally or remotely. Here is a sample LTM Policy configuration where it will log the Source client IP address and port, HTTP header names and its interesting values and the HTTP Payload for all traffic at HTTP request time. From the BIG-IP Configuration Utility: Local Traffic››Policies : Policy List››/Common/log_http_traffic:log_http_request_header_payload log_http_traffic is the name of the LTM Policy log_http_request_header_payload is a rule in the log_http_traffic LTM Policy This view is only in Draft state of the LTM Policy These are configured in the message section of the "Log" action lines. notice the second Log action specifically logs the User-Agent and X-Forwarded-For HTTP headers. Common in HTTP Brute Force Attacks, the HTTP User-Agent header's value may be a slightly modified compared to a legitimate User-Agent value - imitating a legitimate web browser. Similarly, the X-Forwarded-For value may provide some hints where the attack traffic may really be coming from. In scenarios where the attacker may be passing thru web proxies, reviewing the X-Forwarded-For header value can be helpful in determining possible thru source of the attack traffic. Log action 1: tcl: client [IP::client_addr]:[TCP::client_port] -> URL: [HTTP::host][HTTP::uri]: http headers are [HTTP::header names] and the http payload is [HTTP::payload] Log action 2: tcl: client [IP::client_addr]:[TCP::client_port] -> URL: [HTTP::host][HTTP::uri]: interesting HTTP headers are User-Agent:[HTTP::header User-Agent] X-Forwarded-For [HTTP::header X-Forwarded-For] Notice that the Log action references "tcl". LTM Policy actions accepts tcl expression and is commonly used in iRules. iRule commands such as the[HTTP::header] can be used in LTM policies. Important: The iRule commands used in this sample LTM policy are for demonstration purpose only. Complex LTM Policies with iRule commands should be properly tested. F5 Professional Services is the best resource for this type of implementation. log Facility local0 refers to the /var/log/ltm and thus, generated logs will be in /var/log/ltm for this LTM Policy. other log files can be also specified and remote logging is also possible. Here is the Published LTM Policy The sample LTM Policy is then applied to a Virtual Server where malicious traffic is suspected to be hitting Here is the text form of the Published LTM Policy. log_http_request_header_payload 1. Log message 'tcl: client [IP::client_addr]:[TCP::client_port] -> URL: [HTTP::host][HTTP::uri]: http headers are [HTTP::header names] and the http payload is [HTTP::payload]' at request time. 2. Log message 'tcl:client [IP::client_addr]:[TCP::client_port] -> URL: [HTTP::host][HTTP::uri]: interesting HTTP headers are User-Agent:[HTTP::header User-Agent] X-Forwarded-For:[HTTP::header X-Forwarded-For]' at request time. log generated in /var/log/ltm Nov 29 15:24:22 BIG-IP info tmm7[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.7.31:39063 -> URL: 172.16.8.86/: http headers are Host Accept User-Agent X-Forwarded-For Content-Length Content-Type and the http payload is admin:admin Nov 29 15:24:22 BIG-IP info tmm7[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.7.31:39063 -> URL: 172.16.8.86/: interesting HTTP headers are User-Agent:im-a-web-browser X-Forwarded-For:88.88.88.88 Using grep, awk and uniq in the BIG-IP bash prompt, /var/log/ltm can be inspected to check the how frequent a source client IP sent a HTTP request and along with it, HTTP header and Payload information, to the Virtual Server where the LTM Policy is applied. Here, the sample grep/awk/uniq command will count the number of times the client source IP was observed from the sample LTM Policy generated log /var/log/ltm. In the sample LTM Policy, it logs the client source IP twice, as such the sample grep/awk/uniq command was written to take account of this. grep/awk/uniq sample command 1: grep client /var/log/ltm | awk -F " " '{print $9}' | uniq -c | awk -F ":" '{print $1}' | awk -F " " '{print $2}' | uniq -c sample output: root@BIG-IP:Active:Standalone] config # grep client /var/log/ltm | awk -F " " '{print $9}' | uniq -c | awk -F ":" '{print $1}' | awk -F " " '{print $2}' | uniq -c 9 172.16.7.31 grep/awk/uniq sample command 2: grep interesting /var/log/ltm | awk -F " " '{print $18}' | awk -F ":" '{print $2}' | uniq -c This sample command will count the number of time the X-Forwarded-For HTTP Header appeared in the log. the word 'interesting' was initially filtered as it is the line where theX-Forwarded-For information was logged. sample output: root@BIG-IP:Active:Standalone] config # grep interesting /var/log/ltm | awk -F " " '{print $18}' | awk -F ":" '{print $2}' | uniq -c 20 88.88.88.88 The sample grep/awk/uniq command are just simple demonstration commands only. You can build your own scripts, version of the command to get the desired information.The sample output are also for demonstration purpose only. In a real HTTP brute force attack, The sample LTM Policy to log the client source IP and HTTP information in the request will be helpful in describing the possibly malicious traffic. Using F5 iRules to gather HTTP request data From the F5 iRule Home Page, https://clouddocs.f5.com/api/irules/: F5 iRules is a powerful and flexible feature within the BIG-IP® local traffic management (LTM) system that you can use to manage your network traffic. The iRulesTM feature not only allows you to select pools based on header data, but also allows you to direct traffic by searching on any type of content data that you define. Thus, the iRules feature significantly enhances your ability to customize your content switching to suit your exact needs. iRules can also be used to log source IP and HTTP information. In comparison to LTM Policy, iRules are much more flexible and can also be very complex. Depending on how much information you would like to gather during a HTTP Brute Force attack, iRules can be very verbose. iRules are also resource intensive, the more it gets executed as traffic hits a Virtual Server where it is applied, it will consume additional CPU resource as well. If the same information can be gathered using LTM Policy feature, use it instead of iRules. For simplicity we will gather the same HTTP request data as demonstrated in the LTM Policy section. Important: The iRule commands used in the sample iRules are for demonstration purpose only. Complex iRules should be properly tested. F5 Professional Services is the best resource for this type of implementation. In the Configuration Utility,Local Traffic››iRules : iRule List›› can be created. Sample iRule, log_http_request_header_payload The iRule is then applied to a Virtual Server where suspected malicious traffic is hitting Sample iRule: # Log debug to /var/log/ltm? 1=yes, 0=no when RULE_INIT { set static::payload_dbg 1 } when HTTP_REQUEST { set LogString "Client [IP::client_addr]:[TCP::client_port] -> [HTTP::host][HTTP::uri]" if {$static::payload_dbg}{log local0.debug "=============================================" } if {$static::payload_dbg}{log local0.debug "$LogString (request)"} # log each Header. foreach aHeader [HTTP::header names] { if {$static::payload_dbg}{log local0.debug "$aHeader: [HTTP::header value $aHeader]"} } if {$static::payload_dbg}{log local0.debug "============================================="} if {[HTTP::header "Content-Length"] ne "" && [HTTP::header "Content-Length"] <= 1048000} { HTTP::collect [HTTP::header "Content-Length"] } else { HTTP::collect 1048000 } } when HTTP_REQUEST_DATA { # Log the bytes collected if {$static::payload_dbg}{log local0.debug "Collected [HTTP::payload length] bytes"} # Log the payload locally if {[HTTP::payload length] < $static::max_chars}{ if {$static::payload_dbg}{log local0.debug "Payload=[HTTP::payload]"} } } Sample log output in /var/log/ltm: Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: ============================================= Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: Client 172.16.7.31:10761 -> 172.16.8.86/ (request) Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: Host: 172.16.8.86 Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: Accept: */* Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: User-Agent: im-a-web-browser Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: X-Forwarded-For: 88.88.88.88 Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: Content-Length: 11 Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: Content-Type: application/x-www-form-urlencoded Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: ============================================= Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST_DATA>: Collected 11 bytes Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST_DATA>: Payload=admin:admin Using the same grep/awk/uniq commands, we can observe the number of times a client has sent a HTTP request to a Virtual Server using the sample iRule. grep/awk/uniq sample command 3: grep "<HTTP_REQUEST>: Client" /var/log/ltm | awk -F " " '{print $11}' | awk -F ":" '{print $1}' | uniq -c sample output: root@BIG-IP:Active:Standalone] config # grep "<HTTP_REQUEST>: Client" /var/log/ltm | awk -F " " '{print $11}' | awk -F ":" '{print $1}' | uniq -c 13 172.16.7.31 grep/awk/uniq sample command 4: grep "<HTTP_REQUEST>: User-Agent" /var/log/ltm | awk -F " " '{print $11}' | uniq -c The sample command will count the number of HTTP requests a specific User-Agent has sent to a Virtual Server sample output: root@BIG-IP:Active:Standalone] config # grep "<HTTP_REQUEST>: User-Agent" /var/log/ltm | awk -F " " '{print $11}' | uniq -c 13 im-a-web-browser Additional Notes for LTM Policy and iRules HTTP request data gatheringfor HTTP Brute Force attack The sample iRule and LTM Policy provided are just sample approach in observing the traffic to a Virtual Server suspected of being attacked during a HTTP Brute Force attack.There are more LTM Policy and iRule events that can be used to gather data and observe it to qualify certain characteristics of the traffic that can be considered malicious. In both the LTM Policy and iRule samples, we logged HTTP Payload output. During a HTTP Brute Force attack, an attacker - perhaps characterized by a certain source IP or a suspicious User-Agent value,may be sending to the HTTP based application random username and password or, legitimate looking credentials which are normally extracted from a breach,in attempt to guess a legitimate user credentials to compromise and to further do damage. The generated logs can then be used for your analysis and decision making steps in the mitigation of the attack. While the logs that will be generated are useful during a HTTP Brute Force attack, these logs may be excessive and may not be needed when there is noHTTP Brute Force attack event. Removing the sample iRule or LTM Policy from the Virtual Server would prevent further logging. Using F5 Application Visibility and Reporting to gather HTTP request data From the BIG-IP Analytics: Implementations Manual: https://techdocs.f5.com/kb/en-us/products/big-ip_analytics/manuals/product/big-ip-analytics-implementations-14-1-0/01.html#guid-ce2cb6f5-24b4-4570-a827-b8b4db8e19e8 Analytics, or Application Visibility and Reporting (AVR), is a module on the BIG-IP® system that you can use to visually analyze the performance of web applications, TCP traffic, DNS traffic, FastL4, and overall system statistics. The statistics are displayed in graphical charts where you can drill down into a specific time range or system aspect to better understand network performance on certain devices, IP addresses, memory and CPU utilization, and so on. You can further focus the statistics in the charts by selecting dimension entities such as applications or virtual servers. About HTTP Analytics profiles An HTTP Analytics profile ( Local Traffic > Profiles > Analytics > HTTP Analytics ) is a set of definitions that determines the circumstances under which the system gathers, logs, notifies, and graphically displays information regarding traffic to an application. Each monitored application is associated with an HTTP Analytics profile. You associate the HTTP Analytics profile with one or more virtual servers used by the application. Each virtual server can have one HTTP and/or one TCP Analytics profile associated with it. Required License and Provisioning BIG-IP AVR module should be licensed and provisioned for AVR features to be available. Here is a sample screenshot of data gathered from a Virtual Server with HTTP Analytics profile applied. This can be viewed from the Configuration Utility Statistics››Analytics : HTTP : Overview We can clearly see the HTTP data gathered are common HTTP statistics that we would like to observe and have an understanding during a HTTP Brute force attack, Source Client IP address, Countries, User Agents, URLs and more and its corresponding statistics such as Average Transaction per second, Average Server Latency (ms) and more, are helpful information to help deciding what possible mitigation will be useful to quickly mitigate the attack. In the sample screenshot, we can see the User Agents statistics that "curl/7.47.1" sent bulk of the request to the Virtual Server. For this sample scenario, If this client HTTP User Agent is not expected and the HTTP application expects only real web browsers, the User Agent can be considered to be blocked from accessing the Virtual Server. Similarly, specific IP addresses or Countries may be unexpected traffic sources and are sending high volume traffic, these can also be considered to be blocked depending on what is considered expected traffic. These AVR statistics can also be exported in PDF format for offline review. HTTP Analytics profile can created from the Configuration Utility, Local Traffic››Profiles : Analytics : HTTP Analytics Here is a sample screenshot of a custom HTTP Analytics profile and is applied to a Virtual Server. Additional Notes for AVR HTTP request data gatheringfor HTTP Brute Force attack Public facing HTTP application are expected to receive malicious traffic. ProvisioningAVR module, configuring a HTTP Analytics profile and applying it to a HTTP Virtual Server in advance before a HTTP Brute Force attack happens provides advantages. It allows the gathering of HTTP traffic related statistics to understand the HTTP traffic pattern for a particular Virtual Server and helps in defining a traffic baseline and what normal traffic looks like and several other characteristic of traffic such as source IP its geolocation/country information, User Agent information and more. As an example of howAVR HTTP request data gatheredmay be used for HTTP Brute Force attack prevention, an application's login URL may be receiving excessive requestsas observed in Configuration Utility Statistics››Analytics : HTTP : Overview for a HTTP based Virtual Server with the HTTP analytics applied - in AVR, URL statistics can be reviewed and "Ave TPS" and "Transactions" are available - and application owner may define this an attack. Without HTTP Analytics profile applied to a Virtual Server, alternative manual methods or monitoring system would need to have been in place to describe the suspected HTTP Brute Force attack event.HTTP traffic information, very common, the User Agent may be a spoofed and upon reviewing the amount of transaction it sent during the HTTP Brute Force attack event, application owner may decide to block the specific User Agent value. Sample Application of AVR and LTM policy Data gathering techniques In this sample HTTP Analytics screenshot, the login URL /WackoPicko/users/login.php received high volume transaction from a specific source IP address. A closer look at the page information, an unusual looking User Agent was sending the suspicious amount of traffic. An application owner may consider this as an attack to the login URL and have the specific IP address or the User Agent to be blocked. Applying the sample LTM Policy where HTTP request data was logged, in reference to the HTTP AVR data screenshot,it can be observed that a HTTP Brute Force Attack was happening.username=!@#$%25 and random password are being sent to the login URL /WackoPicko/users/login.php. sample logs of a HTTP Brute Force Attack captured using thesample LTM Policy [root@BIG-IP:Active:Standalone] config # grep username /var/log/ltm | head Nov 30 17:54:10 BIG-IP info tmm4[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49133 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=12345678 Nov 30 17:54:10 BIG-IP info tmm5[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49134 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=jessica Nov 30 17:54:10 BIG-IP info tmm6[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49135 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=lovely Nov 30 17:54:10 BIG-IP info tmm4[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49136 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=abc123 Nov 30 17:54:10 BIG-IP info tmm5[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49137 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=monkey Nov 30 17:54:10 BIG-IP info tmm6[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49138 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=123456789 Nov 30 17:54:10 BIG-IP info tmm7[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49139 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=654321 Nov 30 17:54:10 BIG-IP info tmm[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49140 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=12345 Nov 30 17:54:10 BIG-IP info tmm1[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49141 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=daniel Nov 30 17:54:10 BIG-IP info tmm2[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49142 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=princess Using the sample generated logs during the HTTP Brute Force attack, running a combination of grep/cut/awk/uniq/sort command provides insight the top 10 username's approximate number login attempts grep/cut/awk/uniq/sort sample command: grep username /var/log/ltm | cut -d " " -f 27 | awk -F "&" '{print $1}' | awk -F "=" '{print $2}' | uniq -c | sort -rnk 1 | head sample output: [root@sec8:Active:Standalone] config # grep username /var/log/ltm | cut -d " " -f 27 | awk -F "&" '{print $1}' | awk -F "=" '{print $2}' | uniq -c | sort -rnk 1 | head 1006 aberrational 1006 abe 1006 abdul 1006 abduct 1006 abdominally 1006 abdominal 1006 abdomen 1006 abdication 1006 abcdefg 1006 abcdef Similarly, looking specifically for a username's approximate number of login attempts- in this example, '!@#$%25' sample command: grep '!@#$%25' /var/log/ltm | wc -l sample output: [root@sec8:Active:Standalone] config # grep '!@#$%25' /var/log/ltm | wc -l 4024
