HTTP Brute Force Mitigation Playbook: Bad Actor Behavior and Gathering Statistics using BIG-IP LTM Policies, iRules and BIG-IP AVR - Chapter 2
Gathering Statistics and Bad Actor Behavior
In this Chapter of the HTTP Brute Force Mitigation Playbook series, we will review BIG-IP Local Traffic Manager (LTM) and Application Visibility and Reporting (AVR) modules' features to show how it can used to gather statistics and how we can use these statistics as a base line of what an organization may consider as normal traffic and malicious traffic and how it relates to HTTP Brute Force Attacks.
Normal Traffic and Malicious Traffic Definition
Normally, we consider normal traffic as client activities associated to an application that is expected and does not affect availability and performance.
Malicious traffic, on the other hand, are traffic bound to hit any public facing application or device with the goal of impacting availability, performance and its security.
We will focus on malicious traffic related to HTTP Brute Force Attacks.
Bad Actor Behavior
In the case of HTTP Brute Force Attacks, it is common to observe these considered Bad Actor Behavior from malicious traffic:
- A spoofed HTTP User-Agent header
- Traffic from unexpected Geolocation(s)
- Excessive traffic from a certain source IP
- Multiple Login attempts on a web application from the same or distributed source IP using different credentials
- HTTP requests to an application that iterates thru a HTTP parameter's value either from a single or distributed traffic source
The malicious traffic could be sent slowly and in low volume in an attempt to be more stealthy and applying on or more of the Bad Actor Behavior combinations or also be sent thru high volume traffic and will have noticeable impact such as high CPU usage on the application server
Prerequisites
Managing the BIGIP configuration requires Administrator access. Ensure access to Configuration Utility (Web GUI) and SSH is available. These management interfaces will be helpful in configuring, verifying and troubleshooting on the BIGIP. Having access to serial console output of the BIGIP is also helpful.
When working with F5 Support, having an F5 Support/AskF5 account will be helpful. To access your AskF5 account, access the following link:
https://support.f5.com/csp/my-support/home
Another helpful site is F5 ihealth, where qkviews extracted from the BIGIP device can be uploaded and analysed. Logs, statistics, heuristics, graphs and BIGIP configuration extracted thru a qkview are available in ihealth when uploaded and will aid in diagnosing and troubleshooting efforts.
Register an account to these F5 Sites as these will helpful in working with F5 Support. When troubleshooting, generating a qkview or extracting bigip logs or viewing configuration in the GUI require access to the BIGIP and having the administrative access to it and referenced F5 Support sites will be helpful.
Local Traffic Manager (LTM) and Application Visibility and Reporting (AVR) license are required to use the related features. LTM and AVR modules should be provisioned to access related configuration discussed in this chapter.
Using Local Traffic Manager (LTM) features to gather HTTP request data
Local Traffic Manager (LTM) features such as iRules and LTM Policy can be used to log messages generated from a certain stage of the HTTP request. In the case of a HTTP brute force attack, inspecting the source IP, HTTP headers, the HTTP request payload can provide information on what malicious traffic may look like.
Required License and Provisioning
BIG-IP LTM module should licensed and provisioned for LTM features to be available.
A "harmless" HTTP request
The following "curl" command will receive a proper response from a web site it is requesting content from.
curl --silent --output /dev/null -lvk https://172.16.8.84 -H "User-Agent: im-a-web-browser"
The web server does not validate whether this curl request is coming from a legitimate web browser as a normal user would use. the curl "-H" option allows a HTTP header to be included in the curl HTTP request, in this case "User-Agent: im-a-web-browser" will be the HTTP User-Agent and value that the web server will receive. A legitimate HTTP User-Agent value can also be used for this option and the web server will simply accept and process the request without validating whether the request is from a real web browser or an automated client .
Using Local Traffic Manager (LTM) Policy to log HTTP request data
A Local Traffic Manager (LTM) Policy can be applied to a Virtual Server and match a traffic pattern and execute defined actions. LTM Policy configuration is straightforward and is able to log essential data, such as in the case of a HTTP Brute Force attack, gather source IP, HTTP headers and HTTP payload. It can log locally or remotely.
Here is a sample LTM Policy configuration where it will log the Source client IP address and port, HTTP header names and its interesting values and the HTTP Payload for all traffic at HTTP request time.
From the BIG-IP Configuration Utility:
Local Traffic ›› Policies : Policy List ›› /Common/log_http_traffic:log_http_request_header_payload
log_http_traffic is the name of the LTM Policy
log_http_request_header_payload is a rule in the log_http_traffic LTM Policy
This view is only in Draft state of the LTM Policy
These are configured in the message section of the "Log" action lines. notice the second Log action specifically logs the User-Agent and X-Forwarded-For HTTP headers. Common in HTTP Brute Force Attacks, the HTTP User-Agent header's value may be a slightly modified compared to a legitimate User-Agent value - imitating a legitimate web browser. Similarly, the X-Forwarded-For value may provide some hints where the attack traffic may really be coming from. In scenarios where the attacker may be passing thru web proxies, reviewing the X-Forwarded-For header value can be helpful in determining possible thru source of the attack traffic.
Log action 1: tcl: client [IP::client_addr]:[TCP::client_port] -> URL: [HTTP::host][HTTP::uri]: http headers are [HTTP::header names] and the http payload is [HTTP::payload] Log action 2: tcl: client [IP::client_addr]:[TCP::client_port] -> URL: [HTTP::host][HTTP::uri]: interesting HTTP headers are User-Agent:[HTTP::header User-Agent] X-Forwarded-For [HTTP::header X-Forwarded-For]
Notice that the Log action references "tcl". LTM Policy actions accepts tcl expression and is commonly used in iRules. iRule commands such as the [HTTP::header] can be used in LTM policies.
Important: The iRule commands used in this sample LTM policy are for demonstration purpose only. Complex LTM Policies with iRule commands should be properly tested. F5 Professional Services is the best resource for this type of implementation.
log Facility local0 refers to the /var/log/ltm and thus, generated logs will be in /var/log/ltm for this LTM Policy. other log files can be also specified and remote logging is also possible.
Here is the Published LTM Policy
The sample LTM Policy is then applied to a Virtual Server where malicious traffic is suspected to be hitting
Here is the text form of the Published LTM Policy.
log_http_request_header_payload
1. Log message 'tcl: client [IP::client_addr]:[TCP::client_port] -> URL: [HTTP::host][HTTP::uri]: http headers are [HTTP::header names] and the http payload is [HTTP::payload]' at request time. 2. Log message 'tcl:client [IP::client_addr]:[TCP::client_port] -> URL: [HTTP::host][HTTP::uri]: interesting HTTP headers are User-Agent:[HTTP::header User-Agent] X-Forwarded-For:[HTTP::header X-Forwarded-For]' at request time.
log generated in /var/log/ltm
Nov 29 15:24:22 BIG-IP info tmm7[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.7.31:39063 -> URL: 172.16.8.86/: http headers are Host Accept User-Agent X-Forwarded-For Content-Length Content-Type and the http payload is admin:admin
Nov 29 15:24:22 BIG-IP info tmm7[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.7.31:39063 -> URL: 172.16.8.86/: interesting HTTP headers are User-Agent:im-a-web-browser X-Forwarded-For:88.88.88.88
Using grep, awk and uniq in the BIG-IP bash prompt, /var/log/ltm can be inspected to check the how frequent a source client IP sent a HTTP request and along with it, HTTP header and Payload information, to the Virtual Server where the LTM Policy is applied.
Here, the sample grep/awk/uniq command will count the number of times the client source IP was observed from the sample LTM Policy generated log /var/log/ltm. In the sample LTM Policy, it logs the client source IP twice, as such the sample grep/awk/uniq command was written to take account of this.
grep/awk/uniq sample command 1:
grep client /var/log/ltm | awk -F " " '{print $9}' | uniq -c | awk -F ":" '{print $1}' | awk -F " " '{print $2}' | uniq -c
sample output:
root@BIG-IP:Active:Standalone] config # grep client /var/log/ltm | awk -F " " '{print $9}' | uniq -c | awk -F ":" '{print $1}' | awk -F " " '{print $2}' | uniq -c
9 172.16.7.31
grep/awk/uniq sample command 2:
grep interesting /var/log/ltm | awk -F " " '{print $18}' | awk -F ":" '{print $2}' | uniq -c
This sample command will count the number of time the X-Forwarded-For HTTP Header appeared in the log. the word 'interesting' was initially filtered as it is the line where the X-Forwarded-For information was logged.
sample output:
root@BIG-IP:Active:Standalone] config # grep interesting /var/log/ltm | awk -F " " '{print $18}' | awk -F ":" '{print $2}' | uniq -c
20 88.88.88.88
The sample grep/awk/uniq command are just simple demonstration commands only. You can build your own scripts, version of the command to get the desired information. The sample output are also for demonstration purpose only. In a real HTTP brute force attack, The sample LTM Policy to log the client source IP and HTTP information in the request will be helpful in describing the possibly malicious traffic.
Using F5 iRules to gather HTTP request data
From the F5 iRule Home Page, https://clouddocs.f5.com/api/irules/:
F5 iRules is a powerful and flexible feature within the BIG-IP® local traffic management (LTM) system that you can use to manage your network traffic. The iRulesTM feature not only allows you to select pools based on header data, but also allows you to direct traffic by searching on any type of content data that you define. Thus, the iRules feature significantly enhances your ability to customize your content switching to suit your exact needs.
iRules can also be used to log source IP and HTTP information. In comparison to LTM Policy, iRules are much more flexible and can also be very complex.
Depending on how much information you would like to gather during a HTTP Brute Force attack, iRules can be very verbose. iRules are also resource intensive, the more it gets executed as traffic hits a Virtual Server where it is applied, it will consume additional CPU resource as well. If the same information can be gathered using LTM Policy feature, use it instead of iRules. For simplicity we will gather the same HTTP request data as demonstrated in the LTM Policy section.
Important: The iRule commands used in the sample iRules are for demonstration purpose only. Complex iRules should be properly tested. F5 Professional Services is the best resource for this type of implementation.
In the Configuration Utility, Local Traffic ›› iRules : iRule List ›› can be created.
Sample iRule, log_http_request_header_payload
The iRule is then applied to a Virtual Server where suspected malicious traffic is hitting
Sample iRule:
# Log debug to /var/log/ltm? 1=yes, 0=no
when RULE_INIT {
set static::payload_dbg 1
}
when HTTP_REQUEST {
set LogString "Client [IP::client_addr]:[TCP::client_port] -> [HTTP::host][HTTP::uri]"
if {$static::payload_dbg}{log local0.debug "=============================================" }
if {$static::payload_dbg}{log local0.debug "$LogString (request)"}
# log each Header.
foreach aHeader [HTTP::header names] {
if {$static::payload_dbg}{log local0.debug "$aHeader: [HTTP::header value $aHeader]"}
}
if {$static::payload_dbg}{log local0.debug "============================================="}
if {[HTTP::header "Content-Length"] ne "" && [HTTP::header "Content-Length"] <= 1048000} {
HTTP::collect [HTTP::header "Content-Length"]
} else {
HTTP::collect 1048000
}
}
when HTTP_REQUEST_DATA {
# Log the bytes collected
if {$static::payload_dbg}{log local0.debug "Collected [HTTP::payload length] bytes"}
# Log the payload locally
if {[HTTP::payload length] < $static::max_chars}{
if {$static::payload_dbg}{log local0.debug "Payload=[HTTP::payload]"}
}
}
Sample log output in /var/log/ltm:
Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: =============================================
Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: Client 172.16.7.31:10761 -> 172.16.8.86/ (request)
Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: Host: 172.16.8.86
Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: Accept: */*
Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: User-Agent: im-a-web-browser
Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: X-Forwarded-For: 88.88.88.88
Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: Content-Length: 11
Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: Content-Type: application/x-www-form-urlencoded
Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST>: =============================================
Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST_DATA>: Collected 11 bytes
Nov 29 19:46:03 BIG-IP debug tmm4[11545]: Rule /Common/log_http_request_header_payload <HTTP_REQUEST_DATA>: Payload=admin:admin
Using the same grep/awk/uniq commands, we can observe the number of times a client has sent a HTTP request to a Virtual Server using the sample iRule.
grep/awk/uniq sample command 3:
grep "<HTTP_REQUEST>: Client" /var/log/ltm | awk -F " " '{print $11}' | awk -F ":" '{print $1}' | uniq -c
sample output:
root@BIG-IP:Active:Standalone] config # grep "<HTTP_REQUEST>: Client" /var/log/ltm | awk -F " " '{print $11}' | awk -F ":" '{print $1}' | uniq -c
13 172.16.7.31
grep/awk/uniq sample command 4:
grep "<HTTP_REQUEST>: User-Agent" /var/log/ltm | awk -F " " '{print $11}' | uniq -c
The sample command will count the number of HTTP requests a specific User-Agent has sent to a Virtual Server
sample output:
root@BIG-IP:Active:Standalone] config # grep "<HTTP_REQUEST>: User-Agent" /var/log/ltm | awk -F " " '{print $11}' | uniq -c
13 im-a-web-browser
Additional Notes for LTM Policy and iRules HTTP request data gathering for HTTP Brute Force attack
The sample iRule and LTM Policy provided are just sample approach in observing the traffic to a Virtual Server suspected of being attacked during a HTTP Brute Force attack. There are more LTM Policy and iRule events that can be used to gather data and observe it to qualify certain characteristics of the traffic that can be considered malicious. In both the LTM Policy and iRule samples, we logged HTTP Payload output.
During a HTTP Brute Force attack, an attacker - perhaps characterized by a certain source IP or a suspicious User-Agent value, may be sending to the HTTP based application random username and password or, legitimate looking credentials which are normally extracted from a breach, in attempt to guess a legitimate user credentials to compromise and to further do damage. The generated logs can then be used for your analysis and decision making steps in the mitigation of the attack.
While the logs that will be generated are useful during a HTTP Brute Force attack, these logs may be excessive and may not be needed when there is no HTTP Brute Force attack event. Removing the sample iRule or LTM Policy from the Virtual Server would prevent further logging.
Using F5 Application Visibility and Reporting to gather HTTP request data
From the BIG-IP Analytics: Implementations Manual:
Analytics, or Application Visibility and Reporting (AVR), is a module on the BIG-IP® system that you can use to visually analyze the performance of web applications, TCP traffic, DNS traffic, FastL4, and overall system statistics. The statistics are displayed in graphical charts where you can drill down into a specific time range or system aspect to better understand network performance on certain devices, IP addresses, memory and CPU utilization, and so on. You can further focus the statistics in the charts by selecting dimension entities such as applications or virtual servers.
About HTTP Analytics profiles
An HTTP Analytics profile ( Local Traffic > Profiles > Analytics > HTTP Analytics ) is a set of definitions that determines the circumstances under which the system gathers, logs, notifies, and graphically displays information regarding traffic to an application. Each monitored application is associated with an HTTP Analytics profile. You associate the HTTP Analytics profile with one or more virtual servers used by the application. Each virtual server can have one HTTP and/or one TCP Analytics profile associated with it.
Required License and Provisioning
BIG-IP AVR module should be licensed and provisioned for AVR features to be available.
Here is a sample screenshot of data gathered from a Virtual Server with HTTP Analytics profile applied. This can be viewed from the Configuration Utility Statistics ›› Analytics : HTTP : Overview
We can clearly see the HTTP data gathered are common HTTP statistics that we would like to observe and have an understanding during a HTTP Brute force attack, Source Client IP address, Countries, User Agents, URLs and more and its corresponding statistics such as Average Transaction per second, Average Server Latency (ms) and more, are helpful information to help deciding what possible mitigation will be useful to quickly mitigate the attack.
In the sample screenshot, we can see the User Agents statistics that "curl/7.47.1" sent bulk of the request to the Virtual Server. For this sample scenario, If this client HTTP User Agent is not expected and the HTTP application expects only real web browsers, the User Agent can be considered to be blocked from accessing the Virtual Server. Similarly, specific IP addresses or Countries may be unexpected traffic sources and are sending high volume traffic, these can also be considered to be blocked depending on what is considered expected traffic.
These AVR statistics can also be exported in PDF format for offline review.
HTTP Analytics profile can created from the Configuration Utility, Local Traffic ›› Profiles : Analytics : HTTP Analytics
Here is a sample screenshot of a custom HTTP Analytics profile and is applied to a Virtual Server.
Additional Notes for AVR HTTP request data gathering for HTTP Brute Force attack
Public facing HTTP application are expected to receive malicious traffic. Provisioning AVR module, configuring a HTTP Analytics profile and applying it to a HTTP Virtual Server in advance before a HTTP Brute Force attack happens provides advantages. It allows the gathering of HTTP traffic related statistics to understand the HTTP traffic pattern for a particular Virtual Server and helps in defining a traffic baseline and what normal traffic looks like and several other characteristic of traffic such as source IP its geolocation/country information, User Agent information and more.
As an example of how AVR HTTP request data gathered may be used for HTTP Brute Force attack prevention, an application's login URL may be receiving excessive requests as observed in Configuration Utility Statistics ›› Analytics : HTTP : Overview for a HTTP based Virtual Server with the HTTP analytics applied - in AVR, URL statistics can be reviewed and "Ave TPS" and "Transactions" are available - and application owner may define this an attack. Without HTTP Analytics profile applied to a Virtual Server, alternative manual methods or monitoring system would need to have been in place to describe the suspected HTTP Brute Force attack event. HTTP traffic information, very common, the User Agent may be a spoofed and upon reviewing the amount of transaction it sent during the HTTP Brute Force attack event, application owner may decide to block the specific User Agent value.
Sample Application of AVR and LTM policy Data gathering techniques
In this sample HTTP Analytics screenshot, the login URL /WackoPicko/users/login.php received high volume transaction from a specific source IP address. A closer look at the page information, an unusual looking User Agent was sending the suspicious amount of traffic. An application owner may consider this as an attack to the login URL and have the specific IP address or the User Agent to be blocked.
Applying the sample LTM Policy where HTTP request data was logged, in reference to the HTTP AVR data screenshot, it can be observed that a HTTP Brute Force Attack was happening. username=!@#$%25 and random password are being sent to the login URL /WackoPicko/users/login.php.
sample logs of a HTTP Brute Force Attack captured using the sample LTM Policy
[root@BIG-IP:Active:Standalone] config # grep username /var/log/ltm | head
Nov 30 17:54:10 BIG-IP info tmm4[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49133 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=12345678
Nov 30 17:54:10 BIG-IP info tmm5[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49134 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=jessica
Nov 30 17:54:10 BIG-IP info tmm6[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49135 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=lovely
Nov 30 17:54:10 BIG-IP info tmm4[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49136 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=abc123
Nov 30 17:54:10 BIG-IP info tmm5[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49137 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=monkey
Nov 30 17:54:10 BIG-IP info tmm6[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49138 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=123456789
Nov 30 17:54:10 BIG-IP info tmm7[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49139 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=654321
Nov 30 17:54:10 BIG-IP info tmm[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49140 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=12345
Nov 30 17:54:10 BIG-IP info tmm1[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49141 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=daniel
Nov 30 17:54:10 BIG-IP info tmm2[11545]: [/Common/log_http_traffic/log_http_request_header_payload]: client 172.16.5.40:49142 -> URL: 172.16.8.86/WackoPicko/users/login.php: http headers are Host User-Agent Cookie Content-Length Content-Type and the http payload is username=!@#$%25&password=princess
Using the sample generated logs during the HTTP Brute Force attack, running a combination of grep/cut/awk/uniq/sort command provides insight the top 10 username's approximate number login attempts
grep/cut/awk/uniq/sort sample command:
grep username /var/log/ltm | cut -d " " -f 27 | awk -F "&" '{print $1}' | awk -F "=" '{print $2}' | uniq -c | sort -rnk 1 | head
sample output:
[root@sec8:Active:Standalone] config # grep username /var/log/ltm | cut -d " " -f 27 | awk -F "&" '{print $1}' | awk -F "=" '{print $2}' | uniq -c | sort -rnk 1 | head
1006 aberrational
1006 abe
1006 abdul
1006 abduct
1006 abdominally
1006 abdominal
1006 abdomen
1006 abdication
1006 abcdefg
1006 abcdef
Similarly, looking specifically for a username's approximate number of login attempts - in this example, '!@#$%25'
sample command: grep '!@#$%25' /var/log/ltm | wc -l
sample output:
[root@sec8:Active:Standalone] config # grep '!@#$%25' /var/log/ltm | wc -l 4024