Forum Discussion

mrintzler's avatar
mrintzler
Icon for Nimbostratus rankNimbostratus
Feb 04, 2008

iRule causing ever-increasing TMM CPU utilization

I've adapted an iRule written by Deb Allen to rate limit (in count, not in bps) PDF downloads from one of our web applications. It basically counts the number of PDF's downloaded in the past 10 seconds, and gives an error response if they are exceeding the threshold. The rule works as expected. However, as the rule is in place, TMM CPU utilization slowly increases, until performance is affected after about a week. If I re-initialize the iRule (make a small modification and save, or simply remove and reassign to the virtual) the TMM utilization drops back down to normal levels. I recently upgraded from 9.2.3 to 9.4.4 in hopes that a couple memory leak issues were the culprit, but the issue remains. iRule timing shows that the avg CPU time for the iRule steadily increases as time goes on. I'm thinking the issue is due to the array growing in size, despite the lines clearing the older entries out. Is there a way to automate periodically re-initializing the array? Below is the iRule:


 rule pdfapp-abuse-detection-rule
 Limits the rate of PDF downloads for a particular IP client
 Adapted from 'RateLimit_HTTP' on devcentral 
 Adapted by Michael Rintzler
 August 20th, 2007
 Original concept by Deb Allen, F5 Networks
 April 2006
 
when RULE_INIT {
  set ::pdfappmaxRate 10          ;set later per user from class
  set ::pdfappwindowSecs 10           ;global
  init array if non-existent
  array set ::pdfHistory { }
  wipe array if already existent
  array unset ::pdfHistory
}
when HTTP_REQUEST timing on {
  if { [HTTP::method] eq "GET" } {
    if { ([HTTP::uri] contains ".pdf") and not ([HTTP::header exists Range ]) and ([HTTP::cookie exists ERIGHTS])} {
      log local0. "pdfapp PDF Initial download detected."
      Extract clients IP address
      set client_ip [IP::remote_addr]
      set pdfapp_session [HTTP::cookie ERIGHTS]
      set currentTime [clock seconds]
      we need to count requests in last $windowSecs seconds, so mark the cutoff time 
      set pdfappwindowStart [expr {$currentTime - $::pdfappwindowSecs}]
      find GETs for this sessionID
      set pdfCount 1
      set mypdfappMaxRate $::pdfappmaxRate
      count GETs within the window, delete those that are older
      foreach { requestID requestTime } [array get ::pdfHistory ${pdfapp_session}*] {
        count pdf downloadss with start time > $pdfappwindowStart, delete the rest
        if { $requestTime > $pdfappwindowStart } {
          incr pdfCount 1
        } else {
          unset ::pdfHistory($requestID)
        }
      }
       if { $pdfCount < $mypdfappMaxRate } {
        Allow request and add new record to array w/myUserID.rand + currentTime
 set pdfapprequestID to a random number
        set requestID "${pdfapp_session}.[expr { int(10000000 * rand()) }]"
        set ::pdfHistory($requestID) $currentTime
        log local0. "PDF Abuse - download from $client_ip, SESSION $pdfapp_session - Current rate: $pdfCount" 
     } else {
        Reject request with 200 response
        log local0. "PDF Abuse - User $client_ip, Session $pdfapp_session"
        return
      }
    }
      else { log "Partial PDF download detected." }
  }
}

3 Replies

  • spark_86682's avatar
    spark_86682
    Historic F5 Account
    It sounds like you've diagnosed the problem correctly in that your array just grows over time. If a client just downloads a PDF and leaves, I don't see any way that your iRule ever cleans that entry out of the array, so under normal use, the array would just grow and grow.

     

     

    I don't have a perfect suggestion for you; here don't yet exist any timer commands for you to run a block of code every so often. You could do something like keep a counter of how many HTTP_REQUEST events you've seen and completely clear out the array every 100,000 or so.
  • Deb_Allen_18's avatar
    Deb_Allen_18
    Historic F5 Account
    To answer your question, I've used the clock to trigger periodic housekeeping in other circumstances, something like this:
    when RULE_INIT {
      set ::pdfappmaxRate 10              ;set later per user from class
      set ::pdfappwindowSecs 10           ;global
      init array if non-existent
      array set ::pdfHistory { }
      wipe array if already existent
      array unset ::pdfHistory
      set ::housekeeping_clock [clock seconds]
      set ::housekeeping_interval 900
    }
    when HTTP_REQUEST {
      if { [clock seconds] > [expr {$::housekeeping_clock + $::housekeeping_interval}] }{
        wipe array if already existent
        array unset ::pdfHistory
      }
      ...
    }

     

     

    However, this code block should reap the old entries from the array upon every request:

     

          count GETs within the window, delete those that are older
          foreach { requestID requestTime } [array get ::pdfHistory ${pdfapp_session}*] {
            count pdf downloadss with start time > $pdfappwindowStart, delete the rest
            if { $requestTime > $pdfappwindowStart } {
              incr pdfCount 1
            } else {
              unset ::pdfHistory($requestID)
            }
          }
    Not sure why it isn't doing so. Before I added another periodic housekeeping function, esp one that's indiscriminate like that above, I'd try to figure out why this isn't working. I'd start by adding some logging here to see if anything is matching the delete condition.

     

     

    HTH

     

    /deb
  • You're exactly right. That was my problem. When I first implemented a version of this iRule, it was keyed off of the end-users IP address. Most of our customers will have the same IP address indefinitely, or at least for a long time. When I modified the iRule for a different application, I keyed the users by their session cookie, which changes each time they log in. As you pointed out, I only wipe their history from the array when they come back to download, so if a user comes back an hour later with a different session ID, their old entries in the array will be stuck their forever.

     

     

    I've added the following code to periodically wipe the array. Since this is only to prevent extreme robotic abuse situations, it's ok to periodically lose the history for the previous 10 seconds:

     

    
          if { $::totalPDF > $::refreshint } {
    array unset ::pdfHistory
    log local0. "$::refreshint downloads reached.  Resetting array"
    set ::totalPDF 0
          }

     

     

    Thanks for all the responses!