Forum Discussion

scott_sams_8256's avatar
scott_sams_8256
Icon for Nimbostratus rankNimbostratus
Feb 20, 2009

content tracking

we found an irule here by Kirk (thanks Kirk) that breaks content down by count. any genius out there that can take this and add total bytes for each category? our ecomm group wants to see the breakdown in the classification of traffic.

 

 

thanks!

 

 

 

 

Content_ Type_ Tracking

 

Enter a topic name to show or a new topic name to create; then press Enter

 

.

 

Contributed by: kirkbauer - kirk@f5.com

 

Description

 

 

Have you ever wondered what types of content are served by your web servers? Have you wondered how many requests could be offloaded to the BIG-IP if you added the RAM Cache module? Or maybe how much acceleration you may get from WebAccelerator ? Try out this iRule. Most of the time everything other than the HTML pages should be cacheable.

 

iRule Source

 

 

First, create statistics profile named "ContentType" with following entries:

 

HTML

 

Images

 

Scripts

 

Documents

 

Stylesheets

 

Other

 

Now associate this Statistics Profile to the virtual server. Then apply the following iRule.

 

To view the results, go to Statistics -> Profiles - Statistics

 

 

when HTTP_RESPONSE {

 

switch -glob [HTTP::header "Content-type"] {

 

"image/*" { STATS::incr "ContentType" "Images" }

 

"text/html" { STATS::incr "ContentType" "HTML" }

 

"text/css" { STATS::incr "ContentType" "Stylesheets" }

 

"*javascript" { STATS::incr "ContentType" "Scripts" }

 

"text/vbscript" { STATS::incr "ContentType" "Scripts" }

 

"application/pdf" { STATS::incr "ContentType" "Documents" }

 

"application/msword" { STATS::incr "ContentType" "Documents" }

 

"application/*powerpoint" { STATS::incr "ContentType" "Documents" }

 

"application/*excel" { STATS::incr "ContentType" "Documents" }

 

"default" { STATS::incr "ContentType" "Other" }

 

}

 

}

 

 

application delivery
dev
devops
iRules

16 Replies

Sort By
Show More
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Feb 26, 2009
    If the Content-Length header doesn't exist, [HTTP::header value "Content-Length"] will return a null string and STATS::incr will probably throw a TCL error. You might either wrap it in a catch, or check that the HTTP::header result is an integer.

     

     

    Out of curiosity, scottsams, do you see the content length of 1.4k on every response? Can you post the response headers from a response that shows 1.4k for the HTTP::payload length value?

     

     

    Aaron
  • scott_sams_8256's avatar
    scott_sams_8256
    Icon for Nimbostratus rankNimbostratus
    Feb 26, 2009
    thanks i will try that. as for the 1.4k when i sniffed the transaction my MTU was set to 1460. it has since changed and went to 1.2k. i did a tcpdump and saw the frame sizes negotiated to 1260 also. it seemed to correlate. oddly, when i hit one page with a 116k jpg, the stat incremented to 1 with a size of 1.2k.

     

    i am attaching a jpg screen shot with various info. irule, log, website i hit.

     

    here is the log

     

    Feb 26 10:19:30 tmm tmm[955]: 01220002:6: Rule http_stat_tracker : images_len is 1212

     

    Feb 26 10:20:52 tmm tmm[955]: 01220002:6: Rule http_stat_tracker : Header content is image/JPEG; Charset=utf-8 payload length is 1212

     

    Feb 26 10:20:52 tmm tmm[955]: 01220002:6: Rule http_stat_tracker : images_len is 1212

     

    Feb 26 10:21:06 tmm tmm[955]: 01220002:6: Rule http_stat_tracker : Header content is image/JPEG; Charset=utf-8 payload length is 1212

     

    attached is irule, jpg page size
  • Jon_Strabala_46's avatar
    Jon_Strabala_46
    Icon for Nimbostratus rankNimbostratus
    Jul 02, 2011
    Looking at this post I guess I can do the below (simple basic example)

    Question 1) but what would I have to (or should I) modify it work with a STREAM profile ?

    Question 2) I assume that this does NOT impact the basic built in statistics ?

    Question 3) When I make a new stats profile does it basically just build a "rrd" infrastructure ?

    Question 4) there was alot of discussion in this thread BUT no single best practice was identified "hoolio" could you comment on this ?

    As Scott said (above) I assume I have to first define a profile (Profile->Other-Statistics) and create a new named profile "content_and_len", then check "custum) with following entries:

    Html

    Html_len

    Images

    Images_len

    Css

    Css_len

    Other

    Other_len

    Then associate this Statistics Profile to the virtual server.

    Next I would have to the following iRule and then also associate that with the same virtual server, this iRULE will also tests for unset or missing length. 

    when HTTP_RESPONSE { 
  switch -glob [HTTP::header "Content-type"] { 
    "text/html" { 
      STATS::incr "ContentType" "Html"  
      if { [HTTP::header exists "Content-Length"] } { 
        STATS::incr "ContentType" "Html_len" [HTTP::payload length] 
      }
    } 
    "image/*" { 
      STATS::incr "ContentType" "Images"  
      if { [HTTP::header exists "Content-Length"] } { 
        STATS::incr "ContentType" "Images_len" [HTTP::payload length] 
      }
    }

    "text/css" { 
      STATS::incr "ContentType" "Css"  
      if { [HTTP::header exists "Content-Length"] } { 
        STATS::incr "ContentType" "Css_len" [HTTP::payload length] 
      }
    } 
    "default" { 
      STATS::incr "ContentType" "Other" 
      if { [HTTP::header exists "Content-Length"] } { 
        STATS::incr "ContentType" "Other_len" [HTTP::payload length] 
      }
    } 
  }
}

    To view the results, I would then go to Statistics -> Local Traffic -> Profiles/Statistics

    I have not yet pumped any traffic yet but I do see the zero counts.

  • Jon_Strabala_46's avatar
    Jon_Strabala_46
    Icon for Nimbostratus rankNimbostratus
    Jul 02, 2011
    What if I don't care to have the above show up in the GUI F5 interface but want to merely keep a count for every five minutes and write it out to a file and zero the stats

     

     

    (before anyone goes crazy on me - In my use case I will actually track the content_and_len by some key like USER-AGENT)

     

     

    I imagine I use could use the version 10.1.* "TABLE" syntax for my hashes and counters along with "AFTER" and dump my results to file (preferably named ua_use_yyyymmdd.out). If I can't use iRULEs to write to a file I guess I could use a rolling log.

     

     

    rolling log: http://devcentral.f5.com/Tutorials/TechTips/tabid/63/articleType/ArticleView/articleId/1084377/Writing-to-and-rotating-custom-log-files.aspx

     

     

    Here is some simple code using AFTER (I did not use the "table" syntax) to show what I am having an issue getting my head around

     

     

       when RULE_INIT {
      set reqs_last_600_sec 0
      set new_req_count 0
      after 600000 –periodic 

         set reqs_last_600_sec $new_user_count
         set new_req_count 0

          HOW DO I SAVE the reqs_last_600_sec value to a file ?
          Is it better to send it to a rollign log file or is it
          better to make an OOB request to some other server like
          a memcached server.
          }
   }
   when HTTP_REQUEST { incr new_req_count }

     

     

     

    Any help or directions would be appreciated.

     

  • Kevin_Davies_40's avatar
    Kevin_Davies_40
    Icon for Nacreous rankNacreous
    Jul 04, 2011
    when HTTP_RESPONSE {

     

    if { [HTTP::header exists "Content-Length"] } {

     

    if { [catch {STATS::incr "ContentLength" [HTTP::header "Content-Type"] [HTTP::header "Content-Length"]} } {

     

    log local0.info "Counting content-type \" [HTTP::header "Content-Type"]\" failed with an error"

     

    }

     

    }

     

    }

     

     

    Seems simpler.
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Jul 05, 2011
    Question 1) but what would I have to (or should I) modify it work with a STREAM profile ?

     

     

    What do you want to use the stream profile for with this iRule? Do you mean that you're rechunking the response content? If so, I don't think that would affect the Content-Length header. I think HTTP_RESPONSE would be triggered before the HTTP payload is modified anyhow.

     

     

    Question 2) I assume that this does NOT impact the basic built in statistics ?

     

     

    Correct.

     

     

    Question 3) When I make a new stats profile does it basically just build a "rrd" infrastructure ?

     

     

    No, I think the stats data is stored in an internal memory structure. I don't think it's ever written to disk.

     

     

    Question 4) there was alot of discussion in this thread BUT no single best practice was identified "hoolio" could you comment on this ?

     

     

    Best practice for what part? As far as avoiding a runtime error if the Content-Length header isn't there? I'd just add a check to see if [HTTP::header Content-Length] ne "" rather than using catch.

     

     

    Aaron