Scatter Plotting Response Times With iRules And Google Charts

We’ve presented several articles in the past utilizing the Google Charts API with iControl and iRules. A few of these include

I realized we have yet to present an example of a Scatter Chart, so I set to put together one that did just that.

The Chart

To get your attention, I’ll change the format of my tech tip a little bit by presenting the output first, and then I’ll go over the iRule I used to generate it.

The data collected for this example is calculated for a page request to an application page. The HTTP response code (200, 302, 404, etc), the size of the response payload (page size), and the duration (time it took for the backend server to process and return the page). The output of the iRule consists of two charts and a few control commands.

The first chart represents the response time for a page to be processed by the backend server against the response code from the server. The size of the dots in the chart represent the size in bytes of the response. And the second chart plots the response time on the X-Axis with the size of the payload.

The iRule

Now we’ll dig into the iRule used to store the data and build the report and embedded charts.

Data Collection

The data collection component of this iRule is fairly straightforward. in the HTTP_REQUEST event for incoming page requests, we store the system time in milliseconds into the t1 variable. When the response comes back from the server, we store the current time in the t2 variable. A difference is then calculated representing the duration of the request. The HTTP status code and the response size, indicated in the “Content-Length” response header determined as well. The three of these values are combined together with a semicolon separator and then inserted into the data subtable. I also create a subtable to store the various response codes for a reason you’ll see below.

All the stored data is inserted with a default 180 second timeout, forcing the data to be purged from the session table after 3 minutes. This ensures that the data table won’t grow too big and we can stay within the 2000 character URL limit in the resulting graphs.

   1: when HTTP_REQUEST {

   2:   set t1 [clock clicks -milliseconds];

   3: }

4:

   5: when HTTP_RESPONSE {

   6:   set t2 [clock clicks -milliseconds];

   7:   set duration [expr {$t2 - $t1}];

   8:   set code [HTTP::status];

   9:   set clen [HTTP::header Content-Length];

10:

  11:   # Only monitor application traffic

  12:   if { [HTTP::header Content-Type] starts_with "text/html" } {

13:

  14:     # Only allow $MAX_KEYS in table.

15:

  16:     set data "${code}:${clen}:${duration}";

  17:     if { [table incr -subtable $DATA_TABLE -mustexist $data] eq ""} {

  18:       table set -subtable $DATA_TABLE $data 1 $TIMEOUT $LIFETIME;

  19:     }

20:

  21:     if { [table incr -subtable $CODES_TABLE -mustexist $code] eq "" } {

  22:       table set -subtable $CODES_TABLE $code 1 $TIMEOUT $LIFETIME;

  23:     }

  24:   }

  25: }

Global Settings

I’ve got several global variables. These most likely should either be hard coded or stored in global scope in the RULE_INIT method, but I included them in the HTTP_REQUEST event for ease of reading in the article. The size of the charts can be customized as well as the data timeout and the subtable names.

   1: set CHART_WIDTH 400

   2: set CHART_HEIGHT 325

   3: set MAX_KEYS 300;

   4: set TIMEOUT 180;

   5: set LIFETIME indefinite;

   6: set DATA_TABLE "scatterchart";

   7: set CODES_TABLE "scatterchartcodes";

Handling Report Requests

The main page request for this iRule application, is processed when the user requests the “/scatterchart” url on the virtual that this iRule is applied to. A simple HTML page is generated containing images with links back to the iRule for the charts described below. A few other commands are added with links to refresh the page, reset the data, and to popup a HTML table with all the raw data. Each of those commands will be described below.

   1: switch [HTTP::uri] {

   2:   "/scatterchart" {

   3:     set content "<html><head><title>iRule Scatter Chart</title></head><body><center>";

   4:     append content "<img src='/scatterchart/t_by_c' width=${CHART_WIDTH} height='${CHART_HEIGHT}'>";

   5:     append content "<img src='/scatterchart/d_by_s' width=${CHART_WIDTH} height='${CHART_HEIGHT}'>";

   6:     append content "<br/><a href='/scatterchart'>RELOAD</a>";

   7:     append content "&nbsp;|&nbsp;<a href='/scatterchart/reset'>RESET</a>";

   8:     append content "&nbsp;|&nbsp;<a href='/scatterchart/data' target='dtwindow'>DATA</a>";

   9:     append content "</center></html>";

  10:     HTTP::respond 200 Content $content;

  11:   }

  12: }

The Time By Status Code Chart

The first chart plots the duration by the status code. This section of code stores the data series for the chart in the d_x (X-Value), d_y (Y-Value), and d_s (Size of dot) variables. The records in the data subtable are looped over with the foreach command. I have a sanity check with the CUR_KEY variable to make sure the URL doesn’t go beyond the 2000 character limit. The status code, length, and duration are extracted from the record with the getfield command and the various data variables are appended to.

In the data collection portion of the iRule, I created a subtable to contain the various status codes. With this data, then build custom Y-Axis labels for the status codes returned. Since we are plotting by the code and the code is a number that equals the Y-height on the chart, the chart label (chxl) and point (chxp) values are the same.

The URL to the Google Chart is then generated from the data and a HTTP::redirect is issued to the browser to the actual URL at Google’s servers.

   1: switch [HTTP::uri] {

   2:   "/scatterchart/t_by_c" {

   3:     # Time by response code

   4:     set d_x "";

   5:     set d_y "";

   6:     set d_s "";

7:

   8:     set CUR_KEY 0;

9:

  10:     foreach key [table keys -subtable $DATA_TABLE] {

  11:       incr CUR_KEY;

  12:       if { $CUR_KEY > $MAX_KEYS } { break; }

13:

  14:       set     code [getfield $key ":" 1];

  15:       set      len [getfield $key ":" 2];

  16:       set duration [getfield $key ":" 3];

17:

  18:       set len [expr $len / 75];

  19:       if { $len < 300 } { set len 300; }

20:

  21:       if { $d_x != "" } { append d_x ","; }

  22:       append d_x $duration

  23:       if { $d_y != "" } { append d_y ","; }

  24:       append d_y $code;

  25:       if { $d_s != "" } { append d_s ","; }

  26:       append d_s $len;

  27:     }

28:

  29:     set CHXL "";

  30:     set CHXP "";

31:

  32:     # Build Y-Axis labels with the HTTP status codes

  33:     foreach code [table keys -subtable $CODES_TABLE] {

  34:       if { $CHXL != "" } { append CHXL "|" }

  35:       append CHXL $code;

  36:       if { $CHXP != "" } { append CHXP "," }

  37:       append CHXP $code;

  38:     }

39:

  40:     set XMIN 0;

  41:     set XMAX 1000;

  42:     set YMIN 0;

  43:     set YMAX 500;

  44:     set SMAX 1000;

45:

  46:     set chartUrl "http://1.chart.apis.google.com/chart?cht=s";

  47:     append chartUrl "&chtt=Web+Requests+Time+By+Status+Code";

  48:     append chartUrl "&chxt=x,y&chs=${CHART_WIDTH}x${CHART_HEIGHT}";

  49:     append chartUrl "&chco=F1F12F|989866|000000|224499|AA0033";

  50:     append chartUrl "&chxr=0,${XMIN},${XMAX}|1,${YMIN},${YMAX}";

  51:     append chartUrl "&chds=0,${XMAX},0,${YMAX},0,${SMAX}";

  52:     append chartUrl "&chxl=1:${CHXL}";

  53:     append chartUrl "&chxp=1,${CHXP}";

  54:     append chartUrl "&chg=0,10";

  55:     append chartUrl "&chd=t:${d_x}|${d_y}|${d_s}"

56:

  57:     HTTP::redirect $chartUrl;

  58:   }

  59: }

The Duration By Size Chart

The generation for this chart is very similar to the previous one, except that we’ve changed the data values a bit. This time the X-Axis represents the duration and the Y-Axis the size of the response. I’ve added code to grow the X and Y Axis maximum values on the chart if the data exceeds the current maximum values. The records in the session table are iterated upon as in the previous section of code and the chart URL is generated and redirected to the client.

   1: switch [HTTP::uri] {

   2:   "/scatterchart/d_by_s" {

   3:     # Duration by Size

   4:     set d_x "";

   5:     set d_y "";

   6:     set d_s "";

   7:     set rows "";

8:

   9:     set XMIN 0;

  10:     set XMAX 1000;

  11:     set YMIN 0;

  12:     set YMAX 500;

  13:     set SMAX 100;

14:

  15:     set CUR_KEY 0;

16:

  17:     foreach key [table keys -subtable $DATA_TABLE] {

  18:       incr CUR_KEY;

  19:       if { $CUR_KEY > $MAX_KEYS } { break; }

20:

  21:       set     code [getfield $key ":" 1];

  22:       set      len [getfield $key ":" 2];

  23:       set duration [getfield $key ":" 3];

24:

  25:       if { $d_x != "" } { append d_x ","; }

  26:       append d_x $duration

  27:       if { $duration > $XMAX } { set XMAX $duration; }

28:

  29:       if { $d_y != "" } { append d_y ","; }

  30:       append d_y $len;

  31:       if { $len > $YMAX } { set YMAX $len; }

32:

  33:       if { $d_s != "" } { append d_s ","; }

  34:       append d_s 100;

  35:     }

36:

  37:     set chartUrl "http://0.chart.apis.google.com/chart?cht=s";

  38:     append chartUrl "&chtt=Web+Requests+Duration+By+Size";

  39:     append chartUrl "&chxt=x,y&chs=${CHART_WIDTH}x${CHART_HEIGHT}";

  40:     append chartUrl "&chco=F1F12F|989866|000000|224499|AA0033";

  41:     append chartUrl "&chxr=0,${XMIN},${XMAX}|1,${YMIN},${YMAX}";

  42:     append chartUrl "&chds=0,${XMAX},0,${YMAX},0,${SMAX}";

  43:     append chartUrl "&chg=0,10";

  44:     append chartUrl "&chd=t:${d_x}|${d_y}|${d_s}"

45:

  46:     HTTP::redirect $chartUrl;

  47:   }

  48: }

A Raw Data Table

For debugging purposes, I threw in a command to generate a HTML table containing all the records in the table. A 3-column table is then returned to the browser with the Code, Length, and Duration values in the data table.

   1: switch [HTTP::uri] {

   2:   "/scatterchart/data" {

   3:     set content "<html><head><title>Scatter Chart raw data</title></head><body><center>";

   4:     append content "<a href='/scatterchart/data'>RELOAD</a><br/>";

   5:     append content "<table border='1'><tr><th>Code</th><th>Length</th><th>Duration (ms)</th></tr>";

6:

   7:     foreach key [table keys -subtable $DATA_TABLE] {

   8:       set     code [getfield $key ":" 1];

   9:       set      len [getfield $key ":" 2];

  10:       set duration [getfield $key ":" 3];

  11:       append content "<tr><td>${code}</td><td>${len}</td><td>${duration}</td></tr>";

  12:     }

13:

  14:     append content "</table><br/><a href='/scatterchart/data'>RELOAD</a></center></body></html>";

15:

  16:     HTTP::respond 200 content $content;

  17:   }

  18: }

Data Reset

If you are just too impatient to wait 3 minutes for the records to reset, or you are running this on a highly active site and you want to clear out all the data at any given time, I’ve included a command to reset the subtables. Passing in the URI “/scatterchart/reset” will erase all entries in the subtables and let you start clean.

   1: switch [HTTP::uri] {

   2:   "/scatterchart/reset" {

   3:     table delete -subtable $DATA_TABLE -all;

   4:     table delete -subtable $CODES_TABLE -all;

   5:     HTTP::redirect "/scatterchart";

   6:   }

   7: }

Get The Full Source

The source for this iRule can be found in the iRule CodeShare under the topic ScatterChartingResponseTimes.