Forum Discussion

David_Remington's avatar
David_Remington
Icon for Employee rankEmployee
Mar 13, 2008

dilemma: run slow or crash on load

So I have a rule I am writing that has to parse a very large (6000+) entry class file for every URI that traverses the virtual server. Because of the constraints on possibilities for a match I basically have to loop through the class and do a string match (not regexp). So the rule is dog slow... I mean, really really slow. I mean, a single iteration uses about 3% of the CPU giving me a scorching 40 requests per second, give or take. So it is non-viable.

So I decided to take the class file and parse it into an array so that I could just loop through a small portion of the overall class file based on the first couple characters in the requested uri. During runtime this makes a huge difference. Instead of 3% the rule drops to less than .05% which is well within our performance requirements.

Unforutnately, the RULE_INIT event does so much work building the array that tmm's heartbeat times out when you do a 'b load' and the box fails over. It also seems to suspend all network activity during the load, which causes monitors to time out, etc.

After the tmm restart, the rule loads fine and runs great.

Any ideas how I can do the following faster or do something to give tmm enough breathing room to not lock up? Is there a way to break up the work so that tmm can increment it's heartbeat or does the entire event have to be parsed before tmm can do some other task?


   when RULE_INIT {
     set debug 1
     if { $debug > 0 } {log local0.notice "rule initializing"}
     set list_map [lsort -unique -decreasing -ascii $::cl_map_v10]
     array set ::map { }
     array set ::keylist { }
     if { $debug > 0 } {log local0.notice "starting to load array"}
     set idxelement { a b c d e f g h i j k l m n o p q r s t u v w x y z 0 1 2 3 4 5 6 7 8 9 }
     foreach idx $idxelement {
        if { $debug > 1 } { log local0.notice "idx: $idx" }
        foreach idx2 $idxelement {
           if { $debug > 1 } { log local0.notice "   idx2: $idx2" }
           set n 0
           foreach element $list_map {
              if { $element starts_with "/${idx}${idx2}" } {
                 incr n
                 set ::map(${idx}${idx2},${n}) $element
              }
           }
           set ::listsize(${idx}${idx2}) $n
        }
     }
     if { $debug > 0 } {log local0.notice "map(ab,1): $::map(ab,1)"}
     if { $debug > 0 } {log local0.notice "done load array"}
     if { $debug > 0 } {log local0.notice "rule initialized"}
   }

One suggestion I have thought of/been suggested is to make a bunch of class files instead of using the array, but unfortunately the above rule creates what would basically be 36x36 class files. Even if I pared that down a bit I'd at least need several hundred class files which will be a huge pain for the customer to manage.

Thoughts? Suggestions? Condolences?

(ps - if there is tcl magic to do the "set idxelement" line without typing the alphanumerics I'd love to hear it although I think it is not directly related to my core problem.. all I could find were extended tcl commands that don't seem to be supported in irules.)

Thanks
application delivery
dev
devops
iRules

3 Replies

Sort By
  • hooleylist's avatar
    hooleylist
    Icon for Cirrostratus rankCirrostratus
    Mar 13, 2008
    On your last minor question...

    In shell, you could use seq to print out a range of characters. The best I could think of is a for loop with scan:

    
   for {set i 0} {$i<=122} {incr i} {
      if {($i >= 48 && $i <= 57) || ($i >= 97 && $i <=122)}{
         lappend chars [format %c $i]
         log local0. "$i: [format %c $i]"
      }
   }
   log local0. "\$chars: $chars, length: [llength $chars]"

    Aaron
  • spark_86682's avatar
    spark_86682
    Historic F5 Account
    Mar 20, 2008
    Basically, you're doing this in a very inefficient way. Specifically, you're converting your class list into an array by iterating over your class list 1,296 (=36*36) times. You can do the same thing by only iterating over it once:

     

    
  when RULE_INIT {
     set debug 1
     if { $debug > 0 } {log local0.notice "rule initializing"}
     set list_map [lsort -unique -decreasing -ascii $::cl_map_v10]
     array unset ::map
     array unset ::listsize
     if { $debug > 0 } {log local0.notice "starting to load array"}
     foreach element $list_map {
        set idx1 [string index $element 1]
        set idx2 [string index $element 2]
        set count 1
        catch { set count $::listsize(${idx1}${idx2}) }
        set ::map(${idx1}${idx2},${count}) $element
        incr count
        set ::listsize(${idx1}${idx2}) $count
     }
     if { $debug > 0 } {log local0.notice "map(ab,1): $::map(ab,1)"}
     if { $debug > 0 } {log local0.notice "done load array"}
     if { $debug > 0 } {log local0.notice "rule initialized"}
   }

     

     

    This code takes the fist two characters of each element of your class, looks up which index into the "map" array is the next one, stuffs it in there, and increments and stores the new count for that index. The "catch" line is just a clever way of dealing with the fact that if you try to access an unset array key, Tcl gives an error instead of returning 0 or something. So if we haven't set a particular "listsize" key yet, the "count" variable keeps its value of 1.

     

     

    A few other notes: I've made your "array set"s into "array unset"s, since I think that that's what you really meant to do. I got rid of "keylist" since you never actually use it. I did not check if the values stored in "listsize" match your original rule, or are off by 1. If they are off, I think you can just do "set count 0" instead of 1, and move the "incr count" line just after the "catch" line. Basically, this should get you close, and you can make further changes to suit your application.