Append to list stored in table key

What will be the values new_val?

How will you use it?

Will these keys expire? (you do not have set timeout)

you may create one subtable per IP address and define one key per val...

table add -subtable ip-[IP::client_addr] $c_val 1 3600

foreach c_val [table keys -notouch -subtable ip-[IP::client_addr] ] {
    do what you want
}

Hi Piotr,

there is a better way to append a list entry to an existing table...

[table append -subtable ip " [IP::client_addr]"

Note: A list is "SPACE" seperated string. So lappend var_name newentry is basicaly the same as append var_name " newentry"

BTW: But you may still want to consolidate the subtables into classic tables as explained by Stanislas.

Cheers, Kai

Well, so here is my goal, probably what I posted was a bit misleading:

When http req check if given cookie exists and created entry in sub/table using as a key session id value from cookie and as value list with client IP address.

Then for each new connection check if IP in the list is the same, if not add new IP as next entry in the list.

timeout and lifetime for table key is irrelevant here - it will be set depending on situation.

So code could be:

when HTTP_REQUEST {

     set s_id [HTTP::cookie value $my_cookie]
     if { [table lookup $s_id] eq "" } {
        table set $s_id [IP::client_addr]

     } else {

        set cur_v [table lookup $s_id]
        if { [lsearch $cur_v [IP::client_addr]] == -1  } {
           lappend cur_v [IP::client_addr]

           table set $s_id $cur_v
        }
     }

Just wonder if it can be done in more optimized way.

Piotr

Your irule is to store for each request the list of IP addresses of each cookie... what will you do of this list?

If you want to optimize, add a timeout for your table,

It is better to use subtables instead of main table. with subtables, you can list all keys of the subtable and use irule to list add / remove keys of a subtable.

With the main table, you won't be able to list all keys...

Hmm, my goal was to store each unique ip for given session id. So I can then analyze if different ip's were used during one http session - maybe I missed something but it's only storing IP for new session id (no key = session id in table yet) and then check if given ip is already stored in a key - if not append it to the list of ips stored as key value. At least that was my goal but maybe my logic has some flaw?

Piotr

Yes, you store for each session all IP addresses. but you won't be able to list all sessions stored. you may know the session ID to read list of IPs.

The best solution to keep informations readable and manageable is:

• create one subtable for each session
  • create one key per IP address with value containing hit count
  • force keys timeout (even if you force timeout to 1 week)
• create one subtable sessions
  • one key per session

the irule can be:

when HTTP_REQUEST {
     set s_id [HTTP::cookie value $my_cookie]
      New session ID
     if { [table lookup -subtable $session $s_id] eq "" } {
        table add -subtable $session $s_id 1 $timeout
        table add -subtable $s_id [IP::client_addr] 1 $timeout
         Existing session ID / New IP
     } elseif { [table lookup -subtable $s_id [IP::client_addr]] eq "" } {
        table add -subtable $s_id [IP::client_addr] 1 $timeout
         Existing session ID Existing IP
     } else { table incr -subtable $s_id [IP::client_addr] $timeout}
 }

Stanislas,

That part is irrelevant. I will have separate sub/table to store all session ids to be able later on to retrieve all ips per session.

My main concern here was to optimize adding new ips as a list to table key value.

Based on Kai advice I think final solution is:

if {[table lookup $sess_id] eq ""} {

    table set $sess_id [IP::client_addr]
} else {

    if { [lsearch [table lookup $sess_id]] == -1} {

        table append $sess_id " [IP::client_addr]"

    }
}

What do you think?

Piotr

Hi Piotr,

when doing performance optimization of a certain code block of you should at first define an expected traffic pattern.

If you expect just a few request from each individual IP address, then the "optimal code" would look somewhat different, compared to a code which is optimized to support long living session - where you may expect hundreds or thousands request coming from the same IP or even a single TCP connection.

Once you have figured out your specific traffic mix, you should start to optimize the code so that this specific traffic patter can be handled as fast as possible. The remaining traffic which is not matching the defined pattern may be executed within certain (sometimes deeply nested and most likely CPU intensive) exceptions.

For your scenario I would like to assume, that you need to optimize the code for a few long living connections without having that many IP address changes. So the mission would be to check if the session ID and the current IP address is already known in a single request, without doing math or enforce restriction on each individual request.

if { [table incr -mustexist -subtable "IPs_[HTTP::cookie value SESSION_COOKIE]" [IP::client_addr]] ne "" } then {

    
     Here comes the fast data path for most of the request
    
     Outcome:
     Verified that Session ID and IP already exist and it didn't exceed the connection limits.
     Collected additional statistics how often the session id has requested content using this IP 
    

    return

} else {

    
     Here comes the slow data path for only a handful requests

     Crawl the HTTP::cookie(s) a second time but this time store the result 
    If { [set temp(session_id) [HTTP::cookie value SESSION_COOKIE]] eq ""] } then {
         Cookie does not exist!
         Do whatever is needed...
        return
    }

     Crawl the table once and store the result
    set temp(session_id_counter) [table keys -count -subtable "IPs_$temp(session_id)"]

    Evaluate the results and perform action
    if { $temp(session_id_counter) eq 0 } then {

         Session ID is not known yet since no IP is stored.

        table add -subtable "master_table" $temp(session_id) "1" indef indef
        table add -subtable "ips_$temp(session_id)" [IP::client_addr] "1" indef indef

    } elseif { $temp(session_id_counter) < $static::conf(session_id_limit) } then {

        Session ID has a new IP and has not exceeded its limit
        table add -subtable "ips_$temp(session_id)" [IP::client_addr] "1" indef indef

    } else {

        Session ID has a new IP but has exceeded its max IP limit
        drop

    }
    unset -nocomplain temp
}

BTW: Try to avoid variable usage as much as possible. Only use them if consecutive usage would save CPU cycles. When needing them try to use arrays (e.g. arrays(key_name)), they do have a lot of benefits compared to classic variables (e.g. faster unset for a group of variables, less usage of [eval] to build the name on the fly, etc.)

Cheers, Kai

Kai,

I am a bit confused - is your example code related to adding new ips to key or to my needs from another post?

I wonder what exactly statement like $temp(session_id_counter) is doing - sorry, but I am really starting with iRules and last time I did some programming was 20+ years ago - so this skills are rather rusty :-(

I am aware that using variables is not good when performance is main goal, I even learned that variable name length is quite important to save resources.

I was reading all best practice articles about iRules that are available on DevCentral but it's not so easy to sort out all this knowledge right away.

To be honest I was not able to find some hard rules which data is cached by TMM and could be safely retrieved without performance ht and variable usage (like HTTP::host) and which are not, as well as (but maybe I missed this one) when is break even point for placing something in var instead of referencing that via command - I mean how many times is too much and it's better to use var - or maybe there is no limit here?

Is drop indeed working? I saw plenty of discussions suggesting reject as working.

Last one what is the reason to use unset -nocomplain temp?

Thanks for help, I really appreciate your time and effort here.

Piotr

Hi Piotr,

Sorry didn't want to confuse you... 😉

The code above is just a sample code based on some of your previous questions (aka. master table), but also contains the handling of IPs per session_id table entries. The important part of this code is, that you could check a lot of things in a single [table] query to speed up the processing of most of your requests and that you could hide the complex stuff (e.g. counting limits, creation new session_id and master_table updates or new IP_addr entries) as an exception to this initial [table] query. The task of the exception code path is then to analyze why the initial [table] query has failed and then correct this situation, by inserting additional session_id or IP_addr, so that the very next request from the same session_id and IP_addr could be handeled using the fast single [table] query.

Basically you should try the fastest method first (because it should work most of the time), then catch the possible errors/exceptions/empty results followed by an analyses of whats was going wrong and a fix to make sure the next request won't fail again. The opposite approach would be to perform a rather complex error handling on each request to make sure that the used [table] query never fails...

Regarding $temp(session_id_counter). This is not a statement, it an [array] variable that can be used as an alternative to classic variables. The handling would look like this....

set temp(var1) "value1" set temp(var2) "value2" lappend temp(var2) "value3" set temp(var_index) [array names temp -glob "value*"] unset temp(var3) unset temp etc.

And yes, it is somewhat hard to get a feeling of which command is faster than the other. The available information is unfortunately distributed across the internet and spanning a lot articles, different wiki sites and forum posts. But even if you found some information then it may be already outdated... So in the end you may want to measure the performance of each code block by yourself, using Event timing on or [clock clicks] timestamps with or without some additional [while] or [for] loops. Over the time you'll get the experience which code block should be used for a given scenario...

BTW: [drop] is working. But you can't send a [HTTP::respond] and [drop] at the same time. BTW2: The -nocomplain option could be skipped in this case. But I tend to use this option in most cases...

Cheers, Kai