Forum Discussion

JoshuaMessenger's avatar
JoshuaMessenger
Icon for Nimbostratus rankNimbostratus
Feb 27, 2015

How would I use a Table as a global cross tmm variable to be CMP compliant?

I've been asked to handle load-balancing and automatic fail over between an active / passive CIFS / NFS storage system. I have the user_conf setup to monitor the ltm logs and trigger bash scripts on primary pool failure. The issue I currently have is: If i use TCL variables or Global Variables because the irule needs to modify the variable to prevent continuous file system fail overs the VIP is being demoted from CMP causing only 1 core to be used.

Below is the irule. Any help on using tables or sessions would be appreciated as all cores need to be used as Day 1 is 2 million file transactions and 1 year from now its supposed to scale to 50 million.

when RULE_INIT {
    set static::trackingVar 0
    log local0. " Global Variable Initialized $::static::trackingVar"
}
when CLIENT_ACCEPTED {
  if { [active_members "PrimaryStorage"] >= 2 } {
    if {$static::trackingVar == 1} {
        log local0. "Kill All Connections!!!"
        set static::trackingVar 0
    }
    pool PrimaryStorage
  } else {
   if {$static::trackingVar == 0} {
        set static::trackingVar 1
        log local0. "$static::trackingVar"
        log local0. "Kill All  Connections!!!"
        log local0. "Kill file Cluster Sync!!!"
        log local0. "Enable Read Write On Secondary Cluster!!!"
    }
    pool FailoverStorage
  }
}

9 Replies

  • Replace your

    static::
    with a class lookup from:

    CMP Compatibility: Global Variables

    Not CMP compatible:

    set val "$::www_redir https://${host}:${port}[HTTP::uri]\r\n$::www_opts"

    CMP compatible (v9.x):

    set val "[findclass "www_redir" global_vars " "] https://${host}:${port}[HTTP::uri]\r\n[findclass "www_opts" global_vars " "]"

    To support this lookup, you will need to add a new class w/ the appropriate values for each key:

    class global_vars { “www_redir xxxxxx” “www_opts xxxxxx” }

    Note: In 9.4.4 and higher, when referencing the class with the findclass or matchclass commands you should not use
    ::
    or
    $:: prefix
  • If I use a class how would I edit the class once its instantiated? With a class in my scenario I would have something similar to:

    class Trigger_var {
        StorageFarm := 0
    }
    

    If there is a way to either change the value of "StorageFarm" or add additional elements to the class such as "StorageFarm := 1" then I can look at the content as the trigger to force the failover.

  • According to this article you can and since your class is not static it should be ok?:

     

    iRules 101 - 08 - Classes

     

    Can I modify a class real-time with my iRule?

     

    Technically, yes. Once the configuration is loaded into memory, you can technically modify a class with TCL's list commands. Doing so, however, not only converts the data in the class from an efficient, hashed format into a simple list format, thereby slowing down queries; but the changes made are also not permanent, as they cannot be written back to the file that stores the class data. This makes the changes effective only until TMM is restarted. In general, there is usually another way of structuring your code to avoid this that would be preferred.

     

  • Hi Joshua,

     

    in case you want to follow a table based approach you can use the following code in an iRule to modify the value of the table key "storage_pool" to the name of pool you want to balance to:

     

    when CLIENT_ACCEPTED {
        set storage_pool "poolx"
        table set storage_switch $storage_pool indef indef
    }
    

    On the virtual server balancing the incoming connection the following iRule will lookup the table and select the pool based on the table key´s value:

     

    when CLIENT_ACCEPTED {
        log local0. "pool selector: [table lookup storage_switch]"
        if { [table lookup storage_switch] equals "poolx" } {
            log local0. "using poolx"
            pool [table lookup storage_switch]
        } else {
            log local0. "using [table lookup storage_switch]"
            pool [table lookup storage_switch]
        }
    }
    

    Table´s content is available globally.

     

    I´m wondering why you are not handling the failover via putting pool members into forced offline mode?

     

    In case all storage device are in the same pool you can trigger the switchover by administratively changing the pool member state. Please check as well the "Action on service down" parameter (pool configuration settings).

     

    Thanks, Stephan

     

  • I think there is a miss understanding in what I'm been asked to accomplish. I understand the pool selection process can be accomplished by standard F5 configuration. What I need to accomplish is:

     

    1.) Client starts NFS or CIFS connection to VIP

     

    2.) VIP selects primary_pool and sends traffic to that pool. 3.) If primary_pool is down:

     

    3.a) Send kill command for all F5 sessions associated with VIP

     

    3.b) Have F5 send command to Secondary storage pool master to kill backend sync jobs

     

    3.c) wait 5 seconds

     

    3.d) Activate write on secondary storage cluster

     

    4.) Send all connections to secondary cluster.

     

    5.) On recovery of primary cluster manual fail back is desired.

     

    The variable I'm using is a flag so globally the traffic across all sessions only have steps 3.a - 3.d performed once on a fail over. Note: Fail over only occurs if the primary storage pool has <2 pool members.

     

    Everything functions but its not optimized. When I use static TCL variables or Global Variables the iRule is demoted from the CMP meaning only 1 of 8 cores is used for these transactions.

     

    • StephanManthey's avatar
      StephanManthey
      Icon for MVP rankMVP
      Hi Joshua, your current version of iRule will not interrupt existing connections. Existing connections can be closed by marking poolmembers as "down" (requires setting "Action of service down" in pool config to "Reject"). (And of course be manually deleting from connection table from CLI.) Poolmembers can be marked down by monitor, administratively or by iRule. In my sample iRule I tried to answer your initial question regarding a variable (represented by a table value) which is shared and updated across all TMMs. Thanks, Stephan Thanks, Stephan
    • JoshuaMessenger's avatar
      JoshuaMessenger
      Icon for Nimbostratus rankNimbostratus
      partially correct. The irule writes to the log and user_alert.conf monitors the log for text. The file looks like: alert TerminateAllConnections "Kill All Connections!!!" { exec command="tmsh -c 'cd /FCS_STORAGE/ ; delete sys connection'" } alert IsilonClusterSyncStop "Kill Cluster Sync!!!" { exec command="tmsh bash /config/ClusterSyncStop.sh" }
    • StephanManthey's avatar
      StephanManthey
      Icon for MVP rankMVP
      Hi Joshua, haven´t seen your comment yet. Interesting solution to use alertd to delete connection table entries. And 'tmsh delete sys conn' was what I was referring to when saying "manually deleting from connection table". Using it in the context of an administrative partition is indeed new to me. Will check. Thanks for the update as well. Stephan
  • iRule is now working beautifully.

    when CLIENT_ACCEPTED {
        if {[info exists [table lookup storage_switch]]}{
            set storage_pool "FCS_STORAGE_PRIMARY"
            table set storage_switch $storage_pool indefinite
            log local0. "pool selector: [table lookup storage_switch]"
            log local0. "Does Table Exist - it should now? [info exists [table lookup storage_switch]]"
        }
        if { [active_members "FCS_STORAGE_PRIMARY"] >= 2 } {
            if {[table lookup storage_switch] equals "FCS_STORAGE_FAILOVER"} {
                log local0. "Kill All Connections!!!"
                set storage_pool "FCS_STORAGE_PRIMARY"
                table replace storage_switch $storage_pool indefinite
                log local0. "pool selector: [table lookup storage_switch]"
            }
        pool FCS_STORAGE_PRIMARY
        } else {
        if {[table lookup storage_switch] equals "FCS_STORAGE_PRIMARY"} {
            set storage_pool "FCS_STORAGE_FAILOVER"
            table replace storage_switch $storage_pool indefinite
            log local0. "pool selector: [table lookup storage_switch]"
            log local0. "Kill All Connections!!!"
            log local0. "Kill Cluster Sync!!!"
            }
        pool FCS_STORAGE_FAILOVER
    }
    }