Forum Discussion

Kevin_M_182964's avatar
Kevin_M_182964
Icon for Nimbostratus rankNimbostratus
May 21, 2015

Is my persist record getting ignored?

I posted a similar question a few months back, but I still seem to be having trouble with using an iRule to manage persistence between clients and servers. Basically, the situation is the servers provide a transaction ID that I use as the key for a uie persist record. I seem to be capturing the transaction ID correctly, and I am able to add and delete the entries from the persist records table, but for some reason the LTM is occasionally forwarding some of the requests to the wrong server. Here is the iRule that creates the persist record:

when HTTP_RESPONSE {
   make sure Location and transaction exist
  if {[HTTP::header exists Location] && [string match *transactions* [HTTP::header Location]]} {
     get everything right of /transactions/ up to next /
    set Transaction_ID [findstr [HTTP::header Location] "/transactions/" 14 /]
     if persist record doesn't exist, create one
    if {[persist lookup uie $Transaction_ID] equals ""} {
      persist add uie $Transaction_ID 300
      log local0. "[IP::client_addr]:[TCP::client_port], transaction ID: $Transaction_ID, created persist record: [persist lookup uie $Transaction_ID]"
    } else {
      log local0. "[IP::client_addr]:[TCP::client_port], transaction ID: $Transaction_ID, existing persist record: [persist lookup uie $Transaction_ID]"
    }
  }
}

And here is the iRule that watches for subsequent requests to either use the persist entry or to remove it:

when HTTP_REQUEST {
  if { [HTTP::uri] contains "/transactions/" } {
     get field from /transactions/ to next /
    set Transaction_ID [findstr [HTTP::uri] "/transactions/" 14 "/"]
     check if persistence record exists for this transaction
    if {[persist lookup uie $Transaction_ID] equals ""} {
      log local0. "[IP::client_addr]:[TCP::client_port] [HTTP::uri], $Transaction_ID persist record does not exist"
    } else {
       check if transaction is closed by PUT with ContentLength=0
      if {[HTTP::method] equals "PUT" && [HTTP::header Content-Length] equals "0"} {
        log local0. "End of transaction $Transaction_ID method: [HTTP::method], length: [HTTP::header Content-Length]"
        persist delete uie $Transaction_ID
      } else {
         use existing persist record
        log local0. "[IP::client_addr]:[TCP::client_port] [HTTP::uri], transactionID: $Transaction_ID, existing persist record: [persist lookup uie $Transaction_ID]"
        persist uie $Transaction_ID
      }
    }
  }
}

I can see in the LTM log that the records are being added and deleted correctly, but a tcpdump on the servers shows that sometimes the requests are being sent to the wrong server. Am I using the persist statement correctly? Does "persist uie $Transaction_ID" tell the LTM to use this persist record? Is there a log I can check to see why it might be forwarding the request to the wrong server?

Thanks, Kevin

application delivery
devops
iRules
LTM
persistence

7 Replies

Sort By
  • amolari's avatar
    amolari
    Icon for Cirrus rankCirrus
    May 21, 2015

    It should be display using this command format in your iRule

    [persist lookup uie "$Transaction_ID any"]

  • Kevin_M_182964's avatar
    Kevin_M_182964
    Icon for Nimbostratus rankNimbostratus
    May 22, 2015

    I think there are a couple of things going on here. First, I've figured out that I can simply combine the two event handlers into a single iRule, and assign that iRule to a universal persist profile. I also figured out how to create a rule that shows which member of the pool has been selected (LB_SELECTED and LB::server addr). I can see that my persist records are being honored correctly, which is a relief.

     

    The problem is that I have two servers in the pool, and they use an incrementing sequence beginning with 1 for the transaction ID. The transaction IDs occasionally coincide, which is why I put "persist delete uie $Transaction_ID" in the request check. However, once the persist record is deleted, the LB want to use the other server in the pool and sends the final request of the transaction to the wrong server. I've found that if I don't include the delete statement, the transaction IDs of the two servers overlap and an existing persist record is reused. This sometimes causes the LB to send the requests to the wrong server.

     

    Is there a way to delete the record after the decision to forward based on the persist record has been made? All I have to go on for the final ACK back from the server is a 200 response (i.e. no transaction ID to tie it to), so I have to act based on the request.

     

    Thanks.

     

  • nitass_89166's avatar
    nitass_89166
    Icon for Noctilucent rankNoctilucent
    May 24, 2015

    i might be lost but does deleting persistence record after forwarding really help because request with the same transaction id (from another server) may come before (since transaction id is independent on each server)?

     

    just wondering whether persistence method that does not utilize persistence table, such as carp, helps.

     

    sol11362: Overview of the CARP hash algorithm

     

    https://support.f5.com/kb/en-us/solutions/public/11000/300/sol11362.html

     

    • nitass_89166's avatar
      nitass_89166
      Icon for Noctilucent rankNoctilucent
      May 24, 2015
      sorry, please ignore my comment. i was wrong. carp persistence method won't help.
  • nitass's avatar
    nitass
    Icon for Employee rankEmployee
    May 24, 2015

    i might be lost but does deleting persistence record after forwarding really help because request with the same transaction id (from another server) may come before (since transaction id is independent on each server)?

     

    just wondering whether persistence method that does not utilize persistence table, such as carp, helps.

     

    sol11362: Overview of the CARP hash algorithm

     

    https://support.f5.com/kb/en-us/solutions/public/11000/300/sol11362.html

     

    • nitass's avatar
      nitass
      Icon for Employee rankEmployee
      May 24, 2015
      sorry, please ignore my comment. i was wrong. carp persistence method won't help.
  • Kevin_M_182964's avatar
    Kevin_M_182964
    Icon for Nimbostratus rankNimbostratus
    May 26, 2015

    I think I've figured out how to do it. Before I delete the persist record, I grab the node that the packet would have been sent to. After deleting the record, I force the packet to be sent to that node:

       check if transaction is closed by PUT with ContentLength=0
  if {[HTTP::method] equals "PUT" && [HTTP::header Content-Length] equals "0"} {
    log local0. "[IP::client_addr]:[TCP::client_port], transactionID: $Transaction_ID method: [HTTP::method], length: [HTTP::header Content-Length], deleting existing persist record: [persist lookup uie $Transaction_ID]"
     delete the persist record, then send packet to correct server
    set server_addr [persist lookup uie $Transaction_ID node]
    set server_port [persist lookup uie $Transaction_ID port]
     delete the record
    persist delete uie $Transaction_ID
     send to server
    log local0. "[IP::client_addr]:[TCP::client_port], forwarding transactionID: $Transaction_ID to $server_addr:$server_port"
    node $server_addr $server_port

    This seems to do the trick. I think there's still some risk that the same transaction ID could be used by both servers at the same time, but I don't see anyway around that.