Forum Discussion

CCVLo's avatar
CCVLo
Icon for Nimbostratus rankNimbostratus
Mar 26, 2013

MBLB for HSM Messages

 

Hi all !

 

 

I´ve been trying to load balance messages destinated to a Thales HSM. Since HSM client uses a single connection to communicate to HSM I´ll use MBLB to achieve that. The HSM Message has a header, a message length indicator and the message itself. I´ve applied a MBLB profife to my virtual server and created the following iRule:

 

 

when CLIENT_ACCEPTED {

 

 

set client_closed 0

 

 

TCP::collect

 

log local0. "Client Connected !"

 

 

}

 

 

periodically checks if the client connection closed, if so, also closes associated server connection.

 

when CLIENT_CLOSED {

 

 

set client_closed 1

 

 

}

 

 

Triggered when a connection has been established with the target node

 

when SERVER_CONNECTED {

 

 

after 1000 -periodic if {$client_closed} {TCP::close}

 

 

}

 

 

when CLIENT_DATA {

 

scan [TCP::payload] %c%c head rlen

 

 

log local0. "Header = $head -- Length = $rlen"

 

if {($head == 0)} {

 

 

if {[TCP::payload length] < [expr {2 + $rlen}]} {

 

TCP::collect

 

return

 

}

 

}

 

 

TCP::release [expr {$rlen + 2}]

 

TCP::notify request

 

TCP::collect

 

}

 

 

I´ve been using a client to send messages to the virtual server. Checking the traffic with TCPDUMP I can see the messages balancing to members of my pool and they´ve been returning to BigIP but the client application only receives the message back when it is originated from the first server selected to handle the message load balancing. Messages from the second server never come back to client.

 

 

Follows the behavior on my client application:

 

 

Connected!

 

Send: 0000BA1234F843005571922

 

Received: 0000BB0058647

 

Send: 0000BA1234F843005571922

 

Send: 0000BA1234F843005571922

 

Received: 0000BB0058647

 

Send: 0000BA1234F843005571922

 

Send: 0000BA1234F843005571922

 

Received: 0000BB0058647

 

 

VS Config:

 

 

virtual HSM-MBLB {

 

snat automap

 

pool LB-POOL

 

destination 192.168.1.132:1500

 

ip protocol tcp

 

rules hsm-mblb

 

profiles {

 

mymblb {}

 

tcp {}

 

}

 

}

 

 

pool LB-POOL {

 

members {

 

192.168.2.10:1500 {}

 

192.168.2.11:1500 {}

 

}

 

}

 

 

I´ve been using a BigIP LTM Virtual Edition to test this scenario.

 

 

[root@BigIP-VE-001:Active] config bigpipe version

 

Kernel:

 

Linux 2.6.18-164.2.1.el5.1.0.f5app

 

Package:

 

BIG-IP Version 10.1.0 3341.1084

 

Final Edition

 

 

 

Thanks,

 

Carlos Covolo

 

 

6 Replies

  • Hi Carlos,

    I am not sure what could be the reason.

    maybe you can try something like below to see if it helps.

    (modify SERVER_CONNECTED and adding SERVER_DATA)

     
    
    when SERVER_CONNECTED {
        after ...
        TCP::collect
    }
    when SERVER_DATA {
        TCP::release [TCP::payload length]
        TCP::notify response
        TCP::collect
    }
    
    

    perhaps, you may add log message to see if SERVER_DATA is actually invoked

    I think you might also try a different version (10.0 or 11.x)

    HTH 🙂

    Nat
  • CCVLo's avatar
    CCVLo
    Icon for Nimbostratus rankNimbostratus
    Hi Nat,

     

     

    You rock ! I was configured a SERVER_DATA entry but haven´t put on it the TCP::release and TCP::notify response statements.

     

     

    Now I can received data from both servers:

     

     

    Conected!

     

    Send: 0000BA1234F843005571922

     

    Received: 0000BB0058647

     

    Send: 0000BA1234F843005571922

     

    Received: 0000BB0058647

     

    Send: 0000BA1234F843005571922

     

    Received: 0000BB0058647

     

     

    and checking with TCPDUMP I can see the data flow to both servers in the server POOL.

     

     

    Thanks a lot !!!

     

     

    My best regards,

     

    Carlos Covolo
  • CCVLo's avatar
    CCVLo
    Icon for Nimbostratus rankNimbostratus

    Hi Rodrigo,

    Follows the final iRule:

    rule MBLB-HSM {
    
    when CLIENT_ACCEPTED {
    
    set client_closed 0
    TCP::collect
    log local0. "Client Connected !"
    
    }
    
     periodically checks if the client connection closed, if so, also closes associated server connection.
    when CLIENT_CLOSED {
    
    set client_closed 1
    
    }
    
    Triggered when a connection has been established with the target node
    when SERVER_CONNECTED {
    
    after 1000 -periodic if {$client_closed} {TCP::close}
    TCP::collect
    
    }
    
    when CLIENT_DATA {   
      binary scan [TCP::payload] cc head rlen
    
      log local0. "Header = $head -- Length = $rlen"
      if {($head == 0)} {
    
    if {[TCP::payload length] < [expr {2 + $rlen}]} {
                TCP::collect expr {2 + $rlen - [TCP::payload length]}
                return
                }
      }
    
      TCP::release [expr {$rlen + 2}]
      TCP::notify request
      TCP::collect  
    }
    
    when SERVER_DATA {
    
       TCP::release [TCP::payload length]
       TCP::notify response
       TCP::collect
    
       log local0. "Received server respose !"
    
    }
    }
    

    I´ve also created a health monitor for HSM:

    monitor HSM_PING {
       defaults from external
       interval 10
       timeout 31
       args "5"
       run "/usr/bin/monitors/hsm.pl"
    }
    
    
     /usr/bin/monitors/hsm.pl
    
    !/usr/bin/perl
    
    use strict;
    use IO::Socket;
    
    my $IP = $ARGV[0];
    $IP =~ s/::ffff://g;
    
    my $PORT = $ARGV[1];
    
    my $TIMEOUT = $ARGV[2];
    
    my $HSMPING="\x{00}\x{08}0000NO00";
    
    my $res=hsm_probe ($HSMPING);
    
    if ($res =~ m/0000NP00/) {
    
        print "UP" . "\n";
        exit 0;
    }
    
    exit 255;
    
    sub hsm_probe {
    
        my $line = $_[0];
        my $data;
        chomp ($line);
        my $HSM_port        = $PORT;
        my $HSM_host        = $IP;
    
        eval {
    
        local $SIG{ALRM} = sub { die 'Timed Out'; };
        alarm $TIMEOUT;
            my $sock = IO::Socket::INET->new(
                PeerAddr => $HSM_host,
                PeerPort => $HSM_port,
                Proto    => 'tcp',
            )
                || die "Couldn't connect to $HSM_host:$HSM_port: $!";
    
            $sock->autoflush(1);
            print $sock $line;
            $sock->recv($data, 26);
            close $sock;
        alarm 0;
            return $data;
            1;
        } or do { alarm 0; return "ERROR"; }
    
    }
    

    My best regards, Carlos Covolo