Forum Discussion

Chris_Phillips's avatar
Chris_Phillips
Icon for Nimbostratus rankNimbostratus
Oct 10, 2013

Best approach for managing HSL connections

Howdy,

 

I've a reasonably high volume site, and various lower volume ones and am looking to stasrt introducing HSL to a number of iRules. This is on 10.2.4 so no http access logging direct from the virtual etc, so am doing this in iRules old skool. The high volume site is moving behind a CDN, which will pool and aggregate connections from clients down into a smaller number of TCP connections, although we're not yet sure exactly how long a CDN connection is likely to last, and how many reuses etc.

 

What's the recommended approach on how to manage HSL::open calls? on a per CLIENT_ACCEPTED basis per iRule? reuse the same HSL across multiple iRules? As part of the CDN implementation we have a generic iRule that setups up a few standard things, like mangling headers they set, so we could create an HSL connection at that "setup" stage and then use it going forwards through the other functional iRules?

 

Currently we log some low level messages through standard log commands, so connections to our central log server are still just one per LTM, but switching to HSL would potentially be opening what.... 500 connections per box? That seems a hell of a lot, but as it's a per client basis on a huge CDN, which therefore will still open a lot of connections, seems potentially very wasteful if we're not utilizing these connections a lot. Would anyone suggest looking at somehow pooling or aggregating these HSL connections to less that one per CLIENT_ACCEPTED? I can imagine maybe modulusing (sp?!) the source port number to pick one of a defined maximum number of connections to log to, opening it if it doesn't already exist?

 

Thanks!

 

4 Replies

  • Hamish's avatar
    Hamish
    Icon for Cirrocumulus rankCirrocumulus

    I always use the HSL::open per iRule. And always have. It's more of a handle than a real opening of a connection. The HSL system will open real connections as necessary and they're pooled so multiple HSL::open handles will utilise fewer connections to the logging servers.

     

    There's a couple of gotchas around that of course.

     

    1. There's no guarantee of order of delivery with the messages.
    2. There's no guarantee two messages send consecutively will actually got across the same pooled 'connection'. WHich means if you have multiple poolmembers in the HSL pool, sending two messages might arrive out of order and on different servers.
    3. There's no way to limit the number of connections used to a poolmember.

    Most issues only crop up on very high rates. I did some work on a POC a few years ago for DB and using 10.2 we ran into most of these features. They're more annoying than anything else, but your logging and reporting infrastructure just needs to be designed around it.

     

    Of course there's now SIDEBAND connections as well. But you've got to be at 11.something for those. And they're not as lightweight and scalable as HSL.

     

    FWIW I use only HSL logging nowadays. Leaves /var/log/ltm available for what it was originally (i.e. logs of members/pools etc up/down etc).

     

    H

     

  • Ahh yes, I have been seeing some scant references to HSL doing it's own management, but barely enough to actually confirm what's going on under the covers. If it pools effectively, then that naturally removes any scaling issues, and you happily throw opens around when you feel like it? All sounds good to me then. So if we're requesting a handle when is that handle freed? on the close of the connection? Or would that only be if if was opened in the equivalently scoped event? Like.... if we open on HTTP_REQUEST, would the back ground events of HTTP_RESPONSE close it? I imagine that it's more relevant that this doesn't actually matter.

     

    Currently we're only logging small volumes to a syslog-ng service, but hopefully this will be laying the groundwork to heavily expand logging to a Splunk farm so just looking to do the groundwork mostly.

     

    Thanks!

     

  • Hamish's avatar
    Hamish
    Icon for Cirrocumulus rankCirrocumulus

    I believe the handle is freed along with everything else when the connection closes.

     

    Normally (Pre 11.4) I define a handle with HSL::open at the start of every event. Mainly because you can't always rely on RULE_INIT to fire... WIth 11.4 you get functions... Mmm....

     

  • Hamish's avatar
    Hamish
    Icon for Cirrocumulus rankCirrocumulus

    Oh.. The handle is literally that apparently as well. So it's really a local variable that just happens to tell HSL::send where to queue a message for sending.

     

    So doesn't matter if you litter your iRule with them. They won't leak, and nothing to 'close' (The HSL code in TMM will rep the pooled connections at some stage... Actually IIRC that was an issue in one of the 10.x releases where HSL wouldn't reap the opened connections when using TCP for logging... But that was fixed. And it wasn't so much of an iRule issue so much as a real bug in the pooling - but you did have to hit it quite hard to get it to break. I think I was in the thousands of messages a second band to get that on an 8800 IIRC).

     

    H