Forum Discussion

Neonsun_116864's avatar
Neonsun_116864
Icon for Nimbostratus rankNimbostratus
Nov 20, 2017

HTTP health check flapping - potential offenders?

We are seeing random pool members drop out due to TCP errors on an HTTP health check (basic GET against a static HTML page) for a specific pool. Seems to be load related but we're not exactly sure where the resource starvation would be happening. Hosts are not reporting any tcp issues (not sure they would).

 

2200S on 12.1.2 HF1, AFM enabled (Cisco ASA at perimeter). About 35K active connections, 400 new conns/second and just shy of 1k HTTP req/s, ~100 SSL reqs/s. IIS backend (VMware hosted), four servers in pool. Normal app response time of 50-100ms per request. Netstat on host shows 650-700 active connections on port 80, out of which about 20 are from the LB IP at any given time, with TIME_WAIT status. Receive-side scaling disabled on hosts.

 

Monitor request runs the following: "GET /lbcheck.htm HTTP/1.1\r\nHost:\r\nConnection: close\r\n\r\n" Interval 5 secs, timeout 16 secs. Returns about 80 bytes ("Server is up" message)

 

Error from log:

 

Pool /Common/Pool-80 member /Common/10.103.xx.xxx:80 monitor status down. [ /Common/Isalive: down; last error: /Common/Isalive: Tcp read: Connection refused; Unable to connect; No successful responses received before deadline.; Response Code: 200 (OK) @2017/11/14 04:59:45. ] [ was up for 0hr:0min:13sec ]

 

Any suggestions from the community on how to troubleshoot further would be greatly appreciated! (Switches, VMware virtual network, vNIC settings, Windows TCP tweaks, etc.) :)

 

1 Reply

  • First increase your monitor timings, from Interval 5 secs to 10 secs, timeout 16 secs to 31 secs see if that helps.

     

    Second get a tcpdump if you can see what is happening to the connection. It is possible the connection is not being closed correctly so web server (or something on the network) stops allowing any more active connections. Have seen this on older Linux Redhat servers that just run out of available TCP ports from the F5's Self IPs.