F5 is not flushing expired TCP connections
Hi All,
Last days I noticed that due to unknown factor BigIP F5 load balancer is not flushing expired TCP connections on one VIP. Connections stuck in CLOSE-WAIT/LAST-ACK and FIN-WAIT/CLOSING state. As a result connection table grows to the point where there are no free ports available to handle legitimate traffic for new client’s connections. Customer experienced issue when number of connection which stuck in CLOSE-WAIT/LAST-ACK and FIN-WAIT/CLOSING is around 90k. What is strange in my opinion becasue we have two pool members and also use snat pool on this VIP. Snat pool consist of two ip addresses. In my opinion customer should be affected due to port exhoustion when the number of stuck connections will be around 258000: here is my calculation: TCP protocol allows to use 2^16-1=65535 ports minus reserved ports which gives 64511 available port numbers. We use two ip addresses in snat pool. One ip address is used towards two pool members theoretical number of free ports from one ip address should be 2x64511= 129022. Beacuse we have two addresses on snat it should be 2x129022 = 258044. Therefore I have two major questions to resolve. 1) Why connections stuck in CLOSE-WAIT/LAST-ACK and FIN-WAIT/CLOSING state for a long time and are not cleaned 2) Why customer experienced issue when value is similar to 90k.
Moreover due to unknown factor mentioned issue are not only related to one customer but also have affection on other customers (different context, same loadbalancer). On the affected VIP we use standard tcp as Protocol Profile (Client) and no Protocol Profile (Server).
some current statistic from affected vip:
tmsh show ltm virtual ... details
Ltm::TCP Profile: tcp
Connections Open 194 Current in CLOSE-WAIT/LAST-ACK 37.6KCurrent in FIN-WAIT/CLOSING 37.6K Current in TIME-WAIT 2 Accepted 8.4M Not Accepted 0 Established 7.7M Failed 377 Expired 458.2K Abandoned 132
If someone had met with something similar and will be able to help I will be very grateful!