Nimbostratus
NimbostratusThe logs are clean, the pool members did not fail at anytime.
We are using source address persistence but the clients are high number of differing IP addresses so this shouldnt have caused issue. The same setup in alternate DC is working just fine.
The client are NOT behind proxy. In fact the pool members are the proxy servers in this case.
I opened a case with F5 support and they stated this is caused by a glitch in Source Address Persistence going along with Least COnnections method. It happens rarely but no pattern or specific cause provided.
Solution provided by F5 - Do a failover to standby unit - this will close all connections on current device and re-establish new connections on the other device - that will clear up the distribution issue.
Result- It worked as they stated. Now we see evenly balanced traffic accross all 6 pool members after the failover.
Thanks for the replies and assistance!
Nimbostratus***Yes, that was the first recommendation from F5 support, but closing all connections on the VS (and waiting the 180s persistence to timeout) would result in an outage. So failover was the best option.