LB dropping ICMP packets regularly.
I am linux server administrator, and have noticed that for about 4 hours each day (same time each day), a simple ping of the default router results in packet loss, ranging from 10-100 percent. This packet loss can be seen on all servers going though this LB device. The packet loss appears exactly the same time on all servers, and disappears the same time on all servers as well. Unsurprisingly, pinging a server through the LB also show packet loss.
I am told by our network team that the default router for this particular server, is actually just a logical device on the LB device. They claim they cannot see any issue, and the LB device is forwarding on the ping ICMP requests to the destination server, however tcpdump shows the destination machine never receiving some of the ICMP requests.
During this time, a tcpdump from the server, also reveals many ARP who-is request's coming from the default router for itself (so the WHO and the TELL is the ip of the router). In addition, these requests occur multiple times a minute.
As soon as the packet loss stops happening, the ARP requests cease completely. As soon as the packet loss starts occurring again, the ARP requests return.
As a server guy, I am not familiar with the F5 LB device, but it does not make sense to me that it would make so many of these ARP requests each minute, especially for itself. Does anyone have any idea of what might be happening here?