Forum Discussion

silvajai_01_131's avatar
silvajai_01_131
Icon for Nimbostratus rankNimbostratus
Nov 05, 2013

IP/Pools Go Down Unexpectedly Due to No Members Present Message In the Logs

Greetings,

 

I been running 10K's in active/standby HA mode for a few months now and lately am starting to see an issue with unexpected VIPS/Pools being marked down due to no members present. The outage occurs for around 3 seconds or so and then the VIPS/Pools come backup. I am running BIG-IP 11.3.0 Build 3117.0 Hotfix HF5 on the boxes.

 

The problem only seems to occur for pool members that are on the same vlan, in this case vlan24. I have 27 VIPs configured on the boxes and the only impacted VIPS/Pools are the ones that have members in vlan24. The vlan is reachable via the inside trunk, which is made up of (2) 10 Gbe members. All of the inside reachable vlans are carried over this trunk. The external connection is made up of a (2) 1 Gbe member trunk.

 

I have checked the switches that the trunks are connected to and can't find any indication of an issue with arp or the cam forwarding table. I checked the LTM logs and can't find anything going on there either. The health monitors used is the same monitor configuration for all of the pools, the pools that stay up and the impacted pools as well.

 

The response that I got back from TAC is that they don't see any issues in the QKview logs that I provided. I was planning on taking a tcpdump, but the problems happens to quick before I an initiated.

 

I am curious if anyone else on here are see any similar issues running the 11.3.0 code.

 

application delivery
availability
design
management
performance

5 Replies

Sort By
  • What_Lies_Bene1's avatar
    What_Lies_Bene1
    Icon for Cirrostratus rankCirrostratus
    Nov 05, 2013

    How do you know the Pools go down please? Is the F5 the gateway for servers in VLAN24? Are the other VLANs the same L2/L3-wise?

     

    You may be able to use an iRule with the LB_FAILED event to get some logs.

     

  • silvajai_01_131's avatar
    silvajai_01_131
    Icon for Nimbostratus rankNimbostratus
    Nov 05, 2013

    I saw the below message in the log of the offending LTM and my applications using the associated VIPs start to complain.

     

    /var/log/ltm.1.gz:Nov 4 08:21:57 ltm02a err tmm9[15335]: 01010028:3: No members available for pool /Common/am-solr_pool_9983 /var/log/ltm.1.gz:Nov 4 08:21:57 ltm02a err tmm10[15335]: 01010028:3: No members available for pool /Common/ad-solr2ut_pool_9983 /var/log/ltm.1.gz:Nov 4 08:21:57 ltm02a err tmm10[15335]: 01010028:3: No members available for pool /Common/am-solr_pool /var/log/ltm.1.gz:Nov 4 08:21:57 ltm02a err tmm10[15335]: 01010028:3: No members available for pool /Common/am-solr_pool_9983 /var/log/ltm.1.gz:Nov 4 08:21:57 ltm02a err tmm11[15335]: 01010028:3: No members available for pool /Common/ad-solr2ut_pool_9983 /var/log/ltm.1.gz:Nov 4 08:21:57 ltm02a err tmm11[15335]: 01010028:3: No members available for pool /Common/am-solr_pool /var/log/ltm.1.gz:Nov 4 08:21:57 ltm02a err tmm11[15335]: 01010028:3: No members available for pool /Common/am-solr_pool_9983 /var/log/ltm.1.gz:Nov 4 08:22:06 ltm02a err tmm[15335]: 01010028:3: No members available for pool /Common/am-statdb_pool /var/log/ltm.1.gz:Nov 4 08:22:06 ltm02a err tmm[15335]: 01010028:3: No members available for pool /Common/ac-lgdb-w_pool /var/log/ltm.1.gz:Nov 4 08:22:06 ltm02a err tmm6[15335]: 01010028:3: No members available for pool /Common/am-statdb_pool /var/log/ltm.1.gz:Nov 4 08:22:06 ltm02a err tmm6[15335]: 01010028:3: No members available for pool /Common/ac-lgdb-w_pool /var/log/ltm.1.gz:Nov 4 08:22:06 ltm02a err tmm8[15335]: 01010028:3: No members available for pool /Common/am-statdb_pool /var/log/ltm.1.gz:Nov 4 08:22:06 ltm02a err tmm8[15335]: 01010028:3: No members available for pool /Common/ac-lgdb-w_pool /var/log/ltm.1.gz:Nov 4 08:22:06 ltm02a err tmm9[15335]: 01010028:3: No members available for pool /Common/am-statdb_pool /var/log/ltm.1.gz:Nov 4 08:22:06 ltm02a err tmm9[15335]: 01010028:3: No members available for pool /Common/ac-lgdb-w_pool /var/log/ltm.1.gz:Nov 4 08:22:06 ltm02a err tmm10[15335]: 01010028:3: No members available for pool /Common/am-statdb_pool /var/log/ltm.1.gz:Nov 4 08:22:06 ltm02a err tmm10[15335]: 01010028:3: No members available for pool /Common/ac-lgdb-w_pool /var/log/ltm.1.gz:Nov 4 08:22:06 ltm02a err tmm1[15335]: 01010028:3: No members available for pool /Common/am-statdb_pool /var/log/ltm.1.gz:Nov 4 08:22:06 ltm02a err tmm1[15335]: 01010028:3: No members available for pool /Common/ac-lgdb-w_pool /var/log/ltm.1.gz:Nov 4 08:22:06 ltm02a err tmm2[15335]: 01010028:3: No members available for pool /Common/am-statdb_pool /var/log/ltm.1.gz:Nov 4 08:22:06 ltm02a err tmm2[15335]: 01010028:3: No members available for pool /Common/ac-lgdb-w_pool /var/log/ltm.1.gz:Nov 4 08:22:06 ltm02a err tmm3[15335]: 01010028:3: No members available for pool /Common/am-statdb_pool

     

  • What_Lies_Bene1's avatar
    What_Lies_Bene1
    Icon for Cirrostratus rankCirrostratus
    Nov 06, 2013

    Ah, sorry, when you said you'd checked the logs and couldn't see anything I thought you meant no entries at all.

     

    So, a few questions;

     

    1) Are there no log entries about Pool Members going down or up?

     

    2) What monitor is being used at the Pool Level? Is there more than one?

     

    3) What monitor is being used at the Node level?

     

    4) Have you looked at the server logs too?

     

    5) Is there any pattern to these events? Same time each day, every X hours etc. etc.

     

  • silvajai_01_131's avatar
    silvajai_01_131
    Icon for Nimbostratus rankNimbostratus
    Feb 24, 2014

    This information is a bit late but I found the root cause to this problem. As it turns out, someone duplicated the self-ip address of the floater on a box belonging to the impacted vlan. The address is was resolved and the problem corrected.