Forum Discussion

Ken_B_50116's avatar
Ken_B_50116
Icon for Cirrostratus rankCirrostratus
Nov 25, 2015
Solved

Pool member not found error for a node that doesn't exist in the config

I am running LTM 11.4.1 HF7. I am getting this in the LTM log 4 to 6 times a minute:

Nov 25 09:22:02 F5-LTM-TMC-01 err mcpd[7840]: 01020036:3: The requested pool member (/WebTeam/pool_webextorg /Common/198.22.xx.xx http) was not found.

I have xx'd out part of the IP address for privacy.

The problem is that this 198.22.xx.xx address does not exist in the pool, or anywhere I can see in the configuration.

The pool mentioned in the error looks like this:

[root@F5-LTM-TMC-01:Active:Changes Pending] config  tmsh list ltm pool /WebTeam/pool_webextorg
ltm pool /WebTeam/pool_webextorg {
    load-balancing-mode least-connections-member
    members {
        /WebTeam/172.22.0.108:80 {
            address 172.22.0.108
            session monitor-enabled
            state up
        }
        /WebTeam/172.22.0.110:80 {
            address 172.22.0.110
            session monitor-enabled
            state up
        }
        /WebTeam/172.22.0.125:80 {
            address 172.22.0.125
            session monitor-enabled
            state up
        }
    }
    monitor /WebTeam/http_xxxxxxx
    partition WebTeam
}

If I search through all of the config for the offending address, nothing is found:

[root@F5-LTM-TMC-01:Active:Changes Pending] config  tmsh list | grep 198.22.xx.xx
[root@F5-LTM-TMC-01:Active:Changes Pending] config 

What could be triggering this frequent error for an IP address that doesn't even exist? Could something be continually querying the LTM for this pool member?

  • This was finally resolved after 4 months with F5 tech support. Ultimately F5 had to reproduce the issue on a test system. F5 was able to tell me the name of the user account associated with the error being logged. I contacted that person. They were running a .net app using iControl that was monitoring pools statuses. The config in the app had some bad data and was triggering the error messages.

     

4 Replies

  • nathe's avatar
    nathe
    Icon for Cirrocumulus rankCirrocumulus

    What about if you check bigip.conf too?

    grep 198.22.xx.xx bigip.conf?

    You're currently searching the running config, not the stored config.

  • I ended up contacting F5 support for this issue. The advice so far is based around article SOL13030. The theory is (from the article), "The text configuration files (such as /config/bigip.conf) and the running configuration loaded in the mcpd process from the binary configuration image are out of synchronization" and, "In cases where the system is unable to load the binary configuration, or where the binary image is no longer accurate, you can correct the situation by forcibly reloading the configuration."

     

    The claimed fix is a simple command line operation and a reboot of each unit in the cluster. I am planning to carry this out later this week and will follow up here with the results.

     

    • Ken_B_50116's avatar
      Ken_B_50116
      Icon for Cirrostratus rankCirrostratus
      I finally got this resolved. Thanks for F5 support for seeing this through. The problem was caused by someone in our internal development department who was running a .net app using iControl to query the LTMs to get pool status. Their app was querying for objects that did not exist. F5 support was able to tell me the user ID of the person involved with the odd errors, and talking to that person turned up the app and misconfigured query.
  • This was finally resolved after 4 months with F5 tech support. Ultimately F5 had to reproduce the issue on a test system. F5 was able to tell me the name of the user account associated with the error being logged. I contacted that person. They were running a .net app using iControl that was monitoring pools statuses. The config in the app had some bad data and was triggering the error messages.