Decrypting the /var/log/ltm health monitor

Hi Darren~ I'm not sure about the 200 to 500 to 300 issue but this article seems to, at least, explain the error code you are seeing:

SOL10524: Error Message: Per-invocation log rate exceeded - https://support.f5.com/kb/en-us/solutions/public/10000/500/sol10524.html?sr=59308171

and specifically, mcpd[2239]: 01070638:6: Resuming log processing at this invocation; held 22 messages.

Hope that helps?

ps

the error codes themselves are not published, and I'm not sure they'd be terribly helpful to you anyway. Better would be to enable monitor debugging:

tmsh modify sys db bigd.debug value enable

This will give you detailed logs in /var/log/bigdlog

[0][18226] 2016-12-29 19:42:08.333578: ID 12    :(_do_ping): time to ping, now=[1482980316.191702][2016-12-28 20:58:36], status=DOWN [ addr=::ffff:192.168.103.101:80 mon=/Common/http fd=-1 pend=0 conn=0 up_intvl=5 dn_intvl=5 timeout=16 time_until_up=0 immed=0 next_ping=[1482980316.171756][2016-12-28 20:58:36] last_ping=[1482980311.184895][2016-12-28 20:58:31] deadline=[1482980317.397351][2016-12-28 20:58:37] on_service_list=True snd_cnt=23728 rcv_cnt=0 ]
[0][18226] 2016-12-29 19:42:08.333614: ID 12    :(_send_active_service_ping): pinging [ addr=::ffff:192.168.103.101:80 srcaddr=none ]
[0][18226] 2016-12-29 19:42:08.333622: ID 12    :(_connect_to_service): creating new socket (rd0) [ addr=::ffff:192.168.103.101:80 ]
[0][18226] 2016-12-29 19:42:08.333725: ID 12    :(_connect_to_service): connect: Operation now in progress [ addr=::ffff:192.168.103.101:80 srcaddr=::ffff:192.168.103.5%0:58627 ]
[0][18226] 2016-12-29 19:42:08.333755: ID 12    :(_do_ping): post ping, status=DOWN [ addr=::ffff:192.168.103.101:80 mon=/Common/http fd=10 pend=1 conn=1 up_intvl=5 dn_intvl=5 timeout=16 time_until_up=0 immed=0 next_ping=[1482980321.171756][2016-12-28 20:58:41] last_ping=[1482980316.191702][2016-12-28 20:58:36] deadline=[1482980317.397351][2016-12-28 20:58:37] on_service_list=True snd_cnt=23729 rcv_cnt=0 ]
[0][18226] 2016-12-29 19:42:09.845594: ID 12    :(adjust_deadline): from [1482980317.397351][2016-12-28 20:58:37] to [1482980322.703494][2016-12-28 20:58:42] [ addr=::ffff:192.168.103.101:80 mon=/Common/http fd=10 pend=1 conn=1 up_intvl=5 dn_intvl=5 timeout=16 time_until_up=0 immed=0 next_ping=[1482980321.171756][2016-12-28 20:58:41] last_ping=[1482980316.191702][2016-12-28 20:58:36] deadline=[1482980322.703494][2016-12-28 20:58:42] on_service_list=True snd_cnt=23729 rcv_cnt=0 ]
[0][18226] 2016-12-29 19:42:09.845602: ID 12    :(_analyze_pings): visit DOWN, now=[1482980317.703494][2016-12-28 20:58:37] [ addr=::ffff:192.168.103.101:80 mon=/Common/http fd=10 pend=1 conn=1 up_intvl=5 dn_intvl=5 timeout=16 time_until_up=0 immed=0 next_ping=[1482980321.171756][2016-12-28 20:58:41] last_ping=[1482980316.191702][2016-12-28 20:58:36] deadline=[1482980322.703494][2016-12-28 20:58:42] on_service_list=True snd_cnt=23729 rcv_cnt=0 ]

Just make sure you disable the debug when you are done:

tmsh modify sys db bigd.debug value disable

I also have this problem. Please tell me what's the meaning for 'Response Code: 404 (Not Found); Response Code: 500 (Internal Server Error); Response Code: 200 (OK);' in the below alert. Is it received three response at the same monitoring interval? Or it was the record for last three monitoring intervals before DOWN? Thanks.

Apr 10 03:13:24 LB1 notice mcpd[4800]: 01070638:5: Pool /Common/pool_34541 member /Common/172.19.155.11:34541 monitor status down. [ /Common/mon_http1: down, /Common/mon_http2: up; last error: /Common/mon_http1: Tcp read: Connection refused; Unable to connect; Response Code: 404 (Not Found); Response Code: 500 (Internal Server Error); Response Code: 200 (OK); No successful responses received before deadline. @2018/04/10 03:13:24. ] [ was up for 78hrs:30mins:32sec ]

I also have this problem. Please tell me what's the meaning for 'Response Code: 404 (Not Found); Response Code: 500 (Internal Server Error); Response Code: 200 (OK);' in the below alert. Is it received three response at the same monitoring interval? Or it was the record for last three monitoring intervals before DOWN? Thanks.

Apr 10 03:13:24 LB1 notice mcpd[4800]: 01070638:5: Pool /Common/pool_34541 member /Common/172.19.155.11:34541 monitor status down. [ /Common/mon_http1: down, /Common/mon_http2: up; last error: /Common/mon_http1: Tcp read: Connection refused; Unable to connect; Response Code: 404 (Not Found); Response Code: 500 (Internal Server Error); Response Code: 200 (OK); No successful responses received before deadline. @2018/04/10 03:13:24. ] [ was up for 78hrs:30mins:32sec ]

Hi, Is there already some more info about this? We have the same issue for the moment. We were not able yet to take tcpdump and debug logs due to change windows.

Oct 18 14:22:14 slot1/F5 notice mcpd[6750]: 01070638:5: Pool /Common/pool member /Common/pool-member:80 monitor status down. [ /Common/monitor_http: down; last error: /Common/monitor_http: Response Code: 200 (OK); Unable to connect; No successful responses received before deadline.; Response Code: 503 (Service Unavailable); Response Code: 500 (Internal Server Error) @2018/10/18 14:22:14. ] [ was up for 17hrs:53mins:4sec ]

Br, Ben