Forum Discussion

someguy's avatar
someguy
Icon for Nimbostratus rankNimbostratus
Nov 03, 2016

Why do calls to REST API fail?

We are seeing intermittent failures for calls to the rest api on 11.5.1 HF6. The TCP handshake completes, client sends SSL Client Hello, BigIP sends ACK, and no further packets are seen. Is this related to iControl/REST connection limits?

In the icrd log I see:

Nov  1 12:45:34 mylb notice icrd_child[31618]: 31618,31633,             RestRequestSender,   INFO,Connection idle too long fd:9 cached.
Nov  1 12:45:42 mylb notice icrd:  8195,13830,             RestRequestSender,   INFO,Connection idle too long fd:13 cached.
Nov  1 12:50:34 mylb notice icrd_child[31655]: 31655,31670,             RestRequestSender,   INFO,Connection idle too long fd:10 cached.
Nov  1 12:50:52 mylb notice icrd:  8195,13830,             RestRequestSender,   INFO,Connection idle too long fd:13 cached.
Nov  1 12:55:33 mylb notice icrd_child[31717]: 31717,31732,             RestRequestSender,   INFO,Connection idle too long fd:10 cached.
Nov  1 12:56:02 mylb notice icrd:  8195,13830,             RestRequestSender,   INFO,Connection idle too long fd:13 cached.
Nov  1 12:56:08 mylb notice icrd:  8195,13817,                    RestServer,   INFO,Connection idle too long fd:11
Nov  3 13:05:33 mylb notice icrd_child[31936]: 31936,31951,             RestRequestSender,   INFO,Connection idle too long fd:10 cached.
Nov  3 13:05:51 mylb notice icrd:  8195,13830,             RestRequestSender,   INFO,Connection idle too long fd:13 cached.
Nov  3 13:06:08 mylb notice icrd:  8195,13817,                    RestServer,   INFO,Connection idle too long fd:11

The restjavad log doesn't register anything during the failure time, and lsof shows the below.

[myuser@mylb:Active:Changes Pending] ~  lsof -nPu apache | grep -E "(TCP|COMMAND)"
COMMAND    PID   USER   FD   TYPE             DEVICE     SIZE      NODE NAME
httpd    12579 apache    3u  IPv6         1272168705                 TCP *:80 (LISTEN)
httpd    12579 apache    5u  IPv6         1272168710                 TCP *:443 (LISTEN)
httpd    12579 apache   18u  IPv4         1554156729                 TCP 127.0.0.1:52743->127.0.0.1:8100 (CLOSE_WAIT)
...several iterations of the above...
httpd    16246 apache    3u  IPv6         1272168705                 TCP *:80 (LISTEN)
httpd    16246 apache    5u  IPv6         1272168710                 TCP *:443 (LISTEN)
httpd    16246 apache   16u  IPv6         1555163788                 TCP 10.1.1.1:443->10.10.10.10:50000 (ESTABLISHED)
httpd    16246 apache   18u  IPv4         1554419261                 TCP 127.0.0.1:43108->127.0.0.1:8100 (CLOSE_WAIT)
api
application delivery
f5 icontrol
iControl
iControlREST

3 Replies

Sort By
  • someguy's avatar
    someguy
    Icon for Nimbostratus rankNimbostratus
    Nov 03, 2016

    Mentioned TCPDUMP:

    2016-11-01 13:15:34.129681 IP 10.10.10.10.33672 > 10.1.1.1.443: Flags [S], seq 629635135, win 5840, options [mss 1460,sackOK,TS val 528901005 ecr 0,nop,wscale 7], length 0
2016-11-01 13:15:34.129707 IP 10.1.1.1.443 > 10.10.10.10.33672: Flags [S.], seq 2779043284, ack 629635136, win 14480, options [mss 1460,sackOK,TS val 2033090823 ecr 528901005,nop,wscale 7], length 0
2016-11-01 13:15:34.143747 IP 10.10.10.10.33672 > 10.1.1.1.443: Flags [.], ack 1, win 46, options [nop,nop,TS val 528901016 ecr 2033090823], length 0
2016-11-01 13:15:34.152022 IP 10.10.10.10.33672 > 10.1.1.1.443: Flags [P.], seq 1:106, ack 1, win 46, options [nop,nop,TS val 528901025 ecr 2033090823], length 105
2016-11-01 13:15:34.152039 IP 10.1.1.1.443 > 10.10.10.10.33672: Flags [.], ack 106, win 114, options [nop,nop,TS val 2033090845 ecr 528901025], length 0
2016-11-01 13:16:04.130234 IP 10.10.10.10.33672 > 10.1.1.1.443: Flags [F.], seq 106, ack 1, win 46, options [nop,nop,TS val 528931002 ecr 2033090845], length 0
2016-11-01 13:16:04.169465 IP 10.1.1.1.443 > 10.10.10.10.33672: Flags [.], ack 107, win 114, options [nop,nop,TS val 2033120863 ecr 528931002], length 0

    And the corresponding output from SSLDump:

    New TCP connection 1: 10.10.10.10(33672) <-> 10.1.1.1(443)
1 1  0.0223 (0.0223)  C>S SSLv2 compatible client hello
  Version 3.1
  cipher suites
  TLS_DHE_RSA_WITH_AES_256_CBC_SHA
  TLS_DHE_DSS_WITH_AES_256_CBC_SHA
  TLS_RSA_WITH_AES_256_CBC_SHA
  TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
  TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
  TLS_RSA_WITH_3DES_EDE_CBC_SHA
  TLS_DHE_RSA_WITH_AES_128_CBC_SHA
  TLS_DHE_DSS_WITH_AES_128_CBC_SHA
  TLS_RSA_WITH_AES_128_CBC_SHA
  TLS_DHE_RSA_WITH_DES_CBC_SHA
  TLS_DHE_DSS_WITH_DES_CBC_SHA
  TLS_RSA_WITH_DES_CBC_SHA
1    30.0005 (29.9782)  C>S  TCP FIN
  • JRahm's avatar
    JRahm
    Icon for Admin rankAdmin
    Nov 04, 2016

    yes, open a case. Soft issues like this are difficult to track down without access to information you wouldn't want to share in a public forum. What kind of load are you experiencing to the rest interface? Have you tried increasing the resources provisioned against the management process?