Forum Discussion

Davidfisher_345's avatar
Davidfisher_345
Icon for Altocumulus rankAltocumulus
Jul 18, 2018

Bad gateway error 502 on statistic pages

Hey Guys, We are getting this on one of our boxes. It only shows up in the statistics tabs in virtual servers and pools for now. I tried restarting the httpd and tomcat but nothing. Thi...
BIG-IP
big-ip ltm
LTM
security
Samir_Jha_52506's avatar
Samir_Jha_52506
Icon for Noctilucent rankNoctilucent
Jul 20, 2018

As per my knowledge error page and log is no where related. I suspect, 2 or more user login same time and execute their task. httpd service restart will solve issue but i can see its Active box. So take the action accordingly.

 

Check the tmm & httpd service log for more information. If still face issue open support case.

 

Note: Full box restart will solve issue.

 

  • Davidfisher_345's avatar
    Davidfisher_345
    Icon for Altocumulus rankAltocumulus
    Jul 20, 2018

    As I mentioned, the first thing we did was restart HTTPD and TOMCAT, but that did not fix it.

    Now f5 support has answered :

    The issue with the 503 errors observed while trying to view stats in the GUI might be related to the java process out of memory errors seen in the tomcat logs:

/var/log/tomcat/catalina.out
Exception in thread "Thread-2" java.lang.OutOfMemoryError: Java heap space Jul 15, 2018 2:55:33 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
SEVERE: Caught exception (java.lang.OutOfMemoryError: Java heap space) executing org.apache.jk.common.ChannelSocket$SocketAcceptor@189f044, terminating thread
    at java.util.Arrays.copyOf(Arrays.java:2367)
    at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
    at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
    at java.lang.StringBuilder.append(StringBuilder.java:132)
    at com.f5.mcp.io.ConnectionManager.run(ConnectionManager.java:267)
    at java.lang.Thread.run(Thread.java:744)
com.f5.mcp.io.McpQueryException: Empty reply message

Please follow the steps in the solution links below to provision additional memory to the tomcat process. I'd suggest you start with an extra 20MB and verify if the issue persists

K25554628: The tomcat process may experience an out of memory exception, become unresponsive, and fail to automatically restart
https://support.f5.com/csp/article/K25554628

K9719: Error Message: java.lang.OutOfMemoryError
https://support.f5.com/csp/article/K9719

If the issue persists after increasing the memory, we can continue troubleshooting from there

    This is a BIGIP 10000 box and has 2000+ virtual servers, is the load causing this problem?

    Can this cause any problems to the production traffic at all?