I am having an issue with one of my LTM1500 BIG-IP's where the web based management console has become slow to the point of unusable. Pages can take up to 220 seconds to load and sometimes do not load at all without doing a refresh. Load balanced traffic is unaffected - it is just the management console. I have tried 'ping tests' to the IP of the console and they seem fine. Putty and Winscp sessions are also ok - it is just the web interface. Any ideas?

Which LTM version are you running? Do you see any errors in the log files under /var/log/httpd? Does top show httpd running at high CPU or memory percentages? If you restart httpd (bigstart restart httpd) does the latency decrease? Aaron

I ran into this problem using v9.3.1 HF2. Turned out that there was a memory leak in the mcpd process which manifested itself as slow GUI responses. The problem was identified as CR91313. To diagnose, I ran top and looked at the Size column. This indicates how much memory the each process is consuming. In my case, mcpd was eating something rediculous like 450MB of memory, and increased with each ConfigSync. Look at the memory usage of the processes and see if something is out of whack. You can type "M" to sort by memory usage.

I am on 9.4.3 There are some errors in the httpd log but they are old and not up to date. Retsarting the httpd makes no difference at all.

How about restarting Tomcat (bigstart restart tomcat4) and then httpd (bigstart restart httpd)?

I have restarted Tomcat and it made a bit a difference. Looking at TOP there are all sorts of processes poping up mcpd comes up as 30604 Tmm is 966M Tomcat4 Java comes up with 3 processes of 6000+ occasionaly but they soon dissapear. Just cllicked network map and just have a loading prompt. Took 144 seconds to load.... TOP currenlty is PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND 1584 root RT 0 966M 966M 964M S 1.1 85.9 29:01 0 tmm 11174 root 15 0 1100 1100 784 R 0.5 0.0 0:06 0 top 1614 root 15 0 4132 4132 3208 S 0.1 0.3 2:23 0 bigd 2716 root 15 0 284 284 236 S 0.1 0.0 0:11 0 rrdshim 1 root 15 0 552 552 468 S 0.0 0.0 0:03 0 init 2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0 3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd 4 root 15 0 0 0 0 SW 0.0 0.0 0:46 0 kapmd 5 root 39 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0 8 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush 6 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kswapd 7 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kscand 9 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kupdated 11 root 24 0 0 0 0 SW 0.0 0.0 0:00 0 khubd 16 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald 168 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald 169 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald 170 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald 171 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald 181 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 loop0 189 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald 1377 root 15 0 1356 1356 852 S 0.0 0.1 2:02 0 syslog-ng 1434 root 15 0 1436 1436 1192 S 0.0 0.1 0:00 0 sshd 1456 ntp 15 0 2556 2556 2192 S 0.0 0.2 0:02 0 ntpd 1502 root 15 0 940 940 580 S 0.0 0.0 0:00 0 crond 1525 root 15 0 336 336 276 S 0.0 0.0 0:00 0 runsvdir 1526 root 0 -20 5464 5464 4220 S < 0.0 0.4 0:06 0 overdog 1527 root 22 0 456 456 404 S 0.0 0.0 0:00 0 agetty 1547 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv 1548 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv 1549 root 15 0 308 308 272 S 0.0 0.0 0:00 0 runsv 1550 root 15 0 308 308 272 S 0.0 0.0 0:00 0 runsv 1551 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv 1552 root 15 0 316 316 276 S 0.0 0.0 0:00 0 runsv 1553 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv 1554 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv 1555 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv 1556 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv 1557 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv 1558 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv 1559 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv 1560 root 15 0 308 308 272 S 0.0 0.0 0:00 0 runsv

Web Based Configuration GUI - very very slow

11 Replies

hooleylist
Cirrostratus
Aug 26, 2008
Which LTM version are you running?

Do you see any errors in the log files under /var/log/httpd? Does top show httpd running at high CPU or memory percentages? If you restart httpd (bigstart restart httpd) does the latency decrease?

Aaron
smp_86112
Cirrostratus
Aug 26, 2008
I ran into this problem using v9.3.1 HF2. Turned out that there was a memory leak in the mcpd process which manifested itself as slow GUI responses. The problem was identified as CR91313.

To diagnose, I ran top and looked at the Size column. This indicates how much memory the each process is consuming. In my case, mcpd was eating something rediculous like 450MB of memory, and increased with each ConfigSync. Look at the memory usage of the processes and see if something is out of whack. You can type "M" to sort by memory usage.
jondyke_46152
Nimbostratus
Aug 26, 2008
I am on 9.4.3

There are some errors in the httpd log but they are old and not up to date. Retsarting the httpd makes no difference at all.
smp_86112
Cirrostratus
Aug 26, 2008
How about restarting Tomcat (bigstart restart tomcat4) and then httpd (bigstart restart httpd)?
jondyke_46152
Nimbostratus
Aug 26, 2008
I have restarted Tomcat and it made a bit a difference.

Looking at TOP there are all sorts of processes poping up mcpd comes up as 30604

Tmm is 966M

Tomcat4 Java comes up with 3 processes of 6000+ occasionaly but they soon dissapear.

Just cllicked network map and just have a loading prompt. Took 144 seconds to load....

TOP currenlty is

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND

1584 root RT 0 966M 966M 964M S 1.1 85.9 29:01 0 tmm

11174 root 15 0 1100 1100 784 R 0.5 0.0 0:06 0 top

1614 root 15 0 4132 4132 3208 S 0.1 0.3 2:23 0 bigd

2716 root 15 0 284 284 236 S 0.1 0.0 0:11 0 rrdshim

1 root 15 0 552 552 468 S 0.0 0.0 0:03 0 init

2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0

3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd

4 root 15 0 0 0 0 SW 0.0 0.0 0:46 0 kapmd

5 root 39 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0

8 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush

6 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kswapd

7 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kscand

9 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kupdated

11 root 24 0 0 0 0 SW 0.0 0.0 0:00 0 khubd

16 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald

168 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald

169 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald

170 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald

171 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald

181 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 loop0

189 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald

1377 root 15 0 1356 1356 852 S 0.0 0.1 2:02 0 syslog-ng

1434 root 15 0 1436 1436 1192 S 0.0 0.1 0:00 0 sshd

1456 ntp 15 0 2556 2556 2192 S 0.0 0.2 0:02 0 ntpd

1502 root 15 0 940 940 580 S 0.0 0.0 0:00 0 crond

1525 root 15 0 336 336 276 S 0.0 0.0 0:00 0 runsvdir

1526 root 0 -20 5464 5464 4220 S < 0.0 0.4 0:06 0 overdog

1527 root 22 0 456 456 404 S 0.0 0.0 0:00 0 agetty

1547 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv

1548 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv

1549 root 15 0 308 308 272 S 0.0 0.0 0:00 0 runsv

1550 root 15 0 308 308 272 S 0.0 0.0 0:00 0 runsv

1551 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv

1552 root 15 0 316 316 276 S 0.0 0.0 0:00 0 runsv

1553 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv

1554 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv

1555 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv

1556 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv

1557 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv

1558 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv

1559 root 15 0 312 312 272 S 0.0 0.0 0:00 0 runsv

1560 root 15 0 308 308 272 S 0.0 0.0 0:00 0 runsv
smp_86112
Cirrostratus
Aug 26, 2008
At first tmm memory struck me as too high. But then I checked my LTM, and it's reporting 1.7GB. The performance charts only show TMM at ~350MB. So I'm not sure what to trust.
jondyke_46152
Nimbostratus
Aug 26, 2008
No ASM

Straight BigIP LT1500 with 2GB of RAM V9.4.3 Build 1.4 final

I increased the host memory portion as I though it may have been running out of memory (using b db Provision.extraMB 768)

It made little difference.

Info from a qkview dump

MEMORY STATISTICS --

| (Host) Total = 1.100GB Used = 403.1MB

| (TMM) Total = 956.0MB Used = 23.09MB

I have now started a call with F5 support team as this is very odd behaviour. My standby switch (obviously now my active one) is fine and has not even had the above memory increase.
jondyke_46152
Nimbostratus
Aug 26, 2008
I think TOP shows tmm memory that is allocated but I don't think it reflects what is actually in use. Qkview seems to be a better option and this is what i have sent off to the support team.
hooleylist
Cirrostratus
Aug 26, 2008
jondyke, you're correct about top showing allocated CPU and memory. The perf graphs should be accurate. For a CLI tool, tmstat also shows the actual CPU cycles and memory TMM is using.

CPU: 0% busy 1% idle 99% sleep

Memory Allocated

22,269,504 / 1,807,745,024

Aaron
Robert_Sutcliff
Nimbostratus
Aug 27, 2008
Hi,

I would check your DNS settings...

If you have "Display Host Names When Possible" enabled (under preferences on the gui) and the names of pool members/nodes are unresolvable the F5 GUI will sit and wait for a DNS timeout (a few minutes). This happens even if it's not actually displaying the names, but is displaying something that could be resolved in DNS.

I had a similar problem, and found that some sections of the GUI were faster than others (iRules for example). Eventually, I found that my DNS settings were wrong (pointing at valid DNS servers but on an unreachable LAN). Once I corrected this the GUI started responding quickly again.

Rob