Forum Discussion

John_Stewart_10's avatar
John_Stewart_10
Icon for Nimbostratus rankNimbostratus
Dec 22, 2011

BIG3D

We have an LTM running 10.2.3.112 code. We ran the big3d -v command and get the following:

 

 

root@rndltmcore:Active] config big3d -v

 

big3d version big3d Version 10.2.3.112.0 for linux

 

 

We attempted discover this LTM with the upgrade box unchecked and it failed. We ran the discovery again with the box checked and it succeeded . The log message indicated that the big3d was upgraded to 1.4.

 

 

However, when we run the big3d -v command again, we get the following:

 

 

[root@rndltmcore:Active] config big3d -v

 

big3d version big3d Version 10.2.3.112.0 for linux

 

 

What is concerning is that SCOM MGMT Pack told us it upgraded the big3d, but the LTM does not reflect that.

 

 

We also know that when we upgrade the big3d, it will interrupt the communication between GTM's and LTM's, which will affect load balancing for that period. It will also affect communication between Enterprise manager and GTM's and LTM's as they are being discovered.

 

 

We also know that when we do a code upgrade with a new ISO, it will replace the big3d that was manually upgraded. This could potentially impair SCOM monitoring.

 

 

Looking for feedback, or comments on this subject.

 

 

 

management
microsoft
scom

8 Replies

Sort By
  • Richard__Harlan's avatar
    Richard__Harlan
    Historic F5 Account
    Dec 27, 2011
    running big3d -v will return the version of the file in /usr/bin/big3d run the following command ps aux | grep -i big3d. This will show you the full path of the running version of big3d then run path/big3d -v this will show you the running version of big3d.

     

     

    yes you are correct when big3d is updated it has to restart the process and GTM communication will be interrupted while it restarts.

     

  • John_Stewart_10's avatar
    John_Stewart_10
    Icon for Nimbostratus rankNimbostratus
    Dec 27, 2011
    Thanks,

     

     

    Got the the following this time:

     

    [root@rndltmcore:Active] config ps aux | grep -i big3d

     

    root 3374 0.0 0.0 1536 364 ? S Dec22 0:00 runsv big3d

     

    root 3428 0.0 0.1 10596 5064 ? S Dec22 2:18 /shared/bin/big3d

     

    root 8463 0.0 0.0 3036 780 pts/0 R+ 15:32 0:00 grep -i big3d

     

    [root@rndltmcore:Active] config /shared/bin/big3d -v

     

    /shared/bin/big3d version big3d Version [b}10.4.0.1.0 for linux

     

    [root@rndltmcore:Active] config

     

     

    Still need to understand ramifications of a code upgrade to the LTM
  • John_Stewart_10's avatar
    John_Stewart_10
    Icon for Nimbostratus rankNimbostratus
    Jan 13, 2012
    OK, upgraded one of our LTMs that we discovered with SCOM mgmt pack to version 10.2.3. HF1. Now we get this!!!!!!!!!!!!!!

     

     

    Jan 13 04:02:05 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:06 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:07 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:08 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:09 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:10 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:11 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:12 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:13 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:14 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:15 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:16 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:17 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:18 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:19 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:20 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:21 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:22 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:23 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:24 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:25 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:26 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:27 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:28 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:29 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:30 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:31 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:32 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:33 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:34 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:35 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:36 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:37 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:38 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:39 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:40 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:41 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:42 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:43 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:44 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:45 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:46 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:47 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:48 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:49 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:50 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:51 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:52 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:53 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:54 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:55 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:56 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:57 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:58 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:02:59 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:00 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:01 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:02 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:03 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:04 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:05 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:06 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:07 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:08 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:09 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:10 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:11 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:12 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:13 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:14 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:15 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:16 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:17 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:18 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:19 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:20 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:21 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:22 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:23 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:24 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:25 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:26 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:27 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:28 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:29 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:30 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:31 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:32 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:33 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:34 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:35 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:36 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:37 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:38 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:39 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:40 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:41 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:42 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:43 local/rndltmcore emerg logger: Re-starting big3d

     

    Jan 13 04:03:44 local/rndltmcore emerg logger: Re-starting big3d

     

  • Alex__Applebaum's avatar
    Alex__Applebaum
    Icon for Employee rankEmployee
    Jan 13, 2012
    Hi John,

     

     

    Sorry, that sucks. The way it works is big3d startup scripts first look to see if there is a newer version in /shared/bin/. Newer versions are installed by EM/GTM/etc. If one doesn't exist, it will use the default version of the install in /usr/sbin/

     

     

    Historically, it could be starting due to various issues like signatures/permissions, etc. You can use the solution below to debug and see if there's any reason logged.

     

     

    sol10999: Error Message: Re-starting big3d

     

    http://support.f5.com/kb/en-us/solutions/public/10000/900/sol10999.html?sr=18711325

     

     

    Or you could open a case with support to help find out what the exact issue is or if this is a known issue with that particular combination of Mgmt Pack + BIGIP version. I know Dev has been working on this issue so opening a case on this and having it attached to the appropriate bug ID will help draw attention to the issue.

     

     

    if really needed to stop big3d from restarting right now, you can just remove the "newer one" and restart big3d (like below) so it uses the default one.

     

     

    [root@ltm1:Active] config /usr/sbin/big3d -v

     

    /usr/sbin/big3d version big3d Version 10.2.3.112.0 for linux

     

    [root@ltm1:Active] config

     

    [root@ltm1:Active] config ls -lart /shared/bin/big3d

     

    -rwxr-xr-x 1 root root 3869783 Dec 15 06:05 /shared/bin/big3d

     

    [root@ltm1:Active] config /shared/bin/big3d -v

     

    /shared/bin/big3d version big3d Version 10.4.0.12.0 for linux

     

    [root@ltm1:Active] config

     

    [root@ltm1:Active] config

     

    [root@ltm1:Active] config ps -ef | egrep -i big3d | egrep -v grep

     

    root 3736 3553 0 2011 ? S 0:00 runsv big3d

     

    root 3790 3736 0 2011 ? S 3:17 /shared/bin/big3d

     

    [root@ltm1:Active] config rm /shared/bin/big3d

     

    rm: remove regular file `/shared/bin/big3d'? y

     

    [root@ltm1:Active] config ps -ef | egrep -i big3d | egrep -v grep

     

    root 3736 3553 0 2011 ? S 0:00 runsv big3d

     

    root 3790 3736 0 2011 ? S 3:17 /shared/bin/big3d

     

    [root@ltm1:Active] config bigstart restart big3d

     

    [root@ltm1:Active] config ps -ef | egrep -i big3d | egrep -v grep

     

    root 3736 3553 0 2011 ? S 0:00 runsv big3d

     

    root 16996 3736 32 05:40 ? D 0:00 /usr/sbin/big3d

     

     

    -Alex

     

  • John_Stewart_10's avatar
    John_Stewart_10
    Icon for Nimbostratus rankNimbostratus
    Jan 13, 2012
    Alex, This started when I upgraded 10.2.3 LTM to HF1. Said LTM had 1.4 big3d that was intsalled my the SCOM Management pack. I replaced the 10.4 big3d and all is well,,,,,except SCOM :-)

     

     

  • Alex__Applebaum's avatar
    Alex__Applebaum
    Icon for Employee rankEmployee
    Jan 13, 2012
    Yeah, think the problem is SCOM's big3d is technically older than the 10.2.3 HF1 big3d even though it has a higher code (10.4.xxxx).

     

     

    I know you will want your SCOM back up soon so the only thing I can recommend is sending a case to support.

     

     

    TITLE: SCOM's big3d "Version [b}10.4.0.1.0" restarts on 10.2.3 HF1.

     

     

    Referencing this thread.

     

     

    They might be able to give you the newer big3d (ex. the big3d my EM 2.3 installed "10.4.0.12.0" and make everyone happy (SCOM won't complain there's an "older" code and will have the most recent big3d code which works with the latest 10.2.3 HF1). Or you might even be able to put a v11 big3d on. However, you would need to get Dev's recommendation on that one.

     

     

    Like I said, last I heard, they were working aggressively on addressing this versioning issue so we won't have to worry about this anymore.

     

     

     

     

  • Alex__Applebaum's avatar
    Alex__Applebaum
    Icon for Employee rankEmployee
    Jan 17, 2012
    Hi John,

     

     

    I installed the EM big3d (10.4.0.1.0) on 10.2.3 and upgraded to 10.2.3 HF1 and didn't see the restart issue.

     

     

    [root@ltm2:Active] config b version | head

     

    Kernel:

     

    Linux 2.6.18-164.11.1.el5.1.0.f5app

     

    Package:

     

    BIG-IP Version 10.2.3 123.0

     

    Hotfix HF1 Edition

     

     

    Hot fixes:

     

    ID372864

     

    ID372590

     

    ID372804

     

    [root@ltm2:Active] config ps -ef | egrep 'bin/big3d' | grep -v grep

     

    root 3821 3802 0 17:11 ? S 0:00 /shared/bin/big3d

     

     

    [root@ltm2:Active] config /shared/bin/big3d -v

     

    /shared/bin/big3d version big3d Version 10.4.0.1.0 for linux

     

     

    What I think may have happened is that a 2nd big3d process was started at the same time. For example, I accidently ran it again like:

     

     

    [root@ltm2:Active] config /shared/bin/big3d

     

    Jan 16 17:39:31 local/ltm2 emerg logger: Re-starting big3d

     

    Jan 16 17:39:32 local/ltm2 emerg logger: Re-starting big3d

     

    Jan 16 17:39:33 local/ltm2 emerg logger: Re-starting big3d

     

    Jan 16 17:39:34 local/ltm2 emerg logger: Re-starting big3d

     

    Jan 16 17:39:35 local/ltm2 emerg logger: Re-starting big3d

     

    Jan 16 17:39:36 local/ltm2 emerg logger: Re-starting big3d

     

    [root@ltm2:Active] config Jan 16 17:39:37 local/ltm2 emerg logger: Re-starting big3d

     

     

    I hit Ctl-C and things returned to normal (with only default daemon running).

     

     

    There is a Bug ID to make it exit/log if another instance running which might help in the future.

     

     

    Bug 344228 - If big3d detects another big3d already running, exits without log message

     

     

    So short of it is, both big3ds should run just fine on 10.2.3 HF1. You should have been able to bigstart stop big3d & bigstart start big3d and you would have been ok. However in order for MP not to complain again, you need to restore your MP big3d with higher version . Going forward, you should be able to re-discover or even copy from another unit that MP big3d (10.4.0.1.0) back to /shared/bin/ and get your MP back up and running. The one with latest timestamp wins so if you see the 10.2.3 HF1 "10.2.3.112.0" big3d pop back up as the running process, just

     

     

    touch /shared/bin/big3d

     

     

    Otherwise, please open a case (while your logs haven't rotated) so they can investigate further. Dev was looking at this post so just post the case and I can forward it to them.