Getting Started with iRules LX, Part 5: Troubleshooting

When you start writing code, you will eventually run into some issue and need to troubleshoot. We have a few tools available for you.

Logging

One of the most common tools use for troubleshooting is logging. STDOUT and STDERR for the Node.js processes end up in the file

/var/log/ltm
. By temporarily inserting functions such as
console.log
,
console.error
, etc. you can easily check the state of different objects in your code at run time.

There is one caveat to logging in iRules LX; logging by most of most processes on BIG-IP is throttled by default. Basically, if 5 messages from a process come within 1 second, all other messages with be truncated. For Node.js, every line is a separate message so something like a stack trace will get automatically truncated. To remove this limit we can change the DB variable

log.sdmd.level
to the value
debug
in TMSH as such -

root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos)
# modify sys db log.sdmd.level value debug

SDMD is the process that manages the Node.js instances. This turns on verbose logging for SDMD and all the Node.js processes so you will want to turn this back to

info
when you are finished troubleshooting.

Once that is done you can start adding logging statements to your code. Here, I added a line to tell me when the process starts (or restarts) -

var f5 = require('f5-nodejs');

console.log('Node.js process starting.');

var ilx = new f5.ILXServer(); 

The logs would then show me this -

Jun  7 13:26:08 test-ve info sdmd[12187]: 018e0017:6: pid[21194] plugin[/Common/test_pl1.ext1] Node.js process starting.

Node Inspector

BIG-IP comes packaged with Node Inspector for advanced debugging. We won't cover how to use Node Inspector because most developers skilled in Node.js already know how to use it and there are already many online tutorials for those that dont.

Note: We highly recommend that you do not use Node Inspector in a production environment. Debuggers lock the execution of code at break points pausing the entire application, which does not work well for a realtime network device. Ideally debugging should only be done on a dedicated BIG-IP (such as a lab VE instance) in an isolated development environment.

Turn on Debugging

The first thing we want to do is to put our extension into single concurrency mode so that we are only working with one Node.js process while debugging. Also, while we are doing that we can put the extension into debugging mode. This can be done from TMSH like so -

root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos)
# list ilx plugin DC_Plugin
ilx plugin DC_Plugin {
    disk-space 156
    extensions {
        dc_extension { }
        dc_extension2 { }
    }
    from-workspace DevCentralRocks
    staged-directory /var/ilx/workspaces/Common/DevCentralRocks
}
root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos)
# modify ilx plugin DC_Plugin extensions { dc_extension { concurrency-mode single command-options add { --debug } }}
root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos)
# list ilx plugin DC_Plugin
ilx plugin DC_Plugin {
    disk-space 156
    extensions {
        dc_extension {
            command-options { --debug }
            concurrency-mode single
        }
        dc_extension2 { }
    }
    from-workspace DevCentralRocks
    staged-directory /var/ilx/workspaces/Common/DevCentralRocks
}
root@(test-ve)(cfg-sync Standalone)(Active)(/Common)(tmos)
# restart ilx plugin DC_Plugin immediate 

Increase ILX::call Timeout

Also, if you recall from part 3 of this series, the TCL command

ILX::call
has a 3 second timeout by default that if there is no response to the RPC, the command throws an exception. We don't need to worry about this for the command
ILX::notify
because it does not expect a response from the RPC. While we have the debugger paused on a break point you are pretty much guaranteed to hit this timeout, so we need to increase it to allow for our whole method to finish execution. To do this, you will need to add arguments to the
ILX::call
highlighted in red -

ILX::call $ilx_handle -timeout big_number_in_msec method arg1

This number should be big enough to walk through the entire method call while debugging. If it will take you 2 minutes, then the value should be 120000. Once you update this make sure to reload the workspace.

Launching Node Inspector

Once you have put Node.js into debugging mode and increased the timeout, you now you can fire up Node Inspector. We will need to find the debugging port that our Node.js process has been assigned via TMSH -

root@(eflores-ve3)(cfg-sync Standalone)(Active)(/Common)(tmos)
# show ilx plugin DC_Plugin 

<----------------snipped ----------------->

     ---------------------------------
     | Extension Process: dc_extension
     ---------------------------------
     | Status                running
     | PID                   25975
     | TMM                   0
     | CPU Utilization (%)   0
     | Debug Port            1033
     | Memory (bytes)      
     |   Total Virtual Size  1.2G
     |   Resident Set Size   5.8K

Now that we have the debugging port we can put that and the IP of our management interface into the Node Inspector arguments -

[root@test-ve:Active:Standalone] config # /usr/lib/node_modules/.bin/node-inspector --debug-port 1033  --web-host 192.168.0.245 --no-inject
Node Inspector v0.8.1
Visit http://192.168.0.245:8080/debug?port=1033 to start debugging.

We simply need to take the URL given to us and put it in our web browser. Then we can start debugging away!

Conclusion

We hope you have found the Getting Started with iRules LX series very informative and look forward to hearing about how iRules LX has helped you solved problems. If you need any help with using iRules LX, please dont hesitate to ask a question on DevCentral.

Updated Jun 06, 2023
Version 2.0

Was this article helpful?

3 Comments

  • I am using version 13.1.0.5, and attempted to use node-inspector, but that directory doesn't exist (/usr/lib/node_modules/ exists, but the subdirectories .bin/node-inspector don't). I can't find it anywhere. Does node-inspector still ship with the product, or has it been replaced by something else?

     

  • This applies to version 13.1.0.5

    Hi @AlgebraicMirror,

    I just ran into the same issue. I installed node-inspector myself in my home directory on the F5:

    cd
    npm install node-inspector
    

    This puts node-inspector inside your own ~/node_modules directory. The exectutable can be found here:

    ~/node_modules/.bin/node-inspector
    

    I had to do a bit of fiddling to get the node-inspector to work on the F5.

    First step was to go to Local Traffic > iRules > LX Plugins and click on my plugin name.

    Then there's an Extensions section, and you can click on the name of your extension.

    In the Properties for your extension, change the following settings:

    • Concurrency mode = single (this means you only have one nodejs process instead of multiple)
    • Command options = --debug (this enables the nodejs debugger)

    You might need to restart your nodejs process by clicking the "Reload from Workspace" button on your plugin.

    You've now got a nodejs process listening on a port for debug messages. You can find them by:

    tmsh show ilx plugin // extensions | grep Debug
    

    (substitute your partition and plugin name)

    Then you can run node-inspector:

    ~/node_modules/.bin/node-inspector --web-host 10.10.10.1 --no-inject --debug-host 10.10.10.1 --debug-port 20002 --ssl-key /etc/httpd/conf/ssl.key/server.key --ssl-cert /etc/httpd/conf/ssl.crt/server.crt
    

    Note that the above command line has a few configurable parts. I found that node-inspector defaulted to localhost, and so I explicitly specify the IP address for both the place I want node inspector to connect to the debug port, and to host the inspector web site. I have used my management IP address that the F5 BIG-IP Configuration utility runs on.

    The ssl key and cert allow it to run in https mode, which might be necessary depending on which browser you're using.

    node-inspector will start, and print a message with the url you should browse to in your web browser:

    Node Inspector v1.1.2
    Visit https://10.10.10.1:8080/?port=20002 to start debugging.
    

    You can press

    Ctrl-Z
    and type
    bg
    to run that in the background, or put an & on the end of the original command.

    You can also modify the url of the node-inspector web app that starts up to have a different host and port if you need to:

    https://10.10.10.1:8080/?host=10.10.10.1&port=20002
    

    You can find out about other node-inspector stuff on the node-inspector site:

    https://github.com/node-inspector/node-inspector

    (Note that the site suggests using the new

    --inspect
    flag and Chrome dev tools instead of node-inspector. The
    --inspect
    flag appears to be disabled in the build of node on the F5, so this is not an option.)

    When you connect to the website that starts up, it might be really slow to show anything on the screen. I recommend opening the web inspector / developer tools in your browser and checking the network tab is seeing lots of requests, and the Console tab doesn't have any errors.

    Of course, you SHOULDN"T DO THIS IN PRODUCTION!!! Only run debug in a dev environment, VM, etc... as you'll be blocking real web requests and slow down your F5.

    An alternative approach to installing the node-inspector in your home directory on the F5 is to install it on your own computer, and then connect to the F5's debug port in the command line (i.e. a different --web-host than --debug-host). I haven't tried this, but it might be better than polluting your F5 with extra files and processes taking CPU.

    I hope that helps!

    Kirk