BIG-IP : iControl : LocalLBDataGroupFile.set_local_path() : swap large file under load
F5 BIG-IP v11.4.1 (Build 635.0) LTM on ESXi I have a .NET C app that uses iControl to perform following sequence : • transfer data-file to staging location on BIG-IP device • recache data-group with contents of this data-file The specific iControl API used for the recache is : LocalLBDataGroupFile.set_local_path() This operation has been 100% consistently successful in non-prod environments with very low traffic to perform data-group-file updates of up to 2M records. Non-prod config : single stand-alone device ( no HA pair ). The operation has also been 100% consistently successful in prod environments with high traffic to update a non-live data-group-file up to 300K records. Prod config : HA pair consisting of 2-node sync-failover device-group with auto-sync disabled. NOTE: By "live data-group-file" I mean an enabled virtual-server has an assigned iRule that references the data-group ( performs matches against data-group maps ). By "non-live data-group-file" I mean that the data-group exists but either is not referenced by any iRules, or iRules that reference it currently are not assigned to any enabled virtual-server. Here is where the problem occurs : When the operation is run in prod environments with high traffic (40-60% baseline cpu utilization, 200-400 Mbps baseline throughput) to update a live data-group-file ( 100K+ records ) the iRule fails. Exactly how the iRule fails is unknown and currently is under investigation by F5 Support, however here are some data-points : • the file-transfer and data-group recache iControl calls return success to the C caller • requests that the iRule normally conditionally rewrites to various backend pools no longer arrive at those servers • BIG-IP logs contain zero errors related to either the iControl operation of the iRule • public client requests that should be processed by the iRule display generic Akamai 500 error-pages NOTE: I have a test that removes Akamai from the equation, but have not yet had an opportunity to run it. My understanding is that LocalLBDataGroupFile.set_local_path() was re-designed/coded for 11.4 and was lab-tested up to 1M records. However, I wonder if any testing was performed in an environment with significant load ? Through trial-and-error I discovered the following workaround (for an HA pair only) : • create a/b pair of data-groups and corresponding set of a/b iRules that are identical except that "a" iRule references "a" data-group, and "b" iRule references "b" data-group • on active node, initially configure virtual-server to use "a" iRule • use C application to update "b" data-group-file ( NOTE: possibly this could also be accomplished via the admin browser, but above 100K records the time-lags and potential impact on prod operations become concerning. ) • if now swap-in "b" iRule to virtual-server ( effectively swapping-in "b" data-group ) the irule will begin to behave strangely (requests swallowed and never routed to backend pool although no errors present in LTM logs) • however, the following "trick" seems to work : sync active to standby promote standby to active on new active, swap-in "b" iRule to the virtual-server reboot new standby* sync new active to new standby Somehow the sync operation "cures" the issues induced by swapping the live iRule to point to a just-updated data-group. So in summary it seems that for a high-load environment attempting to swap new contents into a live data-group somehow induces a failure-case for iRule lookups against that data-group. The failure symptoms are identical both for the technique of re-caching the live data-group with new contents ( iControl API LocalLBDataGroupFile.set_local_path() ), and for the iRule a/b swap technique. However, an active-to-standby sync operation seems to "cure" whatever bad-state the data-group has been put into. Can anyone provide insights as to why swapping-in new contents to a large data-group-file associated with an iRule assigned to a VIP under heavy load would cause iRule data-group lookup failures ?BIG-IP : sync-failover device-group sync options
F5 BIG-IP Virtual Edition v11.4.1 (Build 635.0) LTM on ESXi For a device-group of type sync-failover, the admin browser provides these options : Automatic Sync Full Sync Maximum Incremental Sync Size (KB) Could someone please explain these options ? Are syncs on a time-schedule ? Or are syncs triggered by any change to the primary ? Or exactly what types of changes will trigger a sync ? Specifically, will a change to a data-group file ( e.g. add/delete a line ) trigger a sync ?
Best tool for managing multiple Big-IPs?
Those of you with multiple Big-IP devices to manage, do you use Big-IQ or some other alternative? We have a couple of 7200s with four VMs each in active/standby pairs, plus a 2200 and a few lab VMs. I'd like to be able to go to one place to see all of my nodes, pools, vservers, etc rather than have to swap back and forth between devices.
regex/grok patterns for BigIP Logs
Hi I'm ingesting logs into the ELK stack. quite nice. Only problem is i'm having to write the regex patterns by hand for each log type (tmm/secure/user). Is there a wiki that lists: log syntax (e.g. fields used for TMM log and field order in the log) regexes for these formats I'd be really pleased if there was even just a page that listed (1) log syntax for each type I've seen in the discussions here that log formats also change between versions... any ideas on how often this happens...?BIG-IP : standby auto-sync to active
BIG-IP 11.4.1 Build 635.0 Hotfix HF2 For a sync-failover device-group in active-standby configuration with auto-sync enabled. What is behavior following making a config-change to standby device ( specifically, update data-group contents ) ? Will the changes be immediately sync'd to the active ? Does auto-sync ensure delta-zero between active/standby ? i.e. is it possible for active to have changes not yet sync'd to standby at time change is made to standby ?