How AI Will Automate Cybersecurity in the Post-COVID World
Widespread remote working is accelerating the trend of digitization in society and a derivative trend of this acceleration is our increased reliance on online applications - which also means cybercrime is becoming more lucrative. Over on F5 Labs, Shuman Ghosemajumder briefly introduces the problem space and links to an article on VentureBeat about how AI will Automate cybersecurity in a Post-Covid world. https://www.f5.com/labs/articles/bylines/how-ai-will-automate-cybersecurity-in-the-post-covid-world
Rate Limiting SSL VPN User Traffic
With lots of people working at home, contention on VPNs is a real problem at the moment - license capacity, device CPU and throughput rate. One way to deal with this is to apply rate limits to user traffic. This can be done in a number of ways - applying a BWC policy in the Access Policy, using Traffic Classifiers, etc but I like simple solutions so i'm going to show you how to do it with virtual servers and iRules, and to take the easy way out you can use my iApp to do it for you! For a start, let's look at an SSL VPN in a bit more detail. Tunnels! The tunnel part of the SSL VPN is based around the Connectivity Profile - this specifies settings like compression and VPN settings. When you create a Connectivity Profile, this also creates a tunnel interface This tunnel interface is used as an internal connector so that outgoing traffic can be managed - the same function is used with HTTP explicit proxies which use http-tunnel. This is a very powerful feature - this means that we can create a virtual server which listens on that tunnel interface to be able to capture VPN user traffic before it leaves the BIG-IP. Bandwidth Controllers We have two types of bandwidth controller policy - Static and Dynamic. A Static policy sets the overall rate of traffic allowed, a dynamic policy allows us to set an overall rate but also a rate per user flow. For instance, we could allow ALL VPN traffic to be 1Gbps but each individual traffic flow within that could be limited to 1Mbps. iRules To apply the policy to the user traffic we are going to use an iRule with the BWC::policy command - this will set the policy on this flow for both uplink and downlink traffic in two different events - CLIENT_ACCEPTED and SERVER_CONNECTED when CLIENT_ACCEPTED { BWC::policy attach /Common/bwc-10M "[IP::remote_addr]:[TCP::remote_port]" } when SERVER_CONNECTED { BWC::policy attach /Common/bwc-10M "[IP::remote_addr]:[TCP::remote_port]" } Obviously the TCP::remote_port would be UDP::remote_port in a UDP virtual server. Putting it together Below you can see the virtual server configuration which I created using my iApp ltm virtual Common/vpn-1.app/vs_bwc_vpn-1_tcp_default { app-service /Common/vpn-1.app/vpn-1 creation-time 2020-04-29:10:26:39 destination Common/0.0.0.0:any ip-protocol tcp last-modified-time 2020-04-29:11:32:33 mask any profiles { Common/tcp { } } rules { Common/vpn-1.app/rule_bwc_vpn-1_tcp_default } serverssl-use-sni disabled source 0.0.0.0/0 source-address-translation { pool Common/snat-1 type snat } translate-address disabled translate-port disabled vlans { Common/connectivity-1 } vlans-enabled vs-index 7 } ltm virtual Common/vpn-1.app/vs_bwc_vpn-1_udp_default { app-service /Common/vpn-1.app/vpn-1 creation-time 2020-04-29:10:26:39 destination Common/0.0.0.0:any ip-protocol udp last-modified-time 2020-04-29:11:32:33 mask any profiles { Common/udp { } } rules { Common/vpn-1.app/rule_bwc_vpn-1_udp_default } serverssl-use-sni disabled source 0.0.0.0/0 source-address-translation { pool Common/snat-1 type snat } translate-address disabled translate-port disabled vlans { Common/connectivity-1 } vlans-enabled vs-index 6 } It works! iperf with no bandwidth controller $ iperf -c 10.20.20.3 ------------------------------------------------------------ Client connecting to 10.20.20.3, TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.20.20.131 port 5957 connected with 10.20.20.3 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 184 MBytes 154 Mbits/sec iperf with a 10Mbps bandwidth controller applied $ iperf -c 10.20.20.3 ------------------------------------------------------------ Client connecting to 10.20.20.3, TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.20.20.131 port 6066 connected with 10.20.20.3 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.2 sec 12.1 MBytes 9.98 Mbits/sec Make it easy I've shown you there how to put together the constituent parts yourself but brought to you by the magic of iApps, you can do it all automagically. You can use the iApp at APM VPN Bandwidth Controller iApp - just create the BWC policy you want to apply beforehand and run the iApp. You can even treat certain protocols differently - imagine if you want to limit HTTPS to 1Mbps but allow other traffic to have 10Mbps ( because a 1 second delay in a web page load is not noticeable but a 1 second delay in a Zoom session is very noticeable! )APM VPN Bandwidth Controller iApp
Problem this snippet solves: Overview This iApp will create a set of virtual servers to apply a Bandwidth Controller policy to VPN tunnel traffic. Example iperf without the iApp: $ iperf -c 10.20.20.3 ------------------------------------------------------------ Client connecting to 10.20.20.3, TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.20.20.131 port 5957 connected with 10.20.20.3 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 184 MBytes 154 Mbits/sec iperf with 10Mbps dynamic policy $ iperf -c 10.20.20.3 ------------------------------------------------------------ Client connecting to 10.20.20.3, TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.20.20.131 port 6066 connected with 10.20.20.3 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.2 sec 12.1 MBytes 9.98 Mbits/sec iperf with 1Gbps dynamic policy $ iperf -c 10.20.20.3 ------------------------------------------------------------ Client connecting to 10.20.20.3, TCP port 5001 TCP window size: 64.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.20.20.131 port 6569 connected with 10.20.20.3 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 190 MBytes 159 Mbits/sec Function This creates a set of virtual servers listening on the VPN tunnel with an iRule assigned which applies the BWC policy to both upload and download traffic. You can change your BWC rate as you require, it will be applied to new flows ie users don't have to reconnect. This has been tested that it deploys and works on v13 but I have not tested this in a production environment, therefore you should test its usage prior to implementation in a production environment. If you have successfully tested it then please PM with details and/or suggestions. How to use this snippet: Usage Instructions This assumes that you already have a VPN configured Create a Bandwidth Controller Policy with the overall bandwidth for the WHOLE VPN, and enable Dynamic if you want to specify the bandwidth for each user. In the example below, the Maximum Rate of 20Mbps is applied to the whole VPN and 10Mbps is applied to each flow. Load the iApp template at iApps>Templates and Import Deploy a new iApp service at iApps>Application Services>Applications and select the apm_bwc_iapp template Select the VPN tunnel and the Default BWC policy Select any SNAT requirements If you want to apply different rates to different traffic then add entries in the Protocol-specific Bandwidth Controller table. If you don't want to add these then click the X to remove the default entry. Hit Finished Objects created: Code : cli admin-partitions { update-partition Common } sys application template /Common/apm_bwc_iapp { actions { definition { html-help { } implementation { set app_dir [tmsh::pwd] set app_name $tmsh::app_name # https://support.f5.com/csp/article/K54955814 set rule_tcp {when CLIENT_ACCEPTED { BWC::policy attach <%=$bwc_policy%> "[IP::remote_addr]:[TCP::remote_port]" } when SERVER_CONNECTED { BWC::policy attach <%=$bwc_policy%> "[IP::remote_addr]:[TCP::remote_port]" } } set rule_udp {when CLIENT_ACCEPTED { BWC::policy attach <%=$bwc_policy%> "[IP::remote_addr]:[UDP::remote_port]" } when SERVER_CONNECTED { BWC::policy attach <%=$bwc_policy%> "[IP::remote_addr]:[UDP::remote_port]" } } if { $::main__use_snat == "Automap" } { set snat "source-address-translation \{ type automap \} " } elseif { $::main__use_snat == "SNAT Pool" } { set snat "source-address-translation \{ type snat pool $::main__snatpool \} " } else { set snat "" } # Create default iRule tmsh::create ltm rule rule_bwc_${app_name}_udp_default [ tmsh::expand_macro $rule_udp -vars "bwc_policy \"$::main__bwc_policy\"" ] tmsh::create ltm rule rule_bwc_${app_name}_tcp_default [ tmsh::expand_macro $rule_tcp -vars "bwc_policy \"$::main__bwc_policy\"" ] # Create default VS tmsh::create ltm virtual vs_bwc_${app_name}_udp_default ip-protocol udp vlans-enabled vlans replace-all-with \{ $::main__tunnel \} destination 0.0.0.0:any mask any $snat profiles replace-all-with \{ udp \} rules \{ rule_bwc_${app_name}_udp_default \} source 0.0.0.0/0 translate-address disabled translate-port disabled tmsh::create ltm virtual vs_bwc_${app_name}_tcp_default ip-protocol tcp vlans-enabled vlans replace-all-with \{ $::main__tunnel \} destination 0.0.0.0:any mask any $snat profiles replace-all-with \{ tcp \} rules \{ rule_bwc_${app_name}_tcp_default \} source 0.0.0.0/0 translate-address disabled translate-port disabled # Create custom ports and iRules foreach {row} $::main__entries { array set cols [lindex $row 0] # protocol, port and bwc_policy set rulename "rule_bwc_${app_name}_$cols(protocol)_$cols(port)" set vsname "vs_bwc_${app_name}_$cols(protocol)_$cols(port)" if { $cols(protocol) == "tcp" } { tmsh::create ltm rule $rulename [tmsh::expand_macro $rule_tcp -vars "bwc_policy \"$cols(bwc_policy)\"" ] tmsh::create ltm virtual $vsname ip-protocol tcp vlans-enabled vlans replace-all-with \{ $::main__tunnel \} destination 0.0.0.0:$cols(port) mask any $snat profiles replace-all-with \{ $cols(protocol) \} rules \{ $rulename \} source 0.0.0.0/0 translate-address disabled translate-port disabled } else { tmsh::create ltm rule $rulename [tmsh::expand_macro $rule_udp -vars "bwc_policy \"$cols(bwc_policy)\"" ] tmsh::create ltm virtual $vsname ip-protocol udp vlans-enabled vlans replace-all-with \{ $::main__tunnel \} destination 0.0.0.0:$cols(port) mask any $snat profiles replace-all-with \{ $cols(protocol) \} rules \{ $rulename \} source 0.0.0.0/0 translate-address disabled translate-port disabled } } } macro { } presentation { section main { # The entry below creates a large text box that must be filled out with a valid IP Address # For details of APL, look at the iApps developers guide: # https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-iapps-developer-11-4-0.html message intro "This iApp will create a forwarding virtual server on the specified VPN tunnel which intercepts the traffic and assigns a BWC policy" choice tunnel display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items net tunnel]" } choice bwc_policy display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items -norecursive net bwc policy]" } choice use_snat display "large" default "None" { "None" => "None", "Automap" => "Automap", "SNAT Pool" => "SNAT Pool" } optional (use_snat == "SNAT Pool") { choice snatpool display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items ltm snatpool]" } } table entries { choice protocol display "large" default "tcp" { "tcp" => "tcp", "udp" => "udp" } string port display "large" required validator "PortNumber" default "443" choice bwc_policy display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items -norecursive net bwc policy]" } } } text { # Entities below set the text for the questions and section names, etc. Make them simple and relevant. main "Main" main.intro "Usage" main.tunnel "VPN Tunnel" main.bwc_policy "Default BWC Policy" main.use_snat "Source Address Translation" main.snatpool "SNAT Pool" main.entries "Protocol-specific Bandwidth Controller" main.entries.protocol "Protocol" main.entries.port "Port" main.entries.bwc_policy "BWC Policy" } } role-acl none run-as none } } description "iApp to create an outgoing VS to apply a BWC policy to VPN user traffic v2" ignore-verification false requires-bigip-version-max none requires-bigip-version-min none requires-modules { apm } signing-key none tmpl-checksum none tmpl-signature none } Tested this on version: 13.0SSL VPN Tunnel and Office 365
Problem this snippet solves: This implements Regan Anderson's script from https://devcentral.f5.com/s/articles/SSL-VPN-Split-Tunneling-and-Office-365 as an iApp for simple deployment. This will install and configure the script and also create an iCall which runs every day at midnight. Also available at https://github.com/pwhitef5/apm-vpn-optimisation/blob/master/o365_iapp.tmpl How to use this snippet: Add the iApp via iApps>Templates Create a new application via iApps>Application Services>Applications, use the o365_optimisation iApp template Select your requirements ( remember to delete additional URLs and IPs if not used ) and hit finished Code : cli admin-partitions { update-partition Common } sys application template /Common/o365_optimisation { actions { definition { html-help { Office 365 Optimisation iCall Version 1 To see this in a wider window, hit the Launch button above For help on the various options, go to https://github.com/f5regan/o365-apm-split-tunnel This script fetches Office 365 URLs and IPs (IPv4 and/or IPv6) from Microsoft's Office 365 IP Address and URL web service, dynamically updates Network Access List "Exclude" properties for one or more Network Access Lists, and applies changes to the affected Access Policies. If the script is running on an HA pair of BIG-IPs then the script will also initiate a ConfigSync to push the updated configuration from the active BIG-IP to the standby BIG-IP. Script Requirements TMOS 12.1.0 or higher BIG-IP must be capable of resolving internet DNS names (ex. via DNS Lookup Server configuration) BIG-IP must be able to reach endpoints.office.com via TCP 443 (via Management or TMM interface) Administrative rights on the BIG-IP(s) Bash shell access on the BIG-IP(s) Things to Note This software is supplied "AS IS" without any warranties or support. This script does not enable “split tunneling” or make any other modifications, other than those mentioned, to the Network Access List(s) that may be required to enable the desired functionality. For guidance relating to Network Access List / Split Tunnelling configuration refer to the BIG-IP APM Knowledge Center. Some split tunneling guidance: Allow Local DNS Servers should be enabled to allow client access to Office 365 when VPN is disconnected DNS Exclude Address Space is not supported on macOS IPV6 Exclude Address Space doesn't currently work on macOS This script should not be used with BIG-IP Edge Client's Always Connected Mode: if Stonewall is configured to block traffic, then the excluded resources are not reachable (this is by design). This script tracks the version of the Office 365 service instance and will only update the exclusion lists if a newer version is detected. If modifications to the script's operating parameters (ex. Network Access Lists, O365 Service Areas, Additional URLs/IPs to Exclude) are made, they will NOT take effect until the script detects a new service instance version. To force the script to run with the updated parameters earlier, remove the o365_version.txt file from the script's working directory OR temporarily set force_o365_record_refresh = 1, then manually execute the script (python /shared/o365/apm_o365_update.py). This script overwrites the contents of the following fields in the defined Network Access Lists when an update is detected: If "use_url" is set to 1: DNS Exclude Address Space If "use_ipv4" is set to 1: IPV4 Exclude Address Space If "use_ipv6" is set to 1: IPV6 Exclude Address Space The aforementioned fields / properties should not be managed via TMSH or the GUI after implementing this script as the script will overwrite any manual changes to these fields when it detects the next Office 365 service instance update. To add non-Office 365 URLs/IPs to or remove any URL/IP (Office 365 or otherwise) from the "Exclude Address Space" properties of the Network Access Lists, use the noimport_* and additional_* fields in the script (see User Options in the Appendix for usage details) . Be sure to apply these changes to both units of an HA pair! Usage of DNS Address Exclusions requires the installation of the DNS Relay Proxy service on the VPN client K9694: Overview of the Windows DNS Relay Proxy service K49720803: BIG-IP Edge Client operations guide | Chapter 3: Common approaches to configuring VPN While the endpoints retrieved by this script handle the vast majority of Office 365 traffic, it is possible some traffic related to Office 365 is not summarized in the endpoint lists and will still go through the VPN. HA: This script must be implemented on both members of an HA pair HA: Updates made to the Python script are NOT synchronized between peers - updates must be made manually to each instance of the Python script HA: Exclusion list updates will only take place on the Active member - changes are synced from the Active to Standby by the script This script relies on iCall to periodically run. See What is iCall? on DevCentral for an overview of iCall. iCall scripts and periodic handlers are documented in greater detail in the F5 TMSH Reference. } implementation { set app_dir [tmsh::pwd] set app_name $tmsh::app_name set script {#!/usr/bin/env python # -*- coding: utf-8 -*- # O365 URL/IP update automation for BIG-IP # Version: 1.1 # Last Modified: 01 April 2020 # Original author: Makoto Omura, F5 Networks Japan G.K. # # Modified for APM Network Access "Exclude Address Space" by Regan Anderson, F5 Networks # Modified for iCall by Peter White, F5 Networks # # This Sample Software provided by the author is for illustrative # purposes only which provides customers with programming information # regarding the products. This software is supplied "AS IS" without any # warranties and support. # # The author assumes no responsibility or liability for the use of the # software, conveys no license or title under any patent, copyright, or # mask work right to the product. # # The author reserves the right to make changes in the software without # notification. The author also make no representation or warranty that # such application will be suitable for the specified use without # further testing or modification. #----------------------------------------------------------------------- import httplib import urllib import uuid import os import re import json import commands import datetime import sys #----------------------------------------------------------------------- # User Options - Configure as desired #----------------------------------------------------------------------- # Access Profile Name(s) - ex. SINGLE ["AP1"] OR MULTIPLE ["AP1", "AP2", "AP3"] access_profiles = [} set accessProfiles {} set naList {} foreach {row} $::main__access_profile { array set cols [lindex $row 0] lappend accessProfiles "\"$cols(profile)\" " lappend naLists "\"$cols(nalist)\" " } append script [ join $accessProfiles , ] append script {] # Network Access List Name(s) - ex. SINGLE ["NAL1"] OR MULTIPLE ["NAL1", "NAL2", "NAL3"] na_lists = [} append script [ join $naLists , ] append script {] # Microsoft Web Service Customer endpoints (ENABLE ONLY ONE ENDPOINT) # These are the set of URLs defined by customer endpoints as described here: https://docs.microsoft.com/en-us/office365/enterprise/urls-and-ip-address-ranges customer_endpoint = } append script "\"$::main__endpoint\"" append script { # O365 "SeviceArea" (O365 endpoints) to consume, as described here: https://docs.microsoft.com/en-us/office365/enterprise/urls-and-ip-address-ranges care_exchange = } append script $config__include_exchange append script { # "Exchange Online": 0=do not care, 1=care care_sharepoint = } append script $config__include_sharepoint append script { # "SharePoint Online and OneDrive for Business": 0=do not care, 1=care care_skype = } append script $config__include_skype append script { # "Skype for Business Online and Microsoft Teams": 0=do not care, 1=care care_common = } append script $config__include_common append script { # "Microsoft 365 Common and Office Online": 0=do not care, 1=care # O365 Record types to download & update use_url = } append script $config__use_url append script { # DNS/URL exclusions: 0=do not use, 1=use use_ipv4 = } append script $config__use_ipv4 append script { # IPv4 exclusions: 0=do not use, 1=use use_ipv6 = } append script $config__use_ipv6 append script { # IPv6 exclusions: 0=do not use, 1=use # O365 Categories to download & update o365_categories = } append script $config__o365_categories append script { # 0=Optimize only, 1= Optimize & Allow, 2 = Optimize, Allow, and Default # O365 Endpoints to import - O365 required endpoints or all endpoints # WARNING: "import all" includes non-O365 URLs that one may not want to bypass (ex. www.youtube.com) only_required = } append script $config__only_required append script { # 0=import all, 1=O365 required only # Don't import these O365 URLs (URL must be exact or ends_with match to URL as it exists in JSON record - pattern matching not supported) # Provide URLs in list format - ex. [".facebook.com", "*.itunes.apple.com", "bit.ly"] #noimport_urls = [] noimport_urls = [".symcd.com",".symcb.com",".entrust.net",".digicert.com",".identrust.com",".verisign.net",".globalsign.net",".globalsign.com",".geotrust.com",".omniroot.com",".letsencrypt.org",".public-trust.com","platform.linkedin.com"] # Don't import these O365 IPs (IP must be exact match to IP as it exists in JSON record - IP/CIDR mask cannot be modified) # Provide IPs (IPv4 and IPv6) in list format - ex. ["191.234.140.0/22", "2620:1ec:a92::152/128"] noimport_ips = [] # Non-O365 URLs to add to DNS Exclude List # Provide URLs in list format - ex. ["m.facebook.com", "*.itunes.apple.com", "bit.ly"] additional_urls = [} set additional_urls {} foreach {row} $::config__additional_urls { array set cols [lindex $row 0] lappend additional_urls "\"$cols(url)\"" } append script [ join $additional_urls , ] append script {] # Non-O365 IPs to add to IPV4 Exclude List # Provide IPs in list format - ex. ["191.234.140.0/22", "131.253.33.215/32"] additional_ipv4 = [} set additional_ips {} foreach {row} $::config__additional_ips { array set cols [lindex $row 0] lappend additional_ips "\"$cols(ip)\"" } append script [ join $additional_ips , ] append script {] # Non-O365 IPs to add to IPV6 Exclude List # Provide IPs in list format - ex. ["2603:1096:400::/40", "2620:1ec:a92::152/128"] additional_ipv6 = [] # Action if O365 endpoint list is not updated force_o365_record_refresh = 0 # 0=do not update, 1=update (for test/debug purpose) # BIG-IP HA Configuration device_group_name = "} append script $config__dg append script {" # Name of Sync-Failover Device Group. Required for HA paired BIG-IP. ha_config = } if { [llength [tmsh::get_config cm device]] > 1 } { append script "1" } else { append script "0" } append script { # 0=stand alone, 1=HA paired # Log configuration log_level = } append script $main__debug append script { # 0=none, 1=normal, 2=verbose #----------------------------------------------------------------------- # System Options - Modify only when necessary #----------------------------------------------------------------------- # Working directory, file name for guid & version management work_directory = "/shared/o365/" file_name_guid = "/shared/o365/guid.txt" file_ms_o365_version = "/shared/o365/o365_version.txt" log_dest_file = "/var/log/o365_update" # Microsoft Web Service URLs url_ms_o365_endpoints = "endpoints.office.com" url_ms_o365_version = "endpoints.office.com" uri_ms_o365_version = "/version?ClientRequestId=" #----------------------------------------------------------------------- # Implementation - Please do not modify #----------------------------------------------------------------------- list_urls_to_exclude = [] list_ipv4_to_exclude = [] list_ipv6_to_exclude = [] def log(lev, msg): if log_level >= lev: log_string = "{0:%Y-%m-%d %H:%M:%S}".format(datetime.datetime.now()) + " " + msg + "\n" f = open(log_dest_file, "a") f.write(log_string) f.flush() f.close() return def main(): # ----------------------------------------------------------------------- # Check if this BIG-IP is ACTIVE for the traffic group (= traffic_group_name) # ----------------------------------------------------------------------- result = commands.getoutput("tmsh show /cm failover-status field-fmt") if ("status ACTIVE" in result) or (ha_config == 0): log(1, "This BIG-IP is standalone or HA ACTIVE. Initiating O365 update.") else: log(1, "This BIG-IP is HA STANDBY. Aborting O365 update.") sys.exit(0) # ----------------------------------------------------------------------- # GUID management # ----------------------------------------------------------------------- # Create guid file if not existent if not os.path.isdir(work_directory): os.mkdir(work_directory) log(1, "Created work directory " + work_directory + " because it did not exist.") if not os.path.exists(file_name_guid): f = open(file_name_guid, "w") f.write("\n") f.flush() f.close() log(1, "Created GUID file " + file_name_guid + " because it did not exist.") # Read guid from file and validate. Create one if not existent f = open(file_name_guid, "r") f_content = f.readline() f.close() if re.match('[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}', f_content): guid = f_content log(2, "Valid GUID is read from local file " + file_name_guid + ".") else: guid = str(uuid.uuid4()) f = open(file_name_guid, "w") f.write(guid) f.flush() f.close() log(1, "Generated a new GUID, and saved it to " + file_name_guid + ".") # ----------------------------------------------------------------------- # O365 endpoints list version check # ----------------------------------------------------------------------- # Read version of previously received record if os.path.isfile(file_ms_o365_version): f = open(file_ms_o365_version, "r") f_content = f.readline() f.close() # Check if the VERSION record format is valid if re.match('[0-9]{10}', f_content): ms_o365_version_previous = f_content log(2, "Valid previous VERSION found in " + file_ms_o365_version + ".") else: ms_o365_version_previous = "1970010200" f = open(file_ms_o365_version, "w") f.write(ms_o365_version_previous) f.flush() f.close() log(1, "Valid previous VERSION was not found. Wrote dummy value in " + file_ms_o365_version + ".") else: ms_o365_version_previous = "1970010200" f = open(file_ms_o365_version, "w") f.write(ms_o365_version_previous) f.flush() f.close() log(1, "Valid previous VERSION was not found. Wrote dummy value in " + file_ms_o365_version + ".") # ----------------------------------------------------------------------- # O365 endpoints list VERSION check # ----------------------------------------------------------------------- request_string = uri_ms_o365_version + guid conn = httplib.HTTPSConnection(url_ms_o365_version) conn.request('GET', request_string) res = conn.getresponse() if not res.status == 200: # MS O365 version request failed log(1, "VERSION request to MS web service failed. Assuming VERSIONs did not match, and proceed.") dict_o365_version = {} else: # MS O365 version request succeeded log(2, "VERSION request to MS web service was successful.") dict_o365_version = json.loads(res.read()) ms_o365_version_latest = "" for record in dict_o365_version: if record.has_key('instance'): if record["instance"] == customer_endpoint and record.has_key("latest"): latest = record["latest"] if re.match('[0-9]{10}', latest): ms_o365_version_latest = latest f = open(file_ms_o365_version, "w") f.write(ms_o365_version_latest) f.flush() f.close() log(2, "Previous VERSION is " + ms_o365_version_previous) log(2, "Latest VERSION is " + ms_o365_version_latest) if ms_o365_version_latest == ms_o365_version_previous and force_o365_record_refresh == 0: log(1, "You already have the latest MS O365 URL/IP Address list: " + ms_o365_version_latest + ". Aborting operation.") sys.exit(0) # ----------------------------------------------------------------------- # Request O365 endpoints list & put it in dictionary # ----------------------------------------------------------------------- request_string = "/endpoints/" + customer_endpoint + "?ClientRequestId=" + guid conn = httplib.HTTPSConnection(url_ms_o365_endpoints) conn.request('GET', request_string) res = conn.getresponse() if not res.status == 200: log(1, "ENDPOINTS request to MS web service failed. Aborting operation.") sys.exit(0) else: log(2, "ENDPOINTS request to MS web service was successful.") dict_o365_all = json.loads(res.read()) # Process for each record(id) of the endpoint JSON data for dict_o365_record in dict_o365_all: service_area = str(dict_o365_record['serviceArea']) category = str(dict_o365_record['category']) if (o365_categories == 0 and category == "Optimize") \ or (o365_categories == 1 and (category == "Optimize" or category == "Allow")) \ or (o365_categories == 2): if (only_required == 0) or (only_required and str(dict_o365_record['required']) == "True"): if (care_common and service_area == "Common") \ or (care_exchange and service_area == "Exchange") \ or (care_sharepoint and service_area == "SharePoint") \ or (care_skype and service_area == "Skype"): if use_url: # Append "urls" if existent in each record if dict_o365_record.has_key('urls'): list_urls = list(dict_o365_record['urls']) for url in list_urls: list_urls_to_exclude.append(url) # Append "allowUrls" if existent in each record if dict_o365_record.has_key('allowUrls'): list_allow_urls = list(dict_o365_record['allowUrls']) for url in list_allow_urls: list_urls_to_exclude.append(url) # Append "defaultUrls" if existent in each record if dict_o365_record.has_key('defaultUrls'): list_default_urls = dict_o365_record['defaultUrls'] for url in list_default_urls: list_urls_to_exclude.append(url) if use_ipv4 or use_ipv6: # Append "ips" if existent in each record if dict_o365_record.has_key('ips'): list_ips = list(dict_o365_record['ips']) for ip in list_ips: if re.match('^.+:', ip): list_ipv6_to_exclude.append(ip) else: list_ipv4_to_exclude.append(ip) log(1, "Number of unique ENDPOINTS to import...") # Add administratively defined URLs/IPs and (Re)process to remove duplicates and excluded values if use_url: # Combine lists and remove duplicate URLs urls_undup = list(set(list_urls_to_exclude + additional_urls)) ## Remove set of excluded URLs from the list of collected URLs for x_url in noimport_urls: urls_undup = [x for x in urls_undup if not x.endswith(x_url)] log(1, "URL: " + str(len(urls_undup))) if use_ipv4: # Combine lists and remove duplicate IPv4 addresses ipv4_undup = list(set(list_ipv4_to_exclude + additional_ipv4)) ## Remove set of excluded IPv4 addresses from the list of collected IPv4 addresses for x_ip in noimport_ips: ipv4_undup = [x for x in ipv4_undup if not x.endswith(x_ip)] log(1, "IPv4 host/net: " + str(len(ipv4_undup))) if use_ipv6: # Combine lists and duplicate IPv6 addresses ipv6_undup = list(set(list_ipv6_to_exclude + additional_ipv6)) ## Remove set of excluded IPv6 addresses from the list of collected IPv6 addresses for x_ip in noimport_ips: ipv6_undup = [x for x in ipv6_undup if not x.endswith(x_ip)] log(1, "IPv6 host/net: " + str(len(ipv6_undup))) # ----------------------------------------------------------------------- # URLs, IPv4 & IPv6 addresses formatted for TMSH # ----------------------------------------------------------------------- if use_url: # Initialize the URL string url_exclude_list = "" # Write URLs to string for url in urls_undup: url_exclude_list = url_exclude_list + " " + url.lower() if use_ipv4: # Initialize the IPv4 string ipv4_exclude_list = "" # Write IPv4 addresses to string for ip4 in (list(sorted(ipv4_undup))): ipv4_exclude_list = ipv4_exclude_list + "{subnet " + ip4 + " } " if use_ipv6: # Initialize the IPv6 string ipv6_exclude_list = "" # Write IPv6 addresses to string for ip6 in (list(sorted(ipv6_undup))): ipv6_exclude_list = ipv6_exclude_list + "{subnet " + ip6 + " } " # ----------------------------------------------------------------------- # Load URL and/or IPv4 and/or IPv6 lists into Network Access resource # ----------------------------------------------------------------------- if use_url: for na in na_lists: result = commands.getoutput("tmsh modify /apm resource network-access " + na + " address-space-exclude-dns-name replace-all-with { " + url_exclude_list + " }") log(2, "Updated " + na + " with latest O365 URL list.") if use_ipv4: for na in na_lists: result = commands.getoutput("tmsh modify /apm resource network-access " + na + " address-space-exclude-subnet { " + ipv4_exclude_list + " }") log(2, "Updated " + na + " with latest IPv4 O365 address list.") if use_ipv6: for na in na_lists: result = commands.getoutput("tmsh modify /apm resource network-access " + na + " ipv6-address-space-exclude-subnet { " + ipv6_exclude_list + " }") log(2, "Updated " + na + " with latest IPv6 O365 address list.") #----------------------------------------------------------------------- # Apply Access Policy and Initiate Config Sync: Device to Group #----------------------------------------------------------------------- for ap in access_profiles: result = commands.getoutput("tmsh modify /apm profile access " + ap + " generation-action increment") if ha_config == 1: log(1, "Initiating Config-Sync.") result = commands.getoutput("tmsh run cm config-sync to-group " + device_group_name) log(2, result + "\n") log(1, "Completed O365 URL/IP address update process.") if __name__=='__main__': main() } # Create directory catch { exec /bin/mkdir -p /shared/o365 # Create the script set scriptname "/shared/o365/apm_o365_update_${app_name}.py" set fh [ open $scriptname w+ ] puts $fh $script close $fh # Make the script executable exec /bin/chmod +x $scriptname } set scriptText "script_${app_name} definition \{ catch \{ exec python $scriptname \} \}" tmsh::create sys icall script $scriptText tmsh::create sys icall handler periodic handler_${app_name} script script_${app_name} interval 86400 first-occurrence [ clock format [clock seconds] -format %Y-%m-%d:23:59:59 ] } presentation { section main { # The entry below creates a large text box that must be filled out with a valid IP Address # For details of APL, look at the iApps developers guide: # https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-iapps-developer-11-4-0.html choice debug display "medium" default "Off" {"Off" => "0", "Low" => "1","High" => "2"} optional (debug != "0") { message message1 "Note: logging output goes to /var/log/o365_update" } table access_profile { choice profile display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items apm profile access]" } choice nalist display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items apm resource network-access]" } } choice endpoint display "medium" default "Worldwide" {"Worldwide" => "Worldwide", "USGovDoD" => "USGovDoD","USGovGCCHigh" => "USGovGCCHigh","China" => "China","Germany" => "Germany"} } section config { choice include_exchange display "medium" default "1" {"No" => "0", "Yes" => "1"} choice include_sharepoint display "medium" default "0" {"No" => "0", "Yes" => "1"} choice include_skype display "medium" default "0" {"No" => "0", "Yes" => "1"} choice include_common display "medium" default "0" {"No" => "0", "Yes" => "1"} choice use_url display "medium" default "1" {"No" => "0", "Yes" => "1"} choice use_ipv4 display "medium" default "1" {"No" => "0", "Yes" => "1"} choice use_ipv6 display "medium" default "0" {"No" => "0", "Yes" => "1"} choice o365_categories display "medium" default "0" {"Optimize Only" => "0", "Optimize and Allow" => "1", "Optimize, Allow and Default" => "2"} choice only_required display "medium" default "1" {"No" => "0", "Yes" => "1"} table additional_urls { string url required display "medium" validator "IpOrFqdn" } table additional_ips { string ip required display "medium" validator "IpAddress" } choice dg display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items cm device-group]" } } text { # Entities below set the text for the questions and section names, etc. Make them simple and relevant. main "Main" main.debug "Logging Level" main.message1 "" main.access_profile "Access Profiles / NA List" main.access_profile.profile "Access Profile" main.access_profile.nalist "NA List" main.endpoint "Endpoint" config "Configuration" config.include_exchange "Exchange" config.include_sharepoint "Sharepoint" config.include_skype "Skype" config.include_common "Common" config.use_url "Use URLs" config.use_ipv4 "Use IPv4 subnets" config.use_ipv6 "Use IPv6 subnets" config.o365_categories "O365 Categories" config.only_required "Required Endpoints Only" config.additional_urls "Additional URLs" config.additional_urls.url "FQDN" config.additional_ips "Additional IPs" config.additional_ips.ip "IP/Mask" config.dg "High Availability Device Group" } } role-acl none run-as none } } description "o365 split tunnel update iCall v1" ignore-verification false requires-bigip-version-max none requires-bigip-version-min none requires-modules { apm } signing-key none tmpl-checksum none tmpl-signature none } Tested this on version: 13.0Hitting the Easy Button: Securing the Remote Desktop on F5 BIG-IP APM
Being able to provide the most effective remote access solution is critical, especially in these turbulent times. In this article printed with permission from authors Lucas Thompson & Michael Waechter, we're going to talk about Remote Desktop Web Access. Solution Brief In short, it enables end-users to access their Remote Desktop applications through the F5 APM Webtop. The benefits of utilizing Remote Desktop Web Access over a desktop can be many. With the only requirement being a compatible web browser, Microsoft RDP application (comes installed with all modern versions of Windows), and a backend server hosting the applications… the solution speaks for itself. When the Full Webtop is displayed, APM will fetch a list of RemoteApps available on the target Terminal Server via HTTPS (using the Server SSL profile on the APM virtual server) and the associated icons. They will then be presented to the end user. The connection is done over HTTPS to the APM, and APM uses RDP (port 3389) to the Terminal Server. In the classic Terminal Server Desktop use case, the user is assigned a ‘native’ type RDP resource. This icon is presented to the user on a Full Webtop. Access is made by selecting the icon. A .RDP file is downloaded to the end user client PC, and the browser will activate the OS’s native RDP client to proceed with the connection. The connection is done over HTTPS to the APM, and APM uses RDP (port 3389) to the Terminal Server. Not only is the client setup simple, but the administration part of the equation is equally as easy to enable. I’m running version 14.1 (LTS) and here are a few screenshots of the setup. To enable the solution, let’s click on Access -> Connectivity/VPN Go ahead and choose VDI/RDP -> Remote Desktops Add the relative information. (Note: It’s always best to have the host name be a FQDN, and add this as a LTM node for health monitoring.) Technical Workflow The user clicks a resource icon on the Full Webtop, an RDP file is downloaded and then executed by the TS client on the user’s PC. The RemoteApp use case has a few differences versus targeting a desktop, or terminal server directly. In this case APM Will: Obtain a list of the RD feeds. The list of RDP App Resources will be derived from the RemoteApp feed. The list of icons will be delivered to the end user’s browser. The end user’s browser will request the icon pictures via a proxy mechanism in the VDI module. Because the RemoteApp feed comes through HTTPS and IIS on the Terminal Server, we have to make sure that: BIG-IP data plane can route to the Terminal Server. BIG-IP can create a HTTPS connection to Terminal Server. Terminal Server is rendering the page correctly. When you browse to it (https://terminalserver/rdweb/) you should see something like this: 2008: 2012: Authenticate with the same credentials that the test user uses in APM, and you should see an App Feed or desktop feed: Solution FAQ What kind of licenses are used for RDP access? APM has two license types: CCU and Access Session. Access Sessions are used for each established session ID. CCU are used for Network VPNs and other things that require more advanced features. Native Mode RDP does NOT use a CCU (connectivity) license. Only a single Access Session license will be consumed by a connecting user. What RDP options are supported? All of them. They’re basically echoed back to the client in the .RDP file. Put your desired parameter into the Custom Parameters area. It’s OK to use session variables in %{session.variable} format as well. RDP Custom Parameters configuration Lists of the RDP options have been compiled by 3rd parties, including the one at https://www.donkz.nl/overview-rdp-file-settings/ which is quite comprehensive. Please note that the following options are reserved for APM RDG use. If you attempt to apply these custom parameters, they will be ignored and/or overwritten by APM: Gatewayusagemethod Gatewayprofileusagemethod Gatewayhostname Gatewaycredentialssource Gatewayaccesstoken authentication level full address server port enablecredsspsupport signscope signature prompt for credentials on client domain username alternate full address gatewaybrokeringtype RDP Window Title The maximized window title for MSTSC inherits the target device name (not the RD gateway host). The medium-sized window title for MSTSC inherits the RDP filename (which is always “launch” -- see RFE 610244). One interesting thing that is possible is to internally-redirect the RDP session so that the client THINKS its connecting to one site, but then re-assign the remote host variable to a different site during the RAP access policy execution. RemoteApps It’s possible to create a lot of apps by using a PowerShell script on a RemoteApp-enabled terminal server. Client Requirements Microsoft Remote Desktop Client is supported for both Windows and Mac. Because the protocol used utilizes the Remote Desktop Gateway functionality, only newer RDP clients work. Legacy clients will likely not be able to create connections. iOS/Android The latest iOS / Android App Store RDP clients from Microsoft are supported. There might be some version conflicts, but for the most part the latest and greatest will work Reconnections / Disruptions Reconnections work the same as normal RDP If the user disconnects and reconnects, the session will be resumed. The client instructs the RD Gateway (APM) to again establish the session. The Remote Desktop session will be resumed also, the same way as with normal RDP. If the session is deleted or timed out or otherwise destroyed, the connection will stop, RDP will try to reconnect, but it will fail, and you will see this message from the MSTSC client.Securing your VMware Remote Solutions to Support COVID-19 Work From Home Scaling
Many of us are now working from home in unprecedented numbers. For infrastructure teams it's putting impressive strain on remote work solution. Building off our primary DevCentral COVID-19 article, our support teams and solution architects are hearing from many of you asking us for new and better ways to expand VMware capabilities with F5 BIG-IP Local Traffic Manager (LTM) and Access Policy Manager (APM). Get started securing your VMware remote working solutions with the field-recommended guides below. F5 with VMware Virtual Desktop Infrastructure (VDI) Solutions (Horizon View, Workspace ONE) How to deploy F5 with Horizon View using iApps This is a comprehensive guide for deploying F5 BIG-IP APM with VMware Horizon. Walk through the F5 iApp to assist in configuring APM with VMware Horizon View. How to use BIG-IP LTM in front of VMware Horizon Unified Access Gateway This guide will show step by step guidance on how to use F5 BIG-IP LTM to increase the scale and resiliency of either greenfield or brownfield VMware Horizon deployments. How to Deploy F5 APM with VMware ONE Providing a step-by-step instruction for setting up F5 BIG-IP APM as a proxy gateway for VMware Horizon with VMware Workspace ONE. How to deploy F5 BIG-IP LTM with VMware Workspace ONE Identity Manager (vIDM) This guide provides step-by-step instructions for setting up the first Identity Manager virtual appliance (Node 1), for production implementations. VMware recommends the deployment of two (2) additional nodes for three (3) total. Nodes 2 and 3 will be cloned from the first node after it's been configured and setup with the F5 BIG-IP to provide a fully load-balanced configuration. Reach Out To Us As our technical teams work with our users to provide continuous COVID-19 coverage, you may still need additional information we haven't surfaced yet. If you can't determine what best meets your requirements, let us know in the comments or reach out to our technical community. Don't forget to check out AskF5, our support knowledge center.F5 Supporting Our Technical Community During the COVID-19 Outbreak
Our community health is always a top priority. That priority extends to all of you who support each other every day here at DevCentral. We're a global community and we know many of you are directly affected by the COVID-19 pandemic and we want to help. Many of us are now required to work from home, and for some of us that's hard to do. The last thing we want you to worry about is technical issues. Speaking with several of you and talking to support and our teams out in the field answering your questions, we're busy gathering content that will help us all during this trying time. Our Support During the Outbreak AskF5 K70811681: F5 response to the global impact of coronavirus - F5 Support published their policy March 4th and our ability to support you remains unaffected. We strive to meet our stringent business continuity management plans to provide you with the service you've come to expect from F5 even during events like this. Troubleshooting and Support for F5 Remote Access Solutions Finding out the limits of your configuration or license during unplanned global issues is stressful to say the least. To help you troubleshoot and get started resolving those issues we compiled the below list based on your questions. AskF5 K21883200: Emerging issues you may experience during the COVID-19 outbreak - Compiled from the incoming support calls received, this will be your best ongoing source of top issues our users are running into with the increased load for remote access functionality. If you're having BIG-IP APM performance issues, start here! AskF5 K20775035: BIG-IP APM Operations Guide - This is the a great place to start if you haven't implemented APM. Consider this your field guide. It provides topics ranging from end user clients, configuration examples, to the ever-important Chapter 10: Troubleshooting. AskF5 K05847240: Troubleshooting BIG-IP APM Networks Access issues related to lease pools - This has been a popular question where lease pools are running out because so many people are connecting in and saturating available IP space. AskF5 K7752: Licensing the BIG-IP System - If you received a new license to bump up your APM client count, here's how install your new license. F5.com: Configuring the BIG-IP APM as a SAML 2.0 Identity Provider for Common SaaS Applications - Long title I know but if you're scrambling to get some federated access up for your systems start here. From the Field With everyone working remotely the need for additional BIG-IP APM answers and solutions is evident. It may be increasing DHCP lease pools for your users, installing a new license to increase your APM client count, or using Per-App VPN App Tunnels, we're working on getting you the information you need. Determining if you're licensed for Remote Access Capabilities - Many of you may have a license where APM and client access licenes are provisioned but you're just not using them. This article will determine if you're ready to start configuring APM for your remote workers. Responding to the Coronavirus - Six Ways to Improve App Availability - It's not just remote access issues people are running into. Ensure your apps are tuned and working for any heightened traffic from the increased remote working requirements. AskF5: Customizing BIG-IP APM access policy error messages - We've heard a lot about this. If you don't have good error messaging pages your user won't know if they entered a bad password or failed the client policy check. Creating an SSL VPN Using F5 Full Webtop - For those of you who have Access Policy Manager (APM) licensed but haven't configured it yet, here's a great starting point. There's a big spike in traffic for this article so we'll make sure it's in your list of to-read content Azure Active Directory and BIG-IP APM Integration - Ease identity and access management by integrating your cloud directory structure. Another high traffic article for people needing help with integrating quickly. Configuring a Per-App VPN Using F5 App Tunnels - Taking the burden off your corporate infrastructure by allowing VPN to a single app. Not many of us need full VPN tunnels so why waste your bandwidth. AskF5 K16680: VoIP through Network Access connections - Our engineering support team asked us to include this AskF5 KB based off the increase in customer questions related to VoIP via VPN. AskF5 K12524516: APM Network Access (VPN) compression causes higher CPU usage - We're also seeing in uptick in customers calling in on this issue. If you're experiencing performance issues with BIG-IP APM, check here for a quick resolution. What to do if you're experiencing an attack? The darker side is malicious users are taking advantage of the business upheaval and trying out new attack vectors and old favorites too. If you're trying to balance out the needs of your user's remote issues, let us help you with managing the influx of bad actors. Contact our F5 Security Incident Response Team (SIRT) . They're here and ready to help. Contact Us We will continue to update this document as we gain more insights from the field. As always if you have questions please login (or sign up if you're new) and hop on over to the DevCentral Q&A where our community of F5 technical professionals are happy to assist. And if you found something useful that helped you manage your remote workers tell us! We'll be happy to spread the news. It's amazing to see the support everyone provides during these difficult times and we're always proud to work and support you.APM DTLS Virtual Server iApp
Problem this snippet solves: TCP VPN tunnels suffer from an issue known as "TCP meltdown" where packets dropped on the tunnel cause both TCP tunnels to backoff and we see a sawtooth throughput pattern. See https://en.wikipedia.org/wiki/Tunneling_protocol This iApp will clone your existing TCP VPN virtual server and create a UDP DTLS virtual server on port 4433. It will also enable DTLS on your selected Network Access profile. Tested on v13.1 but not used in a production deployment Available on Github at https://github.com/pwhitef5/apm-vpn-optimisation/blob/master/apm_dtls_iapp.tmpl How to use this snippet: Install the iApp as normal Deploy a service and select the VPN virtual server and Network Access profile in use The newly created virtual server will be a clone of the original called <VS name>_dtls and listening on UDP Code : cli admin-partitions { update-partition Common } sys application template /Common/apm_dtls_iapp { actions { definition { html-help { } implementation { set app_dir [tmsh::pwd] set app_name $tmsh::app_name # https://support.f5.com/csp/article/K54955814 set vs [lindex [ tmsh::get_config ltm virtual $::main__vsname ] 0] set profiles [ tmsh::get_field_value $vs profiles ] set connectivityProfile "" set clientsslProfile "" foreach profile $profiles { set name [ lindex [ split $profile ] 1 ] set newname "${name}-dtls" # Retrieve the connectivity profile if { ! [ catch { tmsh::get_config apm profile connectivity $name } ] } { set connectivityProfile $name } # Retrieve the client-ssl profile if { ! [ catch { tmsh::get_config ltm profile client-ssl $name } ] } { set clientsslProfile $name } } if { $connectivityProfile == "" } { error "Error! Virtual server $::main__vsname does not have a connectivity profile assigned" } if { $clientsslProfile == "" } { error "Error! Virtual server $::main__vsname does not have a client SSL profile assigned" } set vip "[lindex [split [ tmsh::get_field_value $vs destination ] : ] 0 ]:4433" # Check if VLANs are set if { [catch {set vlans [ tmsh::get_field_value $vs vlans]} err ] } { # VLANS are not configured set vlans "" } else { # VLANS are configured if { $vlans != "" } { set vlans " vlans-enabled vlans replace-all-with \{ $vlans \}" } else { set vlans "" } } # Check if SNAT is set if { [ catch {set snatType [ tmsh::get_field_value $vs source-address-translation.type ] } err ]} { # No SNAT pool set snat "" } elseif { $snatType == "automap" } { # Automap set snat " source-address-translation \{ type automap \}" } elseif { $snatType == "snat" } { # SNAT pool is set set snatpool [ tmsh::get_field_value $vs source-address-translation.pool ] set snat " source-address-translation \{ type snat pool $snatpool \}" } else { # Should never hit this set snat "" } # Create VS tmsh::create ltm virtual $newname $vlans $snat ip-protocol udp destination $vip profiles replace-all-with \{ $connectivityProfile \{ context clientside \} $clientsslProfile \{ context clientside \} \} # Change network access profile tmsh::stateless enabled tmsh::modify apm resource network-access $::main__networkAccessProfile dtls true tmsh::stateless disabled } presentation { section main { # The entry below creates a large text box that must be filled out with a valid IP Address # For details of APL, look at the iApps developers guide: # https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-iapps-developer-11-4-0.html message intro "This iApp will create a DTLS virtual server based on your existing VPN virtual server, and add DTLS support to the network access profile" choice vsname display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items ltm virtual]" } choice networkAccessProfile display "large" tcl { package require iapp 1.1.0 return "[iapp::get_items apm resource network-access]" } } text { # Entities below set the text for the questions and section names, etc. Make them simple and relevant. main "Main" main.intro "Usage" main.vsname "VPN Virtual Server" main.networkAccessProfile "Network Access Profile" } } role-acl none run-as none } } description "iApp to create a DTLS virtual server based on your existing TCP VPN virtual server" ignore-verification false requires-bigip-version-max none requires-bigip-version-min none requires-modules { apm } signing-key none tmpl-checksum none tmpl-signature none } Tested this on version: 13.0APM VPN Optimisation iApp
Problem this snippet solves: This is an iApp which creates an iCall that runs every 5 minutes and checks the 5-minute average CPU rate averaged across all CPUs. Depending on the CPU rate, features are enabled or disabled such as compression. Note that this will make changes to all of your connectivity profiles ( except the built-in `connectivity` profile ) and Network Access profiles so you should have a backup before deploying. Changes are auto-applied to all SSL-VPN APM access profiles Logging is done to /var/log/ltm as shown below. CPU > 90% Compression Off Split-tunneling On Default Rate Class 100K Client-traffic-classifier Enabled CPU > 60% Compression Off Split-tunneling On Default Rate Class 500K Client-traffic-classifier Enabled CPU > 40% Compression Off Split-tunneling Off Default Rate Class 1M Client-traffic-classifier Enabled CPU > 20% Compression On Split-tunneling Off Default Rate Class 1M Client-traffic-classifier Enabled CPU < 20% Compression On Split-tunneling Off Default Rate Class 1M Client-traffic-classifier Disabled I have tested the basic workings of this but it has NOT been tested in a production environment. I would be happy to have some pilot customers to try it out and help develop. Source code is held at https://github.com/pwhitef5/apm-vpn-optimisation/tree/master How to use this snippet: Copy and paste the text below into a document on your PC or jump server Navigate to the BIG-IP GUI iApps>Templates. Click on Import Tick 'Overwrite Existing Templates' and select the file you created in step 1. Click Upload Create a service by navigating to iApps>Application Services>Applications. Click Create Call the service 'APM-VPN-Optimisation' or a suitable name, select the 'apm-vpn-optimisation_icall' template Hit Finished To view the changes made by the iCall, login to the BIG-IP via ssh and run the command `tailf /var/log/ltm` Example logs: Mar 20 14:55:00 apm-1 notice scriptd[9780]: 01420004:5: apm-vpn-optimisation:CPU rate: 1 Mar 20 14:55:00 apm-1 notice scriptd[9780]: 01420004:5: apm-vpn-optimisation:Turning on compression for profile myConnectivity: compress-gzip-level 6 Mar 20 14:55:00 apm-1 notice scriptd[9780]: 01420004:5: apm-vpn-optimisation:Turning on compression for profile networkAccess: compression gzip Mar 20 14:55:00 apm-1 notice scriptd[9780]: 01420004:5: apm-vpn-optimisation:Turning off split-tunneling for profile networkAccess: split-tunneling false Mar 20 14:55:00 apm-1 notice scriptd[9780]: 01420004:5: apm-vpn-optimisation:Creating client-rate-classes and client-traffic-classifier-1. rate: rate_class_1M Code : cli admin-partitions { update-partition Common } sys application template /Common/apm-vpn-optimisation_icall { actions { definition { html-help { } implementation { set app_dir [tmsh::pwd] set app_name $tmsh::app_name set icallTemplate {# Retrieve the CPU usage set cpuStatus [tmsh::get_status sys cpu] set numCpus 0 set totalUsage 0 foreach {cpu} $cpuStatus { incr numCpus set name [tmsh::get_name $cpu] set value [tmsh::get_field_value $cpu cpu-info.${name}.five-min-avg-system ] incr totalUsage $value } set cpuRate [ expr { $totalUsage / $numCpus } ] tmsh::log "apm-vpn-optimisation:CPU rate: $cpuRate" # Set features on or off if { $cpuRate > 90 } { set compression 0 set split-tunneling 1 set rate-class "rate_class_100K" set client-traffic-classifier 1 } elseif { $cpuRate > 60 } { set compression 0 set split-tunneling 1 set rate-class "rate_class_500K" set client-traffic-classifier 1 } elseif { $cpuRate > 40 } { set compression 0 set split-tunneling 0 set rate-class "rate_class_1M" set client-traffic-classifier 1 } elseif { $cpuRate > 20 } { set compression 1 set split-tunneling 0 set rate-class "rate_class_1M" set client-traffic-classifier 1 } else { set compression 1 set split-tunneling 0 set rate-class "rate_class_1M" set client-traffic-classifier 0 } set changed 0 # Set compression set connectivityProfiles [tmsh::get_config apm profile connectivity all-properties] foreach {profile} $connectivityProfiles { set name [tmsh::get_name $profile] if { $name == "connectivity" } { continue } # Get current status set currentStatus [tmsh::get_field_value $profile compress-gzip-level] if { $currentStatus < 1 && $compression > 0 } { # If it is turned off and should be on then turn on # Turn on tmsh::log "apm-vpn-optimisation:Turning on compression for profile $name: compress-gzip-level 6" tmsh::modify apm profile connectivity $name compress-gzip-level 6 } elseif { $currentStatus > 0 && $compression < 1 } { # Turn off tmsh::log "apm-vpn-optimisation:Turning off compression for profile $name: compress-gzip-level 0" tmsh::modify apm profile connectivity $name compress-gzip-level 0 } } set networkAccessProfiles [tmsh::get_config apm resource network-access all-properties] foreach {profile} $networkAccessProfiles { set name [tmsh::get_name $profile] set currentStatus [tmsh::get_field_value $profile compression] if { $currentStatus == "none" && $compression > 0 } { # Turn on tmsh::log "apm-vpn-optimisation:Turning on compression for profile $name: compression gzip" tmsh::modify apm resource network-access $name compression gzip set changed 1 } elseif { $currentStatus == "gzip" && $compression < 1} { # Turn off tmsh::log "apm-vpn-optimisation:Turning off compression for profile $name: compression none" tmsh::modify apm resource network-access $name compression none set changed 1 } } # Set split-tunneling set networkAccessProfiles [tmsh::get_config apm resource network-access all-properties] foreach {profile} $networkAccessProfiles { set name [tmsh::get_name $profile] set currentStatus [tmsh::get_field_value $profile split-tunneling] tmsh::begin_transaction if { $currentStatus != "true" && ${split-tunneling} > 0 } { tmsh::log "apm-vpn-optimisation:Turning on split-tunneling for profile $name: split-tunneling true" tmsh::modify apm resource network-access $name address-space-exclude-dns-name add \{ office.com microsoftonline.com google.com gmail.com facebook.com \} tmsh::modify apm resource network-access $name address-space-include-subnet \{\{ subnet 10.0.0.0/8 \} \{ subnet 172.16.0.0/16 \} \{ subnet 192.168.0.0/16 \}\} tmsh::modify apm resource network-access $name split-tunneling true set changed 1 } elseif { $currentStatus == "true" && ${split-tunneling} < 1 } { tmsh::log "apm-vpn-optimisation:Turning off split-tunneling for profile $name: split-tunneling false" tmsh::modify apm resource network-access $name split-tunneling false set changed 1 } tmsh::commit_transaction } # Create rate class tmsh::log "apm-vpn-optimisation:Creating client-rate-classes and client-traffic-classifier-1. rate: ${rate-class}" tmsh::stateless enabled tmsh::begin_transaction tmsh::create apm resource client-rate-class rate_class_4M \{ rate 4000000 \} tmsh::create apm resource client-rate-class rate_class_2M \{ rate 2000000 \} tmsh::create apm resource client-rate-class rate_class_1M \{ rate 1000000 \} tmsh::create apm resource client-rate-class rate_class_500K \{ rate 500000 \} tmsh::create apm resource client-rate-class rate_class_100K \{ rate 100000 \} tmsh::create apm resource client-traffic-classifier client-traffic-classifier-1 \{ entries add \{ \ entry \{ client-rate-class ${rate-class} dst-ip any dst-mask any dst-port https src-ip any src-mask any \} \ entry0 \{ client-rate-class rate_class_2M dst-ip any dst-mask any dst-port stun protocol 17 src-ip any src-mask any \} \ entry1 \{ client-rate-class rate_class_2M dst-ip any dst-mask any dst-port twrpc protocol 17 src-ip any src-mask any \} \ entry2 \{ client-rate-class rate_class_2M dst-ip any dst-mask any dst-port plethora protocol 17 src-ip any src-mask any \} \ entry3 \{ client-rate-class rate_class_2M dst-ip any dst-mask any dst-port cleanerliverc protocol 17 src-ip any src-mask any \} \ \} \} tmsh::commit_transaction tmsh::stateless disabled set networkAccessProfiles [tmsh::get_config apm resource network-access all-properties] foreach {profile} $networkAccessProfiles { set name [tmsh::get_name $profile] set currentStatus [tmsh::get_field_value $profile client-traffic-classifier] tmsh::begin_transaction if { $currentStatus != "client-traffic-classifier-1" && ${client-traffic-classifier} > 0 } { # Turn on tmsh::log "apm-vpn-optimisation:Turning on client-traffic-classifier for profile $name: client-traffic-classifier client-traffic-classifier-1" tmsh::modify apm resource network-access $name client-traffic-classifier client-traffic-classifier-1 set changed 1 } elseif { $currentStatus == "client-traffic-classifier-1" && ${client-traffic-classifier} < 1} { # Turn off tmsh::log "apm-vpn-optimisation:Turning off client-traffic-classifier for profile $name: client-traffic-classifier none" tmsh::modify apm resource network-access $name client-traffic-classifier none set changed 1 } tmsh::commit_transaction } # Apply profiles if { $changed > 0 } { set accessProfiles [tmsh::get_config apm profile type] foreach {profile} $accessProfiles { set name [tmsh::get_name $profile] # Check type of profile is ssl-vpn if { [tmsh::get_field_value $profile type] == "ssl-vpn" } { tmsh::log "apm-vpn-optimisation: Applying SSL-VPN access profile $name" tmsh::modify apm profile access $name generation-action increment } } } # Left blank } tmsh::create sys icall script "${app_name}_avo_script definition { [tmsh::expand_macro $icallTemplate ] }" tmsh::create sys icall handler periodic ${app_name}_avo_handler interval 300 script ${app_name}_avo_script } presentation { section main { # The entry below creates a large text box that must be filled out with a valid IP Address # For details of APL, look at the iApps developers guide: # https://support.f5.com/kb/en-us/products/big-ip_ltm/manuals/product/bigip-iapps-developer-11-4-0.html message intro "APM VPN Optimisation Version 1 20/3/2020" message usage "Note that this iApp will create an iCall which runs every 5 mins and changes your connectivity and network-access profiles automatically. You should backup your configuration before use to allow rollback to original configuration" } text { # Entities below set the text for the questions and section names, etc. Make them simple and relevant. main "Main" main.intro "" main.usage "" } } role-acl none run-as none } } description "APM VPN Optimisation iApp v2" ignore-verification false requires-bigip-version-max none requires-bigip-version-min none requires-modules { apm } signing-key none tmpl-checksum none tmpl-signature none } Tested this on version: 13.0APM Optimisation Script
Problem this snippet solves: With the current Covid-19 lockdown, many workers are now working from home which is putting stress on existing APM VPN devices. This script looks through the config and suggests some changes to be made to reduce CPU usage, based on https://support.f5.com/csp/article/K46161759 Matthieu Dierick has created a YouTube video showing how to use this at https://youtu.be/F0Z1AnM3L54 Let me know if you have any questions or requirements. Source code is held at https://github.com/pwhitef5/apm-vpn-optimisation/tree/master How to use this snippet: Copy the file to the /var/tmp directory as apm-optimisation Give it permissions with `chmod +x /var/tmp/apm-optimisation` Run with `/var/tmp/apm-optimisation`. Output is to stdout Example: [root@apm-1:Active:Standalone] ~ # ./apm-optimisation APM Optimisation Visibility CPU Usage -------------------------------- Current Average Maximum 52% 30% 93% -------------------------------- Compression -------------------------------- Licensed Hardware unlimited None -------------------------------- --- Partition /Common --- Connectivity Profile Compression -------------------------------- Profile Name Status -------------------------------- myConnectivity Disabled myConnectivity2 Disabled -------------------------------- Network Access Profile Compression ----------------------------------------------------------------------------------------------------------- Name | Compression | Split-Tunneling | Client Traffic Classifier | DTLS ----------------------------------------------------------------------------------------------------------- networkAccess | Enabled | Enabled | Disabled | Enabled networkAccess2 | Disabled | Enabled | Disabled | Disabled ----------------------------------------------------------------------------------------------------------- --- Optimisation Suggestions --- - CPU rate is LOW. Go down the Winchester and wait for it all to blow over - Hardware Compression is not included so consider turning off the feature ------- Partition /Common ------- - To turn off compression in the connectivity profile, run the command 'tmsh modify apm profile connectivity /Common/myConnectivity compression disabled' - To turn off compression in the NA profile, run the command 'tmsh modify apm resource network-access /Common/networkAccess compression none' - To turn on Client Traffic Classifier, run the commands below: tmsh create apm resource client-rate-class /Common/rate_class_2M { rate 2000000 } tmsh create apm resource client-rate-class /Common/rate_class_1M { rate 1000000 } tmsh create apm resource client-traffic-classifier /Common/client-traffic-classifier-1 { entries add { entry { client-rate-class rate_class_1M dst-ip any dst-mask any dst-port https src-ip any src-mask any } } } tmsh modify apm resource network-access /Common/networkAccess client-traffic-classifier client-traffic-classifier-1 - Network Access profile /Common/networkAccess is using SNAT automap. Consider using a SNAT pool - To turn on Client Traffic Classifier, run the commands below: tmsh create apm resource client-rate-class /Common/rate_class_2M { rate 2000000 } tmsh create apm resource client-rate-class /Common/rate_class_1M { rate 1000000 } tmsh create apm resource client-traffic-classifier /Common/client-traffic-classifier-1 { entries add { entry { client-rate-class rate_class_1M dst-ip any dst-mask any dst-port https src-ip any src-mask any } } } tmsh modify apm resource network-access /Common/networkAccess2 client-traffic-classifier client-traffic-classifier-1 - To turn on DTLS, create a duplicate virtual server listening on UDP and enabled DTLS in the Network Access List Network Settings ( see https://devcentral.f5.com/s/articles/APM-DTLS-Virtual-Server-iApp ) - Network Access profile /Common/networkAccess2 is using SNAT automap. Consider using a SNAT pool ----------------------------------------------------------------------------------------------------------- Code : #!/bin/bash # Version 5 8/4/2020 P.White # This is a script to check your APM system and give suggestions to reduce CPU usage # Taken from suggestions at https://support.f5.com/csp/article/K46161759 # v2 - small typo fix line 119 create changed to modify # v3 - updated classifier to only include https as it was causing an error # v4 - loops through admin partitions and prints out for each # v5 - added DTLS check and suggestion suggestions="--- Optimisation Suggestions ---\n" getLicensedCompression () { # Show the licensed compression comp=`tmsh -q show sys license detail|grep perf_http_compression|awk '{print $2}'|sed 's/\[\(.*\)\]/\1/g'` if [ x$comp != "x" ];then echo -n "$comp" else echo -n "Error!" fi } getHardwareCompression () { # Show hardware compression hcomp=`tmsh -q show sys license detail|grep "HTTP Hardware Compression"` if [ x$hcomp = "x" ];then # Hardware compression is not enabled echo -n "None" else echo -n "$hcomp" fi } clear echo "APM Optimisation Visibility" # CPU usage cur=`tmsh -q show sys cpu |grep "Utilization"|awk '{print $2}'` avg=`tmsh -q show sys cpu |grep "Utilization"|awk '{print $3}'` max=`tmsh -q show sys cpu |grep "Utilization"|awk '{print $4}'` if [ $avg -gt 90 ];then suggestions+=" - CPU rate is VERY HIGH! Turn off compression, implement split tunneling and consider more processing\n" elif [ $avg -gt 60 ];then suggestions+=" - CPU rate is HIGH! Turn off compression and consider split tunneling for non-internal traffic\n" elif [ $avg -gt 40 ];then suggestions+=" - CPU rate is MEDIUM. Consider turning off compression where required\n" else suggestions+=" - CPU rate is LOW. Go down the Winchester and wait for it all to blow over\n" fi echo echo "CPU Usage" echo "--------------------------------" echo -e "Current\tAverage\tMaximum" echo -e "$cur%\t$avg%\t$max%" echo "--------------------------------" echo # Compression clic=`getLicensedCompression` chw=`getHardwareCompression` if [ $chw = "None" ];then suggestions+=" - Hardware Compression is not included so consider turning off the feature\n" fi echo "Compression" echo "--------------------------------" echo -e "Licensed\tHardware" echo -e "$clic\t$chw" echo "--------------------------------" # loop through adminstrative partitions for partition in `tmsh -q list auth partition one-line|awk '{print $3}'`;do suggestions+="\n------- Partition /$partition -------\n" echo " --- Partition /$partition ---" echo echo "Connectivity Profile Compression" echo "--------------------------------" echo -e "Profile Name\t\tStatus" echo "--------------------------------" for profile in `tmsh -q -c "cd /$partition;list apm profile connectivity one-line"|awk '{print $4}'`;do if [ $profile = "connectivity" ];then continue fi if [ `tmsh -q -c "cd /$partition;list apm profile connectivity $profile one-line"|grep "compress-gzip-level 0"|wc -l` -gt 0 ];then echo -e "$profile\t\tDisabled" else suggestions+=" - To turn off compression in the connectivity profile, run the command 'tmsh modify apm profile connectivity /$partition/$profile compress-gzip-level 0'\n" echo -e "$profile\t\tEnabled" fi done echo "--------------------------------" echo echo "Network Access Profile Compression" echo "-----------------------------------------------------------------------------------------------------------" echo -e " Name\t\t\t| Compression\t| Split-Tunneling\t| Client Traffic Classifier\t| DTLS" echo "-----------------------------------------------------------------------------------------------------------" for profile in `tmsh -q -c "cd /$partition;list apm resource network-access one-line"|awk '{print $4}'`;do # Compression if [ `tmsh -q -c "cd /$partition;list apm resource network-access $profile one-line"|grep "compression gzip"|wc -l` -gt 0 ];then echo -en "$profile\t\t| Enabled" suggestions+=" - To turn off compression in the NA profile, run the command 'tmsh modify apm resource network-access /$partition/$profile compression none'\n" else echo -en "$profile\t\t| Disabled" fi if [ `tmsh -q -c "cd /$partition;list apm resource network-access $profile one-line"|grep "split-tunneling true"|wc -l` -gt 0 ];then echo -en "\t| Enabled" else echo -en "\t| Disabled" suggestions+=" - To turn on split-tunneling, run the command 'tmsh modify apm resource network-access /$partition/$profile split-tunneling true'\n" suggestions+=" - To configure split-tunneling exclude traffic by DNS name, run the command 'tmsh modify apm resource network-access /$partition/$profile address-space-exclude-dns-name add { office.com microsoftonline.com google.com gmail.com facebook.com }'\n" suggestions+=" - To configure split-tunneling exclude traffic by IP address, run the command 'tmsh modify apm resource network-access /$partition/$profile address-space-include-subnet add { { subnet 10.0.0.0/8 } { subnet 172.16.0.0/16 } { subnet 192.168.0.0/16 } }'\n" fi if [ `tmsh -q -c "cd /$partition;list apm resource network-access $profile one-line"|grep "client-traffic-classifier "|wc -l` -gt 0 ];then echo -en "\t\t| Enabled" else echo -en "\t\t| Disabled" suggestions+=" - To turn on Client Traffic Classifier, run the commands below:\n" suggestions+="tmsh create apm resource client-rate-class /$partition/rate_class_2M { rate 2000000 }\n" suggestions+="tmsh create apm resource client-rate-class /$partition/rate_class_1M { rate 1000000 }\n" suggestions+="tmsh create apm resource client-traffic-classifier /$partition/client-traffic-classifier-1 { entries add { entry { client-rate-class rate_class_1M dst-ip any dst-mask any dst-port https src-ip any src-mask any } } }\n" suggestions+="tmsh modify apm resource network-access /$partition/$profile client-traffic-classifier client-traffic-classifier-1\n" fi if [ `tmsh -q -c "cd /$partition;list apm resource network-access $profile one-line"|grep "dtls true"|wc -l` -gt 0 ];then echo -en "\t\t\t| Enabled" else echo -en "\t\t\t| Disabled" suggestions+=" - To turn on DTLS, create a duplicate virtual server listening on UDP and enabled DTLS in the Network Access List Network Settings ( see https://devcentral.f5.com/s/articles/APM-DTLS-Virtual-Server-iApp )\n" fi # Check for SNAT automap if [ `tmsh -q -c "cd /$partition;list apm resource network-access $profile one-line all-properties"|grep "snat automap"|wc -l` -gt 0 ];then suggestions+=" - Network Access profile /$partition/$profile is using SNAT automap. Consider using a SNAT pool\n" fi echo "" done echo "-----------------------------------------------------------------------------------------------------------" # Check VSs for mirroring for vs in `tmsh list ltm virtual one-line|awk '{print $3}'`;do if [ `tmsh -q -c "cd /$partition;list ltm virtual $vs mirror"|grep "mirror enabled"|wc -l` -gt 0 ];then echo echo "WARNING! Virtual Server /$partition/$vs has mirroring enabled\n" echo suggestions+="Consider disabling Connection Mirroring for virtual server /$partition/$vs with the command 'tmsh modify ltm virtual /$partition/$vs mirror disabled'\n" fi done done echo echo -e "$suggestions" echo "-----------------------------------------------------------------------------------------------------------" Tested this on version: 13.0
