Implementing SSL Orchestrator - Validation & Troubleshooting
Introduction
This article is part of a series on implementing BIG-IP SSL Orchestrator. It includes high availability and central management with BIG-IQ.
Implementing SSL/TLS Decryption is not a trivial task. There are many factors to keep in mind and account for, from the network topology and insertion point, to SSL/TLS keyrings, certificates, ciphersuites and on and on. This article focuses on verification that the solution is working.
This article is divided into the following high level sections:
- How to check if content is being decrypted
- How to check if content is being blocked
- How to check if content is being bypassed
- Logging and Troubleshooting
Please forgive me for using SSL and TLS interchangeably in this article.
Software versions used in this article:
BIG-IP Version: 14.1.2
SSL Orchestrator Version: 5.5
BIG-IQ Version: 7.0.1
How to check if content is being decrypted
Go to a computer that connects to the internet through SSL Orchestrator. Go to an encrypted website like www.cisco.com. The page should load without error. Click the padlock icon in the Address Bar.
Notice that the connection is secure and the certificate is valid. Click the Certificate to view more details. The certificate was issued to www.cisco.com but has been issued by subrsa.f5labs.com. The connection to this web site was decrypted and re-encrypted by SSL Orchestrator.
Switch to the Palo Alto UI and go to Monitor > Traffic. It should look like this.
Click the link on the left in the red box.
This drills into more details.
Notice the IP address of 72.163.10.10 and port 80 (not port 443). A quick lookup of that IP shows it belongs to Cisco.
How to check if content is being blocked
Now go to the website www.eicar.org. This site hosts a harmless test malware file so you can see if your security measures are effective. Click Download Anti Malware Testfile on the right.
Click the Download link on the left.
Scroll down on the next page. You should see a section with 8 different downloads.
There are 4 different file types. The first group uses HTTP. The second group uses HTTPS. Click any of the HTTPS links to verify it is detected and blocked.
You should get a block page like the following.
Notice the URL indicates an attempt to download one of the compressed .zip files.
How to check if content is being bypassed
Next try a Banking website and make sure it is bypassed. On the client open a browser and go to www.chase.com. Click the Padlock icon in the address bar and it should look like this.
Click the highlighted section for more details about this certificate.
This certificate was issued by Entrust CA. The SSL Orchestrator is not decrypting this connection. We can check the Palo Alto logs to make sure it doesn’t have a record of this transaction.
The image below shows ping traffic generated by SSL Orchestrator to test the health of the L2 Service. Notice there is no web-browsing or port 80 traffic.
Logging and Troubleshooting
To enable logging:
-Connect to the WebUI
-Go to SSL Orchestrator and click on Logs
-Enable the required logging levels
Note: By default, logs are stored locally on the BIG-IP.
Below is a reasonably-ordered list of troubleshooting steps.
· If the SSL Orchestrator deployment process fails, review the ensuing error message. It would be impossible to list here all of the possible error messages and their meanings, but often enough the messages will reveal the issue.
· Re-review the lab steps for any missing or misconfigured settings.
· Enable debug logging in the SSL Orchestrator configuration. Tail the APM log from a BIG-IP command line or from the logs page in the management UI. Debug logging will very often reveal important issues. Specifically, it will indicate traffic classification matches, mismatches or deployment issues.
tail –f /var/log/apm tail -f /var/log/restnoded/restnoded.log tail -f /var/log/restjavad.0.log
· If the SSL Orchestrator deployment process succeeds, but traffic isn’t flowing through the environment made evident by lack of access to remote sites from the client:
o Never enable debug logging on the Per-Request Policy logging option in a production environment. If needed, be sure to turn it back off after capture logs. The PRP logging is extremely verbose and WILL affect system performance.
o /var/log/apm is used to log data plane traffic flow
o /var/log/restnoded/restnoded.log is used to log control plane (when SSLO configurations are deployed)
o /var/log/restjavad is similarly used to log control plane restjava functions
o It’s also sometimes useful to tail /var/log/ltm, which is where SSL and generic data plane issues will show up.
o Ensure that the client is properly configured to either default route to the ingress VLAN and self-IP of the BIG-IP for transparent proxy access or has the correct browser proxy settings defined for explicit proxy access.
o Ensure that traffic is flowing to the BIG-IP from the client with a tcpdump capture at the ingress interface.
o Review the LTM configuration created by the SSL Orchestrator. Specifically, look at the pools and respective monitors for any failures.
o Isolate service chain services. If at least one service chain has been created, and debug logging indicates that traffic is matching this chain, remove all but one service from that chain and test. Add one service back at a time until traffic flow stops. If a single added service breaks traffic flow, this service will typically be the culprit.
o If a broken service is identified, insert probes to verify inbound and outbound traffic flow. Inline services will have a source (S) VLAN and destination (D) VLAN, and ICAP and receive only services will each have a single source VLAN. Insert a tcpdump capture at each VLAN in order to determine if traffic is getting to the device, and if traffic is leaving the device through its outbound interface.
o How to insert probes. The services VLANs are wrapped in application services so must be addressed accordingly in the tcpdump. Note that each inline service will create two separate VLANs – one for inbound and one for outbound, so it becomes easy to surgically insert captures at specific points in the flow (ie. to the service, coming from the service). So a tcpdump capture of a service named “FEYE” might look like this:
tcpdump -lnni ssloS_FEYE.app/ssloS_FEYE
Where “ssloS_FEYE.app” is the application service container, which contains the “ssloS_FEYE” VLAN.
o By default the ‘all traffic’ rule doesn’t attach a service chain, so it can be as simple as removing service chains from all of the security rules. If traffic doesn’t flow with no service chains attached anywhere, then the problem is in SSLO proper, likely a routing or decryption issue. If the traffic only fails with a service chain applied, this is when it’s useful to isolate the services. If a broken service is identified, insert tcpdump probes as described above.
o If traffic is flowing through all of the security devices, insert a tcpdump probe at the egress point to verify traffic is leaving the BIG-IP to the gateway router.
o If traffic is flowing to the gateway router, perform a more extensive packet analysis to determine if SSL if failing between the BIG-IP egress point and the remote server.
tcpdump –i 0.0:nnn –nn –Xs0 –vv –w <file.pcap> <any additional filters>
Then either export this capture to WireShark or send to ssldump:
ssldump –nr <file.pcap> -H –S crypto > text-file.txt
o If the WireShark or ssldump analysis verifies an SSL issue:
o Plug the site’s address into the SSLLabs.com server test site at:
This report will indicate any specific SSL requirements that this site has.
o Verify that the SSL Orchestrator server SSL profiles (two of them) have the correct cipher string to match the requirements of this site. To do that, perform the following command at the BIG-IP command line:
tmm --clientciphers ‘CIPHER STRING AS DISPLAYED IN SERVER SSL PROFILES’
Further SSL/TLS issues are beyond the depth of this guide. Seek assistance.
• If all else fails, seek assistance.
Summary
In this article you learned how to verify that SSL Orchestrator is decrypting SSL and passing it to an inline security device. You can verify that SSL is decrypted simply by viewing web site certificates. You can also use logging or reporting capabilities of an inline security device to verify that SSL is decrypted. You learned how to test whether the inline security device was blocking malicious content. You also learned how to verify that a policy to bypass SSL decryption is working. Then you learned how to enable logging and perform troubleshooting steps.
Next Steps
Click Next to proceed to the next article in the series.
