Troubleshooting Lync/Skype for Business Reverse Proxy
At F5’s Agility conference, I spoke briefly about troubleshooting Lync/Skype for Business Reverse Proxy at the request of our support people. Regardless if you deploy Lync/Skype via iApps or manually define your BIG-IP elements, there’s rumbling suspicion of witchcraft around how this works. Well settle down there Salem. Looking at support call metrics, we bundled a rather high percent of support cases into two specific issues; certificate deployment problems and misconfigurations with port redirection. Deploying reverse proxy starting with simple configurations and working up to your more complicated requirements will significantly reduce headaches down the road. Reverse Proxy’s Port Redirection At it’s core, the reverse proxy functionality for Lync/Skype is a simple public endpoint listening on port 80 and 443. External connections are port redirected to internal front end pools, listening on 4443 and 8080 respectivly (if you don't redirect 80 to 443 as we do). Services managed through reverse proxy are required for mobile, autodiscover, and extended collaboration services; presented via the following public DNS entries: Meet.yourdomain.com Dialin.yourdomain.com Lyncwebservice.yourdomain.com (this name is up to you and added to topology builder during deployment) Lyncautodiscover.yourdomain.com (required for mobile) When deploying reverse proxy services at F5 (and certifying our functionality for Microsoft), we simply CNAME’d the meet, dialin, and lyncautodiscover namespaces to the FQDN of the web service IP. Fig. 1 shows the mobile client initiating two separate connection points into our Lync/Skype environment. Primary client login is achieved via SIP registration through edge services (access:443). The client also initiates a secondary connection to the web services namespace (lyncwebservices.fqdn listening on 443/80 in Fig. 1). You could add authentication mechanisms and security features with ASM/APM, but when we certified BIG-IP for Lync Reverse Proxy, we achieved validation using a FastL4 virtual translating 443 to 4443 only. Microsoft’s certification process ensures the device(s) responsible for reverse proxy functionality allow their services to work, any extra features added are secondary and up to the admin for ownership and support. Remember, we’re starting simple to make sure we get it working first. From here let’s build certs to get this listening on 443. Building/Installing your Certificate We now have published public DNS records, all used by the reverse proxy, only one name requires a host record binding, the FQDN defined in topology builder for the external web services (in our example lyncwebservice.domain.com). You’ll still add all other CNAME’s to the SAN certificate and Lync/Skype’s cert builder will configure this for your public and private domains. I purchased my Entrust certificate and it included two bonuses, the CA root and intermediary chain certs; they're super important later. Seeing that I use a fastL4 for the reverse proxy functionality, there’s no SSL termination/bridge required at the external LTM in Fig. 1 and instead I’ll bridge SSL at the internal front end pool virtual listening on 4443; again for simplicity and testing. We also install this certificate chain on the external web services defined within the Lync/Skype installation wizard, allowing us to separate certs for internal and external features. Microsoft makes this fairly easy so don’t try to deviate TOO far from recommended configurations. If you want to do this on two separate certificates (one for web services and one for edge services) that’s fine, if you want to create one mega SAN cert, go for it. Microsoft’s cert creation wizard will potentially add more names than you need, so just pay attention. Places of Potential Failure (read: where everything blows up) The previously mentioned secondary connection our client makes to the Lync/Skype web services in Fig. 1 houses a potential hidden evil; A SECONDARY AUTHENTICATION! Breath deep! It’s ok. The secondary authentication creates a client session token from a front end server via the web service's Certificate Provisioning Services published URI (a great write up on this is found here). This authentication is critical to all reverse proxy web services and will cause very odd behavior in a failure scenario, I’ll give examples later. To validate this URI, initiate a browser connection to https://lyncwebservices.yourdomain.com/CertProv/CertProvisioningService.svc and if it’s listening properly, you’ll get a login dialog. It will accept your domain credentials and give you a basic landing page stating “You have created a service”. If you can authenticate from a public connection, you’re most of the way there. If this doesn’t work, check the following: Are you SNATTING the reverse proxy (or is outbound traffic for web services is going back out via the same device; MSFT isn’t going to help with this) Is the node for the reverse proxy pool is listening on 4443 (configure a monitor for this port because the same IP will listen for services on 443 and could cause false positive/negative results) The perimeter LTM cannot reach the internal front end pool; time to check/blame the firewall if you have one separating internal and DMZ networks; Lync/Skype’s planning tool will show you all of the ports required between various entry points. 443 isn’t listening on the public IP or your DNS is pointing to a black hole. …. you get the idea.... basic pathway troubleshooting Start with networking basics; NMAP, Dig/Host/NSLookup, and curl are your friends here. It takes 5 minutes to run NMAP to validate listening ports, Dig public records to validate DNS is publishing proper namespaces, and curl to validate URL’s responses (even if auth prevents full HTTP connections, for that you could OpenSSL s_client). Once you get the web service public URL responding, the next step is testing with a client, and here's where certificates have the potential to cause major headaches. Lync/Skype is one of the few applications who's client logs are actually useful and will prevent hours of packet capturing at your LTM and front end servers. Enable your client logs from within the app and locate them in the following locations: OSX: /Users//Library/Logs/Microsoft-Lync-0.log (OSX provides a generic log location for applications so there’ll be quite a few) Windows: C:\Users\\AppData\Local\Microsoft\Office\15.0 (or whatever)\Lync\Tracing\ Login to Lync/Skype and your newly generated logs will display a plethora of piñatas… a.k.a. client communication data. Search for one of two things: the FQDN of the published web services, OR the phrase WWW-Authenticate. Both will get you to the web service authentication token creation as shown below: WWW-Authenticate: NTLM realm="SIP Communications Service", targetname="feserver.fqdn", version=4 WWW-Authenticate: TLS-DSK realm="SIP Communications Service", targetname=“feserver.fqdn", version=4, sts-uri="https://lyncwebservices.yourdomain:443/CertProv/CertProvisioningService.svc" WWW-Authenticate: TLS-DSK opaque="3AFFA71F", gssapi-data="FgMBDKsCAA ......AAA4AAAA=", targetname="feserver.fqdn", realm="SIP Communications Service", version=4 WWW-Authenticate: TLS-DSK opaque="3AFFA71F", gssapi-data=”XXXXXXX", targetname="feserver.fqdn", realm="SIP Communications Service", version=4 This log shows us the client connecting to public web service URI and completing authentication. If you validated previous steps 1-5… and you see the connection completing… but it still doesn't work.... turn your anger towards certs. Remember that Entrust certificate I purchased that included a cert chain file and CA root file? If those are not installed within the client OS, the authentication will fail because Lync/Skype will run it’s own internal certificate validation (similar to issues with Lync/Skype requiring a valid non-self-signed certificate for Exchange UM). Lync/Skype is not a dumb client and will ONLY work if the certificate being used is a proper CA-owned certificate; this can be an internal CA or public CA (choose a CA who hasn't run afowl of the internet law). Troubleshooting this stage of reverse proxy is best done with Microsoft’s Remote Connectivity Analyzer. Run the Lync/OCS Autodiscover Web Service Remote Connectivity Test and the results will show your certificate information, including cert chain information. If you have cert issues, they’ll show up here. This should be the end of the story, but you’re a smart admin and nothing just “works”. In F5’s case, we had a lot of OSX users complaining of the inability to search for users within Lync for Mac. Well, smart admin, we know client search policies default to file download and web search (for the record, MSFT corporate uses web search only). In my case, the Lync for Mac client was ONLY trying to use web search functionality, and because these were non-standard machines, they didn’t include the Entrust L1K intermediary chain certificate within the OSX keychain. This prevented the web services certificate from validating against the CA root of the public certificate and only in certain connection behaviors, caused search to fail because the previously mentioned web service authentication failed. In this case, the client log only showed the failure to authenticate and we had to determine the cert as the root cause on our own, but the logs did narrow down the issue significantly. In all, this wordy article condenses down to 4 primary troublshooting points: Make sure the public listener 443 redirects to 4443 on the internal front end pool Make sure you have valid certificates answering for internal and external web services defined in Lync/Skype Make sure your clients HAVE THE WHOLE CERT CHAIN for internal and external services Make sure to validate the network path between client and front end server; it can be a few hops or complicated 10+ hop massive network pain in some cases The mentioned certificate issue I troubleshot with MSFT Tier III Premier support went much longer than I care to admit and from that ticket they did add improved logging to the Lync for Mac client but really it was an oversight on my part. We did our due diligence on a specific set of client machine connection variables, but we didn’t have several alternate connectivity scenarios in our testing plan. Oops. ITIL to the rescue and the new admins won’t make that mistake again. Let me know your experiences and thoughts on this subject; I’ve spoken to several people at various conferences and even hopped on a few support calls for F5 to help out with this specific feature of Lync/Skype. Our awesome PD team added support for reverse proxy in our Lync/Skype iApp templates so deployments should go much smoother (check out the current release candidate here). I hope this helps your troubleshooting, if it does(n’t), let me know and I'll be glad to help where possible Cheers!
