Troubleshooting Kerberos Constrained Delegation: Strong Encryption Types Allowed for Kerberos
Over the past week, I was approached by two separate customers who happened to be enabling smart card authentication and ran into the same issue with Kerberos Constrained Delegation (KCD). While this was a rare coincidence, over the past few years enabling KCD has definitely been one of the most common issues I am engaged with. This is not because it is difficult to do but rather it’s not something you do every day and you must take into account the interactions between the BIG-IP, DNS, Active Directory, delegation account, Kerberos encryption types, time and the application itself. Because of that, I wanted to share my recent experience and troubleshooting tips. I am by no means the Kerberos expert but luckily I know a few who were able to provide the why after assisting the customers with the how. If you haven’t configured KCD and would like to before continuing through this guide, you can find a step by step walkthrough here. So with that, let’s begin with the use case.
Use case:
- Customers successfully deployed smart card authentication with Kerberos Constrained Delegation using the DevCentral article provided above on BIG-IP 13.1.1.
- To comply with organizational security policies, the use of AES 256/128 for Kerberos was enforced using group policy on their on their Windows 2016 Domain Controllers.
Computer Configuration >> Windows Settings >> Security Settings >> Local Policies >> Security Options >> "Network security: Configure encryption types allowed for Kerberos" to "Enabled" with only the following selected: AES128_HMAC_SHA1 AES256_HMAC_SHA1 Future encryption types
Enabling this policy results in the following registry setting located in \SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\Kerberos\Parameters\.
Value Type: REG_DWORD
Value: 0x7ffffff8 (2147483640)
- Immediately KCD began to fail which began the conversation around F5 supporting strong encryption types for Kerberos. Based on the F5’s KB article K18315582, this was a known issue on versions 12.0 and lower. However, with what we were experiencing it seemed to be the exact same issue.
"Kerberos: can't decrypt S4U2Self ticket for user 12345679@SITEREQUEST.COM - Decrypt integrity check failed (-1765328353)."
Kerberos Constrained Delegation Pre-requisites
- DNS Reverse Lookup Zones with appropriate A and PTR record(s)
- All of the Active Directory domains involved in Kerberos Delegated authentication must be at a Microsoft Windows Server 2003 Functional Level or higher
- Reliable time source to be used by Active Directory and the BIG-IP
Service Location (SRV) Records
You must configure the BIG-IP system with a DNS server that contains the proper Kerberos SRV records. SRV records are in the form of _kerberos._udp. or _kerberos._tcp.. These DNS entries are queried when the KDC field (Key Distribution Center) from the Kerberos SSO profile is empty and the dns_lookup_kdc value in the /etc/krb5.conf file is set to true. This behavior is how DNS advertises which KDC handles the authentication for a given realm. If these entries are not configured, SSO will likely fail.
If multiple entries exist for a service, UDP is used by default. In the event that the response contains too many KDC records (more than 512 bytes), DNS will switch to TCP on port 53 to request the same entries.
Note: For Kerberos to function correctly, in addition to SRV records, you must also configure A and PTR records for the web servers.
Note: Within this guide, I have defined the KDC in the Kerberos SSO Profile. However, as a best practice I highly recommend validating these records exist.
APM Delegation Account Configuration
Below are screenshots of a delegation account that has been configured to successfully support Kerberos Constrained Delegation.
For demonstration purposes only, when configuring the delegation account with “Trust this user for delegation to any service (Kerberos only)” or “Trust this user for delegation to specified services only (Use Kerberos only)”, I receive the following error.
Kerberos: can't get S4U2Proxy ticket for server HTTP/SP.siterequest.com@SITEREQUEST.COM - Requesting ticket can't get forwardable tickets (-1765328163)
While evaluating the delegation account, I also removed the servicePricipalName(s) on the delegation account to view the error it would generate.
Kerberos: can't get S4U2Self ticket for user 12345679@SITEREQUEST.COM - Server not found in Kerberos database (-1765328377)
Duplicate SPN’s
Based on Microsoft documentation, starting in Windows Server 2012 R2 Domain Controllers will block the creation of duplicate SPN’s though it is still possible to have duplicate SPN’s on domain controllers running 2012R2 and later as described in this article. In my current environment I am running Windows 2016 at a 2016 Domain and Forest functional levels. Because of this I was not able to generate duplicate SPN’s to determine the error it would generate on the BIG-IP. In future testing I would like to provide the error this will generate.
For validation purposes, use the setspn –X command to ensure there are no duplicate SPN’s.
F5 BIG-IP Kerberos Single Sign-On Profile
Below is a BIG-IP Single Sign-On profile that has been configured to successfully support Kerberos Constrained Delegation.
Note: To support AES encryption, you MUST configure the Account Name in SPN format as shown in this screenshot.
Ex. Account Name: host/apm-svc.siterequest.com
Based on my testing, if the account name is not provided in SPN format the following error will be generated in the APM logs.
Kerberos: can't get TGT for apm-svc.siterequest.com@SITEREQUEST.COM - Client 'apm-svc.siterequest.com@SITEREQUEST.COM' not found in Kerberos database (-1765328378)
F5 BIG-IP Visual Policy Editor Configuration for Smart Card Authentication and KCD SSO
Below is a configuration from my APM Visual Policy Editor that has been configured to successfully support Kerberos Constrained Delegation.
Output from F5 BIG-IP krb5.conf File
Below is a configuration from my krb5.conf file that resides on my BIG-IP successfully supporting Kerberos Constrained Delegation. A question was recently asked if because my realm in this guide is SITEREQUEST.COM, shouldn’t the default realm be configured in the configuration file? I can only speak based on testing that defining a default realm in the krb5.conf file had no impact on my KCD SSO. This is likely due to me defining a KDC and realm in my Kerberos SSO profile as shown later in this article.
Delegation Account Encryption Settings
Because the encryption type has been defined using group policy, I have not modified the KCD delegation account to use Kerberos AES encryption as shown in the screenshot below. Applying the encryption type using group policy allows organizations to define a global setting that will affect all the accounts on the computer where the policy is applied.
DNS Settings
One of the most common issues of KCD is the use of incorrect reverse lookup records or no reverse lookup record created at all. Using Microsoft documentation, I found the clearest documentation that I could on this subject.
“Determine whether you are connecting to the Web site by using the actual NetBIOS name of the server or by using an alias name, such as a DNS name (for example, www.microsoft.com). If you are accessing the Web server by using a name other than the actual name of the server, a new Service Principal Name (SPN) must have been registered by using the SetSPN tool. Because the Active Directory directory service does not know this service name, the ticket-granting service (TGS) does not give you a ticket to authenticate the user. This behavior forces the client to use the next available authentication method, which is NTLM, to renegotiate. If the Web server is responding to a DNS name of www.microsoft.com but the server is named webserver1.development.microsoft.com, you must register www.microsoft.com in Active Directory on the server that is running IIS. To do this, you must run the SetSPN tool on the server that is running IIS.”
With that, below is a screenshot of my reverse DNS zone for the service SP.SITEREQUEST.COM.
For demonstration purposes only, I deleted the PTR record for SP.siterequest.com to view the error I would receive.
The Kerberos event generated was “Kerberos: Failed to resolve IP address: ::ffff:10.1.20.111.”
NTP Settings
As with any other client, to prevent "replay attacks," the Kerberos v5 protocol uses timestamps as part of its protocol definition. For timestamps to work properly, the clocks of the client and the domain controller need to be in sync as much as possible. In other words, both devices must be set to the same time and date. While I have validated my time is in sync with my domain controller for this use case, the error message of Kerberos: can't get TGT for host/apm-svc.siterequest.com@SITEREQUEST.COM - Clock skew too great (-1765328347) would be generated in the APM logs if they had not been within the default 5-minute tolerance for Kerberos.
To validate NTP, you can use the command ntpq –pn. In the example below, I configured the BIG-IP’s time source to use the KDC.
- The delay field is the current estimated delay; the transit time between these peers in milliseconds.
- The offset field is the current estimated offset; the time difference between these peers in milliseconds.
- The jitter field is the current estimated dispersion; the variation in delay between these peers in milliseconds.
Clearing the Kerberos SSO Cache
When troubleshooting, it is recommended to clear the Kerberos SSO cache each time. In order to do this, simply run bigstart restart websso.
Reproducing the Issue
I was able to successfully reproduce the issue in my own environment running 14.1 when enforcing AES encryption and changing nothing else. Enabling Debug logging on my access policy allowed me to view the reason SSO was failing.
Testing with kinit and kvno
In my testing, I was still able to obtain a ticket on behalf of myself using the delegation account.
[root@bigip01:TimeLimitedModules::Active:Standalone] config # kinit -f host/apm-svc.siterequest.com
Password for host/apm-svc.siterequest.com@SITEREQUEST.COM
[root@bigip01:TimeLimitedModules::Active:Standalone] config # kvno -C -U steve host/apm-svc.siterequest.com
host/apm-svc.siterequest.com@SITEREQUEST.COM: kvno = 4
Using Wireshark on the Domain Controller
After validating the error in APM logs, I attempted to follow the Kerberos authentication flow using Wireshark on my domain controller.
KRB_AS_REQ
KRB_AS_REP
KRB_TGS_REQ
KRB_TGS_REP
BIG-IP APM SSO Logs
Based on the Wireshark captures show above, I am indeed obtaining an S4U2Self ticket which cannot be decrypted by the BIG-IP. This is also validated using my APM logs when enabling Debug logging.
Resolution
After testing many different methods and validating all settings based on F5 documentation as well as my own, I decided to attempt and reset my password based on a lot of references in Kerberos documentation around the client’s master key being generated from its password. Sure enough, this ended up resolving the error message "Kerberos: can't decrypt S4U2Self ticket for user 12345679@SITEREQUEST.COM - Decrypt integrity check failed (-1765328353)."
Now that I had a how, I wanted to know why this was required. After reaching out to my local Kerberos experts the following explanation. As I gather additional details on the why, I will continue to add to this document.
“The principal (account) is created using the system-default enctype. When you change the enctype, you must also recreate the principal, or at least update the principal’s password.”
References
https://docs.microsoft.com/en-us/windows/desktop/secauthn/ticket-granting-service-exchange
https://docs.microsoft.com/en-us/windows/desktop/secauthn/key-distribution-center
https://www.stigviewer.com/stig/windows_server_2016/2018-03-07/finding/V-73685
