Forum Discussion
Just my 2c, might not be relevant to your situation.
I experienced something similar when I was trying to set up an office online server and attach it to our SharePoint VIP with smart card auth. Turns out I didn't need to mess with SPNs/configure Kerberos or anything. SharePoint ACLs were handling the access to the files and the IIS site used anonymous authentication.
- eric_haupt1Oct 12, 2018Nimbostratus
action_; My deployment definitely requires SSO with KCD variables. My APM essentially does client-cert auth from any user anywhere and then proxies this user through into an internal sharepoint on a private domain/realm. We use identifiers within the cert the client provides to validate the client within the AD domain. All they need to provide is their cert and PIN to use it. APM takes care of the rest. From Sharepoint's perspective, all the IIS front-ends see is the F5 float IP making connections and sending TGS tickets on behalf of domain users.
I went from 11.6.1 HF2 to 13.1.1 last night with no change to this particular issue, but I really didn't expect things to change. My suspicion is something in this tenant domain, but I can't point fingers just yet...
- Kevin_StewartOct 12, 2018Employee
By tenant domain, do you a separate trusted domain for user accounts?
If so, there are a few things you need to do:
- You must ensure that the domains have a full two-way transitive trust (it wouldn't work at all if this wasn't the case)
- The APM SSO account and the target service (assuming IIS) must be in the SAME domain.
- APM must be able to DNS resolve (SRV records) the trusted domain and must have direct connectivity to it.
- To avoid ambiguity, the APM SSO account should use a full SPN that identifies what domain it's in (ex. host/f5kdc.example.com). This same string is needed in three places: the APM SSO username, the AD delegation account's userPrincipalName and servicePrincipalName.
- eric_haupt1Oct 29, 2018Nimbostratus
Kevin,
There won't be any trust between these domains. I call this a "tenant" domain from a network perspective. It exists completely separate from our primary infrastructure for security directives a long time ago.
-
APM SSO account and the target service are in the same domain
-
DNS resolution... I configured host entries for this domain (6 servers) on the LTM/APM but it would be better to be able to define DNS "domains" in TMOS for things like this. I have this cluster leveraging a GTM DNS Delivery listener and I use an irule to select the DNS pool based on DNS Question name... so requests with names containing "*tenant.domain" go to the two tenant DCs. I need to revalidate this. This domain/Realm has also been defined in krb5.conf.
The APM SSO account is specific to this domain, but does not contain the full domain name. Is this a requirement or are you suggesting simply for best practice? This will be my only application that is off of my primary domain.
I opened a TAC case on this... but so far nothing has been identified as erroneous in the config.
-
- Kevin_StewartOct 29, 2018Employee
There won't be any trust between these domains
I'm confused now. The trust I'm referring to would be between whatever domain the target server is in, and whatever domain the users are in. You absolutely MUST have a two-way transitive trust between these domains to enable cross-domain access through APM Kerberos SSO.
And if this is a multi-domain environment, you should use full SPN values everywhere to remove any ambiguity. Yes, the APM SSO account is specific to this domain, but you should still refer to it by its full SPN name (not a short name).
- eric_haupt1Oct 30, 2018Nimbostratus
Kevin, The APM is performing auth gateway functions for users with two things: 1 - a PKI cert, 2 - a user account in the tenant domain.
The users must have a app domain account to access the service. What APM will do is simply take the cert ID (UPN) and then proxy these users to the service via KCD. I do not care where the user is in the world as long as they have a valid cert with a UPN for a valid account within the app domain.
There is no cross domain here at all except for the LTM/APM being configured to support SSO on two domains depending on the APM policy configs. For this application, you can consider it single-domain - but the F5 itself is supporting two domains but not at the same time for the same application. From the Kerberos realm perspective, these users are all local and the communication is from the serverside float and F5 SPN to the IIS front-end.
- Kevin_StewartOct 31, 2018Employee
Now it's all starting to come together. ;)
So a few additional questions,
- Is it still producing the original error, or has it changed (working after the 3rd attempt)?
- When it works, do you actually see good APM logs to indicate it worked ("S4U == OK!")?
- In APM-to-KDC captures, do you see any differences between the Kerberos traffic of good and bad tests?
- Is there more than one server, and does it work consistently with one but not another?
- Is there more than one KDC, and have you looked to see if APM is talking to more than one of them between tests?
- Have you set the APM SSO account to use the full SPN syntax, vs. short name?
- Have you tried the KRB5_TRACE option detailed above?
- And just in case you've stumbled on a bug, have you opened a support case?
- eric_haupt1Oct 31, 2018Nimbostratus
Kevin,
We can replicate the issue every time. When it works I see the "S4U==OK" - note that I see this after the first attempt. It's just on the first attempt I see the "Cannot get forwardable" ... the second time I get "S4U==OK" but I don't process through fully without a "401" back. The third time I get "S4U==OK" and process through to the webapp with "200 OK" everytime.
In the packet captures we've taken, we see nothing out of the ordinary except a "response too big" or something along those lines for the TGT initially.
I can't allow APM to define the SPN using DNS so my patterns are always fully crafted in the SSO profile - This is just our standard.
I'm working to get access to the KDCs and IIS myself to perform packet captures do hopefully do a more thorough inspection. The config on the F5 side should work... everything is in place and correct and a similar config has worked flawlessly for use for over 2 years with hundreds of clients per day.
Now: the only difference I can think of is the service account. On the primary domain I can use host/service.account and it works fine. That is how I have the other domain configured. I will set a new service account with the host/service.account.domain.name and retest.
I have a case open... but nothing noted just yet. I'm getting ready to go from 13.1.1 to 13.1.1.2 in the next few days as well.
Thanks
- Kevin_StewartOct 31, 2018Employee
Hmm.
Can you also look at the clear text traffic to the server? I’m assuming, but it’d be good to validate that on every attempt APM actually does pass a Kerberos AP_REQ ticket to the app, and that for some reason the app doesn’t accept it. If you can get a wireshark view of the APM-server traffic you should be able to see the full Kerberos details. And if that’s true, it might imply that the server is at fault, the KDC is issuing a ticket to the wrong service, or there’s something wrong with the ticket.
- eric_haupt1Nov 02, 2018Nimbostratus
Kevin, we have time set aside next week to dig back into this and I'll let you know what we see. I'm working another app issue (the 401 response thing I have another question out for)
Thanks for all your guidance.
- eric_haupt1Nov 20, 2018Nimbostratus
I stood up an i2800 on TMOS 13.1.1.2 configured just for this domain and application. Same indications so it appears Kerberos within this domain either has issues or has a configuration outside the bounds of F5's recommended KCD logic. Every application I configured on F5 in my primary domain works without issue for Kerberos. I was kind of hoping something pointed to F5 under my control, but it appears to be domain specific or related to a configuration attribute outside of the baseline for KCD SSO. Frustrating.... I hate this application.