SharePoint authentication issue with third party client application

It's helpful to first understand how APM initiates an access session. When a user first connects to an APM VIP, APM sends the client an immediate 302 redirect to /my.policy and embeds its session cookie. The (browser) client then follows the redirect to /my.policy which starts the "visual" policy evaluation. When the visual policy is complete, a second 302 redirect is issued to the original URL. So it basically looks like this:

-> HTTP request to /foo
<- 302 redirect to /my.policy (and set-cookie)
-> HTTP request to /my.policy (with cookie) and starts visual policy
<-> ... visual policy processing ...
<- 302 redirect to /foo
-> HTTP request to /foo (with cookie) and access to the application
-> ... subsequent requests contain the cookie and no further policy processing happens

This interaction is perfectly acceptable for the average browser session, but doesn't always work for other clients - clients that either don't support cookies or HTTP redirects. For clients that don't support HTTP redirects, there's a slightly different option called "clientless-mode" which essentially disables all of the redirects and sends the session cookie to the client in the first response from the server. Clientless-mode does place limitations on the types of agents that you can use in the visual policy though, specifically anything that creates a dialog with the user (ex. logon page, message box, etc.).

Since the web service client is trying to talk to /_vti_bin/Authentication.asmx with XML data, it sort of implies that APM can't interfere with the direct client-server communication. If you also need to use this VIP for interactive (browser-based) access, then I'd suggest to look for the User-Agent HTTP header (or any other tell-tale sign of web service client) and

1. Enable clientless-mode for these requests
2. Send the visual policy down a separate path that does not present interactive agents (ie. logon forms, message boxes).

The next question then becomes, how do web services clients authenticate? Are they literally trying to authenticate to the web service via XML (SOAP)? or can they pass other types of authentication (ie. Kerberos, NTLM, HTTP Basic, etc.)?

Thanks Kevin, yes food for thought. I suspected the re-direct was at least part of the problem after reading this article on dev central. There's a script there that enables "clientless-mode" for certain user-agents. So am testing this at the moment and will look at the visual policy path as well. Thank you for your thorough response.