ASM Brute Force when app success/fail isn't the first response?

No form parameters is a potential problem - you need at least username for brute force detection to work as ASM needs to associate the session with a user in order to count the number of failed login attempts for that username from the same IP address or within the same HTTP Session.

You might need a bespoke iRule if you can't have a reliable way to associate a username with a failed login response.