Nimbostratus
CirrocumulusASM Brute-force protection does not delay attackers who exceed the configured thresholds of failed login attempts, it blocks them (if in Blocking mode of course), so there is no "Progressive delay" however there is a "blocking duration", e.g. if an attacker exceeded 20 failed login attempts per second you block them for 10 minutes. If they come back after 10 minutes and carry on the attack you block them for further 10 minutes.
If you want some sort of "progressive delay" using time delay as per your description it can be done by developing a custom iRule, but in this case I struggle to understand why you need it as ASM has a built-in mechanism for dealing with such cases using statistical analysis of failed login attempts. I believe these will achieve the same or a similar goal you are after.
Here are the Brute Force Detection settings:
Minimum Failed Login Attempts: X/second
Indicates an attack if, for all IP addresses tracked, the number of failed login attempts is equal to, or greater than, this number. This setting prevents false positive attack detection. The default value is 20 login attempts per second.
Failed Logins Attempts increased by X%
Indicates an attack if, for all IP addresses tracked, the ratio between the detection interval and the history interval is greater than this number. The default value is 500 %.
Failed Login Attempts Rate reached X/second
The system considers unsuccessful login attempts to be an attack if, for all IP addresses tracked, the login attempt rate reaches this number. The default value is 100 login attempts per second.
Re: Blocking error message/HTTP response - can be anything you want to configure, it is absolutely flexible