Hello Omer,
When using ASM with APM, you will need to use the 'virtual' command in an iRule on the layered ASM VS to point at the APM VS.
This would not work with a pool member pointing at an APM VS due to how APM fires related to ASM in the hudchain.
Please review the following article:
K54217479: How to protect APM virtual server with ASM
Additionally, there is a little more in depth article using Brute Force Protection:
K13315545: Configuring a BIG-IP ASM virtual server to protect a BIG-IP APM login page with brute force prevention
Hope that helps!