You would need to use an iRule and session/tables to store client IP addresses, then every time a new connection takes place look up the IPs in the table, if it exists then close the new connection down.
When a connection closes you would need to make sure you removed it from the table.
This would only slow a brute force attack down if someone is always using the same source address not if someone has a way of connecting using multiple IP addresses. And this also could block valid traffic as multiple users connecting from behind the same NATed private network and so would share a common IP address seen by the F5 as the source address.
Assuming this is for a SSH connection it would be better to get the backend system to lock accounts after a set number of failed login attempts and even look if a user can be limited to a single authenticated session.