Formatting seems to have messed things up in your posting... Are you using the very first rule listed above (Reject or SNAT choices only).
Are your connections that work coming from multiple IP's? I see you're SNAT'ing everything in the iRule behind a single IP. Perhaps your SMTP server is limiting the connections for a single IP? (Most do to prevent a single remote site taking up all the inbound connections).
When you try 6 parallel connections and 3 time out, do they make it all the way to the SMTP servers? What do you see on a tcpdump? Does the banner come back to each client? Does the banner even get sent from all the server connections?
Do you know how to use tcpdump? It would be useful for you to verify all 6 connections coming into bigip, then whether all 6 connections get established to the SMTP servers, and what happens on both the client-side and server-side for each one.
H