limit user account access to specific virtual servers or pools.

I don't think this is feasible in 11.x or current 12.x code version. You can provide different levels of access using user-roles but that is valid for all the configuration items and can't be set per-VS.