In an SP initiated scenario, if you look at the SAML request that comes through (which you can do in Firefox using an addon called SAML Tracer, for example) you'll notice a couple of fields in the request:
Issuer
(which seems to match to the
Entity ID
you set in your SAML config in APM) and
Assertion Consumer Service URL
among others. I'm not exactly sure which one it uses (I think it's the Assertion Consumer Service URL), but the APM matches one of those values with the corresponding External SP Connector to figure out which one to use, and then does it's processing based on that.
Also, within the policy - through the VPE - you can assign resources to users that they should be allowed to access.
Hope this helps.