Portal Access Link Rewriting all External links that don't have same hostname.

I am trying to migrate our web links over to Portal Links, this does take a lot of time but the big issue I've got at the moment is having external css and js files getting rewritten to the APM. This is causing the websites to take extremely long to load (around 5 minutes).

Does anyone know how I can configure things to avoid rewriting external libraries? Just a word of warning there is a lot of the F5 I don't know so keeping the info simple would be extremely helpful.

Thanks in advance.