iRules how can I get to custom error message?

Firstly, you should not be using global variables, these are prefixed with the double colon. These cause a CMP demotion

https://support.f5.com/csp/article/K13033

# do not set variables in the global namespace
set ::foo bar
 
# set in the local scope
set foo bar

Additionally I would be cautious setting variables in RULE_INIT as they will only be initialised when TTM or the iRule is reloaded. As you've used global variables, $::active_connections_site will only reset to 0 under these two conditions.

The error message you received is because you are using the event HTTP_REQUEST and attempting to use an HTTP response without an HTTP profile assigned to the virtual server . You will also need to add an client-ssl profile to the virtual server if this is using port TCP/443

Lee