Try this for your hash variable:
set variableHash [sha512 $variable]
set lastTenHash [string range $variableHash [expr [string length $variableHash] - 10] end]
For your response you would need to do a
HTTP::collect [HTTP::header Content-Length]
in your when HTTP_RESPONSE event and then in the HTTP_RESPONSE_DATA event you would need to inspect the HTTP::payload for your string probably using scan. Be weary, that this will collect the whole payload of every response which could be a performance hit to the application.
https://devcentral.f5.com/wiki/iRules.HTTP_RESPONSE_DATA.ashx