iRules and robots.txt question

I think you're really talking about two different things: 1) putting a robots.txt file on the BIG-IP for robots to consume, and 2) limiting robot access.

2 is generally covered by the link you provided.

1 is pretty straight forward:

when HTTP_REQUEST {
    if { [string tolower [HTTP::uri]] equals "/robots.txt" } {
         HTTP::respond 200 content "content of robots.txt..."
    }
}