Nimbostratus
The problem you are describing is what we call "web scraping". It is a different problem than L7 DOS, although, sometimes they are related.
Other solutions in the market provide very limited functionality to prevent it, this kind of functionality and much more could be implemented in an iRule.
For example: what you can do is once you identified a session like that (based on an HTTP header/value or or by counting for example the number of requests on a session) route that bot to a different web server using an HTTP class that is sending all the traffic to a different destination pool (you can configure a different pool for a class) , the pool's address can be locally or if you want to be bad - externally.... This way you keep the bot busy crawling, but the content it gather is the content that you want it it to get...
Did that bot cause a DOS on your server? if it did, you may want to look at version 10, available for beta.
Cheers,
Ido
