Forum Discussion

Fletcher_Cocquy's avatar
Fletcher_Cocquy
Icon for Nimbostratus rankNimbostratus
May 14, 2012

iRule to redirect crawlers and bots

Hi - by policy we are open to crawlers and bots to get our sites indexed.   Sometimes they take up all our apache threads (despite robots.txt settings)     Is there an iRule to key off the us...
application delivery
dev
devops
iRules
nathe's avatar
nathe
Icon for Cirrocumulus rankCirrocumulus
May 14, 2012
Fletcher

 

 

Check out this Codeshare link using the switch command: https://devcentral.f5.com/wiki/iRules.ControllingBots.ashx

 

 

You can amend accordingly, of course, to control which brand of bots.

 

 

Rgds

 

N