Forum Discussion

Fletcher_Cocquy's avatar
Fletcher_Cocquy
Icon for Nimbostratus rankNimbostratus
May 14, 2012

iRule to redirect crawlers and bots

Hi - by policy we are open to crawlers and bots to get our sites indexed.

 

Sometimes they take up all our apache threads (despite robots.txt settings)

 

 

Is there an iRule to key off the user agent (eg bingbot)

 

 

User agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

 

 

to segment off this traffic to a separate pool from user traffic

 

 

Thereby isolating the users from bots taking up all the threads?

 

 

thanks!

 

1 Reply

  • nathe's avatar
    nathe
    Icon for Cirrocumulus rankCirrocumulus
    Fletcher

     

     

    Check out this Codeshare link using the switch command: https://devcentral.f5.com/wiki/iRules.ControllingBots.ashx

     

     

    You can amend accordingly, of course, to control which brand of bots.

     

     

    Rgds

     

    N