Forum Discussion

MihirJoshi_3137's avatar
MihirJoshi_3137
Icon for Nimbostratus rankNimbostratus
Mar 14, 2017
Solved

Allow user agents to scrape my website

I am trying to write a iRule which allows facebopot to scrape my webpage

 

when HTTP_REQUEST { if { [HTTP::header "User-Agent"] contains "Valid_Agents_for_Facebook" } { ASM::disable } else { ASM::enable /Common/abc_Sec_Policy } }

 

Valid_Agents_for_facebook is the list of agents. I tested but not working.

 

application delivery
ASM Advanced WAF
BIG-IP
security
  • MihirJoshi_3137's avatar
    MihirJoshi_3137
    Mar 14, 2017

    Finally got it working,

     

    when BOTDEFENSE_ACTION { if { [class match [string tolower [HTTP::header "User-Agent"]] contains "Valid_Agents_for_Facebook"] } { BOTDEFENSE::action allow } }

     

2 Replies

Sort By
  • Vijay_E's avatar
    Vijay_E
    Icon for Cirrus rankCirrus
    Mar 14, 2017

    Are the values within the datagroup case-sensitive and have you added them as they are expected to show up from facebook ?

     

  • MihirJoshi_3137's avatar
    MihirJoshi_3137
    Icon for Nimbostratus rankNimbostratus
    Mar 14, 2017

    Finally got it working,

     

    when BOTDEFENSE_ACTION { if { [class match [string tolower [HTTP::header "User-Agent"]] contains "Valid_Agents_for_Facebook"] } { BOTDEFENSE::action allow } }