bots

3 Topics

Google bot Unreachable on 301 redirect iRules
Hello, premise: I'm not managing directly the F5, I'm just the web designer of the company, so I'm pretty new with this kind of stuff; I’m migrating all European websites (ccTLDs) of the company I work for to a global one (.com), and I have some issues with the google boot and 301 redirects. For example, what I'm doing: Example.fr > example.com/fr Example.co.uk > example.com/en-gb …etc.. My system eng. suggested me to implement the redirect with F5 iRules. This means that the actual flow is: Local domain name DNS changed to the F5 IP, inside F5 I put the iRules to redirect the pages to the new global website: i.e. example.fr > F5 > script for the redirect 301 > example.com/fr The iRules code I used is the following: when HTTP_REQUEST { if { ( [HTTP::host] eq "www.kerrdental.co.uk") or ( [HTTP::host] eq "kerrdental.co.uk" ) } { if { ( [HTTP::uri] contains "/AccessoriesForFillingMaterials/AmalgamAccessories/AmalgamGuns" ) } { HTTP::respond 301 Location "https://www.kerrdental.com/en-uk/dental-restoration-products/amalgam-guns-accessories" } elseif { ( [HTTP::uri] contains "/AccessoriesForFillingMaterials/CompositeRestorativeSystems/CompoRoller" ) } { HTTP::respond 301 Location "https://www.kerrdental.com/en-uk/dental-restoration-products/comporoller-accessories" } } } Now, the problem I have concerns the “Change of address” feature in Google Search Console (Webmaster tool): Search Console notify me that “Cannot confirm that 301-redirects work properly, we couldn’t crawl your site. Check that Googlebot can crawl the root of your site using the Fetch as Google tool” Then, when I go under “do Fetch as Google” the system returns me the status “Unreachable”. When I use RexSwain on the HTTP version of the old URL, I get a 503. Receiving Header: HTTP/1.1·503·Service·Unavailable(CR)(LF) Content-Type:·text/html;·charset=UTF-8(CR)(LF) Content-Length:·931(CR)(LF) Connection:·close(CR)(LF) P3P:·CP="CAO·PSA·OUR"(CR)(LF) Expires:·Thu,·01·Jan·1970·00:00:00·GMT(CR)(LF) Cache-Control:·no-store,·no-cache,·must-revalidate,·post-check=0,·pre-check=0(CR)(LF) Pragma:·no-cache(CR)(LF) (CR)(LF) Can someone help me to understand why google boot cannot find anything? Thank you in advance, Best regards
Solved
Oeg87_310913 Nov 22, 2022 Place Technical Forum
504Views
0likes
2Comments
Lightboard Lessons: What are Bots?
Humans account forless than 50% of internet trafficand the rest is spread between the good bots and bad ones. In this Lightboard Lesson, I light up some #basics about internet bots and botnets. ps Related: The Facts about Botnets The state of botnets in late 2015 and early 2016 What are bots? Configuring BIG-IP Application Security Manager (ASM)
PSilva Oct 18, 2017 Place Technical Articles
321Views
0likes
0Comments
Controlling Bots
Problem this snippet solves: Webbots, you can't live with them, you can't live without them... This iRule determines if a webbot is accessing your systems and assigns them to a lower priority resource. The first example includes the bot list inside the rule and uses the switch statement to find a match. Code : when HTTP_REQUEST { switch -glob [string tolower [HTTP::header User-Agent]] { "*scooter*" - "*slurp*" - "*msnbot*" - "*fast-*" - "*teoma*" - "*googlebot*" { # Send bots to the bot pool pool slow_webbot_pool } default { # Send all other requests to a default pool pool default_pool } } } ### or if you prefer data groups ### ---- String Class ---- class bots { "scooter" "slurp" "msnbot" "fast-" "teoma" "googlebot" } ---- iRule ---- when HTTP_REQUEST { if { [matchclass [string tolower [HTTP::header User-Agent]] contains $::bots] } { pool slow_webbot_pool } else { pool default_pool } } Tested this on version: 10.0
bwolmarans_1284 Mar 06, 2017 Place CodeShare
577Views
0likes
1Comment