Banning G**gle

Eagle

Free Member
Oct 3, 2004
5,235
587
UK
I'm having serious (and very expensive) bandwidth drain - and it appears it's mostly all G**glebot's fault.

How can I effectively ban Googlebot (and other big offenders) from my forum? Does manually banning an IP address in the forum software CP work? Will that damage the Adsense?

I've heard that a robots.txt file isn't effective. :(
 
D

DotNetWebs

Earlier in the year I had a similar problem with a certain spider. Have a look at this thread here:

http://www.ukbusinessforums.co.uk/forums/showthread.php?t=14183&highlight=spiders

The only way I could stop it was to block the IP on the server. The offenders actually acknowledged this and to be fair to them where quite helpful and eventually removed me from their database.

I have to say though I would very, very, reluctant to do this to the Google spider. I have always found it very well behaved and is vital to bring visitors to my site.

MSN is my favourite though. Last night I started a new Ancestry section on my forum and this morning I rank no1 on MSN for 'Horsham Ancestry'. Google always takes a while to catch up.

Regards

Dotty
 
Upvote 0

epiphany

Free Member
May 15, 2005
793
0
Scotland
Eagle, I think there is a tool in the update to Google's webmaster tools that allows you to alter the frequency of the GOOGLEBOT scans.

However, I think the best option with PHPBB would be to create a sitemap which tells Google not to scan certain pages. A PHPBB install with just a few dozen threads will generate thousands of pages in an automatically generate sitemap. In creating your sitemap you need to only point links to content items, the generator I use allows you to block URL's with specific bits of code in it. i.e. "posting.php" in the URL on PHPBB will chop hundreds of pages off of a sitemap. There are a few other files and bits of code to chop off depending on the version you are using, eventually you should be left with googlebot just scanning content

Hope that helps old boy :)
 
Upvote 0

Latest Articles