Banning G**gle

[Eagle]

I'm having serious (and very expensive) bandwidth drain - and it appears it's mostly all G**glebot's fault.

How can I effectively ban Googlebot (and other big offenders) from my forum? Does manually banning an IP address in the forum software CP work? Will that damage the Adsense?

I've heard that a robots.txt file isn't effective.

 

[PeteYoung]

The robots.txt file should keep the Googlebot in check if configured correctly. However you run the risk of your site not being regularly crawled and thus impacting on rankings.

You may find this useful as well
http://forums.searchenginewatch.com/showthread.php?t=14603

 

[Chris H]

Is this a phpBB forum?

 

[DotNetWebs]

Earlier in the year I had a similar problem with a certain spider. Have a look at this thread here:

http://www.ukbusinessforums.co.uk/forums/showthread.php?t=14183&highlight=spiders

The only way I could stop it was to block the IP on the server. The offenders actually acknowledged this and to be fair to them where quite helpful and eventually removed me from their database.

I have to say though I would very, very, reluctant to do this to the Google spider. I have always found it very well behaved and is vital to bring visitors to my site.

MSN is my favourite though. Last night I started a new Ancestry section on my forum and this morning I rank no1 on MSN for 'Horsham Ancestry'. Google always takes a while to catch up.

Regards

Dotty

 

[Optegris]

Is this a phpBB forum?

Yes it is a phpBB forum

 

[Eagle]

I might just lose the Adsense...

 

[Chris H]

Yes it is a phpBB forum

Worth following what dotnetwebs said regarding search spiders. If it's signup bots then try

http://www.phpbb.com/phpBB/viewtopic.php?t=383788

Chris

 

[Eagle]

It's definitely G**gle.

66.249.65.12

Just had another look and the f*@!%*s are still there... about 40 instances.

 

[epiphany]

Eagle, I think there is a tool in the update to Google's webmaster tools that allows you to alter the frequency of the GOOGLEBOT scans.

However, I think the best option with PHPBB would be to create a sitemap which tells Google not to scan certain pages. A PHPBB install with just a few dozen threads will generate thousands of pages in an automatically generate sitemap. In creating your sitemap you need to only point links to content items, the generator I use allows you to block URL's with specific bits of code in it. i.e. "posting.php" in the URL on PHPBB will chop hundreds of pages off of a sitemap. There are a few other files and bits of code to chop off depending on the version you are using, eventually you should be left with googlebot just scanning content

Hope that helps old boy

 

[PeteYoung]

It's definitely G**gle.

66.249.65.12

Just had another look and the f*@!%*s are still there... about 40 instances.

Is there any pattern to the instances ie certain directories?

 

[Eagle]

Thanks for the help, guys. Good suggestions, I'm sure, but I simply don't have time for all this so I think I'll just ban G**gle altogether.

 

[epiphany]

Remember they have 20+ datacentres with different ips

 

[Eagle]

Shutting it down periodically seems the best answer for now. G**gle wins.

 

[Optegris]

Shutting it down periodically seems the best answer for now. G**gle wins.

Not necessarily

I've sent you an email Mark...