3000 Google spiders a day

[Nuno]

On a small WP site, 32 pages, 20 posts but active Twitter feed I'm getting between 2000 and 4000 google spiders every day.

Statpress has an average of all spiders at 3900/day for June, Google webmaster has an average of 2310 for the last 90 days.

Unique visitors are about 100 a day, but page views is around 1000 which I think is high for this site.

What is happening?
What should I do?

 

[Subbynet]

Go to Google Webmaster Tools... Choose your domain, then click the gear icon top right > Site Settings... Then at the bottom change the crawl rate.

That should help.

 

[Nuno]

Cheers.

Taken to the minimum which is 333 second intervals that's about 260 a day.

Wait and see.

Any ideas as to why it happened?

Thanks again.

 

[Subbynet]

Its a guess but it might be down to googlebot trying to fill in the page parameters itself and causing a loop, so it keeps on hitting away like its viewing a new page.

But like I say that's a complete guess, you'd have to check the raw logs to find out exactly what its doing.

 

[Nuno]

Its with the hosting company now, and they say they'll get back. I'll be asking Google too. I'll post anything of interest here, (even if it is down to my muppetry.)

 

Cheers.

Taken to the minimum which is 333 second intervals that's about 260 a day.

Wait and see.

Any ideas as to why it happened?

Thanks again.

Is the site about flies.:|

 

[Nuno]

Earl, go and troll elsewhere.

 

Earl, go and troll elsewhere.

You seem to be confusing humor with something else.

Are your origins German.:|

The answer to why so many spiders is probably a link from an east european site.

 

[OldWelshGuy]

Are you serving up the same info on multiple urls? (unusual for a WP ste, but possible if it is configured badly).

 

[tommytx]

the massive spider hits... what pages are they reading... the same page 50 times a day... in other words what are the hits feeding on... I have never seen google hit a pg more than 1 or 2 times in sequence and rarely do they hit the same page more than once in a day...
So are the urls being hit the same 2o pages over and over.
IF you need a spider monitor that lets you see what the suckers are doing every instant let me know.. I have one.

 

[Nuno]

@OWG No, it's a really quite simple, vanilla site.
WP members is the only plugin I don't use elsewhere but the site has been up for 15 months with no change in architecture (?-right word?) and this started a few months back.
No other scripts, so I'll check plugins today and tomorrow, (time consuming to then measure Googlebot rate) and see what transpires.
Cheers.

 

[OldWelshGuy]

Check your stats for unusual URL's you might have been hacked and files deposited, look for unusual referrers etc

 

[Nuno]

I (and the host) had already checked for injections and inserted code, (base64 code), as well as checking the usual subjects manually, (.htaccess, index.php etc)

Statpress loads very slowly suggesting the problem is connected to that. It is not storing stats forever so thats not it, but gets better when another plugin is deactivated, (add multiple users. AMU).

I'm leaving AMU deactivated and watching the stats. Thanks for pointing to the (hopefully) right area.

 

[Subbynet]

If you login via FTP you should be able to download the raw log files. Then open that in Excel or whatever, filter it by the Googlebot User agent string and you'll see in an instant what its doing by the URL's its trying to reach.

Or download them and email them over to me and I'll have a look for you, shouldn't take long.

 

[Nuno]

I could only access them via CP. They seem to my (Noob) eye that the events plugin is getting too much attention. Would you mind having a quick look? gz folder or unzipped, (about 28kb)?

 

[Nuno]

Should have sorted this before asking so apologies. All in One Events calendar has a known problem with multiple Google crawls. I have changed the robots.txt file as they say here:
http://support.time.ly/limiting-excessive-google-crawls/
Thanks all for time and expertise.

 

[Nuno]

Little update. The amount of Googlebots has gone way on down, but it seems that our Chinese chum, Baidu, ignores the robots.txt file and has hit over 200 times today.
I have now tried to ban it using .htaccess so I'll see over the next few hours. Good game, good game...

 

[Nuno]

Well that hasn't worked. Can someone tell me where my mistake is please?


RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^Baiduspider [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Sogou [NC]
RewriteRule ^.*$ - [F]

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress


I sourced this from Stackoverflow.

 

[OldWelshGuy]

Why not just block the Baidu IP address completely?

 

[Nuno]

Why not just block the Baidu IP address completely?

In .htacess?
would that be:
RewriteCond %{HTTP_USER_AGENT} ^Baidu.com [NC,OR]

I thought the numeric address was dynamic so didn't go there.