Excess site use

[gibby]

Wondering if anyone can figure this out & help

Our hosting company has informed us today that 2/3rd of their server resources are being used today due to excess access to our site.
this has happened once before but we couldnt figure out why

basically we just update web orders & processing etc & not uploading or backing up anything, although they are telling me abnormally high usage levels are being recorded from our IP address.

they are also concerned that spiders may be causing a problem & Im trying to find out how to stop the spiders or slow them down.

any ideas?

Thanks

G

[Aspect Investments]

If they cant cope, change your host.

[Steve2507]

If they cant cope, change your host.I agree, your host in theory shouldn't have a problem.

However it may be worth looking at your logs to see if you are being attacked in some way.

[glencooley.com]

Couple of things, make sure your robots.txt file is telling the spiders the right things to visit etc.

Secondly if you have google webmaster tools, turn your crawl rate down.

Thirdly make sure that any of the processing pages you use on the site arent doing anything silly like not closing connections to a database etc.

HTH

[Online Trader]

I agree with Steve, check your logs, we had an attack a few weeks ago from an IP address in France that was causing us all sorts of grief, we blocked the IP address and the problem was sorted.

[gibby]

Couple of things, make sure your robots.txt file is telling the spiders the right things to visit etc.

Secondly if you have google webmaster tools, turn your crawl rate down.

Thirdly make sure that any of the processing pages you use on the site arent doing anything silly like not closing connections to a database etc.

HTH

Thanks for the replies

I have slowed down the crawl rates

How do I make sure the robots.txt file is telling the spiders what to do & not to do

Im also trying to find the logs in google webtools but having trouble

cheers

G

[glencooley.com]

Hi Gibby

Your logs will be on your server not googles. Ask your ISP for the logs if they arent available in your site folders.

As for the robots.txt have a quick google and just make sure that you are only letting them crawl the content you want them to crawl, exclude things like image directories etc.

Thanks

[office man]

I'd change host asap - your usage could get even bigger.

[NuBlue]

I think it is a touch unfair to blame your host. From the sounds of it something is wrong with your website and your host has bought it to your attention.

A few things worth checking:

- Any database connections are not set to persistant (will be in the config of your cms/application)
- Check your web stats and see if anything suspicous is shown (someone streaming content from your site etc..)
- Ask your host to check the logs for your domain to see if they notice anything suspicous
- Ask for clarification as to exactly which 'resources' you are using a lot of as this may help pinpoint the area to look at.

[gibby]

Thanks, the hosting co are really good.
Im glad they brought it to my attention, but what ever it was - was using 2/3rds of the entire shared server & slowing other sites down.

Looking at the logs an ip was dragging alot of data down - which they have now blocked.
It also turns out that a competitor (of sorts) has copied quite alot of our content and even changed categories etc to match our exactly.

G

[NuBlue]

Looking at the logs an ip was dragging alot of data down - which they have now blocked. It also turns out that a competitor (of sorts) has copied quite alot of our content and even changed categories etc to match our exactly.
G

Thought it may be something like that! Take it as a compliment that your competitors think so highly of you :) ......... and then sue them!! :p