Multiple Requests From IP Address Maxing Out CPU

[a1anm]

My server (VPS) was getting constant requests from IP range 72.21.217.*

It was maxing out my CPU so I blocked this range of IP's and things are back to normal.

Now I'm worried that it could have been something important (search engine etc).

Is there anyway to verify where the requests where coming from?

There is some talk about this IP address here but I am unsure what it means:
http://www.webmasterworld.com/search_engine_spiders/3828718-9-30.htm

Any ideas?

 

[yorkukhosting]

The IP addresses are from an Amazon managed block:

http://whois.arin.net/rest/net/NET-72-21-192-0-1/pft

I presume they were hitting Apache on your VPS, what does your httpd activity logs show for the period you noticed the high CPU usage?

 

[Faevilangel]

The IP address resolves to Amazon's hosting service (EC2)

It seems from that thread, someone is using Amazon's services to run bots / spiders which go viewing websites.

It could be clean but if it's maxing your cpu, then it's probably not something good, as they should only be doing checks sporadically.

Can you check your server logs to see where they were viewing, as it might be a brute force attack to try and login to your website admin panel

you don't use WordPress do you?

 

[a1anm]

Hi,

Yes, it was apache that was causing the high CPU.

I'm unsure how to check the access logs now but I could see from my Magento log that they were just viewing/downloading thousands of images. I think it had been going on for about 24 hours.

Not wordpress....just Magento.

 

[Ryan McFadyen]

Technically it belongs to Amazon but being outbound it's probably someone running something on their hosting service and not Amazon themselves. Someone is using it to access your server. Why is a tricky question to answer, a user of yours might have built themselves an alternate way to access and hasn't properly controlled it or it's someone scraping data and not caring.

Any legitimate (or good intentioned) access should have the common decency to throttle itself so as it's not I would assume its illegitimate access. So block it but keep an eye out for users reporting you've broken something of theirs.

FYI if a user/company has built their own tool of some kind that's just out of control you are in no way obligated to make sure that continues to work but obviously it's good customer service to see if you can help them fix it if that is the case.

 

[Faevilangel]

Hi,

Yes, it was apache that was causing the high CPU.

I'm unsure how to check the access logs now but I could see from my Magento log that they were just viewing/downloading thousands of images. I think it had been going on for about 24 hours.

Not wordpress....just Magento.

hmmm, it could be an image search system then?

This is where my knowledge gets lacking (server stuff) so will let the pro's help you more

 

[yorkukhosting]

The log locations depends upon your Apache config but you could try the following locations as a starting point:

/var/log/httpd
/var/log/apache2

If you are not too familiar with Linux try using WinSCP to access those paths and then down the 'access log' file to your desktop and have a trawl through. The log may at least provide an agent string that may give an indication of what it is.

 

[edmondscommerce]

You did the right thing to block it. The bottom line is that if it was bringing down your server then you had to block it

As a rule, real search engine spiders and other 'good' bots are written to have good manners, if they detect that your server is struggling they back off politely and let your server pull itself together again.

If they were doing directory traversal which would explain a large amount of activity in the magento images directories (they are very broken up into lots of nested folders) then that means it was looking for ways to hack in.