Can I bar an IP address from my website

[Ian J]

I regularly look at the stats on my websites and there is a lot of activity from 82-33-124-30.cable.ubr04.stav.blueyonder.co.uk who every couple of days will visit every page of all five of my sites for reasons completely unknown to me but he / she is obviously not a potential customer as having read my site they will normally either contact me or go away.

I know about .htaccess but my main site is hosted with UK2 and access to PHP or CGI is an extra that I don't curently pay for and I was wondering if there is any other way to bar access to the site for specific IP addresses other than through .htaccess

All other sites are hosted with compila.com which does allow PHP and CGI but I can't get the .htaccess to work with them.

Any ideas please ?

[Top Hat]

It is possible. but I'm not sure how. How about robots.txt you may be able to ban it from there.

Are you paying for bandwidth? if not worry about it when you are.

Probably a bad spider, harvesting email addresses, but you need to check before you ban it it may be good.

[microbe]

Are you sure that this is not a proxy server?

[Ozzy]

Nah that looks like a persons BlueYonder ADSL connection. As you host your site with UK2 it would seem you are unable to do anything about it. You could put some Javascript together to redirect elsewhere based on IP, but Javascript can be blocked by the browser.
Either move to a host that allows you to use .htaccess or PHP, or don't worry about it as Tim says.

Its probably a competitor, and I wouldn't worry about it. My competitors are all over my websites all the time I take that as a complement - so should you :)

[Ian J]

My main site is rated in the top slot by both Yahoo and Google so I expect competitors to have a close look but this chap keeps coming back several times a week.

A couple of months ago I spotted similar unusual activity coming from Brazil and shortly thereafter my site appeared in Portugese singing the praises of a Brazilian factoring company. They hadn't even bothered to change the coding which still contained links to my webdesigner and tracker code etc.

They ignored my requests to change their site until I added an extra page to my own saying "XYZ Factors are dishonest. They stole my website, what will they steal from you" and as that appeared in Google Brazil higher ranked than their own site they changed it straight away.

Anyway, thanks all for your advice

[Rob Holmes]

Hi,

I have never tried the options I found for you so if you do it its up to you. I'm not recommending it from a sales point of view and can't recomend it from a technical point of view as I haven't written or tested them. I would strongly suggest you install on a test domain first and get to know the scipt and how it behaves. Then IF you're happy move it across to your main site.

Heres details of a javascript that I like the look of ..
http://www.javascriptkit.com/script/script2/blockip.shtml

And ..
http://javascript.internet.com/user-details/ip-filter-2.html

They both look similar and like they will do what you want

Remember their IP is probably dynamic so you'd need to keep an eye on your logs (which you do anyway) and re-identify the user and update your script if their IP changes.

Hope this helps,

Rob

EDITED: To amend links

[Ian J]

Thanks - I'll give that a try.

[Rob Holmes]

Hi Ian

If you need some help testing then let me know and we can work on it live with you trying to block me to your site :)

Email is support (at) matrixxhosting dot com

Rob

[Ian J]

The best laid plans of mice and men :(

Someone from Blueyonder in the Staverton area has been having a nose around again late last night but the IP address is slightly different. It is too much of a co-incidence so I assume that he doesn't have a static IP address.

The command used in Google was site:www.factoringsolutions.co.uk which is not a command that I have come across before and it returns details of every page on my site that Google has indexed and shows that he is not just a potential customer as they wouldn't be interested in that sort of information.

I don't see what this person can be achieving by visiting every page on a weekly basis. My site ranks well with all of the major search engines but that is due to a mixture of longevity, site design & SEO, text, inbound links and probably other things too so no matter how many times he looks it won't benefit him.

[Rob Holmes]

hmm - Ian is the rest the same just IP different?

Rob

[Top Hat]

Sounds like they have some program that analyses your site in some way.

Keywords, images, SE placement who knows.

Keep an eye on your competition make sure they are not steeling content.

They are probably just monitoring you, trying to figure out how its done.

[microbe]

This is a very useful command, I frequently use it on its own or in combination to keep track of the pages on my own site which are indexed.

I thought this could be a competitor harvesting text from your site to use in a search engine campaign. I tried searches for large blocks of text from your site to see if they were being "recycled". I tried THIS (http://www.google.co.uk/search?q=%22It+is+food+for+thought+that+whilst+the +number+of+Administrators+appointed+by+factoring+c ompanies+should+be+roughly+in+proportion+to+the+nu mber+of+clients+they+have+on+board%2C+it+frequentl y+isn't%22&sourceid=mozilla-search&start=0&start=0&ie=utf-8&oe=utf-8&client=firefox-a&rls=org.mozilla:en-US:official) but I can't see an obvious problem. You might want to repeat this experiment to make sure that a copy of your site doesn't pop up some time in the future though.

Some SEO software uses this command to test positioning and could be used as part of a competitive analysis. This, though, wouldn't normally require visiting your site, just capturing the results of the search itself.

It seems likely that this person is attempting to track new content and changes to your site. Whatever their reason, my guess is that you are unlikely to benefit from it.

[Ian J]

hmm - Ian is the rest the same just IP different?

Rob

Last week it was 82-33-124-30.cable.ubr04.stav.blueyonder.co.uk and this week it is 82-46-102-109.cable.ubr04.stav.blueyonder.co.uk so whilst the IP is different, they are both from the same area and both display the same pattern of looking at every page whether related or not.

The other interesting thing that has just ocurred to me is that last night's vist was just before midnight but other visits are normally at the weekend so i would guess that the mystery visitor is employed fulltime during the week and doing whatever he does in his free time.

On what is probably an unconnected matter but may be connected, my main competitor in terms of page ranking on Google has just regained the prized top slot for the search term "factoring" and being part of a huge financial conglomerate they do spend a lot of money on SEO and pay per click and some of their SEO tricks are rather unorthodox.

If you have a look at who links to them according to Google there seem to be 400 inbound links from one site and that site is rather interesting. It has 9,000 pages (mainly hidden) and each one is chock full of links to other sites which are either filled with useless garbage or don't lead anywhere at all. It would seem that other sites in the banking group also benefit from this SEO technique as do other sites that use the same web company yet everything I read tells me that this is a Google nono and could lead to the site being banned from the index.

Are there different rules for household names :?:

[Rob Holmes]

Hi,
Google will just look at links to blue yonder thats why the figures are high ;)

If you can see your apache log files you'll be able to establish whether it's a human or a robot by the speed that they look at your whole site.
A robot will steam through your whole site in a matter of a few seconds probably, whereas a human will obviously take longer.

For example I have a script that can take a total copy of your site by me just typing in your domain name. We use it for legitimate reasons like taking copies of sites for customers whose webhost has gone AWOL.

If it's a robot we should be able to come up with a script that blocks an IP if it views more than say 3 to 5 poges a second. If it's human it's just as easy - you can just block connections from blue yonder - but the consequences of locking out anyone using blue yonder as an isp are slightly larger.

Let me know and I'll have a rummage around and see what I can do ;)

Rob

[microbe]

If you can catch them spamming a search engine then you could get them barred. Business.com got caught at it and Google barred them.

The things to look for are cloaking (serving a different page to the Google spider to the page for humans), intellectual property theft, etc.

[microbe]

Had a look at the sites you referred to (I think I found the right ones anyway). My view would be that yes someone is actively building junk pages, in large numbers, to attempt to drive search traffic.

However, I couldn't see anything much that would qualify as spam and get them banned by Google.

I tracked the company behind this site and I agree with you there is some fairly underhand stuff going on. Nothing that I could see that is criminal/illegal/immoral though.

My guess is that the guy visiting you is an SEO person monitoring your content for one reason and another. If you bar his IP then he will just come in some other more anonymous way so you will lose track of his activities.

[Ian J]

If you bar his IP then he will just come in some other more anonymous way so you will lose track of his activities.

Unfortunately you're probably right and I should spend less time worrying what he's up to and more time maintaining and developing my own site.

Thanks all for you help and comments

[Ian J]

The plot thickens. This is what I have extracted from the site stats tonight :-

08 Apr, Fri, 20:50:04 82-33-124-85.cable.ubr04.stav.blueyonder.co.uk Netscape 6 Windows XP
08 Apr, Fri, 21:19:12 82-33-124-85.cable.ubr04.stav.blueyonder.co.uk MSIE 6 Windows XP
08 Apr, Fri, 21:53:07 82-33-124-85.cable.ubr04.stav.blueyonder.co.uk Netscape 6 Windows XP
08 Apr, Fri, 21:53:53 82-33-124-85.cable.ubr04.stav.blueyonder.co.uk Netscape 6 Windows XP

Any ideas what the change of browser might signify :?:

[Rob Holmes]

Just someone seeing your cross browser compatability I suppose.

Rob

[Ozzy]

From a quick Google it seems they are in the Gloucester area if that helps identify who it is.

[WSC-Will]

If you can see your apache log files you'll be able to establish whether it's a human or a robot by the speed that they look at your whole site.
A robot will steam through your whole site in a matter of a few seconds probably, whereas a human will obviously take longer.

Unfortunately, that's not a reliable indicator as it's very easy to add a random pause into a robot. There' no way to distinguish between a human and a robot just by looking at log files.

[WSC-Will]

Any ideas what the change of browser might signify :?:

There's only one person who can answer that for you...

User agents (browser types) can be spoofed (forged) very easily so there's nothing concrete you can draw from it.

I wouldn't worry about it too much. Unless there visits start to hamper the operation of your website then there's not a lot you can do to stop someone who's determined to get through. Saying that, you may be able to use a blocking/redirecting cookie. If the browser (or whatever it is) is accepting cookies then you could have a way of singling them out or at least redirecting them to custom page asking them to stop what they're doing.

[Ian J]

Sorry to resurrect this one again but my old friend has visited my site several times every weekend since I started this thread with half a dozen visits yesterday and a few on Saturday as well.

The odd thing is that instead of coming directly to my site he searches on Google for the keyword and enters that way.

Would Google not give me extra credit for the number of visitors to my site, even though they are all him ?

[microbe]

You don't get credit from Google for people visiting your site, since it has no way of tracking how many people follow links in search results.

[Rob Holmes]

Hmm I was under the impression Google could tell to a certain extent. But I can't remember where I read it or when.

Rob

[Top Hat]

Hmm I was under the impression Google could tell to a certain extent. But I can't remember where I read it or when.

Rob

Maybe they can if you have the toolbar

[microbe]

Yes, that is true.

FWIW, this goes into the subject in detail.

http://searchenginewatch.com/sereport/article.php/2175251#spyware

I am still not convinced that you get "credit" as such even if someone using the toolbar visits your page.

[Webstuff]

You don't... google ranks based solely on in-links and content (inc; meta information, image alts, etc). Unless this person is using lots of bandwidth, don't worry. Does your site display customers / whoevers emails? It might be a harvester trying to steal your customers.

[Ian J]

There are no customer details or even a contact form on my site - just a telephone number, address and email details.

I have a tracker on most of my sites pages yet he seems only to visit the home page but several times a day every weekend.

Perhaps what he is doing is harmless but it sure has my curiosity roused

[Webstuff]

maybe he likes your site and keeps checking back for updates?

[Ian J]

All is revealed :D

I am an editor at DMOZ and a brand new and half finished website has just been submitted into the category that I edit with an instantly recognizeable IP address.

Whilst he hasn't copied my site he has copied certain aspects of the structure but it still doesn't answer the question of why he felt it necessary to keep logging onto my site several times a day as it only took me a couple of minutes using Quadsucker to download his whole site to see how he had copied me.

As I now have his email address perhaps I should ask him :lol:

[Rob Holmes]

Maybe you could post it here and we could all ask him ;)

Rob

[microbe]

Have you tried looking up the domain name http://whois.sc/rippedofffromukbusinessforums.co.uk is a good place to start.

If he is hosting in the US and is breaching your copyright, you can rattle his cage nicely by serving a DMCA notice on his web hosts.

[Ian J]

He hasn't breached any copyrights but just copied certain aspects of the way my website is structured. I have just sent him an email saying that if it helps I am happy to send him copies of any changes that I make to my site to save him continually having to look himself.

I would love to be a fly on his wall when he reads my email as he will be wondering how I have traced him :lol:

[SillyJokes]

Look for copied content using
http://www.copyscape.com

You enter the page URL that you think might be copied and copyscape looks for very similar pages.

I have caught a couple of people by using this tool.

[microbe]

The guy seems like a complete loser. I can only think he must be a slow reader judging from the amount of time he has invested in NOT copying your site.

[Ian J]

Look for copied content using
http://www.copyscape.com

Thanks for that. I have just had a quick try with one of my inside pages and found copied lumps of text already.

[Rob Holmes]

wow - I found our closest competitor has ripped off our email marketing sales copy (not on the matrixx site) on another site.

Thanks for that link!

Rob

[microbe]

Look for copied content using
http://www.copyscape.com

Thanks for that. I have just had a quick try with one of my inside pages and found copied lumps of text already.

Would this be the guy who comes top in the Copyscape search? I see my DMCA suggestion is no use as he hosts in the UK.

He seems to have been quite thorough with Ctrl+C, doesn't he.

[Ian J]

Would this be the guy who comes top in the Copyscape search?

That's another of my sites :D

I tested it out on the second page (http://www.factoringsolutions.co.uk/the_factoring_company.htm) from my site and the site that came top has copied huge lumps of text from my site. I have emailed him asking him to remove them or I will ask his ISP to do so for breach of copyright.

[microbe]

Would this be the guy who comes top in the Copyscape search?

That's another of my sites :D

In that case I wish I had commented on how attractive and professional it looks. :)

This one is hosted in the US. PM me if you decide you want to do the DMCA thing with his hosts, I have copies of standard letters and the like.

[Ozzy]

Oh my! I paid good money for the drafting my my T&C's and Privacy Policy and the amount of people who have ripped them off is amazing. That's a good and enlightening site though.