View Full Version : robots
The Dreaded Lurgy
11th February 2009, 16:34
How important is it to try and restrict robots.
Obviously I dont want Bots indexing each product for every section it appears in, some products appear in 4 sections and then I think google has also indexed some pages which are generated as a custom search within the site.
Will all this be seen as duplicate content and what are the dangers of getting over restrictive with them?
Cheers Jon
iamlijo
11th February 2009, 16:50
Hi ,
just add a no- follow in the pages you dont want to index .
regards
lijo
iamlijo
11th February 2009, 16:54
You dont need to make chages in robots.txt
make nessary chages in each page header. Hope the quote below helps.
Meta Tags ‘NOINDEX,NOFOLLOW’
Filed under: Articles (http://www.linktutorial.com/read/category/articles/)
The meta tags are in the header of a web page, that is, between the <head> tags. There are some webmasters out there that use the meta tag <meta name="ROBOTS" content="NOINDEX,NOFOLLOW"> on their links page or directory. This means that search engine robots will not index (NOINDEX) or follow (NOFOLLOW) the links on that page.
The use of <meta name="ROBOTS" content="NOINDEX,NOFOLLOW"> is most commonly seen during link exchanges where webmasters try to retain their PageRank by instructing search engine robots not to index or follow their links pages or links directory.
Impact of this meta tags on link exchanges
A link on a web page using robots meta tag is not completely useless. If your link is on a web page that uses this robots meta tag, your link would not be counted towards your web site’s link popularity (http://www.linktutorial.com/read/articles/link-popularity). However, visitors may still find your web site through this link.
Detecting meta “NOINDEX,NOFOLLOW”
Detecting this robots meta tags is quite simple. The meta tag can be clearly seen in the source code of the web page. Look within the <head> tags near the top of the source code.
source : link tutorial . com
The Dreaded Lurgy
12th February 2009, 13:12
Thank you for the advice, much appreciated.
ken_uk
12th February 2009, 13:35
I would personally block the pages using robots.txt instead of meta tags.
Not all search engines treat meta's the same way and also when you use the robots file you are telling the search engine what pages NOT to read.
If you robot block a range of pages then the search engine spider will not crawl and read those pages.
If you meta block those pages, the search engine HAS to read every page in order to see that it is blocked.
That means your site is being hit by all those search engine spiders, slowing things down for real customers, and using up your bandwidth - just so they can read those pages in order to decide to ignore them.
Robots on the other hand, is a small file, it gets read once per crawl say and all pages on its ignore list are simply not read, saving bandwidth and resources.