PDA

View Full Version : Issue in Robots.txt


snareshiver
15th February 2010, 05:47
Hi,

I have added a canonical Tag in the Homepage to avoid crawling of Session ids.

But recently, we added a Article page with the following url

Myhomepage/ArticleControl.do?parameter=getArticleList

The above link is restricted in Robots.txt file. Please tell for what reason the url is restricted in robots file.

Andy Walpole
15th February 2010, 06:13
The only person who can answer that madeline is the person who disallowed it in the robots.txt file in the first place.

If you want the search engines to crawl the page then just remove it