What is robots.txt?

[dotsquares]

Why to use robots.txt?

 

[cornish hosting]

http://www.webconfs.com/what-is-robots-txt-article-12.php

 

[OldWelshGuy]

Why to use robots.txt?

Why do you ask? no reply = deleted thread

 

[arunjha]

We use robots.txt to tell search engine not to crawl a particular link which is harmful for our website.

 

[vanny12]

We use robots.txt to tell search engine not to crawl a particular link which is harmful for our website.

thanks for share your knowledge.

 

[bdw]

not another one!

.

 

[danieldavis]

Robot txt. tell us the search engine not a crawl particular link which can used harmful. this txt. is very intrested.
Thanks..

 

[jhardy2008]

Your impression is correct that the purpose of robots.txt is just to control what obedient bots will and won't crawl.

If the bot doesn't find a robots.txt, it assumes it's okay to crawl anything and everything it finds under that domain. If you're okay with that, then no worries.

 

[HFE Signs]

You can also use a robots.txt to tell the bots to crawl your complete site:

User-agent: *
Disallow:

This means disallow none.. some might say whats the point in that.. the point is its better to have a robots.txt file than not to..

 

[alexander0076]

The robot.txt file tells the search engines whether to crawl or not? We can also restrict the search engines to crawl a particular page or not.

 

[james127]

Robot txt File is the set of instruction for the visiting spiders of search engines. It provides a map for the spiders what they can and what they can not index. This file resides in the root directory of website.

 

[Redalkemi]

When we don’t want search engine to crawl particular pages of our website, we use robots.txt file to let the search engine know that these pages are not to be crawled. It is placed in the root directory.

 

[bdw]

Errrrm, this has been answered many times. Why bother repeating it? It could not be something to do with post count, could it?

 

[bogdanbridgewest]

The robots.txt is a text that tells google not to crawl particular links

 

[bdw]

AAAaaaaaaarggghhhh!

.

 

[lalitbansal004]

A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search..

Thanks.
Lalit

 

[TechDon]

I won't asnwer your question as I can see its already correctly been answered a few times although I would just like to mention that you should make certain that you don't put links to pages that contain sensitive information or login pages within your robots.txt file. This would be the first place that someone would look to comprimise the security of your website or gather pages that may contain sensitive information.

 

[gilchrist]

A robots.txt file prevents access to your site by search engine robots that crawl the web.It is generator designed by seo for public use.

 

[finejewellerys]

robots.txt is the guide for the search engines's bots to crawl our site.where to start firstly,easily and fast.like googlebot,if you place a robots.txt in your site foot.When google crawl your site,it will show which files are not allowed to crawl, which is allowed etc. If you put your sitemap address in the robots .txt file,i think it will crawl and index your site quickly.

 

[S-Marketing]

We could use this thread as a kind of honey trap for spammers couldn't we. They just can't leave it alone.