Is it possible to create the Crawl path

[snareshiver]

Hi,

I would like to know whether it is possible to tell the bots to tell from where it should start crawling and which urls it should crawl.

 

[OldWelshGuy]

No it isn't.

I did explain in the other thread how 'crawling works' and that they don't crawl as such any longer.

But I will try again

The spiders request a url, the datacentre sends a url to them, they visit the url grab the code and send it home, then request a new url.

Google algorithmically decides which 8urls to crawl and then they run their quality algorithm over the data collected to decide if the pages are worthy of inclusion.

You can encourage google to spider and index pages, which is whay the hieracael structure and link architecture of a site is so important.

Again, what is it you are trying to achieve, as wh3en you only give a tiny amount of information it makes it difficult to reply fully.

You can also encourage google to spider a non spidereable site by using a sitemap.

 

[snareshiver]

Hi,
You said that the spider requests for a url, datacenter sends it, they grab the contents and send it back.

After that request, will the spider not discover other links in the same url(domain)?

Or it will send a second request for a url from a different domain which it might have in the queue?

 

[commodityconsultant]

As far i consider your question that you are talking about .htaccess file
in which you can mention which url should index or not index by search engines.

 

[snareshiver]

I read this interesting Link which made me to think about configuring Crawl Access.