How long for Google to notice robots.txt changes?

[DesignerNick]

I was blocking a website from being indexed while I worked on it.

I have just edited the robots.txt to allow it, I have gone into webmaster tools and when trying to do a "fetch as googlebot" it still says it is denied by robots.txt.

I just wondered how long it should take before it updates?

 

[OldWelshGuy]

once per day normally with Google.

 

[DesignerNick]

Ah so it will take a while before Google sees that my robots.txt has changed

 

[OldWelshGuy]

How it used to work is that they would call for the robots.txt file once every 24hours, or upon the time of visit.
e.g if thery visit 5 times in 34 hours it will be called once, if they visit again 25 hours later they call again. if they vistit today they call it, but if they don't visit again for a month, they don't call it, as it is at the time of visit that they call it.

They also use proxy caching of the various bots to save resources.

 

[DesignerNick]

Bugger. So there are no tips and tricks to help it along so I can get the site indexed? It is just sit and wait

 

[Mystro]

Bugger. So there are no tips and tricks to help it along so I can get the site indexed? It is just sit and wait

If you have twitter post about the new site with a link and normally Google will follow the other bots

Try that

 

[DesignerNick]

Thanks. Have tweeted it and put it on a busy site.

Webmaster tools is still saying no access though

 

[MASSEY]

When google finds a link pointing to another site it doesn't always follow the link for some reason. This is what i find anyway. Or the effect of following the link takes time to kick in.

I thought robots.txt was a bit of fluff considering i had site not allow and still get indexed.

 

[Mystro]

I tweeted it too for you ..

don't worry about WMT that wont update instantly.

But what you can do is got through WMTs and get to the tab and fetch as Google bot do this on various inner pages and the home page to see if anything is getting blocked then you can click the tab to resubmit to the index.

Google is strange lately and wont rush,

I think you may have a redirect loop on the home page, the first test should give you a 200 response but the tool says redirecting , check yours then check others to see what i mean

I checked your header codes and they say 200 which is what they are ment to so i suspect there is something wrong with your htaccess 301

http://www.ragepank.com/redirect-check/

Fetch as Google bot will show you the problem

 

[ibm@i3media]

usually within a few hours. however you can request your website to be crawled too.