Google Cache

aidan1980 · Oct 17, 2009

Not sure if there is a problem but just checking my webmaster tools and google hasn't cahced my homepage since 2nd october. According to the stats this is a long time as in previous months they have visited every 4-5 days. I change things quite often and have submitted various pages to sites such as digg etc but still no joy. I have no robots txt so that isn't blocking it but it does say robots txt is unreachable if that means anything?

crossdaz · Oct 17, 2009

thebookiesoffers said:
Not sure if there is a problem but just checking my webmaster tools and google hasn't cahced my homepage since 2nd october. According to the stats this is a long time as in previous months they have visited every 4-5 days. I change things quite often and have submitted various pages to sites such as digg etc but still no joy. I have no robots txt so that isn't blocking it but it does say robots txt is unreachable if that means anything?

Has it actually changed - they don't always refresh the cache if only minor changes have occurred.
So long as googlebot is making regular visits I wouldn't worry about it?

aidan1980 · Oct 17, 2009

crossdaz said:
Has it actually changed - they don't always refresh the cache if only minor changes have occurred.
So long as googlebot is making regular visits I wouldn't worry about it?

I'd say quite a few changes have taken place with quite a few new pages being created that haven't been picked up.
in the WM tools the graph for the last 3 months is like a chart for an earthquake then just flat lines from oct 2nd

QVA - Emma · Oct 17, 2009

thebookiesoffers said:
Not sure if there is a problem but just checking my webmaster tools and google hasn't cahced my homepage since 2nd october. According to the stats this is a long time as in previous months they have visited every 4-5 days. I change things quite often and have submitted various pages to sites such as digg etc but still no joy. I have no robots txt so that isn't blocking it but it does say robots txt is unreachable if that means anything?

Hiya,

Don't worry about the homepage for now - check the internal pages that are new cache:www.thebookiesoffers.co.uk/(pagename)

If they've been cached recently don't worry too much.

If you get an xml sitemap done for now and submit in your webmaster tools - it will pick changes up a bit quicker.

Try this tool http://www.xml-sitemaps.com/ and if you need a hand drop me a line.

Emma

aidan1980 · Oct 17, 2009

QVA - Emma said:
Hiya,

Don't worry about the homepage for now - check the internal pages that are new cache:www.thebookiesoffers.co.uk/(pagename)

If they've been cached recently don't worry too much.

If you get an xml sitemap done for now and submit in your webmaster tools - it will pick changes up a bit quicker.

Try this tool http://www.xml-sitemaps.com/ and if you need a hand drop me a line.

Emma

Hi Em,

I am abit cautious about site maps as I have added 1 before but i suffered a little drop in rankings. Noticed a thread that sirearl did about them affecting rankings so deleted it and recovered again. Finding it a little frustrating as I have made some changes that I hope will improve the site yet they haven't been picked up:|

crossdaz · Oct 17, 2009

thebookiesoffers said:
Hi Em,

I am abit cautious about site maps as I have added 1 before but i suffered a little drop in rankings. Noticed a thread that sirearl did about them affecting rankings so deleted it and recovered again. Finding it a little frustrating as I have made some changes that I hope will improve the site yet they haven't been picked up:|

an xml sitemap can be just a list of your urls, eg:

<url>
<loc>http://www.yoursite.co.uk/page-one.htm</loc>
</url>
<url>
<loc>http://www.yoursite.co.uk/page-two.htm</loc>
</url>
<url>
<loc>http://www.yoursite.co.uk/page-three.htm</loc>
</url>

How can that be detrimental?

The cache issue looks like just one of those things tbh? It'll probably update soon enough.

QVA - Emma · Oct 17, 2009

thebookiesoffers said:
Hi Em,

I am abit cautious about site maps as I have added 1 before but i suffered a little drop in rankings. Noticed a thread that sirearl did about them affecting rankings so deleted it and recovered again. Finding it a little frustrating as I have made some changes that I hope will improve the site yet they haven't been picked up:|

Yep,

I don't have one either - not full time anyway.

I only upload one when there are major changes - wait for it to cache then remove it after a couple of weeks.

(there is one there at the moment for any smart comments

)

Google seems to be on a bit of a go slow generally at the moment. It even seems to have changed it's showings of map listings on certain keywords/phrases.

Emma

aidan1980 · Oct 17, 2009

crossdaz said:
an xml sitemap can be just a list of your urls, eg:

<url>
<loc>http://www.yoursite.co.uk/page-one.htm</loc>
</url>
<url>
<loc>http://www.yoursite.co.uk/page-two.htm</loc>
</url>
<url>
<loc>http://www.yoursite.co.uk/page-three.htm</loc>
</url>

How can that be detrimental?

The cache issue looks like just one of those things tbh? It'll probably update soon enough.

http://www.ukbusinessforums.co.uk/forums/showthread.php?t=102689
im no expert but its all here

crossdaz · Oct 17, 2009

thebookiesoffers said:
http://www.ukbusinessforums.co.uk/forums/showthread.php?t=102689
im no expert but its all here

neither am I, let's see what matt says:

http://www.youtube.com/watch?v=hi5DGOu1uA0&feature=PlayList&p=841CB8F9F31BF5D5&index=104

take your pick mate

yes, I know it's all lies and propaganda. Evil google will smack yer ass for telling them the urls of your pages - why wouldn't they?

aidan1980 · Oct 17, 2009

crossdaz said:
neither am I, let's see what matt says:

http://www.youtube.com/watch?v=hi5DGOu1uA0&feature=PlayList&p=841CB8F9F31BF5D5&index=104

take your pick mate

yes, I know it's all lies and propaganda. Evil google will smack yer ass for telling them the urls of your pages - why wouldn't they?

well we wot find out from my site as every time i try and submit a sitemap it doesnt work due to errors:|

aidan1980 · Oct 17, 2009

just looking into it a bit more and it seems google has been visiting but all of a sudden it seem the robot txt is unreachable (although i have never had any robots txt ever) and because of this googlebot just clears off with checking the page? does this sound about right?

aidan1980 · Oct 17, 2009

thebookiesoffers said:
just looking into it a bit more and it seems google has been visiting but all of a sudden it seem the robot txt is unreachable (although i have never had any robots txt ever) and because of this googlebot just clears off with checking the page? does this sound about right?

this is just a snippet of what it say in WM tools

Crawl errors

Issues Google encountered when crawling your site.
Web Mobile CHTML Mobile WML/XHTML

Show URLs: HTTP ‎(0)‎ - In Sitemaps ‎(0)‎ - Not followed ‎(0)‎ - Not found ‎(0)‎ - Restricted by robots.txt ‎(0)‎ - Timed out ‎(0)‎ - Unreachable ‎(57)‎
URLDetailDetectedhttp://www.thebookiesoffers.co.uk/ robots.txt unreachableOct 11, 2009
http://www.thebookiesoffers.co.uk/32red-casinorobots.txt unreachableOct 12, 2009
http://www.thebookiesoffers.co.uk/32redpokerrobots.txt unreachableOct 8, 2009
http://www.thebookiesoffers.co.uk/888-casino-bonusrobots.txt unreachableOct 12, 2009

crossdaz · Oct 17, 2009

thebookiesoffers said:
this is just a snippet of what it say in WM tools

Crawl errors

Issues Google encountered when crawling your site.
Web Mobile CHTML Mobile WML/XHTML

Show URLs: HTTP ‎(0)‎ - In Sitemaps ‎(0)‎ - Not followed ‎(0)‎ - Not found ‎(0)‎ - Restricted by robots.txt ‎(0)‎ - Timed out ‎(0)‎ - Unreachable ‎(57)‎

At a rough guess I'll say your site has been having some downtime?
Robots text tells the bots where they can and can not go - if it isn't there then all bots will crawl everything.

Check with your host and see if any outages correspond with the dates given by wmt's.

If you are worried about robots text (you needn't be) just open notepad
paste this into it

Code:

User-Agent: *
Allow: /

then save as

robots.txt

and upload to your server.

"Allow: /" means nothing is blocked

aidan1980 · Oct 17, 2009

crossdaz said:
At a rough guess I'll say your site has been having some downtime?
Robots text tells the bots where they can and can not go - if it isn't there then all bots will crawl everything.

Check with your host and see if any outages correspond with the dates given by wmt's.

If you are worried about robots text (you needn't be) just open notepad
paste this into it

Code:

User-Agent: * Allow: /

then save as

robots.txt

and upload to your server.

"Allow: /" means nothing is blocked

I've dropped a robots txt in so hopefully this may sort it. Cheers

Ali-v-8 · Oct 18, 2009

you are being cached but some pages haven't been touched since 5th september.
but you have pages cached on the 10th of october.
simply improve the internal linkin and bobs your uncle.

aidan1980 · Oct 18, 2009

Ali-v-8 said:
you are being cached but some pages haven't been touched since 5th september.
but you have pages cached on the 10th of october.
simply improve the internal linkin and bobs your uncle.

What pages have been mate? The ones I have checked haven't since Oct 2nd at the latest. webmaster tools suggests the same aswell mate

Ali-v-8 · Oct 18, 2009

do site:www.thebookiesoffers.co.uk and go through manually
th thing is if you do a good internal structure then when one page gets cached the all should.

thebookiesoffers said:
What pages have been mate? The ones I have checked haven't since Oct 2nd at the latest. webmaster tools suggests the same aswell mate

aidan1980 · Oct 18, 2009

Ali-v-8 said:
do site:www.thebookiesoffers.co.uk and go through manually
th thing is if you do a good internal structure then when one page gets cached the all should.

just been through all them and not 1 was cached after oct 3rd:|

aidan1980 · Oct 19, 2009

just checked the WMT again today and its now been updated and 97 pages are robots.txt unreachable. It seems Google tried to have a big visit friday but didn't cache anything.
I found this in the help files:
"robots.txt file unreachable - Before we crawled the pages of your site, we tried to check your robots.txt file to ensure we didn't crawl any pages that you had roboted out. However, your robots.txt file was unreachable. To make sure we didn't crawl any pages listed in that file, we postponed our crawl. When this happens, we return to your site later and crawl it once we can reach your robots.txt file. Note that this is different from a 404 response when looking for a robots.txt file. If we receive a 404, we assume that a robots.txt file does not exist and we continue the crawl."

I have now added the robots.txt but have i done it correctly now and is it a matter of just sitting back and waiting till the next visit?

crossdaz · Oct 19, 2009

thebookiesoffers said:
"

I have now added the robots.txt but have i done it correctly now and is it a matter of just sitting back and waiting till the next visit?

http://www.thebookiesoffers.co.uk/robots.txt

First time it timed out and second time it was OK?

The explanation you gave above is a new one on me - I have plenty of sites with no robots.txt and they get crawled and indexed OK?

"If we receive a 404, we assume that a robots.txt file does not exist and we continue the crawl."

This suggests to me that there is a problem on your server.

aidan1980 · Oct 19, 2009

crossdaz said:
http://www.thebookiesoffers.co.uk/robots.txt

First time it timed out and second time it was OK?

The explanation you gave above is a new one on me - I have plenty of sites with no robots.txt and they get crawled and indexed OK?

"If we receive a 404, we assume that a robots.txt file does not exist and we continue the crawl."

This suggests to me that there is a problem on your server.

The info I got was from google itself http://www.google.com/support/webmasters/bin/answer.py?answer=35147.

Its a new poblem aswell as I have never had crawling problems before

crossdaz · Oct 19, 2009

thebookiesoffers said:
The info I got was from google itself http://www.google.com/support/webmasters/bin/answer.py?answer=35147.

Yes, I know - but if you read the respnse highlighted - they are saying if no robots.txt exists then they'll just assume it's OK to crawl away - which is what I always assumed.

The fact that they run into problems while crawling your site suggests a problem on the server itself - which would explain the random error reports. In other words, they can't even determine your site doesn't (or didn't) have a robots.txt which suggests something is getting in the way.

Unless someone can suggest a better reason for this then I am going to say that your host is at fault here?

You need to get it sorted because they aren't going to keep indexing a site they can't crawl.

aidan1980 · Oct 19, 2009

crossdaz said:
Yes, I know - but if you read the respnse highlighted - they are saying if no robots.txt exists then they'll just assume it's OK to crawl away - which is what I always assumed.

The fact that they run into problems while crawling your site suggests a problem on the server itself - which would explain the random error reports. In other words, they can't even determine your site doesn't (or didn't) have a robots.txt which suggests something is getting in the way.

Unless someone can suggest a better reason for this then I am going to say that your host is at fault here?

You need to get it sorted because they aren't going to keep indexing a site they can't crawl.

seems you are bang on the money mate, just had a chat with them and some sort of error has shown up in the log, just got to wait for them to fix it now

Search

Search

Google Cache

aidan1980

crossdaz

aidan1980

QVA - Emma

aidan1980

crossdaz

QVA - Emma

aidan1980

crossdaz

aidan1980

aidan1980

aidan1980

crossdaz

aidan1980

Ali-v-8

aidan1980

Ali-v-8

aidan1980

aidan1980

crossdaz

aidan1980

crossdaz

aidan1980

Similar threads

Latest Articles