Google only indexes certain products

[movietub]

I'm looking for advice on product catalogue format and SEO.

The site in question is www.warehouse-aquatics.co.uk

I posted about this ages ago and took some advice. We have removed the cascading menu and hard linked to each product category (the logic being that the cascading menu was presenting about 300 links on each page which would confuse Google). We have also shortened the number of sub categories to a max of 3.

We are adding a random product box to the home page to cycle products through there. We have noticed that any product on the home page is indexed almost straight away.

The problem is that initially G indexed about 500 of 2500 products. Snce then any new product that is featured on the home page is added so we are up to 650ish.

But why did it only find 500/2500 original products and why stop there? As far as I can see these products pages have nothing in common.

I know we need to start in-linking products and categories. The problem is so much is suggested and I want to focus on the more obvious and effective things first...

Any help would of course be appreciated.

[fisicx]

Probaly because it only has limited resources. Google would rather get a reasonable handle on 100 sites than spend the same amount of time fully indexing 1 site for what it may consider very niche products. It's also worth considering that it may compare your site to the many other fishy resources and decide that adding your 2500 products to the index doesn't add any value to the visitor (it's all in the google guidelines).

[emerchant]

Get more backlinks, get deeper backlinks :)

[movietub]

Probaly because it only has limited resources. Google would rather get a reasonable handle on 100 sites than spend the same amount of time fully indexing 1 site for what it may consider very niche products. It's also worth considering that it may compare your site to the many other fishy resources and decide that adding your 2500 products to the index doesn't add any value to the visitor (it's all in the google guidelines).

Some products it lists are very niche, others are sold by at least 30 competitors. There seems to be no distinguishing factor here.

I appreciate that Google will only take what it needs, but the same product catalogue on the EKM site ended up with around 1500 products listed. Clearly something is holding back Google indexing the new site.

It's been up and running for around 6 months now with no major change.

[Kev Jaques]

Fix the site errors and use proper linking format for your product ids.
Also left align the page text for product info, centre alignment is not good for reading.

[Planck]

Also left align the page text for product info, centre alignment is not good for reading.

Ah, yes, that'll get your pages indexed...

WTF??

[fisicx]

...but the same product catalogue on the EKM site ended up with around 1500 products listed. Clearly something is holding back Google indexing the new site.
New site new rules. Just becuase Google liked the old site doesn't mean it will index the new one in the same way. If it hasn't got past 500 products in 6 months then it's very unlikely you are going to many more in the index unless you change your internal linking and write loads of new content (giving G something to index).

[movietub]

New site new rules. Just becuase Google liked the old site doesn't mean it will index the new one in the same way. If it hasn't got past 500 products in 6 months then it's very unlikely you are going to many more in the index unless you change your internal linking and write loads of new content (giving G something to index).

New site new rules... well yes! Thats the point of the thread, before I make changes which rules should I follow?

On EKM there was zero internal linking and zero unique content. Google does index 100% of new products added to the site within about 6 hours typically.

All new products are displayed on the homepage for a day or so.

So surely that indicates something is physically putting G off going any deeper than the homepage very often?

I'm aware the product pages need a hell of a lot of work and this is happening. But all I'm striving for at the moment is to be close to where I was 6 months ago with no SEO or effort whatsoever!

[Kev Jaques]

Ah, yes, that'll get your pages indexed...

WTF??

If you looked closely I didn't say it would! ;)
But it's not just search engines you gotta please is it? They don't actually want to buy a fish tank and fishes do they ;)

[movietub]

If you looked closely I didn't say it would! ;)
But it's not just search engines you gotta please is it? They don't actually want to buy a fish tank and fishes do they ;)

Technically if you had read the opening post properly you would have seen that this thread is very much just about pleasing the search engines!

We have people working on the actual product page copy, layout, content and to an extent SEO.

This is more about how to organise so many products in a way that allows Google to find as many as possible.

[Kev Jaques]

And as usual with some forum posts people miss things and are quick to focus/complain on the points that may not be useful and miss the important ones and still expect google to index their sites when parsing errors on some of your product links could affect your lack of indexed products.
Fix the site errors, then come back if your products are not picked up.

[movietub]

And as usual with some forum posts people miss things and are quick to focus/complain on the points that may not be useful and miss the important ones and still expect google to index their sites when parsing errors on some of your product links could affect your lack of indexed products.
Fix the site errors, then come back if your products are not picked up.

Can you show me an example of a product related parsing error?

We add all the products in the same manner. If some products have errors then G is still adding them. Every product we add now is added to G, however the historic products we scraped and imported into the new CMS are not.

I'm on FF at the moment and see no errors. I thought I had cleared the last of them up (although the site code and CSS need tifying and optimising when I have time and have finished playing with the site).

[Kev Jaques]

http://validator.w3.org/check?verbose=1&uri=http%3A%2F%2Fwww.warehouse-aquatics.co.uk%2F

[emerchant]

Quick question are you using Sitemaps / Webmaster tools?

[movietub]

Quick question are you using Sitemaps / Webmaster tools?

Yes, and analytics.

[emerchant]

Yes, and analytics.

So does the store output a compatible sitemap feed with all URLs on? and has these been submitted to webmaster tools?

[movietub]

So does the store output a compatible sitemap feed with all URLs on? and has these been submitted to webmaster tools?

Yes, it can be viewed here:

http://www.warehouse-aquatics.co.uk/index.php?p=sitemapxml

G uploads every 24 hours.

This is another concern actually. The old EKM shop was never submitted properly to Google! Its crazy that with no effort we got 1000 uniques a day, now we get 500-600.

Webmasters says only 300 pages from the sitemap are currently indexed. No errors.

:|

[emerchant]

Yes, it can be viewed here:

http://www.warehouse-aquatics.co.uk/index.php?p=sitemapxml

G uploads every 24 hours.

This is another concern actually. The old EKM shop was never submitted properly to Google! Its crazy that with no effort we got 1000 uniques a day, now we get 500-600.

Webmasters says only 300 pages from the sitemap are currently indexed. No errors.

:|

"we got 1000 uniques a day" - do you mean traffic?

have you changed the domain the store is on?

[movietub]

"we got 1000 uniques a day" - do you mean traffic?

have you changed the domain the store is on?

1000 unique visitors per day. Around 1200 visits in total most days. That was with the same products, images, names, descriptions as we have now.

That was with no SEO effort, sitemap or any marketing. Google just indexed more prooducts from the EKM site and I need to find out why :D

I am aware there are 101 things we could do to improve general SEO. But first I have to tackle the main problem. Clearly something is fundamentally putting Google off indexing the same products on the new site.

EDIT: No not changed domain. Just swapped from EKM to these guys: http://www.thisispearl.com/

[Planck]

Hang on, let me get this straight. You changed from EKM to Pearl, and lost half your traffic, and presumably around half your turnover? And you're still running the Pearl shop and trying to figure out what's wrong?

If we made a change that resulted in our sales halving, the first thing I'd do is revert to the old version, then ask questions later.

[movietub]

Hang on, let me get this straight. You changed from EKM to Pearl, and lost half your traffic, and presumably around half your turnover? And you're still running the Pearl shop and trying to figure out what's wrong?

If we made a change that resulted in our sales halving, the first thing I'd do is revert to the old version, then ask questions later.

Thats the problem with only looking at one statistic and asking questions after taking action! You would have lost out.

The other statistic is conversion rate. Pearl is one of the few solutions powerful enough to share inventory across website and our physical shop. It also auto orders stock and maintains minumum levels whilst highlighting stock that is sold at a higher rate than it is re-ordered. This means we can list stock levels online which brings a HUGE increase in conversion (providing you invest in the stock of course!).

Old conversion 4%, new conversion 8%. So halving the traffic has not harmed us.

Crucially I now know that if I can increase traffic to former levels with the new conversion rate then the business will be around £8k a month more profitable. So I would perfer not to switch back to the more basic EKM solution simply to avoid solving a problem that I'm sure is possible to solve.

[emerchant]

1000 unique visitors per day. Around 1200 visits in total most days. That was with the same products, images, names, descriptions as we have now.

That was with no SEO effort, sitemap or any marketing. Google just indexed more prooducts from the EKM site and I need to find out why :D

I am aware there are 101 things we could do to improve general SEO. But first I have to tackle the main problem. Clearly something is fundamentally putting Google off indexing the same products on the new site.

EDIT: No not changed domain. Just swapped from EKM to these guys: http://www.thisispearl.com/

Firstly have you ask your new guys?

Second, you say you didnt change the domain but did you change all the product URLs? If so did you 301 redirect the old to the new.

[movietub]

Firstly have you ask your new guys?

Second, you say you didnt change the domain but did you change all the product URLs? If so did you 301 redirect the old to the new.

Inevitably product URLs changed, but we did setup a redirect. All older G listings have gone now and only current, accurate links remain (except a handful of links from other sites - less than 10).

The guys at Pearl say that compared to their other clients 500 products indexed out of 2500 is not un-common. Then again I would say that Pearl are more foccused on the CRM/Accounting & B2B side of their software than ecommerce historically. That said big leaps are now being made on the ecommerce side.

I'm hoping that someone can spot something on a mechanical level thats stopping the products being indexed as easily as before. If this required a Pearl fix-around to solve then I'm sure they would do that.

[SFD]

This may not provide the answer but may give you an inight into the problem.

Your homepage has urls

http://www.warehouse-aquatics.co.uk
http://www.warehouse-aquatics.co.uk/index.php
http://www.warehouse-aquatics.co.uk/index
http://www.warehouse-aquatics.co.uk/home-c-285.php

Google has indexed the fourth as the homepage url.

I think this will have affected your rankings and also changed how pagerank is spread throughout the site.

[movietub]

This may not provide the answer but may give you an inight into the problem.

Your homepage has urls

http://www.warehouse-aquatics.co.uk
http://www.warehouse-aquatics.co.uk/index.php
http://www.warehouse-aquatics.co.uk/index
http://www.warehouse-aquatics.co.uk/home-c-285.php

Google has indexed the fourth as the homepage url.

I think this will have affected your rankings and also changed how pagerank is spread throughout the site.

I cleared the 285.php link. Also the all but one link to/index or index.php , the breadcrumb trail defaults to directing to /index.php for the homepage at the start of the trial. I cannot edit this.

Are the first three an issue? Don't most sites have homepage url of with/without 'index' and or the file type extension?

Great that you spotted it though!

[fisicx]

Looking at your site map shows that most of the products have priority 0.5. This is telling google that the pages aren't really that important compared to the few you have set at priority 1.

You can assume that google isn't going to index any more of the site than it already has unless you make some changes so here are my suggestions.

1. Get rid of the site map as a temporary measure.
2. Don't add any new products for a while.
3. Add some permanent links on the homepage to the lower level categorties (the one's not indexed).
4. You sort out the page structure so that it gets to the actual product before line 500.
5. do a bit of SEO work - page titles, h1 and product decriptions will help. And there is still up to 5 clicks to get to the product listing.

Basically you give google something to index, at the moment all you have on the page is exactly the same as your competitors - there is no incentive for G to index the pages. Maybe you didn't work on the old site but as far as G is concerned, your new site is new so it is going to reindex everypage, all the 301 did was tell it where the new page existed.

[SFD]

I cleared the 285.php link. Also the all but one link to/index or index.php , the breadcrumb trail defaults to directing to /index.php for the homepage at the start of the trial. I cannot edit this.

Are the first three an issue? Don't most sites have homepage url of with/without 'index' and or the file type extension?

Great that you spotted it though!

Some sites do go to index.php but I beleive it to be better when everything goes back the the domain.

I have seen in the past where a redirection error on the homepage has confused Google and there is no pagerank showing. I presume this is the case with you as the Google toolbar is showing your site has 0PR. I wouldn't have thought this to be the case.

These are just things I have been told and have worked for me, just passing on the info, I'm no expert.

[emerchant]

Thats the problem with only looking at one statistic and asking questions after taking action! You would have lost out.

The other statistic is conversion rate. Pearl is one of the few solutions powerful enough to share inventory across website and our physical shop. It also auto orders stock and maintains minumum levels whilst highlighting stock that is sold at a higher rate than it is re-ordered. This means we can list stock levels online which brings a HUGE increase in conversion (providing you invest in the stock of course!).

Old conversion 4%, new conversion 8%. So halving the traffic has not harmed us.

Crucially I now know that if I can increase traffic to former levels with the new conversion rate then the business will be around £8k a month more profitable. So I would perfer not to switch back to the more basic EKM solution simply to avoid solving a problem that I'm sure is possible to solve.

FWIW I wouldnt switch back either especially if you have seen a conversion rate improvement, worse case when you go back to the more basic store / system your conversion rate may drop back to what it was and then to top it off it might take a while for Google to reindex anyway.

[SFD]

You also have another problem with your navigation which you should be aware of:

If you navigate to a product such as:

http://www.warehouse-aquatics.co.uk/aquarium-filtration/media-for-external-filters/fluval-media-c-203_323_336.html?p=shop

Then click on your header logo, it takes you to

http://www.warehouse-aquatics.co.uk/aquarium-filtration/

instead of

http://www.warehouse-aquatics.co.uk/

but also displays the homepage text, so for each category you are creating another 'homepage' to add to the first 4 which I mentioned.

If you did want it to navigate to the 'home' of the current category then it should go to

http://www.warehouse-aquatics.co.uk/aquarium-filtration-c-203.html

not

http://www.warehouse-aquatics.co.uk/aquarium-filtration/

I think google could see this as quite a big problem with your site and the reason you are not getting the attention you expect.

[movietub]

You also have another problem with your navigation which you should be aware of:

If you navigate to a product such as:

http://www.warehouse-aquatics.co.uk/aquarium-filtration/media-for-external-filters/fluval-media-c-203_323_336.html?p=shop

Then click on your header logo, it takes you to

http://www.warehouse-aquatics.co.uk/aquarium-filtration/

instead of

http://www.warehouse-aquatics.co.uk/

but also displays the homepage text, so for each category you are creating another 'homepage' to add to the first 4 which I mentioned.

If you did want it to navigate to the 'home' of the current category then it should go to

http://www.warehouse-aquatics.co.uk/aquarium-filtration-c-203.html

not

http://www.warehouse-aquatics.co.uk/aquarium-filtration/

I think google could see this as quite a big problem with your site and the reason you are not getting the attention you expect.

Now thats progress!

I just changed the links on the home buttons so they are all ../index.php and still it creates an instance of the homepage at domain/productpage/index.php :rolleyes:

I could fix it by changing to non relative links but this causes major security problems. If I hard link in https then any forms (such as search) will flash up a security warning. If I hard link as http://domain/index.php then the site will not go in to full secure mode as these links will be stuck in http not https.

So I have to use relative links (as I should do anyway) and it looks like this is a fundamental problem with the way Pearl creates pages.

Any advance on that theory?

[SFD]

Maybe look at some other sites which use the same system and see what they've chosen to do?

[fisicx]

Always good to sort out your nav but it's not going to increase the number of indexed pages. Google has already been to the site, found the pages it wants to index and is quite content to leave it there.

If the wonky navigation was a problem you wouldn't have any product pages in the index.

Help google by providing something new to index. New internal links, pages titles, headers and descriptions. If you leave the pages alone then there is no incentive to add anything to the index.

Note: none of your product images have alt text, titles or captions. Image search can pull in a shedload of visitors so you are probably losing out here as well.

[movietub]

Maybe look at some other sites which use the same system and see what they've chosen to do?

This is another - doesn't suffer the same!

http://www.moonclimbing.com/

having looked more closely it only does this when viewing a product page within a second sub category.

filtration>sub>sub>products causes the problem.


filtration>sub>products does not.

The other site has a shorter category tree (less products to organise).

[movietub]

Always good to sort out your nav but it's not going to increase the number of indexed pages. Google has already been to the site, found the pages it wants to index and is quite content to leave it there.

If the wonky navigation was a problem you wouldn't have any product pages in the index.

Help google by providing something new to index. New internal links, pages titles, headers and descriptions. If you leave the pages alone then there is no incentive to add anything to the index.

Note: none of your product images have alt text, titles or captions. Image search can pull in a shedload of visitors so you are probably losing out here as well.

We are updating product pages and renaming all images for this reason.

I hear what you are saying and we are acting upon it. But what I don't want to do is find a way of improving on the current situation when I know that there is an underlaying issue that did not exist with the EKM site.

Isn't it better to make sure that Pearl/new site is configured to work as well as the EKM site with the same amount and quality of data, links etc and THEN build up from there?

[fisicx]

The other site has a shorter category tree (less products to organise).
And much easier for G to index.

[movietub]

And much easier for G to index.

Yes! but we have over 2500 products, we just can't do it with shorter nav. If we did we would be sacrificing human usability in favour of search bots.

And again, EKM had the same nav except each category tree was one longer.

[SFD]

I just looked at lakesclimber.com

They seem to have got round it by not having a header image and not using https

That's one way round it I suppose.:D

Their site is shocking TBH, seems like there are some quite major bugs in the navigation as standard.

On there site, go to a product, then click on 'my account' and it brings up a 404.

[movietub]

I just looked at lakesclimber.com

They seem to have got round it by not having a header image and not using https

That's one way round it I suppose.:D

Their site is shocking TBH, seems like there are some quite major bugs in the navigation as standard.

On there site, go to a product, then click on 'my account' and it brings up a 404.

Ahh but they don't use Google checkout. It took us ages to get Pearl to get it working, loads of problems with Pearl and the security certificate.

If I used http links it would invalidate the SSL certificate (and inform the user!).

I have raised a ticket with Pearl off the back of this thread. Navigation does seem shaky in areas.

But then what is the perfect all bells and whistles solution for ecommerce? All seem to have very weak areas...

[fisicx]

Isn't it better to make sure that Pearl/new site is configured to work as well as the EKM site with the same amount and quality of data, links etc and THEN build up from there?
Well yes but moving to the new site means G has to start all over again. And as I suggested way back on page one, it doesn't appear to be too impressed. There may have been some little jigger on the old site that google really liked, something about the page structures or navigation or whatever that is missing from the new.

This is always the problem with migrating from old to new - change anything and the SE can decide it doesn't like you any more. I had exactly the same thing happen on an old hobby site. Changed the domain name, redirected and when google reindexed the site my rankng dropped away even though the content/navigation hadn't changed.

So the answer is to keep working on reducing the links from homepage to products and sort the page structure so the product information isn't buried 500 lines down the page.

[movietub]

Well yes but moving to the new site means G has to start all over again. And as I suggested way back on page one, it doesn't appear to be too impressed. There may have been some little jigger on the old site that google really liked, something about the page structures or navigation or whatever that is missing from the new.

This is always the problem with migrating from old to new - change anything and the SE can decide it doesn't like you any more. I had exactly the same thing happen on an old hobby site. Changed the domain name, redirected and when google reindexed the site my rankng dropped away even though the content/navigation hadn't changed.

So the answer is to keep working on reducing the links from homepage to products and sort the page structure so the product information isn't buried 500 lines down the page.

Thats the purpose of this thread though, to try and establish what that 'little' thing might have been.

EKM site was well indexed in less time than current site has been up for.

Why reduce links form homepage to products?

[movietub]

Turns out that Pearl isn't creating new homapages from deeper categories. It is in fact a 404 error with a redirect to the homepage which Google should recognise. I have checked and it is not indexing any of the non existant home page urls.

Not sure what affect is of G seeing 404 errors though? Does it just discard or does it penalise?

[SFD]

I would think ~25 internal 404s are quite serious, external 404s not so much.

[movietub]

Sorted the problem but uncovered another!

Pearls wysiwyg editor was corrupting the code (even though I code in DW and copy over) and replacing '/index.php' relatative with '../index.php' relative extension. This is why it was only affecting 3 deep categories - it would only ever go back relative 2 folders.

Apparently Pearl cannot fix the wysiwyg problem and its impossible to get to the site code for future edits with it being corrupted each time! I will have to keep a very up to date copy offline. Annoying as small edits now become a pain.

I don't even want the wysiwyg!

So how effective is internal linking between products? For example if G moves through a seres of products, all of which are relevant to one another, share keywords etc, is it more likely to index them?

It's just I still have a nasty feeling there is something about Pearl that is making Googles life difficult, and all I would be doing is compensating for it.

Has anyone else experience of certain carts being indexed much more succesfully than others - and maybe the causes?

[SFD]

Can you not modify Pearl's code so it doesn't corrupt? Do they allow this?