I Need a Plan

[Sparetoolparts]

The Problem
Duplicate content across my site caused by numerous issues one being products appearing in multiple categories and products not shown in any categories at all

Solution
Get an Audit done including a crawl of the website and find out exactly what is indexed and what isn't then address issues after audit

Problem
Site audit attempted and the structure lead to it being unfeasible to carryout a crawl of the site due to the resources needed.

Solution
Amend site so the audit can be completed

Possible Plan
1. Change category pages from .html to / to help with crawling
2. Move Products to appear in only 1 category (help crawl and duplicate content)
3. Add products that aren't appearing in categories to newly created categories specifically for them
4. Turn on Category Path on products (currently turned off)
5. Allow Index:Follow on all categories (currently set to index top level categories only)

Help/Advice Needed
Guys I need help with planning on what to do and in what order, things that I'm concerned with include if/when I turn on category paths for instance what affect will this have on my rankings if I choose a granular approach to updating the site section by section over a few months so I can monitor the outcome of the changes. I might improve one category but the others will suffer until I get them fixed, or shall I fix it all at once and hope for the best?

Anyone done anything similar b4 what would you do? can anyone recommend someone to help with this?

 

[BenKirkwood]

Not sure anything in your possible plan will help you audit the site any easier, turning on category path will change the URLs so you'll need to make sure you cover this with 301s (so you still need the existing URLs).

Ran a couple of tests in Screaming Frog - 64bit Java - Maxed RAM, strip automated pages, exclude images,css,js using regEx in the Exclude options as follows:

.*sendfriend.*
.*product_compare.*
.*/media/catalog/product/cache/.*
.*jpg$
.*png$
.*gif$
.*jpeg$

But, you have tens of thousands of URLs, so allow several hours to run through.

Rather than crawling the public site, consider whether you can extract the URLs directly from the DB, found this (might be out of date but...) :

http://t3jsot.blogspot.co.uk/2011/08/magento-export-products-with-full-urls.html

or perhaps look at extracting from the sitemap.xml?

Once you have the URLs, you still have the problem of checking the indexing on thousands of pages in Google/Bing, this will be a slow process as you'll need to use proxies - you could try SEOTools for Excel and cross-check the google cache date (I'm sure there are faster solutions).

Due to the volume of products, this is going to be a nightmare.

I'd have to consider if duplicate content is really an issue? Having products appear in multiple categories is normal for E-Commerce sites, Google is quite aware/tolerant of this normally. You already exclude sub-categories. It is also normal for Google not to index everything especially as the majority of end-points would be considered as 'thin' content.

I'd be interested in how you resolve this!

 

[Alan]

If I recall from a previous thread, it is some 1.4 million URLs

 

[Sparetoolparts]

The changes proposed were/are to allow for a screamfrog crawl

As the machine we were going to use couldn't handle the site in it's current config, but not only that, the individual product parts should be added to the category structure to allow for easier indexing too.

Yes I know I would need to 301 all the categories from .html to "/" but again this was to help with crawling with screamingfrog and the google site command

 

[Sparetoolparts]

P.S

I have extracted all the URL's from the DB already

 

[justinaldridge]

This is classic Magento issue! The problem with Magento is that out of the box it's not SEO friendly and the default "products in any category" set up is a killer.

We took on a client last year with 1.5 million indexed pages...and they only have about 10,000 products. We had to use a huge amount of noindexing throughout the site to bring it down to their current 10,000 indexed pages.

It's a big job and it needs some developer time, but it is possible.

I think you should handle fixing the site in stages, maybe even one category at a time.

Whatever you do, it's going to take a lot of time!

 

[Sparetoolparts]

Hi

Again thanks for the advice, I know it is going to be a long hard graft to get it sorted, I am willing to spend money when it is needed,but with a limited budget I'll need to do as much as possible myself.

yea I've looked at Bruceclays Silo several times and that's why I'm turning category paths back on, but I before I do, I need to fully grasp the consequences, of it doing this, hence why I need a good plan and how to execute it.

Every time I feel like I have it, it keeps slipping through my fingers, so I'm hoping advice here will help it all sink in!