Should I Remove 250k Products from my Website

[Sparetoolparts]

Hi Guys

I've got a Magento 2 website with over 1 Million SKU's, the problem I face is when I go re-index the site, it constantly falls over and the indexes freeze due to the sheer number of my products. Eventually it does Re-Index everything but this can take up to a 1 x day to resolve.

I'm on a dedicated M2 server with resources coming out of my ears so the answer of just add more this or that isn't going to resolve this. I do have over 250k products which are no longer available which I'm contemplating removing from my site so reducing products and hopefully fixing the indexing issues. However I'd be worried that I would lose the SEO benefit of having these products appearing for me organically in Google.

I realise these products won't generate income themselves but they still may bring customers to me site to order something else.

I'd be curious to hear your thoughts on this.

I plan to remove the products from my staging site and then run some tests just to see if in the indexing issues are resolved but the SEO question is what I'm interested in finding out more about

 

[Solve My Problem]

I suspect that Google doesn't rate long term out of stock products very highly when they are all about the customer experience.

Google Analytics will tell you what traffic these pages bring in and their worth, use the data rather than a feeling/or guess

Darren

 

[fisicx]

Google is all about UX these days. Out of stock or discontinued products aren’t good and could be adversely affecting your ranking.

 

[WebDesires]

it always amazes me how shockingly bad Magento is yet is still somehow highly favoured by people... dont get it.

As for your out of stock/discontinued... I dont agree with the other comments and for good reasons.

I work with a few big ecommerce clients who have this predicament of many discontinued or out of stock items and I know from experience there is very good reason for them to persist.

out of stock items - might get some stock at some point, so this is a no-brainer to keep them so pages are not de-indexed and then have to be re-indexed as stock fluctuates.

discontinued - a weird one because they will never exist again for sale, BUT sometimes there are very good reasons to keep them findable, for example if you have a parts website, people will be looking for a specific part and its USEFUL for them to be able to find it instead of frustrated they cant, from there you can recommend an alternative or give advice on why they can not get that item anymore.

Of Course you as the seller should know the importance of items and if people are really looking for them, and if you are recommending alternatives or work-arounds or etc. Obviously if they are straight up dead and you are getting little to no traffic to those pages from search engines, there is absolutely no reason to have them, get rid.

Ultimately what you should do with magento is replace it... yeah i know probably not feasible, the other suggestion is to setup crons for clearing the caches or when clearing caches run the cammands through ssh instead of from the store, its quicker and less likely to fail.

php path/to/magento/bin/magento indexer:reindex

 

[ORDERED WEB]

You don't mention what version of Magento you are using. The Commerce and Cloud (Enterprise) editions of Magento 2 deal with indexing differently. (the enterprise version is more geared up for larger catalogues).

There are modules out there that help index products more efficiently, and some workarounds that a developer can ping in to stop the re-indexing system from re-indexing things it doesn't need to. and effectively keep on top of indexing in near real time as opposed to "doing it in one go"

You can offload some indexing to SOLR, etc.

 

[DontAsk]

Google is all about UX these days. Out of stock or discontinued products aren’t good and could be adversely affecting your ranking.

How does google know if an item is OOS?

 

[fisicx]

How does google know if an item is OOS?

Google can read.

 

[UKSBD]

there was a tweet last week where someone had something along the lines of
"we do not deliver to this post code area" on every product page and it caused problems with Google dropping the pages.

The text wasn't even visible, but once they removed it things improved.

John Mueller tweeted about it too, I'll see if I can find it.

 

[UKSBD]

just found some tweets relating to this.

It was more to do with search console returning soft 404's

You can read some of it here

https://twitter.com/x/status/1476587474642092035

 

[antropy]

it always amazes me how shockingly bad Magento is yet is still somehow highly favoured by people... dont get it.

As above, you really should move away from Magento for these reasons: https://www.antropy.co.uk/blog/the-lure-of-magento/

Consider a rebuild in something light, efficient and fast like OpenCart, or Prestashop.

Paul.

 

[AlanJ1]

As above, you really should move away from Magento for these reasons: https://www.antropy.co.uk/blog/the-lure-of-magento/

Consider a rebuild in something light, efficient and fast like OpenCart, or Prestashop.

Paul.

Can Prestashop / OpenCart handle 1 million + SKUs? (Genuine question).

 

[antropy]

Can Prestashop / OpenCart handle 1 million + SKUs? (Genuine question).

Of course, they run on the same type of database as Magento, just Magento is 10x more complicated and bloated.

So they can handle that number of products * more * efficiently not less.

Paul.

 

[WebDesires]

How does google know if an item is OOS?

Well you should have JSON+LD Structured Data so google knows your product pages, prices and etc.

 

[WebDesires]

just found some tweets relating to this.

It was more to do with search console returning soft 404's

You can read some of it here

https://twitter.com/x/status/1476587474642092035

I'm highly sceptical this was the issue, Google knows all about hidden content controlled by JavaScript, Heck these days they can do some user-interaction themselves and figure some things out.

I mean classic example you've put "we dont deliver" on this thread, does that mean google will instantly treat this page as a soft 404?

They must of had another issue they accidentally or silently fixed (if I was the dev who de-indexed a whole site I would be sweating; looking for an easy excuse), or perhaps another GWT glitch as is so very accustomed.

 

[WebDesires]

Can Prestashop / OpenCart handle 1 million + SKUs? (Genuine question).

Yes - I have one client who pulls a bunch of product feeds and before we refined what he was selling he was importing everything, was over a million items (loads of tit-tat parts and crap).

OpenCart scales pretty well without bottlenecking.

 

[thetiger2015]

You will need a plan for dealing with that many OOS products - the URLs may hold value in search results etc. Bulldozing that many pages is not recommended.

You will also need a plan going forward, no matter if you re-platform, to deal with on-going legacy products. Perhaps a scheduled pruning of the product pages but 301 redirecting to closely related items.

250k product pages, even with just 1 or 2 visits per month, is a lot of traffic to chop off. Handle with care.

 

[Solve My Problem]

You will need a plan for dealing with that many OOS products - the URLs may hold value in search results etc. Bulldozing that many pages is not recommended.

You will also need a plan going forward, no matter if you re-platform, to deal with on-going legacy products. Perhaps a scheduled pruning of the product pages but 301 redirecting to closely related items.

250k product pages, even with just 1 or 2 visits per month, is a lot of traffic to chop off. Handle with care.

If the traffic results in bounces it's worse than worthless, it's costly.

If the traffic is converting then it's valuable, if not, it's a SEO killer

Darren

 

[SEODEV#338055]

If the traffic results in bounces it's worse than worthless, it's costly.

If the traffic is converting then it's valuable, if not, it's a SEO killer

It all depends on whether the search is repeated or not

A bounce in itself is not a killer for SEO, despite many opinions to the contrary

If you need a weather forecast this search will normally result in a bounce

If you need a word definition this too will normally result in a bounce

If you are simply trying to find out the market price of a product again this will normally result in a bounce

If the user on an ecommerce website bounces but does not repeat the search in Google or try an alternative phase on the same search then Google may consider the search to have been satisfied (or abandoned for any number of reasons) so it's the whether the user repeats the search and finds an alternative website where the data suggests a more favourable user experience including feedback from the Google Chrome User Experience Report

I have been wondering for some time if Google's algorithm understands conversions ie if a search ends with a thankyou.php basket order confirmation page is the algorithm able to understand this as a satisfied or completed search?

discontinued - a weird one because they will never exist again for sale, BUT sometimes there are very good reasons to keep them findable, for example if you have a parts website, people will be looking for a specific part and its USEFUL for them to be able to find it instead of frustrated they cant, from there you can recommend an alternative or give advice on why they can not get that item anymore.

On the original point about out of stock but not out of productions products it is always better to leave the landing page online because "you can recommend an alternative or give advice on why they can not get that item anymore" and by deleting the landing page you also delete the product image in Google images and it's landing page which is a major source of traffic for many ecommerce websites

I realise these products won't generate income themselves but they still may bring customers to me site to order something else.

That's exactly right and as @WebDesires correctly says you can recommend alternative products

When recommending alternative products you can do this automatically and program your site to display a default message along the lines of "this product is currently out of stock, however you may be interested in....." and then auto populate your alternative product tiles which link to the alternative product's landing page

It's difficult enough to get ecommerce search traffic so I wouldn't recommend deleting huge volumes of product pages en masse but rather finding smart ways to market other products

 

[Chris Hanson]

If you have products that are no longer available then get rid as soon as possible. Like real space it costs to keep space for something that doesn't exist with this instance being your customers valuable time that has been wasted. The product should not be the strong page so put greater emphasis on SEO for the category page which they link to. That way if your category page is indexed then new/old products will be visible.

 

[WebDesires]

If you have products that are no longer available then get rid as soon as possible. Like real space it costs to keep space for something that doesn't exist with this instance being your customers valuable time that has been wasted. The product should not be the strong page so put greater emphasis on SEO for the category page which they link to. That way if your category page is indexed then new/old products will be visible.

You really should read the thread before commenting.

 

[Chris Hanson]

Your point is?

 

[14Steve14]

I realise these products won't generate income themselves but they still may bring customers to me site to order something else.

I'd be curious to hear your thoughts on this.

We keep all our out of stock products listed on the site, for the very reason that they may bring someone into the website. The OOS items are set to not show to customers as there is no point. They get moved into a hidden category. What we then do is to add a redirect to either a similar product, or the main category that the product was in. People can then see that the product is out of stock, and also get to alternative items.

Seems to work.

 

[SEODEV#338055]

If you have products that are no longer available then get rid as soon as possible. Like real space it costs to keep space for something that doesn't exist with this instance being your customers valuable time that has been wasted. The product should not be the strong page so put greater emphasis on SEO for the category page which they link to. That way if your category page is indexed then new/old products will be visible.

What would you link the new dead pages to? Remember you're giving advice that will affect 250,000 product pages

1. Nothing 404 error page?
2. Alternative product page?
3. Category page?
4. Homepage?
5. Discount page?
6. Apology page?
7. Contact page?

 

[Chris Hanson]

14Steve14 could see where I was coming from. Mentioning category page twice might have given it away.
When you think that robots have a finite time on sites you would want them to index actual products. Duping visitors to other products will not yield results as they are after the price for a long tailed search phrase.

 

[SEODEV#338055]

Mentioning category page twice might have given it away.

Fair enough