Robots.txt Blocking Pages, How do I find out why?

[Sparetoolparts]

Hi Guys

My Robots.txt file seems to be blocking pages that it shouldn't here it the file:

https://www.sparetoolparts.co.uk/robots.txt

According to Search Console it is blocking pages like these:

https://www.sparetoolparts.co.uk/dewalt-dw711-scale-861923-00-spare-part-type-3.html

I've looked at the robots file and I can't see how or why this would be blocked.

Any ideas?

 

[UKSBD]

Hi Guys

My Robots.txt file seems to be blocking pages that it shouldn't here it the file:

https://www.sparetoolparts.co.uk/robots.txt

According to Search Console it is blocking pages like these:

https://www.sparetoolparts.co.uk/dewalt-dw711-scale-861923-00-spare-part-type-3.html

I've looked at the robots file and I can't see how or why this would be blocked.

Any ideas?

it redirects to a /catalogsearch/ page which is blocked

 

[Sparetoolparts]

hhmmmm

I just spotted that myself how embarrassing.

Now my next round of questions will be around why is it redirecting at all!

 

[UKSBD]

hhmmmm

I just spotted that myself how embarrassing.

Now my next round of questions will be around why is it redirecting at all!

looks like you have every page that doesn't exist redirecting to a /catalogsearch/result/?q page rather than returning a 404

 

[UKSBD]

My Robots.txt file seems to be blocking pages that it shouldn't here it the file:

It's actually doing what it should do and blocking pages that don't exist

 

[Sparetoolparts]

but the issue here is, that page should and does exist but with a different url:

Buy A Dewalt DW711 RIP GRADUATED SCALE DW711 861923-00 Spare Part Type 1

Dewalt DW711 RIP GRADUATED SCALE DW711 861923-00 Spare Part. Fix Your Dewalt TABLE SAW Today, All Spares are Ready For Quick Cheap Delivery Our Spares are Genuine Dewalt Parts and Come with a Full Dewalt Guarantee.

www.sparetoolparts.co.uk

Now why on earth, has my site map created that other url!

 

[Sparetoolparts]

When I inspect the URL I get this:

Discovery
Sitemaps
No referring sitemaps detected
Referring page
None detected
URL might be known from other sources that are currently not reported
Crawl
Last crawl
Nov 15, 2023, 8:03:22 PM
Crawled as
Googlebot smartphone
Crawl allowed?
info
No: blocked by robots.txt
Page fetch
info
Failed: Blocked by robots.txt



Very odd,

 

[UKSBD]

Every page that doesn't exist redirects to a /catalogsearch/ page

https://www.sparetoolparts.co.uk/click-here-for-free-icecream

if there isn't a search page for click-here-for-free-icecream it redirects to another catalogsearch page

the end catalogsearch page returns a 200, but is blocked by your robots.txt

can you not 404 catalogsearch pages rather than return a 200?

 

[Sparetoolparts]

sorry for being silly what's the difference?

 

[UKSBD]

sorry for being silly what's the difference?

https://www.sparetoolparts.co.uk/click-here-for-free-icecream doesn't exist and redirects to a page that doesnt exist which then redirects to an internal search page.

You are doing the right thing by blocking the page with your robots.txt but some of the pages are already indexed.

It's not really a problem, but if some of the pages that don't exist are in your sitemaps you could get Google Console errors, false flags.

You have to remember Search Console is just a guide

Edit to add. when I say not really a problem that was with regard Google, a problem for your server is another thing.

 

[DontAsk]

Every page that doesn't exist redirects to a /catalogsearch/ page

https://www.sparetoolparts.co.uk/click-here-for-free-icecream

Chrome, for me, just hangs on loading that page, no redirect.

 

[UKSBD]

Chrome, for me, just hangs on loading that page, no redirect.

Did you give it long enough? It takes well over a minute

 

[Sparetoolparts]

Guys I've a fix on the horizon, as I figure a lot of these issues are coming from referring pages which have an old out dated url, which will hopefully be fixed shortly.

Thanks for the input, once I get all the products updated and validated in Search Console, I'll give an update ;-)

 

[UKSBD]

Guys I've a fix on the horizon, as I figure a lot of these issues are coming from referring pages which have an old out dated url, which will hopefully be fixed shortly.

Thanks for the input, once I get all the products updated and validated in Search Console, I'll give an update ;-)

This is why sitemaps (or referal pages) can be a bad thing (if not kept up to date).

Old pages which may no longer exist and which may never get visits are getting visited because Google is still finding links to them, and beacuse you have error pages set up to redirect to another page Google could be getting lost and your server over worked.

I would be querying whether it's doing more harm than good redirecting to a search page rather than returning a 404 page with a search box on it.

 

[DontAsk]

Did you give it long enough? It takes well over a minute

Why does it take so long to redirect?

I have redirects in .htaccess to replace pages from my old site before I switched to openCart and they are instantaneous. I could probably remove them now as it's been a while, but...

 

[UKSBD]

Why does it take so long to redirect?

Because it's not actually redirecting to a page that exists.

It's redirecting to a page that is assuming you are searching for something and queeries the database looking for it.

If it doesn't find it, it redirects again to a page searching for something similar it thinks you are looking fo.

it eventually redirects to a search results page that is returning a 200 (this is blocked by robots.txt though)

 

[goivvy]

hhmmmm

I just spotted that myself how embarrassing.

Now my next round of questions will be around why is it redirecting at all!

It's probably a third-party extension that redirects 404 to catalog search pages. I had the same redirects on a website I worked on, I just forgot what the plugin was.

Check Stores > Configuration and see if you can find a settings. As I recall, it could be turned on and off.