Crawl stats - pages crawled down/downloading page up

DigitalDarren

Free Member
May 14, 2016
6
0
I've noticed a worrying trend in Google Search Console recently.

Over the last 90 days the pages crawled per day has been on a downward trend since June (where we were around 150-200 pages per day with some peaks and lows), to August (where we have dropped under 100). In July we were under 100 most days with a few peaks where it jumped to 366 for example.

At the same time, time downloading a page has increased. We were well under 500 milliseconds most days in June. Since then we are close to or above 1000 most days.

I'm unsure what has caused this issue.

A few things that have happened during this period:
- Wordpress updates (new version/plugins)
- Additional pages added to website
- Additional content added to some key pages (2/3 more images and a few hundred words text)
- Sign up rows added to all key pages (i.e. email/name form to capture visitors emails)

We use a marketing automation platform called Pardot, so for sign up rows there is a form handler. Likewise, where we've added additional content, there are trackable URLs from Pardot added to the CTA buttons.

Would a third party site like Pardot cause issues.

I should note that despite the above concerns, our organic traffic during the same period has gone up. We are still appearing high for our target keywords. And page speed tests have shown that the site isn't slow (and is much faster than our competitors for example).

Any advice would be appreciated.
 

fisicx

Moderator
Sep 12, 2006
46,772
8
15,418
Aldershot
www.aerin.co.uk
Over the last 90 days the pages crawled per day has been on a downward trend since June (where we were around 150-200 pages per day with some peaks and lows), to August (where we have dropped under 100). In July we were under 100 most days with a few peaks where it jumped to 366 for example.
Has the content on all of those pages changed? Or if they are new pages is Google able to find those pages. Google doesn't keeping coming page to a site unless there is a reason. If you haven't added anything of value then Google won't need to reindex.

With regards page loads speed, wordpress plugins and third party content can slow things down. Every time you add more functionality there will be new server calls which can add up to a significant delay. Pardot is probably one of the biggest culprits.
 
Upvote 0

NathanPayne

Free Member
Aug 2, 2016
18
2
Yeah what @fisicx has said is my thoughts exactly, with one little addition though.

YouTube videos... Have you removed any YouTube embeds from the website? YouTube in order to place ads through a YouTube video has to visit the site to crawl it in order to server relevant ads.

I used to run a website that had lots of embeds and the site was being crawled almost constantly. Then we upgraded the video hosting to another platform and crawling fell through the floor. Ranking stayed the same though. Lots of movie trailer websites have commented on the same types of issues as it can really impact on server load if Google is constantly crawling numerous pages.
 
Upvote 0

DigitalDarren

Free Member
May 14, 2016
6
0
Yes, our scores are really high/positive. Everything says our pages are really fast, but I just checked the time spent downloading the page for saturday and it shot up to 1818. In analytics though it says av. page load speed for that day was 0
 
Upvote 0

DigitalDarren

Free Member
May 14, 2016
6
0
With regards page loads speed, wordpress plugins and third party content can slow things down. Every time you add more functionality there will be new server calls which can add up to a significant delay. Pardot is probably one of the biggest culprits.

Ok, thanks. We have increased the coverage for Pardot sign up forms/form handlers recently across the website, so that's suggesting it's the culprit. However, we did also update to the latest wordpress during this period, so I'm not sure if that's had an impact
 
Upvote 0

fisicx

Moderator
Sep 12, 2006
46,772
8
15,418
Aldershot
www.aerin.co.uk
There is all sorts of chaff on your site that could be culled. You don't need hall the scripts that get loaded with the theme. And why do to have to use pardot? There are lots of other lighter and faster ways to collect the necessary data.
 
Upvote 0

Latest Articles