Screaming Frog continues to be the most handy SEO tool I can think of. If the existing features weren’t enough, I came across their blog post on how to crawl an entire website to bulk crawl PageSpeed Insights scores for individual pages.

PageSpeed Insights tool only allows you to plug a single page in order to get scores.

This obviously presents a challenge. Say, you’re preparing for a website pitch and need to compare SEO performance for brand vs competitors. You can plug in the homepages of the client vs competitors and compare the scores for the homepage. However, this only addresses the homepage. What about category/sub-category/product pages? These could be equally [if not more important for your website]

I’ll summarize the instructions given in the Screaming Frog blog post.

Append “https://developers.google.com/speed/pagespeed/insights/?url=” to all your website URLs to convert them to: https://developers.google.com/speed/pagespeed/insights/?url=http://analyticslog.com/. Repeat this for all pages of your website.

Box plot to find out what are the PageSpeed scores within which 50% of the page lie, highest/lowest values and if there are any outliers. Check this blog post If you’d like to generate box plots R Studio.

Based on URL structure, find out the key category pages and quickly pull up the scores.

Show the distribution of values rather than a single score to bring attention of required SEO changes

Here are the results for my website.

Immediately post crawling, I can see that FCP/DCL metrics are missing for my blog [across all pages, desktop and mobile]. Why is this so?

These two metrics are generated from Chrome User Experience report.

Quote:

The median value for each metric (FCP or DCL) is compared to all the pages monitored by the [Chrome User Experience] CrUX report. Each metric is assigned a category, depending on where it falls in the distribution:

Fast: The median value of the metric is in the fastest third of all page loads.

Slow: The median value of the metric is in the slowest third of all page loads.

Average: The median value of the metric is in the middle third of all page loads

TL;DR: The bigger your website, the more likely it is to have FCP and DCL metrics available.

Putting these into a pivot > Chart quickly shows you a histogram.

The mean score comes out to be 76. As the distribution is skewed, the median is a better metric. This comes to 83 [for mobile optimization score].

Depending on the number of sites you’re crawling, you could put very easily highlight the key values and quickly draw attention to SEO. Nothing gets more attention of your clients than these two topics: Customers and Competitors.

Using this feature in Screaming Frog adds immense value to your SEO audit as it shows the depth of homework done in understanding the website.