Crawler Settings

When setting up a new Sitebulb audit, the first thing you need to do is select what kind of audit you wish to carry out, and which crawler you wish to use.

The Audit type offers two options: Standard Audit or Sample Audit. In most cases, the Standard Audit will be suitable. However, if you are crawling a particularly large site, or crawling a site for the first time, the Sample Audit may be more suitable.

Setting up a Sample Audit allows you to define some sampling rules, and then Sitebulb will crawl a portion of the website according to those rules, which you appear as soon as you select 'Sample Audit' from the dropdown.

The Crawler Type also has two options:

HTML Crawler - this is the default option, and will be suitable for most websites. This uses 'traditional' HTML extraction, and is a lot quicker.

Chrome Crawler - select this option if you need to crawl a site that uses a JavaScript framework, or has a lot of content dependent on JavaScript. The Chrome Crawler will render the page using a version of headless Chrome (essentially, a Chrome browser without a user interface). In order to render, the Chrome Crawler will need to download all the page resources, so it takes a lot longer.

Additionally, there are some data analysis options that are only available if you use the Chrome crawler, so if you want that data you will need to select the Chrome Crawler, regardless of whether you need to crawl the website with the Chrome Crawler in the first place.

The options in question are:

Page Speed, Mobile Friendly and Front-end - Sitebulb can collect more data when using the Chrome Crawler

Code Coverage - Sitebulb can only process this report with the Chrome Crawler

Accessibility - Sitebulb canonlyprocess this report with the Chrome Crawler