TL;DR

Scrapers

In 4.3, I’m introducing a concept that I call Scrapers. Scrapers are written using a new XML format that allows you to easily extend SeoTools. A scrapers can given a set of parameters, fetch a web page, locate a particular part of the page and then parse a return value.

Updated how delays between requests are managed. In HttpSetting you can write:
<IntervalBetweenRequests RandomFrom="1000" RandomTo="1500" IfSame="Host"/> This will ensure that requests to the same host will be executed with a delay of random ms between 1000 and 1500. IfSame can be "Host" (default), "Domain" or "Url". This is used in some scrapers as a strategy to not get blocked too quickly.

Added a default user agent in HttpSettings as some sites returns a http error when none is set. Request are made to look like Chrome 39 on Windows 8.1.

Fixed some Spider issues with GooglePageSpeed and GooglePageRank columns. Some other stability improvements.
*Updated available metrics and dimensions in Google Analytics Core Reporting. Also treating all dimensions as strings.

You can now fetch more than 1000 rows (if you have Pro) using Analytics formulas.

Saving SeoTools.config.xml in AppData directory if failing to save SeoTools dir. If SeoTools.config.xml is not found in SeoTools installation path one is created.

Added some stats to the Spider progress windows.

Fixed problem in Spider when creating report and there’s no active workbook.

HttpStatus and UnshortUrl functions now uses global HttpSettings. HttpStatus column uses HttpSettings from wizard if defined otherwise global HttpSettings. This required when scraping mobile sites where you need to set the User-Agent.

HttpStatus column in the Spider now has an option to show the “final” http response status code or the “first”.