Sitebeam Introduces Real Browsing

By Oliver Emberton

Posted on May 29, 2012

Today we’re pleased to introduce a new technology to Sitebeam which will ultimately affect everything we do: real browsing.

Until now, we’ve downloaded pages like a traditional ‘bot’, which means all Sitebeam can do is see your code. The problem with this is there’s now a lot of clever things that code can do – like AJAX – which are too complex to simulate. We used clever ways of ‘guessing’ these exceptions, but they were always potentially imperfect.

Real browsing technology (technically known as headless browsing) allows Sitebeam to see webpages inside a real web browser. We can load pages and see exactly how they look, how long they take to load, what content they contain in Javascript and more. We’re pretty excited by this, as it will allow us to test for a lot more in the future, and with perfect accuracy.

Today we’re rolling out real browsing for the Cookie Law test, which has been completely rewritten. We’ll be introducing it into other tests in the future.

Cookie Law test

The new Cookie Law test (formerly the EU ePrivacy Law test) can now detect all Javascript cookies perfectly as a result of real browsing, with no ambiguity. We also now detect Flash cookies.

Note that scoring for this test now ranges from 0 to 10 (previously it was 5 to 10). This reflects the law coming into effect as of May 26th 2012 in the UK.

Disabled tests

Today we’re also disabling two existing tests by default: Design variations and URL chopping. You can still run these tests, but if you run a report they won’t include them by default – you have to enable them manually (under Account > Tests).

The Design variations test is one of the least used tests, and has no effect on any Summary scores. It’s also one of the slowest to run. We therefore decided that removing it would speed up reports (which most people want) at the cost of removing a test they probably wouldn’t use.

The URL chopping test was introduced about 5 years ago and is now essentially redundant. It’s also one of the least used and least influential tests.

Other changes

We fixed an extremely annoying bug that could cause a report to randomly fail instantly (you would request a report and it would fail right away – it never even started).

We’ve also made improvements in how we test sites that struggle/refuse to handle lots of incoming connections – you can now control exactly how many connections Sitebeam keeps open at one time (Site settings > Pages to test > Max connections).

What’s coming next

Our number #1 most requested improvement is to the PDFs we create. A future version will create PDFs natively (i.e. without a plugin) and to a much higher standard than we do currently. We’ve been working on this for some time.

We hope you find these changes useful – if you’ve questions or suggestions please leave a comment below.