In my team, we use Selenium for automation of functional testing. We also do a lot of manual regression testing to make sure the appearance of the rendered web pages is ok. I would like to develop a framework, that would allow to compare screenshots of the "base" live web page, and the web page on the test server after applying the software update. I believe this will drastically reduce the amount of manual regression testing we need to do.

I would like the framework to be able to detect portions of web pages that were rendered differently from each other. An example of such functionality can be seen here:
http://expression.microsoft.com/en-us/dd565874 (first image in the article)

I have never worked with image processing, and would need ideas on what algorithms are already publicly available that would help me with the task. From what I can see, simple pixel-by-pixel comparison would not work well, because it does not account for the concept of elements.

15 Answers
15

I'd take a look at Google QualityBots. It's generally used for comparing websites on multiple versions of Chrome, but looks very similar to what you are trying to do. I personally haven't had time to try it out myself mainly because of its use of EC2 machines. Other than that, it is open source.

This is terrific! Although I will not be using this system as it is designed, their approach to image comparison is exactly what I need. I'll be implementing it to work with our current Selenium based framework. Thanks!
–
osjakJan 24 '12 at 3:41

I'm pretty sure I would like to implement this in our selenium stack... +1
–
yoosibaJan 24 '12 at 16:22

@osjak - Did you get it implemented? I need to build something similar.
–
9ikhanApr 30 '13 at 11:38

Imagemagick, a cross-platform imaging library and command line tool, has functions that can be used to compare images. We're using it with pretty good success to determine if two images differ. We then have a GUI built with the same library that can pull the two images up side by side and highlight the differences for a human to decide whether the differences are significant.

sikuli is good to automate things based on images, but comparing images as osjak needs is not something you get out of the box. Look at blog.sikuli.org there you can see that image recognition in sikuli is used to find images on desktop, so you can use desktop areas as methods parameters. Calculating differences of images, marking differences is totally different topic. I would argue that sikulii is not much better to that task, than selenium (or basically language with which you use selenium).
–
yoosibaJan 23 '12 at 14:43

sikuli uses simple pixel-by-pixel comparison, from what I can see. It works for them, but I don't think it will help me. Thanks for the suggestion though.
–
osjakJan 24 '12 at 3:45

Well, unless you're comparing against exact match or some specific comparison algorithm or DOM structure, then Sikuli is useful for comparing images or any object/element on the screen/page represented by an image. The method to do this is to capture a reference element/object (say just a button) on the screen (crop it out from a screenshot) or if you need it relative to other objects, then capture a larger portion of screen or even the whole screen. Save the (cropped) capture as reference image. Then use Sikuli API in test to try and find if object exists anywhere on given screen.
–
DavidOct 4 '12 at 2:44

The Sikuli API has parameters for setting level of difference in finding images as well, though it might be less feature rich in how you specify differences compared to other tools or libraries.
–
DavidOct 4 '12 at 2:46

We're using fighting layout bugs. Quite tricky to set up - the out-of-the-box config is very sensitive in that it gives loads of false negatives. there are several parameters for reducing sensitivity though.
–
DuncNJan 23 '12 at 13:30

We also use a tool for the comparison between the 'base' image (oracle) & the newly rendered page. Our Programmers wrote it in-house based on some code they found on the web - I'll post a link when they get back to me.
–
DuncNJan 23 '12 at 13:33

This seems a bit too complicated for what I need. Thanks though!
–
osjakJan 24 '12 at 3:42

Just a quick question though - are you sure this is an avenue you want to go down? How many pages are you looking to check? Over a certain amount of pages, the time it takes to run the checks negates the value of the checks. This on top of tweaking the config to strip out false negatives. Add to this any dynamic data will throw out the comparison - one of our bugbears is googlemaps being slightly different each time the page is rendered

I agree with DuncN in many regards. Maybe a proof of concept would be helpful to try out before committing a great deal of time to it? There are paid services that have almost zero setup time and would tell you quickly whether the idea itself will be helpful like MogoTest, which I have used in the past.

It isn't free, but is fairly cheap and might be good to pull examples from for your own development should you choose to develop on your own.

I'm the QA manager of Learni an education startup, and I'm a customer of applitools. Their brilliant app/sdk allows me to do exactly what you're describing, e.g adding one code of line, every time I want to take a screenshot, on the current selenium/appium code test you're already using, send it on the fly to their server, and when the test ends I go to their website where i see all the screenshot taken. Then I can approve what passed and what failed, so next time the same test runs, it already knows which test/ screen passes and which not.
You may "tell" which parts of the screen to ignore (like dates which are changing), and it even knows to ignore this section on other screenshots (so you don't need to change it manually on 100's of screenshots). There are other many interesting features, but bottom line, there is no need to invent the wheel. Applitools almost literally, using state of the art technology, putting eyes in front of your tests screens, saving you the need to build such an app, and the need to watch it your self.
Good luck, thanks,
Avi.

There's also a simple old school method of comparing a given screenshot/image to a reference one in checking that they have the same MD5/SHA1 hash. That unfortunately likely may not work for screenshots of whole web pages. But it will at least work for comparing of individual images on a web page. Thought I'd point it out though.

It's a npm package used to compare images/screenshots. If you take a look into the examples folder you will see an examples of how to take those screenshots dynamicly with casperjs and then you can compare it using automated-screenshot-diff.

automated-screenshot-diff generates an perceptual diff from image A - image B and I think that is pretty much what the question is about.

Also... the project is able to generate a comparison report in JSON and HTML (to store as a Jenkins Artifact).

How it works?

Then automated-screenshot-diff will scan your screenshot's folder and calculate differences between your pre-production (stage) release and your production release. All generated image diffs will be in the same directory as your screenshots. If you don't known how to generate those screenshots, take a look at examples folder.

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

Hi, Igor. Could you give some more information about how this project could help the poster and what it adds that hasn't been addressed in the other answers?
–
Kate PaulkDec 16 '13 at 12:33

@KatePaulk just edited my original answer to make it better.
–
Igor EscobarJan 3 '14 at 16:18

It's probably worth noting that you will still need to manually check the diff to make sure it looks ok. You could trigger a failure based on the % difference but generally that is not very reliable.
–
ArdescoJan 6 '14 at 12:14

If you are using python+selenium, there is a useful package, called needle:

It checks that CSS renders correctly by taking screenshots of portions
of a website and comparing them against known good screenshots. It
also provides tools for testing calculated CSS values and the position
of HTML elements.

You can watch a short demo at Julien Phalip's talk from PyCon US 2014:

What you want to achieve is called automated visual regression / validation tests.
Test automation tools such as Selenium or its wrappers (Protractor) only provide the automation part - and you are correct in wishing for visual validation of the page display.
As you mentioned - you need the following elements:

Version control - to define a baseline that represents approved results and the sets of current results to compare against

Integration with automation framework - essentially you want to keep your existing tests untouched - and only take a screenshot now and then

Image comparison - you want to avoid primitive pixel by pixel comparison as most of your tests will fail. You also want different comparison modes - i.e. - ignore changes that are undetectable to the human eye, or ignore changes that are proportionate to the page layout (picture a case when you change the language in the page - why should the test fail if everything behaved properly?)

Ability to define dynamic regions that should be ignored

support for iFrames

Last but not least: easy maintenance. If you made a legitimate change to the page - and all the tests failed (as they should) - you want to approve the change only once and have it automatically approved across all the steps. Otherwise you're facing the nightmare of going through all your tests and approving it.

As @user3210346 mentioned - pay a little money and use Applitools - as they are currently the only company that provides all of what I mentioned above.

I've had great experience with their product and they are very inclined to listen and add capabilities.

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

2

Broken link. It would be more valuable to summarize the article in your answer.
–
user246Dec 24 '14 at 2:25