A picture is worth a thousand words David Tolley CI Architect Build/Release.

Similar presentations

1
a picture is worth a thousand words David Tolley CI Architect Build/Release

2
A little about me And maybe something about Groupon Engineering Page 2 Married… sorry for all the… lady… out there Contact information: dtolley@groupon.com Twitter: DavidTolley Groupon 400-20,000 engineers 300+ commits in a day to main app Each commit triggers 9 jobs Each job has between several hundred to several thousand tests Many other jobs not associated with out primary application https://engineering.groupon.com/ Were always hiring: http://www.groupon.com/techjobs/

3
A Little Selenium Timeline Completely skewed from only my experiences Page 3 1.I run tests 2.I get failures 3.Developer says “ Works for me” 4.I print out the stack trace of failures from now on 5.Developer says “Only fails on Jenkins, that stack trace is a red herring” 6.I take a screenshot of the Browser at point of failure, archive it 7.Developer looks at stack trace 8.Developer finds the corresponding screenshot in the build’s artifacts 9.Developer says “This is cool but… I don’t like having to search for a screenshot” 10.I develop a plugin to insert screenshots into test failure details page 11.Developer says “Awesome! Now I easily know what to fix”

4
So what did I learn? Data is awesome Page 4 But a selenium failure with out visual proof, or a picture without context is hard to digest

5
Words describing a picture My “stack trace” Page 5 Giant Monstrous Absurd 15 Feet tall by 20 Feet Wide Brown stuffed cat crammed into UFO Hanging from the celling on 4 th floor of Groupon’s Head Quarters Giant tongue sticking out of its mouth Cat haters worst nightmare I feel bad for the person who has to clean it’s litter box WTF?!?! This has to be a joke…

14
Selenium can take screenshots… And Jenkins can interact with them… Page 14

15
Screenshot Comparison with Jenkins & Selenium Best thing since apple pie and ice-cream Page 15 Biggest number of bugs in a typical backlog are cosmetic Hardest bugs to find are small cosmetic bugs Manual testing doesn’t fit well in a lean, fast-paced environment Impossible to fully cover UI manually on a Continuous Deployment schedule

16
Screenshot Comparison Some drawbacks… I mean guidelines. Page 16 Only ran on release candidates Need to continually update the “image repo” – so automate this Need’s to run with the same screen resolution each time it’s ran Other then that it’s awesome!

17
Screenshot Comparison Step 1: Auto-magicaly build an image archive Page 17 Create a selenium “crawler” – capture and store endpoints 1.Selenium already crawls your website, simply add the option to take a screenshot at the end of every test, name it what the test is named 2.SCP to repo (I chose userContent/imageArchive on our master)

18
Screenshot Comparison Step 2: Create a build that runs on release candidates Page 18 Create a Jenkins job that can automatically run on your release candidates 1.Much like your crawler job, except tied into your release pipeline 2.Takes the same picture of the same endpoints, and names them the same way 3.Configure ScreenshotComparison plugin, giving it your archive repo to compare against, and location of screenshots on the current run 4.Optionally set a % difference threshold

19
Screenshot Comparison Step 3: I’ve got the magic in me Page 19 1.Gets lists of files in the archive repo 2.Gets list of files in current workspace 3.Compares the files with the same name 4.Prints and stores the percent different Percent Different = Different Pixels / Total pixels