Ari Shamash from Google talked about the consistent issue of non-deterministic (flaky) automated tests and how Google use hermetic environments to highlight these tests. This involves creating 5-20 instances of an application and running tests repeatably to identify inconsistent results.

James Waldrop from Twitter discussed their ongoing strive to eliminate the fail whale through performance testing. He discussed production testing techniques: canaries (small subset of users provided new functionality), dark traffic (use existing app but send some traffic to new version and throwaway response), and tap compare which is comparing dark traffic to actual. He then talked about his tool homegrown performance tool Iago (commonly called Lago because of the capital I in sans-serif fonts).

Malini Das and David Burns from Mozilla discused automated testing of the FirefoxOS mobile operating system and how it uses WebDriver extensively to test the inner context (content) and outer context (chrome) of FirefoxOS. They have a neat Panda Board (headless devices) device pool which can cause non-determistic test failures due to hardware failure. One key point was how important volume/soak testing as people don’t turn off their phones – they expect them to run without rebooting them or turning them off.

Igor Dorovskikh and Kaustubh Gawande from Expedia discussed Expedia’s approach to test driven continuous delivery. Interestingly they use ruby for their automated integration and acceptance tests even though the programmers write their web application in Java. Having a green build light is critical to them which means a failed build rolls back automatically after 10 minutes: giving someone 10 minutes to check in a fix. To enable this, they have created a build coach role which is shared amongst the team, even project managers and directors can take on this role to keep the build green. They also stated that running mobile web app tests on real devices and emulators (using WebDriver) has been beneficial, as well as standard browser user agent emulation to get around issues with multiple windows for features like Facebook authentication.

David Röthlisberger from YouView demonstrated automated set top box testing which uses a video capture comparison tool that compares expected images – similar to Sikuli. These images are stored in a library should the application change in look and feel.

Ken Kania from Google discussed ChromeDriver 2.0 and its advanced support for mobile Chrome browsers.

Vojta Jina from Google demonstrated Karma JS Test Runner (formerly contentiously known as Testtacular) and how it runs JavaScript tests in real browsers as you write code. Neat stuff.

Simon Stewart from Facebook talked about Android application testing at Facebook. Originally Facebook used Web Views in Android & iOS which enabled frequent deployment but resulted in a terrible user experience. They have since started developing native applications for each feature. Interestingly, every feature team has responsibility for all platforms: web, mobile web, Android and iOS. This enables feature parity across platforms. Facebook use their own build tool BUCK which enables faster builds. Simon also pointed out that engineers are entirely responsible for testing at Facebook: they have no test team, no QA department or testers employed. Some engineers are passionate about testing, like some others are passionate about Databases. Dogfooding is very common amongst engineers which results in edge cases being discovered before being released to Production. A highly entertaining talk.

Google really know how to run a conference. It’s hands-down the smoothest one I’ve attended; from the sign-in process to the schedule being adhered to. They even have stenographers and have sign language interpreters.

Oh, and NYC is great. I went to the top of the Empire State Building yesterday: the view to lower Manhattan was amazing.