The Future of Testing (GTAC 2008)

Google Test Automation Conference (GTAC) is my all-time favorite conference. It's free. It's on a single track — this means you don't miss any talks and everyone experiences the same journey of thought. Also, since you have to apply for admittance with a short essay, everyone who attends is really passionate about testing. It's still sort of "underground" which keeps it small and very social.

Last year, I made some kind of attempt to live blog summaries of the GTAC talks but I never made it past part 1. We'll see how far I get this year, stay tuned.

The videos for 2008 aren't online yet but check youtube often because last year they were up in less than a day.

The Future of Testing was the first talk of the GTAC 2008 conference on Thursday Oct 23rd given by James A. Whittaker, a very entertaining speaker who works for Microsoft. His talk was excellent and I highly recommend keeping a lookout for the video. Here are my notes.

Magic of software

Decoding DNA, discovering planets in the galaxy that may be earth like

Showed a bug in a hotel wifi page with a query string containing fee=8.95. Hmmm, how about entering fee=0.0 ? Or fee=-100 !

Ironically, during the Blue Hat Conference where Microsoft invites hackers to break their software, a user sent an email to the distribution list using Outlook but then recalled the message. This triggered a bug where several exchange servers spiraled for hours trying to retract the message from the 1275 users who were on the distribution list.

Showed a video produced by Microsoft that showed a diabetic who was using some heart monitor while running; you see her doctor use a computer screen then send her a message, she enters her house and picks up an E-Ink tablet, starts a video conference with her doctor. There is another scene where a doctor remotely triggers a red circle to appear around a pill bottle on the table in a patient's home. The first woman enters the hospital, thumbs through a digital wallet (virtual credit cards), scans her thumb print.

James said a bunch of engineers were shown this video. One turned to him and said: "this shit's never gonna work."

Crowdsourcing (pioneered by http://www.utest.com/ ?), just toss it into the crowd, vendors provide testers

Said this is the future: vendors provide tests.

There could be a library of common test cases.

Some service could provide tests for all possible cross-site scripting exploits; just add a web app and run.

Testers would become designers not implementors.

Environments would have to be completely portable.

Must be able to reproduce a bug anywhere

Virtualization

The Vista problem

Has many tests, used in-house, used in beta program, used by early adopters, but as soon as it shipped, many new major bugs were discovered. Biggest problem was many environments were not tested (i.e. permutations of configuration).

Visualization

Told a story of his father while working at Ford. If a car was missing a bumper, everyone agreed it needed a bumper. Fixed. The difference in software is that you can't see it. What is missing? What does it look like?

Showed a very cool heat map of Vista components, in squares relative to significance in the system. The darker squares had more lines of code

When he started at Microsoft he used this to determine where to focus testing efforts. Components with more dense code need more tests.

Showed Xbox test tool, a separate window that shows a tester where he/she is in a game (hierarchical tree like view) and what possible scenarios could be entered next.

Game testers just want to fight monsters, need to help them actually test the game application

Visualization that showed what parts of the game were most frequently tested by manual game testers (pin points on a map)

Discovered that everyone liked the wishing well and no one wanted to go on the boats

The boats were boring! Game designers discovered they needed to work more on the boats. They also temporarily disabled the wishing well during testing.

Metrics

There exists a countably infinite number of things to test

Wants metrics for what's going on right now. I.E. See the javascript code that is currently being executed while hovering over a button. [Titus Brown actually hacked on an interesting experiment like this: a web interface to view runtime code coverage of the Sugar GUI framework for OLPC]

Education problem

When engineers are hired out of college they have to be re-trained on how to test code. This is not good. Testing must be taught in computer science programs.

Your name:

Email (not displayed):

Note: HTML tags will be stripped. Hit enter twice for a new paragraph.