#8106 - a testing case study

http://dev.laptop.org/ticket/8106 makes an interesting, relatively
easy-to-follow (even for nontechnical folks) case study on how testing
works right now, since improving QA processes for the next release cycle
has been a topic of discussion lately.
Note that the snags we hit were minor and easily fixable, but took a
while to resolve - how can we fix this for next round?
Here goes. Mel's Ideal scenario (vs what happened):
1. There'd be a test case attached to the ticket saying "Do this! If it
passes, you've proved the feature works."
(Reality: No test case. No problem - I wrote one up. Getting good
testcases to happen is something the QA team should do in any case.)
2. But wait. To write the test case, we needed to find an out of date
content bundle that we could upgrade. Ideally, there would be a list of
available content bundles, you could find different versions of each
content bundle, and install an old one.
(Reality: I hunted around in vain; SJ, bless him, finally stepped in and
gave me direct URLs to 2 bundle files, simultaneously noting that there
was not, in fact, a good way to find that kind of download/history
information yet.)
3. Ok, great - we've installed the old version of the library bundle, we
run the software updater, and we see a clear difference in behav
(Reality: the fault was mine for making a test case that didn't include
the last verification steps for "is this bug fixed?" I'd expected some
sort of visible indication on the screen saying "your content bundles
are updating." There wasn't a message like that, but after a few minutes
of confusion, I realized I should check the library.info file manually.
This indicated that the bundle had indeed been upgraded. I filed
http://dev.laptop.org/ticket/8805 suggesting that we make this visible
to users next time.)
4. Close bug.
(Reality: Close bug. Yay!)
--
Mel Chua
QA/Support Engineer
mel at laptop.org