WGTQA

To begin with individual tests were written by hand in a WGTQA git
repository. My employer
Aplix supported the work (~1 working day a week) as
it would be useful to partake in compliance efforts for both
BONDI Widget runtime efforts and our own commercial
widget runtime offerings.

I made a couple of mistakes. They were mostly interactive tests and didn't make
use of Javascript to automate them in several cases. For example:

Firstly I tried to pin the name of the test on the section of the
specification it was testing.
Unfortunately the numbering changed, so 5.7 had worse than no meaning. It
became confusing and a huge pain to track to keep updated.

index.html just has a simple PASSED rendering. Great for interactive
tests, though useless for automation. The "start file" should have
been a simple Javascript call to indicate the test case passed.

Testing Error conditions

The next mistake was assuming a malformed config would halt processing. In
reality a Widget runtime will ignore bogus values. I wrote several negative
tests for
config.xml,
which wrongly assumed that if the index.html was seen, it would be a FAIL.
Assuming the UA would halt and show the widget has been rejected instead, as a
PASS.

A better test would be check the default values were set when a bad one was
given. Unfortunately Widget APIs are
required to read back and test such values. This API is not available on any
current widget runtime like BONDI or Opera AFAIK.

So you can't really do negative tests, instead you need to check default values
instead.

The "Widget Testing Framework" (WTF) is a PHP Zend
powered contribution by Opera, which basically deals
with the earlier automation problem. WTF comes without any substantial
tests, though Opera has stated they will share some of their existing Widget tests.

Robert Græsdal from Opera is working on WTF also just one day a week.

For example, lets look at a test which tests the order in which index files
are run. So the start files are
modified from the index template from data/std_includes/index.html which
references testpass.js:testcase.result(true); on the first case and
testfail.js:testcase.result(false); on all the others:

index.htm -- PASS

index.html -- FAIL

index.svg -- FAIL

index.xhtml -- FAIL

index.xht -- FAIL

When the widget is downloaded a
testlib.js & style.css is also included in the widget. testpass.js executes
the function implemented by testlib.js which reports the test result with the
magic of AJAX to a configurable reporturi. Unfortunately the backend
service for this isn't provided yet by Opera, though it shouldn't be hard to
recreate.

So the framework builds the widget on the fly. Sooner or later, you'll realise
this is good testing framework for mainly checking to see if the Javascript
APIs e.g. BONDI &
W3C for different
features works correctly within a widget.

Though WTF is not good for testing things like broken
zips and
odd filenames and screwy encodings and such things, before a widget is
actually running. So WTF is not a "silver bullet", a solution for all problems
related to ensuring W3C Widget runtime conformance.

Non-interactive interactive tests

In terms of processing a test, there are 3 types:

Automated

Visual

Interactive

Automated tests are best. You could run all you automated tests via a test
harness (which you need to write) after every change to your Widget runtime for
example. Hence this would become also known as a regression test suite.

Visual tests are generally written as interactive tests at
first. The tester is basically
looking out for some green blob and then to mark it off with a button control.

You do not want expensive, labour intensive and mind numbing interactive tests.
Imagine for every test run after a source check in, having to mark whether
you saw a green blob or not?

Opera has an übercool internal system called SPARTAN which hashes these values
(those green blobs), so that these interactive (bad) tests can become like
automated (good) visual tests. Since SPARTAN isn't public, 'visual tests' will
remain 'interactive tests' for the time being.

Therefore it's a good idea to concentrate on writing automated tests for the
time being.

How a widget is built up by WTF

The widget.txt is not just a description of the test. It also allows some easy feature switches in an INI format.

For example setting fileio=1 should enable opera.io by building an
appropriate config.xml to enable that feature.

The idea behind that, is that it's easier for tests to be maintained that way,
than relying on the feature
element.