Status of Shepherd integration / Test The Web
Forward

DS: heycam wanted to work on
that, not sure if there's any progress on it
... the focus for svg is for css transforms, we can do it with
the css testsuite at the moment
... and then transfer the tests later

ED: is there any info on how to
contribute on the TTWF site?

DS: there will be a presentation
on how to do that

ED: just making sure the
materials will be available online as the event takes place, to
enable people participating even though they're not physically
there

DS: I will publish them right
after doug's presention

TB: peter linss is looking at
integrating some of the changes I made for converting html/css
test to svg
... it's perhaps not generic enough, my code was for the
submitted adobe tests

ED: so there was a question about
whether a pass on a test (regardless of the format) should be a
pass for that feature or not

TB: i think you have to have
separate results, e.g for transforms in svg and for transforms
in html/css
... I don't think any of the browsers support the new
transforms things for svg

ED: anything else needed from us
in time for the event?

TB: would be good to have a
couple of approved tests in our repo

DS: don't think that's
necessary
... we can use the same process as the csswg uses for
review/approval
... can=need

TB: we used to require tests to
have a reviewer, are we giving up on that?

DS: no, css requires that
too
... you have a creator, a reviewer, and a third person to
approve it
... we could say reviewer and approver could be the same person
if we want

TB: how does the test become
approved?

DS: the shepherd tool moves the
approved tests to another directory

TB: right now we have nothing in
our approved directory
... shouldn't we have a few in there?

DS: I think so yes
... only a few people have committed tests so far
... I'll look at reviewing and approving some of the tests

BB: did we come to a conclusion
on the format for the reference images?

CL: we did discuss that
... if possible we agreed that if it's easy to do solid green
for pass for example then that's preferred, but there are cases
where that can give false positives and cases where it's very
difficult, like filters

BB: right... another suggestion
is to use one standard text string for tests with text

TB: I have objections to having
just green rects

BB: in gecko we use tens of
thousands of tests, it's easy to quickly see pass if the pass
images are always green

DS: if you see green it's passed,
if you see red it's failed, basic principle

DS: right, because then the
automated engine doesn't care what it's testing, just compares
the results
... it's up to the author to provide the information
... in the metadata, but it should be there
... I think it's quite clear what it's testing

TB: what do people think?

CL: valuable to run automated
tests, but when you get the list of failed tests it's useful if
a human can quickly tell whats wrong
... and then it's useful to know what those tests are testing
exactly
... this means we should have welldocumented testcases

DS: that's why we do review on
them

TB: maybe we should have them
separate? it's nice to have to some visual tests (like in SVG
1.1)

DS: we have reftests that can do
that, two images have to look the same, otherwise it's a
fail
... I agree that visual tests are good, and that if you see red
it's failed

DS: this next test fails in all
browsers
... anyway, we can also discuss further on the mailinglist

<ChrisL> the 'show reference'
link is broken btw

DS: and it means others can
follow the discussion

CL: i'd like to say another thing
about the template
... the template doesn't use a particular font
... which means every implentation may use a different
font
... suggest we standardize on a particular WOFF font

DS: but we don't need that with
reftests, because it ensures the font is the same on both
reference and testcase
... AHEM is often used in css tests

CL: ahem is not always useful
though

DS: I think it's wrong to require
a particular font

CL: why is consistency bad?

DS: but it doesn't matter,
because we have reftest

CL: what is the problem? we've
had unreadable text, and people assuming a particular default
font
... if you're testing svg it's not going to reflow text for
example

DS: if you add more dependencies
then that's an additional thing that can fail

CL: all browsers support this,
explain why the pass criteria would make it fail?

DS: but you add unnecessary
complexity

CL: so should we also take out
the metadata?

DS: that's different

ED: I think it's quite nice to
have consistent fonts used throughout the testsuite
... looking back at the SVG testsuite, yes svgfonts/webfonts is
additional complexity, but it's also nice to get consistent
rendering across platforms

DS: webfonts are a requirement or
just something we support in svg?

TB: inkscape doesn't support
webfonts

DS: the problem is that if
webfonts isn't a requirement for svg

CL: we have resolved that
webfonts, and woff, are a requirement for svg

ED: so, can we agree on having a
consistent font used when possible?

DS: don't want to require
webfonts for the tests

TB: dirks tests are simple, don't
require labeling, but svg1.1 tests are more visual
... with labels and so on

nikos: it's a risk if the layout
obscures the text
... you don't necessaryly know the output you're going to
get

CL: right, you might get
unexpected results

TB: but the risk is pretty small,
but I prefer the svg11 tests though
... all it would take is to put in a style section in the
template to use a webfont as the default font

<ChrisL> yes it would just
take one @font-face rule plus a font family and size on the
main group

TB: inkscape would ignore it

(discussion on inkscape and testing with
references)

<ChrisL> so you would no
longer need to make speccial inkscape test versions with all
text elements removed

DS: i'm not strongly opposed to
adding WOFF fonts, I just think that we should reduce the tests
as much as possible

CL: that's why I pushed hard for
pass criteria, because it doesn't matter if the WOFF is
supported or not if the only thing that needs to be done is to
render a rect for example
... unless the pass criteria says you have to look exactly like
a given font

DS: for automated tests it's
still additional complexity, requirements for passing the
test

CL: but if we have flags, then we
should just not add the flag "need webfont" if it's not
necessary

DS: how do you style the
elements?

CL: that's why it should be in
the template

DS: but we should only have that
if it's needed for passing the test
... don't think we can agree on having this right now

CL: I don't accept your arguments
that unflagged tests would fail

TB: don't care so much either
way

nikos: no strong opinion from me,
but it should be in the template i think so that it's no risk
to be missed

ED: i'd be fine with having two
templates, one for tests that use text in the visual output,
and one that doesn't (where the webfont isn't needed)

CL: that would be ok with me
too
... would that be fine with you DS?

DS: yes

TB: ok, so two templates, one for
automated tests (without the webfont), one for visual tests
(that have the webfont)
... I'll make those templates, what font do we want to use?

CL: freesans would be good

DS: do we also want to have other
fonts, serif, bold etc?

CL: probably, but not in the
template maybe, but it's good to have a library of fonts that
can be used