Evaluation
of
software
can
take
many
forms
ranging
from
algorithm
correctness
and
performance
to
evaluations
that
focus
on
the
value
to
the
end
user.
This
paper
presents
a
discussion
of
the
development
of
an
evaluation
methodology
for
visual
analytics
environments.
The
Visual
Analytics
Science
and
Technology
(VAST)
Challenge
was
created
as
a
community
evaluation
resource,
that
is,
a
resource
available
to
researchers
and
developers
of
visual
analytics
environments
that
would
allow
them
to
test
out
their
designs
and
visualizations
and
compare
the
results
with
the
solution
and
the
entries
prepared
by
others.
Sharing
results
allows
the
community
to
learn
from
each
other
and
to
hopefully
advance
more
quickly.
In
this
paper
we
discuss
the
original
challenge
and
its
evolution
during
the
seven
years
since
its
inception.
While
the
VAST
Challenge
is
the
focus
of
the
paper,
there
are
lessons
for
many
involved
in
setting
up
a
community
evaluation
program,
including
the
need
to
understand
the
purpose
of
the
evaluation,
decide
upon
the
right
metrics
to
use
and
the
appropriate
implementation
of
those
metrics
including
datasets
and
evaluators.
For
ongoing
evaluations
it
is
also
necessary
to
track
the
evolution
and
to
ensure
that
the
evaluation
methodologies
are
keeping
pace
with
the
science
being
evaluated.
The
discussions
on
the
VAST
Challenge
on
these
topics
should
be
pertinent
to
many
interested
in
community
evaluations.