Reviewing Guidelines

Reviews have a direct and important impact on the quality of a
conference. They also help the community as a whole to improve the quality
of its research. The aim of this guide is to ensure that the review process
is fair and leads to the best possible selection of papers. All reviewers
should make sure that they follow the basic principles outlined in this text.

Be Timely

The EuroVis reviewing process has a very tight schedule, and an important
part of making the commitment to review is that you have agreed to carry
out the work before the stated deadlines. All reviewers need to finish
reviews on time, and program committee members need to bid and declare
conflicts on time.

Protect Ideas

As a reviewer for EuroVis, you have the responsibility to protect the
confidentiality of the ideas represented in the papers you review.
Submissions are by their very nature not published documents. The work is
considered new or proprietary by the authors; otherwise they would not
have submitted it. Of course, their intent is to ultimately publish to
the world, but most of the submitted papers will not appear in the
proceedings of the conference. Thus, it is likely that the paper you have
in your hands will be refined further and submitted to some other journal
or conference, or even to the same conference next year. Sometimes the
work is still considered confidential by the author's employers. These
organizations do not consider sending a paper for review to constitute
a public disclosure. Protection of the ideas in the papers you receive
means:

Do not show the paper to anyone else, including colleagues or students,
unless you have asked them to write a review, or to help with your review.

Do not show movies or other supplemental material to non-reviewers.

Do not use ideas from papers you review to develop new ones.

After the review process, destroy all copies of papers and supplemental
material that are not returned to the senior reviewer and erase any
implementations you have written to evaluate the ideas in the papers, as
well as any results of those implementations.

Avoid Conflict of Interests

As a reviewer of a EuroVis paper, you have a certain power over the
reviewing process. It is important for you to avoid any conflict of
interest. Even though you would, of course, act impartially on any paper,
there should be absolutely no question about the impartiality of review.
Thus, if you are assigned a paper where your review would create a
possible conflict of interest, you should return the paper and not submit
a review. Conflicts of interest include (but are not limited to) situations
in which:

You are a co-author of the work.

You have a strong affiliation with the same institution as one of the
authors. This includes (but is not limited to) your current employment
as a professor, adjunct professor, visiting professor, or similar position,
in the role of a consulting or advisory arrangement, previous employment
with the institution within the last 12 months, being considered for
employment at the institution, any role as an officer, governing board
membership, or relevant committee, or the current enrollment as a student.

You have been directly involved in the work and will be receiving
credit in some way. If you're a member of the author's thesis committee,
and the paper is about his or her thesis work, then you are involved.

You suspect that others might see a conflict of interest in your involvement.
For example, even though Microsoft Research in Seattle and Beijing are in
some ways more distant than Berkeley and MIT, there is likely to be a
perception that they are "both Microsoft" and folks from one should not
review papers from the other.

You have collaborated with one of the authors in the past three years
(more or less). Collaboration is usually defined as having written a paper,
book or grant proposal together, although you should use your judgment.

You were the MS/PhD advisor of one of the authors or the MS/PhD advisee
of one of the authors. Funding agencies typically consider advisees to
represent a lifetime conflict of interest.

You are related to one of the co-authors.
This includes (but is not limited to) being the spouse, a child, sibling,
or parent, as well as any affiliation or relationship of your spouse, of
your minor child, of a relative living in your immediate household or of
anyone who is legally your partner that you are aware of.

Other relationship, such as close personal friendship or significant
animosity between you, that you think might tend to affect your judgment
or be seen as doing so by a reasonable person familiar with the relationship.

The blind reviewing process for some of our conferences will help
hide the authorship of many papers, and senior reviewers will try
hard to avoid conflicts. But if you recognize the work or the author
and feel it could present a conflict of interest, send the paper back
to the senior reviewer as soon as possible so he or she can find someone
else to review it. If you are in doubt about any conflict, you should
discuss it with the editor/paper chairs or the person that assigned the
paper to you. You should never contact the authors directly.

Be Specific

The publishing of scholarly work is essential for the academic community
specifically and the research community in general. Therefore careers
and reputations hinge on publishing in the proceedings, academic tenure
decisions are based on the proceedings, and patent infringement cases
discuss whether something is considered novel enough to publish in the
proceedings.

Therefore, it is our duty to do a careful, objective and scholarly review.
The emphasis of our reviews should be to help the authors on how to
improve their work and therefore to improve the overall quality of the
work in our research community. In general, it will not be helpful to
anyone - neither the program chair, nor the authors:

to do a quick or superficial review, to say that the work is good
or bad without giving clear reasons

to state that the work has been done before without giving clear
citations of previous work

to complain about the structure of the paper without making
suggestions on how to improve it

to dismiss the evaluation method without being specific about the
flaws

A casual or flippant review of a paper that the author has seriously
submitted is not appropriate and may be rescinded from the reviewing
process. In the long run, casual reviewing is a most damaging attack
on our conference. There is no dishonor in being too busy to do a good
review, or to realize that you have over-committed yourself and cannot
review all the papers you agreed to review. But it is a big mistake to
take on too much, and then not back out early enough to allow recovery.
If you cannot do a decent job, give the paper back and say so. But
please, do it early so that the senior reviewer has time to select
another reviewer before the deadline.

Be Helpful and Constructive

Have an open mind, or at least reveal your biases.
If you are a die-hard algorithm-driven researcher, and you are assigned a
user study paper, and the call for papers welcomes both types of papers,
don't bash the paper simply for its methodology. It's not fair and it
benefits no one. Either assess the paper according to the appropriate
user-study standards, or admit that you are not capable of doing so.
At most, you might discuss why the algorithmic approach does not provide
a suitable answer to the research question. But don't force the author
to ask a different question that can be answered using algorithmic
methods. Many of us have been frustrated by reviews in which the judge
basically told the researcher to do a different study. Work within
the author's goals.

Look for the good in a paper.

No matter how much you dislike the paper, try to find the good aspects.
Perhaps there is a different approach to a problem, a novel application,
a promising evaluation methodology, or even a potential twist which
could be studied further during the revision of a paper. Be helpful to
the authors and point out the potential positive aspects. Let the
authors know if you think their work would be better suited for a
different conference, journal or venue. Be encouraging.

Be Tactful

Belittling or sarcastic comments may help display one's wit, but they
are unnecessary in the reviewing process. The most valuable comments
in a review are those that help the authors understand the shortcomings
of their work and how they might improve it. If you intensely dislike a
paper, give it a low score. That makes a sufficient statement.
While we will assure the anonymity of the reviewing process, read over
your review and ask yourself whether you would be able to recite your
critique in front of the authors? If the answer is 'no' you might want
to work on some of the wording of the criticism. A scathing review is
usually not only unhelpful to the authors and paper chairs, but it
also tends to alienate typically new researchers that would like to
enter into our community. Put out a helping hand.

In Summary

Adherence to ethics sometimes requires a more careful analysis of the
work as well as a well-thought out response. However, while it appears
that we might be losing some time, this will be made up by improving
the culture in our community overall, by helping each other to put out
better results and overall increasing the quality of our conferences and
journals. That is what we are striving for in our visualization community
and it is well worth the effort.
Adapted from the IEEE Visualization Ethics Guidelines