The evaluation process in short

Each paper will be evaluated by three independent experts who do not have any conflict of interest with the authors or their institutions. The reviewers are going to indicate their familiarity with the paper’s subject, evaluated the paper along four evaluation criteria and provide comments for the authors. After the individual review a consensus score and ranking is done by the program chairs. They also suggest an acceptance threshold based on the overall consensus score. They carefully examine all aspects of the papers at the borderline, and consider the reviewers comments of high ranked papers. Finally they produce a list of accepted papers.

Reviewer’s familiarity with the paper's subject

The familiarity is scored as:

excellent: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature.

very good: The reviewer is confident but not absolutely certain that the evaluation is correct. It is unlikely but conceivable that the reviewer did not understand certain parts of the paper, or that the reviewer was unfamiliar with a piece of relevant literature.

good: The reviewer is fairly confident that the evaluation is correct. It is possible that the reviewer did not understand certain parts of the paper, or that the reviewer was unfamiliar with a piece of relevant literature. Mathematics and other details were not carefully checked.

fair: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper.

poor: The reviewer's evaluation is an educated guess. Either the paper is not in the reviewer's area, or it was extremely difficult to understand.

Evaluation Criteria

A quality paper is defined as a paper with high scores along the following criteria. The criteria reflect independent aspects of the paper's quality are a hence are also scored independently. Each criterion has an acceptance threshold and extended description of the issues to be evaluated. All marks indicated by green color are above the threshold.

The interpretattion of the scores are:

excellent: This paper is of outstanding quality and in the top 10% of accepted papers

very good: The paper is of very good quality and in the top 25% of accepted papers

good: The paper is of average quality and in top 50% of accepted papers

fair: The paper is of fair quality but below acceptable threshold

poor: The paper is of poor quality and should definitely be rejected

Relevance to conference call and to which degree the paper is a timely contribution

Score:

Excellent, very good, good, fair, poor

Interpretation:

Is the paper within the scope of the workshop. Are the results important and timely?

Scientific/technical originality and potential impact

Score:

Excellent, very good, good, fair, poor

Interpretation:

Are the problems or approaches new? Where possible, reviewers should identify submissions that are very similar (or identical) to versions that have been previously published. Is this a novel combination of familiar techniques? Is it clear how this work differs from previous contributions? Are other people (practitioners, researchers or commercial sector) likely to use these ideas or build on them? Are the results likely to have an impact on the research community or commercial sector?

Scientific/technical content and advances beyond the state-of-the-art

Score:

Excellent, very good, good, fair, poor

Interpretation:

Is the paper technically sound? Is related work adequately referenced? Are claims well-supported by theoretical analysis or experimental results? Is this a complete piece of work, or merely a position paper? Are the authors careful (and honest) about evaluating both the strengths and weaknesses of the work. Does the paper address a difficult problem in a better way than previous research? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions on existing data, or a unique theoretical or pragmatic approach?

Quality and clarity of the presentation

Score:

Excellent, very good, good, fair, poor

Interpretation:

Is the paper technically sound? Are claims well-supported by theoretical analysis or experimental results? Is this a complete piece of work, or merely a position paper? Are the authors careful (and honest) about evaluating both the strengths and weaknesses of the work.

Is the paper clearly written? Is it well-organized? Does it adequately inform the reader? (A superbly written paper provides enough information for the expert reader to reproduce its results.)

Comments for the Authors

Interpretation:

Provide an overall summary and detailed comments related to every evaluation criterium. If appropriate, make suggestions to improve the work . Please mak sure that high marks are reflected by
positive comments, and low marks by negative comments. Avoid offending comments and anything which could reveal your identity.