Paper Review Guidelines

General Guidelines

Here are some recommendations for writing reviews of submitted papers that help the authors and improve the quality of the symposium.

Your job as a reviewer is to write detailed reviews, even for excellent papers. In addition to telling the authors what you didn’t like about their paper, be sure to include what you did like as well.

Even if your opinion is that the paper is poorly written or poorly thought-out, you can still provide constructive criticisms to help the authors, and in the long run, the conference. Think of your goal as convincing the authors of the paper you’re reviewing to submit something else next year, but of such high quality that it will be well-reviewed and easily accepted. Give the authors advice on how to do that.

The best reviews clearly justify the reviewer’s choice of rating. The least valuable review gives a low score with no written comments. That simply tells the authors that they have been unsuccessful, with no indication of how or why.

The focus of your review should be on content.

Papers that you review are supposed to be anonymous. Your review should be based on the merits of the paper, not the reputation of the authors or their institutions. Therefore, we have asked the authors to remove all identifiable references to themselves. We realize that reviewers sometimes know the work and can guess who the authors of the papers might be.

If you recognize the work, it is your responsibility to give a fair and unbiased review, using only the information in the paper. If you do not feel that you can give a fair, unbiased review of the paper and not the authors or institutions, please contact the program chairs immediately.

Your review should not include comments to the authors about the anonymization (or lack thereof) in the paper. If you feel that it is necessary to comment on this, please use the text box, ‘Confidential comments to the committee.’

We realize that occasionally anonymization requires the authors to remove information that affects your review (information that otherwise the paper appears to lack). As a reviewer, you can give the authors the benefit of the doubt. Use the ‘Confidential comments to the committee’ box to indicate this to us. Example “This paper should reference Matt Jadud’s work, unless those references were removed for anonymity”).

Please point out typographic and grammatical errors; if there are too many to list, please state so in your review.

Although SIGCSE requires all submitted papers to be polished work, all accepted authors get a brief opportunity to improve the presentation of their paper before camera-ready copy is due. Your detailed feedback may help improve a paper, and in a small way, improve the conference.

Substandard Recommendations

SIGCSE uses a meta-review process after reviews have been submitted. Reviews that do not objectively, accurately, and clearly assess a paper’s suitability for publication at SIGCSE, founded in the reviewer’s disciplinary expertise and on the basis of the written paper’s originality, technical soundness, contribution to CS education, and clarity of presentation, may be deleted.

For example, an unacceptable review might:

be incoherent, unreadable, or irrelevant to the paper;

focus on the paper’s topic area or presumed authors at the expense of assessing the paper itself; or

provide no justification for its numeric ratings. (Even in “obvious” cases, reviewers should briefly justify ratings.)

Please note that a difference in rating or opinion with other reviewers or PC members will NEVER be cause for deletion of a review.

Examples of Good Reviews

To help reviewers better understand the qualities of good, useful reviews, here are several example comments, organized by review category:

Summary of Submission

Please summarize the submission in 2-4 sentences in your own words. Please DO NOT copy/paste the abstract into this section.

Strengths of this Submission

This paper makes a very good argument in the introduction for why this course is needed. It is timely, and addresses a topic outside of the norm often seen at SIGCSE.

I can’t recall ever seeing something similar at SIGCSE. In spite of the previous problems, I would urge acceptance of this paper on a topic that we rarely see at SIGCSE.

This paper should generate a lot of discussion and have a good audience. It is a topic that many schools are trying to address (including mine.)

Good level of detail on your approach. Table 2 is very handy. Under Section 2, it seems like log analysis and auditing may fit in your column two. How will you ensure additional security emphasis is implemented?

The organization is faultless. It is very clear what the paper is going to say and how. The paper follows through with crystal clear subject headings and a logical flow of information.

Comments for Authors / Areas for Improvement

The hypotheses are too obvious and the validation of them is not enough. Therefore, the contribution of this paper is quite limited.

Hard to judge given the writing organization problems, but I do not see a lot of significance here. The verification that the laboratory helped more than on-line component alone is a nice result, if it is supported by the data. Having taught this course already and collected feedback on your approach makes the paper stronger.

It is important for those who might be considering this approach to know that it can be successful. If I were considering this approach I would want to know if the students could understand the code, and how deeply I could get into the material given time constraints.

I would have liked to see some discussion and references setting this work in the context of other studies of student learning and knowledge retention. While I don’t know of other studies that have examined exactly the phenomenon this paper does, a short search in the ACM digital library turned up these examples that are relevant…

The paper could use additional proofing and polishing. I suggest finding a non-robotics person to read for both language and communication. Some sentences are poorly formed (e.g., sent. 1 of last par. in sec. 1). Some content seems misplaced (e.g., discussion of mobility in section 3).

Paper Track Guidelines

There are three tracks for papers. Reviewers and APCs will be assigned to review papers in one of the three tracks. Please ensure that the Program Chairs know your preferences for the track(s) where you can provide the most expertise and best feedback.

Authors must choose the track that they feel best fits their submission. Review the submission using the guidelines for the track the submission is in; not the track you would prefer it to be in. The Program Chairs will not move papers between tracks.

CS Education Research Paper Track

CS Education Research papers should adhere to rigorous standards, describing hypotheses, methods, and results as is typical for research studies. These normally focus on topics relevant to computing education with emphasis on educational goals and knowledge units/topics relevant to computing education with statistical rigor; methods or techniques in computing education; evaluation of pedagogical approaches; and studies of the many different populations that are engaged in computing education, including (but not limited to) students, instructors, and issues of gender, diversity, and underrepresentation. We welcome replication papers and papers that present null or negative results that meet the criteria below.

For a typical paper in this track, here are some key factors to include (as an author) and to look for (as a reviewer):

Are there one or more clearly stated research questions? Since the rest of the paper will be organized around these, it’s often good to put them in the abstract and in the first section of the paper.

Are the questions of interest to the SIGCSE audience?

Related work in computing education

Is the relevant work in computing education included? If not, a good review must give references to missing material. Simply saying “The related work section is incomplete” is not enough.

Do the authors clearly describe the relationship between the previous work and the current research questions? In what ways does the current project build on the previous work, and how is it different?

Related work in educational theory

Is the project based in educational theory? If not, should it be and what are some theories the authors should consider?

Is the theory described clearly, with appropriate citations?

Is the theory’s relationship to the current project clearly described?

Is the data gathering sufficiently clearly described so that the reader could reproduce it? Some key information to include:

About the data: why this particular type of data is relevant to your research questions

About the participants: how many, what was their background (are they instructors, students, alumni, etc.); what if any formal coursework have they had in computing; how many were men and how many women; and any other factors that are relevant to the author’s project

About the person(s) gathering the data: What is their relationship to the participants? For example, if the data were collected from students in a class, was the instructor one of the researchers or not?

About the data gathering process: did the project use surveys, interviews, samples of student work, other; If surveys or interviews, exactly what questions were asked.

Six pages may not be sufficient to provide all the necessary details. Authors may link to supplemental materials that should be blinded for review.

Is the data analysis process/methodology sufficiently described so that the reader could reproduce it?

What methodology was used?

Is the methodology described, with an appropriate citation?

Is the implementation of the methodology clearly enough described? How many people were involved? What process was used to resolve any disagreements?

Is the analysis process/methodology appropriate for answering the research questions?

Is the analysis methodology something new to computing education research that might be a contribution itself?

Are the results of the analysis clearly summarized?

Are the results thoroughly discussed, including:

Their relationship to the research questions

Their relationship to previous work

The implications of the results for teaching

The implications of the results for future research

Are threats to validity discussed?

Experience Reports and Tools Paper Track

Experience Reports and Tools papers should carefully describe a computer science education intervention and its context, and provide a rich reflection on what worked, what didn’t, and why. This track accepts experience reports, teaching techniques, and pedagogical tools. All papers in this track should provide enough detail so that others could adopt the new innovation.

For a typical paper in this track, here are some key factors to include (as an author) and to look for (as a reviewer):

Are there one or more clearly stated goals in this paper? Since the rest of the paper will be organized around these, it’s often good to put them in the abstract and in the first section of the paper.

Is the experience or tool of interest to the SIGCSE audience?

Related work in computing education

Is the relevant work in computing education included? If not, a good review must give references to missing material. Simply saying “The related work section is incomplete” is not enough.

Do the authors clearly describe the relationship between the previous work and the current research questions? In what ways does the current project build on the previous work, and how is it different?

Are the observations and/or findings from the experience or the use of a tool clearly summarized?

Are the findings thoroughly discussed, including:

Their relationship to previous work

The implications of the results for future use

The implications of the results for teaching

Information on how to adopt or adapt teaching techniques and/or pedagogical tools in other contexts or institutions.

Curricula Initiatives Paper Track

Curricula initiatives should describe new curricula, programs, and degrees, the motivating context before the new initiative was undertaken, what it took to put the initiative into place, what the impact has been, and suggestions for others wishing to adopt it. This track may also include position papers, which are meant to engender fruitful academic discussion by presenting a defensible opinion about a CS education topic, substantiated with evidence.

Is the innovation clearly stated? Since the rest of the paper will be organized around this, it’s often good to put it in the abstract and in the first section of the paper.

Description of the problem or need being addressed.

Is the curricular innovation or position paper of interest to the SIGCSE audience?

Related work in computing education

What prior solutions to this problem exist?

Is the relevant work in computing education included? If not, a good review must give references to missing material. Simply saying “The related work section is incomplete” is not enough?

Do the authors clearly describe the relationship between the previous work and the current research questions? In what ways does the current project build on the previous work, and how is it different?

Examination for discussion

How is the curricular innovation or position paper addressing this problem or need?

How is the curricular innovation or position paper different from previous ideas?

Future Success Indicators

How could the curricular innovation or proposed idea be assessed if adopted or implemented?

In what context can the curricular innovation or proposed idea be used (large research institutions, community colleges, high schools)?

How difficult would the curricular innovation or proposed idea be to adopt it? For example, the human and financial resources needed.