Notes on Chairing Program Committees
April 2013

I have served as program committee chair for CGO 2013, ISMM 2012, PLDI
2007, PACT 2005, and ASPLOS 2004, and served on over 50 conference
program committees, and served as the Editor of TOPLAS. This document
contains reflections on my experiences and advice for program chairs.

Conferences and program committee (PC) meetings serve critical
social and research functions in the computer science
community. Ensuring a fair process that encourages, guides, and
rewards high quality and forward looking research is the
responsibility of the reviewing process.

The reviewing process and PC meetings serve to develop
and express the shared values of the research community. The PC chair
is the most critical person in this process and should seek to
ensure a fair and high quality process that encourages and builds
the community.

Poor review quality and poor process is very damaging to our
community. It alienates all community members and is especially
damaging because it disproportionately discourages new community
members (i.e., graduate students and young researchers), since they
have no or many fewer good previous experiences to counter a bad
experience. PC members and chairs should take their job very
seriously.

PC organization

To improve review expertise, reviewer accountability, handle PC
submissions, and reduce PC member workload, I recommend a moderately
sized Program Committee (PC) with an in person PC meeting and a named
External Review Committee (XRC), as pioneered in the SIGPLAN and
SIGARCH communities by Steve Blackburn at ISMM 2008. ASPLOS, ISCA,
ISMM, PLDI, POPL, and other conferences now regularly use this
structure.

Another key to high quality reviews and moderating the workload
(which are related!), is two round reviewing, in which each paper gets
two PC reviews and one XRC review in a first round. With this
information in hand, the committee rejects at least half the
submissions. The second round adds one or two reviews, such that every
paper that will be considered further will have at least 3 PC
reviews.

Your committee should be large and diverse enough to provide
expert reviewers on a paper with one conflict in the topic area.
Large committees decrease the work load, but sometimes increase the
probability one or more members wont do their work or do it on
time. Make sure everyone agrees to attend the PC meeting. PC
attendance and applying no limit on paper acceptance are key
to a diverse and forward looking program.

Selecting PC members

As the program chair, you want to create a committee that covers
all the technical areas for the conference and that write useful
reviews in a timely fashion (timely is key to reducing your stress).
To get high quality reviews, you need responsible and knowledgeable
reviewers. Your personal experience and those of prior PC chairs are
a great source of this information. Ask them who does their work? Who wrote
the best reviews? Who was productive in the PC meeting, e.g., made
thoughtful comments and moved papers to their conclusion in a
principled fashion?

Secondary, but very important should be diversity of institution,
rank, and experience. I always use a spreadsheet that includes any
previous service on this conference's PC, gender, institution,
career stage (early, middle, late), and topic coverage. The
conference should be tracking all PC members in the past in a
spreadsheet. To create the topic list and a source of program
committee members beyond my personal network, I use and recommend
extracting topics and researchers from the past 5
years of the conference. I also look at related conferences for
emerging topics and researchers.

Asking early or late? For ASPLOS, I asked very early (more than 1
year out from the work), but then several people had personal
conflicts. For PACT, ISMM and CGO, we asked late (4 months out),
which made it more difficult to obtain senior members but yielded a
committees without turnover, but still the occasional emergency
prevents PC attendance.

Software

I recommend a change to the current system, where everyone turns
over every year and thus the PC chair is left to choose the software
submission and reviewing system, and then populate the entire
process. I recommend a three year term for a web submission
czar for each conference to configure and manage the submission
software: one year in training, one year running it by herself or
himself, and one year training the next person. Besides the obvious
benefit that the PC chair has someone to help them who has a lot of
experience with the software, the key benefit is that if the PC wants
to change process, topics, conflicts, review form, etc., they still
can, but at least the change is intentional instead of random.

Paper assignation

Conflicts. I prefer that the authors specify their PC conflicts and
institution conflicts with a check box, and others in a list which
greatly eases paper assignation. However, this mechanism gives authors
the ability to veto reviewers, without conflicts, and several PC
chairs have discovered this problem. Reversing the process to avoid
this problem reveals
authors to PC members, slightly unblinding the process. Reminding
authors to only submit conflicts based on your policy should be sufficient.

Matching. I recommend bidding for the papers by the PC
and XRC, and then doing paper assignments with this information and
topic expertise in hand. I used this system for all the PCs I chaired
except ASPLOS. For CGO, Lieven Eeckhout and I together spent about 10
to 15 hours refining the automated system to make sure each paper had
good expertise. For ASPLOS, I made all the paper assignments by
matching paper topic areas to PC member topic areas. I found that I
did not do a great job (with respect to PC interest and expertise without their
input) and for 169 papers and 18 PC members, and highly recommend
bidding, which is now broadly used.

I used two round reviewing at PLDI, ASPLOS, and CGO. I first used
two round reviewing at ASPLOS by necessity rather than design due to
an unexpected quantity of submissions. In the first round, I
assigned each paper two committee member reviewers (about 20-25 hours
of work for me), and each of them specified one or more reviewers from
the PC. For
the 112 papers with a significant number of positive reviews or
conflicting reviews, I then assigned an additional committee member
reviewer (another 10 hours of work) and in a few cases, external
reviews. Thus, each paper had at least 4 reviews and all papers
discussed at the committee meeting had 5 reviews, with 3 from
committee members. Most of these reviews came in before the rebuttal
period. In addition, each PC member had a higher density of papers in
the top group. I recommend this two tier strategy for better focus
with a large number of submissions, but it is more work for the PC
chair.

For all the other PCs, I instead gave the PC members a list of all
their non-conflict papers, and let them indicate their preferences.
With this information, I achieved a 100% match of submissions to one
or more PC and XRC members that wanted to review the submission, and it took
less time. As a result, I
believe we had better and more qualified reviewing than we would have
otherwise.

Blind Submission, Rebuttal, & Timely Reviewing

I highly recommend blind submission to reduce bias, and the
rebuttal process to encourage constructive and accurate reviewing.

I have seen the rebuttal process change several, but not numerous
numbers of reject and accept decisions. These decisions are however
important and it has the following additional benefits. Reviewers
submit their reviews before the rebuttal period, instead of doing them
on the plane on the way to the PC meeting. Reviewers are more likely
to do a careful job for that reason, and because the authors can
correct them and their peer reviewers are watching. Furthermore,
rebuttal helps build community, when reviewers and authors communicate
through rebuttal, especially when the reviewers add a final statement
about the accept/reject result in light of the committee discussions
and rebuttal comments.

Running the Meeting

I have a few points of possible interest about the purpose of a PC
meeting and how to run the meeting itself smoothly.

PC meetings help build research community. I
recommend holding a PC dinner the night before the meeting. Part of
the PC experience is building community and understanding and
developing shared research evaluation values. Hosting a dinner the
night before helps build a more collegial atmosphere, shared
values, and trust when it comes to discussing the relative strengths
and weaknesses of particular papers, research directions, and
evaluation methodologies.

You need help. At the meeting, I highly recommend
that you ask a graduate student, colleague, or assistant to help
you. This person can keep records, enter decisions in the software
and/or on the board, track conflicts, fetch people who leave the room
for conflicts, and handle unforeseen requests. This assistance will
enable you to move quickly from one paper to the next, instead of
pausing one or more minutes between papers.

Schedule a long day, but build in breaks to refresh
everyone. Stick to your break schedule. In 8 hours of discussion
and 1.5 hours of breaks, it is possible to discuss no more than 50
papers. Adjust either the papers or the meeting length accordingly.

Select the PC member with the highest score on the paper
to lead the discussion. This advocate system is now widely used.
The point of the meeting is to accept papers, and this selection means
each discussion starts with the most positive review. The leader
should also be tasked with representing all reviews of reviewers not
in the room.

Keeping the PC on task. Post the next ten papers and
their PC lead for discussion to help the PC members refresh their
memories. Post the lists of accepted,
rejected, and tabled papers as you go (widely used).

Watch the clock. I budgetted 10 minutes per paper,
and limitted many discussions to five
minutes. Online discussion before the meeting can reduce the time for
each paepr at the meeting
(widely used).

Ordering. I have used a variety of orderings, but I
think random within each groups of 10 to 15 papers works best
because the PC cannot just start rejecting papers and never stop!

I used the following group ordering and random within group
in most of my meetings: top groups (all Accept, two accepts, one
accept) until bogged down, bottom up (reject 10 at time until the PC
saved more than one paper in the group), PC papers, back to groups
based on one top score, and in the end, when you are running out of
time, only the advocate system (widely used).

Closing. When there was only about two hours left, I
moved to a system in which a PC member must volunteer to advocate for
the paper in order for it to be considered further. We quickly listed
one-by-one all the remaining "middle" papers, and asked for a PC
advocate volunteer. The remaining discussion was limited to these
papers led by the advocate (used successfully at CGO 2013, ASPLOS 2004, ISCA
2005, and PACT 2004).

Etiquette and Ethics. I suggest the following meeting and
post-meeting etiquette. Make all conflicts (advisor, advisee, close
personal friend,
family, co-author or grant collaborator in past 5 years) leave the
room. Keep all specific PC comments private. I assigned a secondary
chair for the papers with which I had a conflict for paper assignment
and discussion (I left the room).

In particular, I followed all the guidelines in Patterson's document
on ethics, PC selection, PC submission, double-blind submission,
conflicts in reviewing and in the PC meeting, etc.
I hope you found my experience and advice useful.