tag:blogger.com,1999:blog-6555947.post1677215100084111397..comments2014-01-12T10:46:48.153-07:00Comments on The Geomblog: On PC members submitting papersSuresh Venkatasubramaniannoreply@blogger.comBlogger32125tag:blogger.com,1999:blog-6555947.post-15012000705363457262012-01-10T16:53:14.764-07:002012-01-10T16:53:14.764-07:00I am so happy to read this. I am never a PC member...I am so happy to read this. I am never a PC member, but soon to be. There is no this rule in my head (maybe in some corner of hard disk, but not in the memory). Maybe people, including myself, are always lazy to build a new possibly better system, because it takes much less work to just revise current system. Also, better always means more complicated. We have to balance fairness and efficiency. <br /><br />1. Generally, the ideal system I think. <br /><br />There is an ideal and quite fair system after published, that is, good papers will be cited and noticed. <br />The ideal system is that, there is no submitting and reviewing, and every authors and readers can publish papers and reviews somehow freely. (Like blog? :) The authors and the refrees=readers are the same set. At this sense, I think, PC members should be every researcher, which are also latent authors. )<br /><br />2. Some details of this system. <br /><br />It needs rules about: <br />How to defined the upper bound function of frequency of publishing papers by an author. <br />How to balance the areas researched by a lot of people and a few people (they will not get<br />as many as readers and reviews. we can not simply say more good reviews means good paper). <br />How to distribute the probability of each paper that appears before readers. <br />How to organize the review of review, and the review of review of review, ...<br />How to discuss the system for building this system, and the system for building for building ...<br />How to design some index, shown as scores or curves, for a person as author and reader. <br />(We must be careful about this. If we simply say that a review which is similar to the majority reviews of this paper is a good one, this gives a simple cheating method. We shall also praise the review which is quite opposite to the previous ones but supported by the following ones.)<br />How to forbid cheating in this system, for example, a clique which always give<br />high scores only to its numbers. (One method for this, briefly, when a reader want to see<br />newest papers, he only get a random subset of newest papers. If a member want to<br />praise another member&#39;s paper soon after its appearance, some search operations are needed,<br />which is easy to detect. We analyze this kind of behavior by algorithms and design it as an index. )Mingji Xiahttp://www.blogger.com/profile/14554280160068385310noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-17426940074302794902011-12-29T13:39:20.060-07:002011-12-29T13:39:20.060-07:00Moreover, the NSF has a 4-year exclusion rule, as ...Moreover, the NSF has a 4-year exclusion rule, as opposed to a 2 year rule that many conferences use. I sense in the objections a general chafing at &quot;making us constrained in any way shape or form&quot;. And for those folks, it should be noted that all proposals would be adapted to the needs of theoryCS, not applied in a cookie cutter form.Suresh Venkatasubramanianhttp://www.blogger.com/profile/15898357513326041822noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-957613894345765632011-12-29T13:27:01.999-07:002011-12-29T13:27:01.999-07:00I&#39;m perfectly comfortable with a CoI model tha...I&#39;m perfectly comfortable with a CoI model that &quot;prohibits recent collaborators&quot;. If I can&#39;t convince people I haven&#39;t worked with that my work is interesting, novel, and correct, then I haven&#39;t done my job.<br /><br />Is that a gun in my hand? Why is it suddenly aimed at my foot?!<br /><br />But almost every American theorist with a PhD is *already* comfortable with the &quot;no recent collaborators&quot; COI model, because that&#39;s the *precise* model that NSF uses when it evaluates grant proposals. And yet NSF&#39;s funding decision are mostly sensible [citation needed].Jeffehttp://www.blogger.com/profile/17633745186684887140noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-47341365874319113832011-12-29T11:54:03.028-07:002011-12-29T11:54:03.028-07:00Good discussion.
As far as electronic PC meeting...Good discussion. <br /><br />As far as electronic PC meetings, I am not agnostic on this front - in my experience electronic meetings work <em>better</em>. In part this could be a personal thing - my first PC meeting experience in the early 2000s was electronic, and I think it worked well. My several experiences with SODA was also good. <br /><br />My main gripe about physical PC meetings is that, invariably, they contain long stretches of time where everyone is pretty wiped out. This includes the time after meals, before the end of the day etc. During those times the quality of the discussion drops, and the decisions can become suboptimal. This is particularly true towards the end of the meetings. <br /><br />Other negatives include the time and effort spent on physical travel, often cross-country. As the world becomes more and more linked, this seems harder and harder to justify.<br /><br />For completeness: I do enjoy the personal aspects of physical meetings (meeting people, interactive discussions, etc). But I just think that the cons out-weight the pros.Piotrhttp://people.csail.mit.edu/indyk/noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-29517353789172050102011-12-28T23:38:33.969-07:002011-12-28T23:38:33.969-07:00Hi Paul
Thanks for the detailed and thoughtful c...Hi Paul<br /> Thanks for the detailed and thoughtful comment (which should really be a blog post on its own!). Others will have different views on this, but let me weigh in with my take on your questions:<br /><br />Firstly, regarding electronic PC meetings, I am agnostic on this front: there are pros and cons to this format, and the reduced travel/cost is quite appealing, but I agree that things often reduce to a &#39;star&#39; network of communication between the PC chairs and the pc members/area chairs/senior PC.<br /><br />The idea of limiting the top-level PC to no submissions could work as only a small difference from our current model. <br /><br />In regards to how the PC could be constructed, I&#39;ve seen different models. one is the area chair model where the area chairs choose their sub-PC. another is the area chair model where the PC members are still chosen by the PC chairs. Finally, there&#39;s the &#39;vice chair&#39; model where the vice chairs are not area-specific. In one conference where I&#39;m a vice-chair, the PC members asked us to nominate 5 people each for potential inclusion on the PC. The vice-chair model might be best for theory conferences, so that there&#39;s still a holistic view of the submissions, rather than letting things get balkanized. <br /><br />How the CoI rules are developed is indeed a matter of discussion, and customized for theory needs. The CCC model sounds reasonable, but I haven&#39;t thought deeply about the exact nature of CoI rules, to be honest.Suresh Venkatasubramanianhttp://www.blogger.com/profile/15898357513326041822noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-80361576734828243682011-12-28T23:27:50.706-07:002011-12-28T23:27:50.706-07:00Suresh, you raised several issues:
1. Reduce the w...Suresh, you raised several issues:<br />1. Reduce the work of being on a large conference PC.<br />2. The prohibition of PC members submissions stands in the way. A two-tier PC is a solution and we can safely handle PC member submissions at large conferences by a suitable COI rule.<br />3. An all-electronic PC meeting would help.<br /><br />Though the size of the PC has grown, the workload for a FOCS/STOC PC member has not changed dramatically since the first electronic-only PCs back in the mid-late 1990&#39;s. (Going to on-line reviews and interaction prior to decisions significantly increased the workload versus write-once reviews and in-person only PC meetings.)<br /><br />The electronic-only FOCS/STOC PCs of the 1990&#39;s were pretty awful. It wasn&#39;t a software problem; it was a social problem. PC members disappeared from discussions. Accept/reject decisions stretched over days when some PC members could not participate. PC members did not have much basis for making accept/reject decisions for papers that were not in their purview because there was no incentive for overall discussion vs discussion between two or three experts. I therefore strongly object to suggestion 3.<br /><br />SODA seems to be in a different place from FOCS/STOC. The workload has gotten very high and I have heard that the review process is quite broken. (It seems to suffer from some of the problems of electronic-only FOCS/STOC PCs.)<br /><br />At a broad conference like FOCS/STOC and probably SODA, the core PC should be involved in an overall discussion of papers as opposed to a narrow area-based discussion. It is not just the &quot;PC members can&#39;t submit&quot; rule that pushes against increasing the size of this group. It is also having a group small enough that an overall discussion is not unwieldy.<br /> <br />A two-tier PC might work. I agree that we aren&#39;t that far from a second tier with our sub-reviewers. It seems that SODA might be the right place to start because it seems more out of hand. For STOC/FOCS, I think that we are on the edge - if the submission numbers grow we will be in trouble. I would reduce the size of the top tier from the current PC of 22-24 to 15 or so - big enough to have coverage in expertise but not necessarily volume.<br /><br />With a two-tier PC I would keep the &quot;PC members cannot submit&quot; rule for the top tier. The cost is small and it frees the top-tier PC members a lot. It is common in other communities for the top tier PC to be similarly prohibited from submitting.<br /><br />4. How would we actually organize the two tiers of the PC? The &quot;area chair&quot; format just doesn&#39;t seem to fit our conferences. How would the second tier PC be chosen?<br /><br />I assume that the top tier PC would be involved along with the second tier PC in some number of reviews, since they are likely to be experts on many papers. I suspect that this would be at most half the current numbers that PC members handle.<br /><br />I don&#39;t think that strict COI rules alone do the job. I prefer the CCC rules on COI: All conflicts must be disclosed clearly, not just what they are but how they are expected to impact a review. The review form suggests a range of potential conflicts that are not necessarily disqualifying. These include items such as &quot;I am a good friend of&quot; or &quot;I have competing/closely related work&quot; not usually covered by strict COI rules. When I want a review on a paper I often want to hear from somebody who has such a conflict, but it can be as much or more of a conflict than a co-author or institutional conflict.<br /><br />BTW: At theory PC meetings advisor/advisees or people who are at the same institution are always expected to remain out of the discussion on (and certainly not make arguments for) a paper they are conflicted on. The only thing that Michael did differently was suggest that conflicted people not see reviews (not enforced) and require that the conflicted people leave the room during the PC discussion. The only situation where I would enforce a strict &quot;no-communication&quot; rule would be for an author.Paul Beamenoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-576981665816474972011-12-28T15:10:27.467-07:002011-12-28T15:10:27.467-07:00I&#39;ll take the liberty of saying that we&#39;re...I&#39;ll take the liberty of saying that we&#39;re converging. If the main issue involved in installing CoI as part of expanding a PC is deciding on a &#39;recent author&#39; rule, then we&#39;re pretty far along towards a workable solution. I agree that there are subareas of TCS that are (1) small and (2) technically dense, and that one has to be careful about recent-author rules so as not to disadvantage such areas.Suresh Venkatasubramanianhttp://www.blogger.com/profile/15898357513326041822noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-37606913395454480062011-12-28T14:50:25.290-07:002011-12-28T14:50:25.290-07:00P.s. The areas I mentioned are not my own: I just ...P.s. The areas I mentioned are not my own: I just threw out lattice based crypto, arithmetic circuits, cell probe data structures as random examples of areas that I&#39;d imagine a typical PC will have 0-2 members knowledgeable about, and you&#39;d want them in the room for discussions. <br /><br />But the truth is that I&#39;m not sure what&#39;s exactly the controversary about, if indeed nobody is proposing adding a &quot;recent collaborator&quot; conflict for theory PC&#39;s and subreviewers. (Though I think such collaborations should be disclosed.)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-33260782244285623912011-12-28T14:41:02.250-07:002011-12-28T14:41:02.250-07:00Michael&#39;s posts mention systems conferences im...Michael&#39;s posts mention systems conferences implenting a &quot;coauthor in last x years&quot; (where x is sometimes 2, sometimes 5), and its unclear if he does or does not propose tcs adopt this rule. All tcs conferences I&#39;m aware of have always considered advisor/advisee and family relations as a conflict which people left the room for. So, I&#39;m not sure what exactly michael changed at stoc and whats the big debate about. (Maybe about the &quot;same institution&quot; rule?)<br /><br />Secondly, I didn&#39;t claim theory is unique in having specialized subareas with few people inside each one. I just claim that whenever there is a situation like that, adding a &quot;recent coauthor&quot; conflict doesn&#39;t make sense.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-79482053136971731702011-12-28T14:14:31.257-07:002011-12-28T14:14:31.257-07:00Anon12:36pm -- I would have responded, but Suresh ...Anon12:36pm -- I would have responded, but Suresh aptly did so for me. <br /><br />I would add, though, that in my experience, the arrogance that goes with &quot;my area is so difficult only a small number of specialists can properly evaluate it&quot; correlates with the opinion &quot;so many papers in my area are so good, they really should get in&quot; -- a troublesome issue for (what should be) broad conferences like FOCS/STOC/SODA.Michael Mitzenmacherhttp://www.blogger.com/profile/02161161032642563814noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-13858339402680591722011-12-28T14:00:23.548-07:002011-12-28T14:00:23.548-07:00@Anon12:36pm: A conflict of interest policy that p...@Anon12:36pm: <i>A conflict of interest policy that prohibits recent collaborators will lose the most informed opinions, so a disclosure rather than prohibition based model is more appropriate for tcs.</i><br /><br />I think this is a bit of a strawman argument. No one has proposed a CoI model that &quot;prohibits recent collaborators&quot;. The usual CoI models include same institution, advisor/advisee relationship, and so on. These can always be tweaked for TCS if the community of researchers working in an area is small enough that &quot;collaborator in the last two years&quot; rules out everyone. But there aren&#39;t that many cases of this kind of thing, and I&#39;m saying that from personal experience soliciting reviews from people for papers. <br /><br />Secondly, I&#39;m sorry to say this, but it *is* arrogant to assume that the existence of complicated subareas is something unique to theoryCS. there&#39;s plenty of specialization in machine learning and databases (two areas I&#39;m familiar with). And &quot;systems&quot; is an almost meaningless term: do you mean architecture, operating systems, networking, programming languages, verification, concurrency, databases, .....Suresh Venkatasubramanianhttp://www.blogger.com/profile/15898357513326041822noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-90444196795108049252011-12-28T12:36:57.844-07:002011-12-28T12:36:57.844-07:00A paper should be *readable* by non experts but it...A paper should be *readable* by non experts but it doesn&#39;t meant they can *evaluate* it. I don&#39;t know how its in systems, but theory has many subareas where evaluating a paper requires specialized expertise that few posses (think lattice based crypto, arithmetic circuits, cell probe data structures, etc etc). A conflict of interest policy that prohibits recent collaborators will lose the most informed opinions, so a disclosure rather than prohibition based model is more appropriate for tcs. What part of this argument is arrogant or self serving?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-464422447766268042011-12-27T13:21:13.707-07:002011-12-27T13:21:13.707-07:00@Jeffe: &quot;Also, reviewers of (even theoretical...@Jeffe: &quot;Also, reviewers of (even theoretical) physics papers generally do not verify correctness in detail, but only judge whether the paper is &quot;interesting&quot; and look for egregious errors.&quot;<br /><br />If so, there is no difference, the same mess, you are right. But I have &quot;old fashioned&quot; views at what does it mean &quot;doing science&quot;. Results should be archived, checked, not evaluated. Evaluation is the task of the entire community, and of TIME! Not of PCs or of editors. They should only handle the procedures, check for correctness, and for novelty. But please - no &quot;stamps&quot;, no &quot;certificates of truth&quot;, no advertising, no &quot;main streams&quot;. In this (my) world, TCS conferences look like aggressive markets. Far away form just a silent presentation of scientific findings.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-56073800062488884562011-12-27T09:33:32.736-07:002011-12-27T09:33:32.736-07:00Having served on multiple systems committee confer...Having served on multiple systems committee conferences, I&#39;ve found PC members submitting papers works well in that context. Conflicted people (for a reasonably broad definition of &quot;conflict&quot;, not just authors of a paper) have to leave the room during discussions (or are prevented from viewing the discussion if its an online meeting). And yes, efforts are made to make sure that papers that have PC members on them are not even listed as being in/out until the end of the process, though of course that&#39;s not always perfect. <br /><br />One can argue whether such approaches remove the bias one might expect for (or against) PC-authored papers. In my experience it works well, but that&#39;s an opinion; I don&#39;t have empirical evidence. I should point out even if it doesn&#39;t work perfectly, it may still be better than a system that doesn&#39;t allow PC-authored papers, as Suresh is pointing out here. <br /><br />I&#39;ve heard people say this won&#39;t work for theory conferences, since such COI policies would mean only &quot;non-experts&quot; would read the paper, and decisions would be worse. I find this argument arrogant and self-serving, personally. If you can&#39;t make a paper understandable and interesting to people beyond your friends on the committee, maybe it shouldn&#39;t be accepted. But I should point out that that was the argument I heard when I tried to put in COI policies. <br /><br />Suresh, you might want to reference my old, dramatic posts on this issue: <br /><br />http://mybiasedcoin.blogspot.com/2009/02/stoc-pc-meeting-part-iii-conflicts-of.html<br /><br />http://mybiasedcoin.blogspot.com/2009/02/stoc-pc-meeting-part-iv-conflicts-of.html<br /><br />And this less dramatic one.<br /><br />http://mybiasedcoin.blogspot.com/2010/02/conflicts-of-interest-yet-again.htmlMichael Mitzenmacherhttp://www.blogger.com/profile/02161161032642563814noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-68281054982506734592011-12-27T08:34:41.440-07:002011-12-27T08:34:41.440-07:00Larger PCs will reduce the reviewing load per revi...Larger PCs will reduce the reviewing load per reviewer. Independent of everything else, that is a good thing.Suresh Venkatasubramanianhttp://www.blogger.com/profile/15898357513326041822noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-90622710362410390542011-12-27T08:29:58.181-07:002011-12-27T08:29:58.181-07:00Switching to journals won&#39;t cause less papers ...Switching to journals won&#39;t cause less papers to be written, or more referees to magically appear, and if journals strive for any reasonable turnaround time, then time pressures will also stay the same. (Of course today TCS journals can afford long refereeing processes precisely because their low importance means that not all papers get submitted, and authors don&#39;t deeply care about speedy decisions.)<br /><br /><br />So, the main difference between conferences and journals is the physical meeting and presentations. People can argue if this meeting is a good thing or bad thing, but if someone claims that there is any other essential difference in issues such as quality of refereeing, conflict of interests, etc.. then I&#39;d like to see proof of that.<br /><br />Regarding PC submissions, I&#39;m not sure that not allowing PC submission is the reason theory PC&#39;s are not larger. Nor am I sure that larger PC imply a higher quality reviewing process. (I think there are pros and cons to both large and small PC&#39;s, and unfortunately &quot;quality of reviews&quot; is not easily measurable.)Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-51969173079444562162011-12-27T06:53:48.967-07:002011-12-27T06:53:48.967-07:00I&#39;ve had papers spend years in the journal rev...I&#39;ve had papers spend years in the journal reviewing process only to have them accepted without significant revision. Conferences have the advantage of time-certain decisions.<br /><br />For a once-per-year conference from a smaller community I can easily see allowing Pc members to submit. I am not sure that it would make much of a difference for FOCS/STOC and maybe not for SODA. <br /><br />BTW: In some systems PCs, I&#39;ve heard that the outcomes of decisions on PC members papers are kept secret from the PC members until the entire program is determined, though one can&#39;t completely keep this secret if one is hitting the limit on the number of accepted papers.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-68548415695119238562011-12-27T06:07:45.525-07:002011-12-27T06:07:45.525-07:00&quot;...just paper screening instead of a normal ...&quot;...just paper screening instead of a normal 2-3 months refereeing.&quot;<br /><br />The standard timeline for journal publications in physics is six weeks, not 2-3 months. Also, reviewers of (even theoretical) physics papers generally do not verify correctness in detail, but only judge whether the paper is &quot;interesting&quot; and look for egregious errors.<br /><br />Sound familiar?Jeffehttp://www.blogger.com/profile/17633745186684887140noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-26543627580773274132011-12-26T23:37:40.844-07:002011-12-26T23:37:40.844-07:00Hi Suresh,
I agree that there is nothing holy a...Hi Suresh,<br /><br /> I agree that there is nothing holy about the rule that PC members cannot submit, AS LONG AS other conflict of interests rules are applied. Specifically, the applied CS conferences that I am aware of have pretty strict conflict of interest rules which are being taken rather seriously. This allows them to be more relaxed about PC reviewing, since there are mechanisms to deal with the resulting conflicts. In contrast, until recently, TCS conferences were pretty lax about conflicts. Fortunately this is changing now.Piotrhttp://people.csail.mit.edu/indyk/noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-85591707413410485252011-12-26T21:37:23.791-07:002011-12-26T21:37:23.791-07:00@Anon9:26pm: If the meeting is physical, then they...@Anon9:26pm: If the meeting is physical, then they leave the room (like NSF panel deliberations). If the meeting is electronic, then the system prevents them from commenting/viewing any papers they have a COI with. They can still submit though. In fact, often conferences have two PC chairs so that even the PC chairs can submit, and the each one handles the other&#39;s submissions.Suresh Venkatasubramanianhttp://www.blogger.com/profile/15898357513326041822noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-68413291407242926032011-12-26T21:26:01.856-07:002011-12-26T21:26:01.856-07:00In your overview you say that &quot;there is nothi...In your overview you say that &quot;there is nothing stopping a PC from opening up the discussions to all vice-chairs&quot;. I agree that there is value in the overall discussion but once you open it up like this, how do you handle the COI of vice-chairs who have submitted papers? If vice-chairs are prohibited from submission (which is the case on some non-theory PCs) then it seems like an issue of semantics. <br />Two-tier vice-chairs = PC members<br />Two-t4er PC members = subreviewersAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-58692450151844882162011-12-26T15:08:07.849-07:002011-12-26T15:08:07.849-07:00@Suresh: Unfortunately we cannot come to a &quot;s...@Suresh: Unfortunately we cannot come to a &quot;stable state&quot; incrementally. The error is in the ROOT of of this &quot;science making&quot; PRINCIPLE. One cannot correct it by improving &quot;conference reviewing&quot; or the like. We need another principle.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-58303862861259871112011-12-26T14:34:38.480-07:002011-12-26T14:34:38.480-07:00@Anon2:30 I don&#39;t disagree. But I think that&#...@Anon2:30 I don&#39;t disagree. But I think that&#39;s a different discussion. And I think that any discussion of that nature needs to discuss not the end-goal but how to get there incrementally. <br /><br />This discussion is about a much more mundane issue, but one that can help improve conference reviewing as it currently stands, by reducing reviewer load.Suresh Venkatasubramanianhttp://www.blogger.com/profile/15898357513326041822noreply@blogger.comtag:blogger.com,1999:blog-6555947.post-15460612517222684332011-12-26T14:30:53.776-07:002011-12-26T14:30:53.776-07:00@Suresh: I still wonder WHY we stick at conference...@Suresh: I still wonder WHY we stick at conferences? (And on all implied problems.) 500 submissions to SODA! 500 quickies? And 500 quick decisions. I think (and hope) science is done other way around.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6555947.post-16967998227486423452011-12-26T13:51:56.887-07:002011-12-26T13:51:56.887-07:00I should also add that very few conferences have p...I should also add that very few conferences have physical PC meetings (I think STOC and FOCS are the only ones at this point). Given the logistics and expense of travel, I don&#39;t mind moving towards all-electronic meetings, and with the right software (hotCRP comes to mind) it&#39;s not hard to still get a global view of the discussions.Suresh Venkatasubramanianhttp://www.blogger.com/profile/15898357513326041822noreply@blogger.com