Yesterday I blogged about the accepted artifacts at ISSTA. I think it is worthwhile noting that out of the ten papers that got accepted and which had artifacts submitted there seven for which the artifacts checked out. That is a good thing!

What worries me, however, are the three papers for which Artifact evaluation failed. For those three papers, we were largely unable to reproduce their results, and yet the papers made it into the program. Moreover, for two of those three, the fact that they failed artifact evaluation was already known before the PC meeting, i.e., there would have been a chance to reject them.

The reason for why the PC did not is that only positive reviews were taken into account this time, in order not to discourage people from submitting artifacts in the future. We as a community should really think about whether we cannot find a way to make artifact evaluation the default so that people have no other chance than to submit all the evidence they have to back up their claims.

I was very pleasantly surprised when I received the ICSE Distinguished Reviewer Award today, especially given that out of the 19 papers I reviewed only two were finally accepted. I guess there must also be a helpful way to reject papers after all. So thanks a lot to the ICSE authors for their positive vote! And more thanks go also to Alexandre Bartel, Mauro Baluda, Philipp Holzinger, Siegfried Rasthofer, Stephan Huber and Steven Arzt for assisting my reviews. Thanks a lot guys, we all share this award!