The culture of computer science emphasizes novelty and self-containment, leading to a fragmentation where each research project strives to create its own unique world. This approach is quite distinct from experimentation as it is known in other sciences — i.e. based on observations, hypothesis testing, and reproducibility — that is based on a presupposed common world. But there are many cases in which such experimental procedures can lead to interesting research results even in computer science. It is therefore proposed that greater acceptance of such activities would be beneficial and should be fostered.

The Canon

Let's build an "Experimental Evaluation in Software and Systems Canon", a list of readings on experimental evaluation and "good science" that have influenced us and that have the potential to influence the researchers coming after us.