Background: Programming language design is not usually informed by empirical
studies. In other ﬁelds similar problems have inspired an evidence-based paradigm
of practice. Such a paradigm is practically inevitable in language design, as well.
Aims: The content of evidence-based programming design (EB-PLD) is explored,
as is the concept of evidence in general. Additionally, the extent of evidence
potentially useful for EB-PLD is mapped, and the appropriateness of Cohen’s
kappa for evaluating coder agreement in a secondary study is evaluated. Method:
Philosophical analysis and explication are used to clarify the unclear. A systematic mapping study was conducted to map out the existing body of evidence.
Results: Evidence is a report of observations that affects the strength of an argument. There is some but not much evidence. EB-PLD is a ﬁve-step process for
resolving uncertainty about design problems. Cohen’s kappa is inappropriate for
coder agreement evaluation in systematic secondary studies. Conclusions: Coder
agreement evaluation should use Scott’s pi, Fleiss’ kappa, or Krippendorff’s alpha. EB-PLD is worthy of further research, although its usefulness was out of
scope here.
...