Online Submission, Paper presented at the Annual Meeting of the American Educational Research Association (San Francisco, CA, Apr, 2006)

Summative evaluations have often been undertaken to determine the impact of educational programs on student academic achievement employing a quasi-experimental design. The summative finding is expected to be less misleading if a statistical model is performed on a dataset including a sound matched sample as a control group. This is because an extreme or untrustworthy extrapolation is not necessary to be applied and a regression artifact will be migrated. Empirical results showed that with the use of a single set of matched samples, the estimated program's effect might be unstable, however. A research methodology incorporating the components of multiple sets of matched samples in addition to the statistical control modeling (e.g., ANCOVA) is expected to mitigate this problem because this proposed method is expected to consistently reduce the selection bias in the quasi-experimental designs since almost all possible sets (e.g., 1000) of matched samples are drawn from the non-treated population. The proposed multiple-sets-of-matched-samples method creates a condition in which a single treatment group has multiple sets of matched samples as control groups. If 1000 matched samples are drawn from the non-treated population and the effect size (ES) measure is then estimated for each of 1000 comparisons, the mean as well as the distribution of those 1000 ES measures can be a more reliable measure to assess the efficacy of educational programs. This enhances our confidence level to determine whether or not a program is effective. A summative evaluation study of a technology-based reading intervention program that has utilized this research methodology was introduced in this paper to enunciate the appealing features of this research methodology. Appendix: (1) A Set of Matched Samples Created by the Greedy Matching Method with the Propensity Score. (Contains 6 figures and 6 tables.)