Measuring Reflective Cognitive Capacity: A Methodological Recommendation for Accounting Research of Feedback Effects.

Practical Implications:

The primary finding from the study is that average effects can mask real differences inparticipants’ cognitive capacity. Thus, the fundamental issue is not whether reflective cognitivecapacity is malleable. Rather, the issue is this: can participants whose thinking dispositionspredispose them to avoid being reflective—to avoid reevaluating their initial responses andsubsequently consider alternative theories (rules)—enhance their ability to engage in reflectivethinking? Future accounting behavioral research, especially studies that provide participants withfeedback and an opportunity to learn, should include measures of reflective cognitive capacity(either the Need for Cognition scale or the Cognitive Reflection Test) in order to improveexplained variance and more rigorously test techniques used to train accounting professionals.

This study investigates whether measures of reflective cognitive capacity can differentiate whichparticipants are more or less likely to benefit from feedback intervention. This is importantbecause if participants systematically differ in their ability to reflect, and accounting researchersomit controlling for such variation, then accounting academe’s recommendations regarding theeffectiveness of various feedback intervention techniques are likely to be overstated. In otherwords, such recommendations might not be applicable to those accounting professionals who areless inclined to engage in reflective thinking. This potential methodological issue relates tostudies in managerial accounting settings and financial information processing, as well as auditjudgments.

Design/Method/ Approach:

This study provides results from four separate experiments of feedback effects. These studieswere conducted across a two-year period, utilizing four separate accounting participant pools, allenrolled in a Master’s of Science in Accounting program, employing different feedbackmechanisms, and examining different measures of performance. In each of the four studies,reflective cognitive capacity is measured using the Need for Cognition scale; in the latter twostudies, the Cognitive Reflection Test, recently reported in the behavioral economics literature isused. The evidence was gathered prior to April 2014.

Findings:

Across all four experiments, the results consistently document that variations in participants'reflective cognitive capacity explain differences in post-feedback performance. Based on fourdifferent experiments, conducted across a two-year period, the results provide strong evidencethat the NFC and CRT measures could reasonably partition participants into two groups: thosethat are more likely, versus those that are less likely, to benefit from feedback intervention. Theincremental benefit derived from controlling for differences in reflective cognitive capacitycertainly exceeds the incremental cost. Based on an analysis of adjusted means, participants withrelatively high reflective cognitive capacity improved their performance after receiving summaryoutcome feedback, whereas participants with relatively low reflective cognitive capacity did notimprove.