Researchers in organizational behavior, human resource management, entrepreneurship, strategy, sociology, psychology, education, and many other fields now explicitly recognize that lower-level entities are usually nested within higher-level collectives. For example, employees are nested within jobs and teams, establishments within companies, and firms within industries. However, how do we know whether there is variability of a lower-level relationship across higher-order units? The answer to this question has important implications in terms of the appropriateness of using usual data-analytic techniques based on ordinary least squares regression, or whether a multilevel modeling approach should be used. Moreover, the presence of variability in lower-order relationships across higher-order units leads to an examination of potential contextual factors that may serve as moderators of such relationships. In addition, practitioners are particularly interested in such effects because they provide information on the contextual conditions and processes under which interventions focused on individuals (e.g., selection, leadership training, performance appraisal and management) result in more or less positive outcomes.

Our article which appears in Organizational Research Methods titled “An Expanded Decision Making Procedure for Examining Cross-level Interaction Effects with Multilevel Modeling” offers a new index to assess variability of lower-level relationships across higher-order processes and units. This new index is labeled intraclass correlation beta (i.e., ρ_β). We illustrate the computation of ρ_β using previously published articles and also a Monte Carlo study. Our results suggest that researchers contemplating the use of multilevel modeling, as well those who suspect nonindependence in their data structure, should expand the decision criteria for using multilevel approaches to include ρ_β. To facilitate this process, we offer illustrative data sets and the icc_beta R package for computing ρ_β in single and multiple-predictor situations and make them available through the Comprehensive R Archive Network (i.e., CRAN).

We are very excited about the potential of ρ_β to allow us to uncover the presence of variability in lower-order relationships across higher-order process and units and look forward to discoveries that can be made based on information provided by ρ_β. Moreover, ρ_β can also be used as an index of effect size and used to synthesize previously published research to understand which may be more or less fruitful research domains in which cross-level moderating effects may exist.

Herman Aguinis (http://mypage.iu.edu/~haguinis) is the John F. Mee Chair of Management and Founding Director of the Institute for Global Organizational Effectiveness in the Kelley School of Business, Indiana University. His multi-disciplinary, multi-method, and multi-level research addresses human capital acquisition, development, and deployment, and research methods and analysis. He has published five books and more than 120 articles in refereed journals. He is a Fellow of the Academy of Management, former editor-in-chief of Organizational Research Methods, and received the 2012 Academy of Management Research Methods Division Distinguished Career Award for lifetime contributions.

Steven Andrew Culpepper (http://publish.illinois.edu/sculpepper/) is an assistant professor in the Department of Statistics at the University of Illinois at Urbana-Champaign. He completed a doctorate in educational psychology from the University of Minnesota in 2006. His research focuses on statistical methods in the social sciences and includes the development of new methodologies, evaluation of existing procedures, and application of novel statistical techniques to substantive questions in demography, education, management, and psychology.

Every time we review existing literature on the effect of management methods on organizational performance, we find it hard to compare results across studies. The contradictions between studies are mostly caused by different concepts and measurement approaches of organizational performance. If, due to completely different concepts and measurement systems, we are not able to combine study results, how can we as researchers even pretend to contribute to management research by the newest study applying a new construct measurement approach on organizational performance? Consequently, the interest into measurement approaches, construct validation and conceptual nature of organizational performance was triggered in our research team. After reviewing previous literature on this subject we recognized that no construct validation study addressing jointly the conceptual level of organizational performance and the construct validity of a comprehensive set of indicators at the operational level had been published before. This was the gap we wanted to close with our study.

Following Combs, Crook, and Shook (2005)1 we distinguish between operational and organizational performance. In this framework operational performance combines all non-financial outcomes of organizations. Furthermore, the conceptual domain of organizational performance is limited to economic outcomes. On this basis, we identify four organizational performance dimensions: profitability, liquidity, growth, and stock market performance. For each of these dimensions, we propose and test a set of construct valid indicators on a large panel data set with 37,262 firm-years for 4,868 listed US-organizations.

Interestingly, the growth dimension is troublesome under conditions of high environmental instability (e.g., in 2002 after the dotcom bubble or at the beginning of the financial crises in 2008). We perceive two possible explanations for this finding. First, growth is examined based on three aspects of size: sales, employees, and assets. These aspects differ in their reactivity with regard to increasing environmental instability (e.g., although sales might decrease immediately, investments already under way will be finished, thus increasing an organization’s assets base). Second, Higgins (1977)2 introduced the concept of a sustainable growth rate that must be in alignment with overall organizational performance, the financial policy, and the dividend payout ratio. If an organization grows at a rate above its sustainable growth rate, the other aspects (e.g., other dimensions of organizational performance) will eventually decrease. Fully developing these two arguments was beyond the scope of our article. However, they pose interesting research questions for future research on the growth dimension of organizational performance.

In summary, we propose a validated set of measurement indicators for the organizational performance construct for future management research. Furthermore, we highlight situations, in which construct validity is hampered.

Our article describes a surprising commonality in empirical results relying on agency theory in general and in the particular domain of corporate governance research: Mean estimates of relationships between measures of boards of directors’ independence, CEO duality, equity holdings, and corporate financial performance are disappointingly low. Typically, measures of board independence explain only about 1% of variance in relevant outcomes. We use meta-analytic evidence to document that this disappointing result is highly consistent across very diverse bodies of research. Our article was motivated by our desire to understand the reason(s) for such small effect sizes, because this knowledge would allow us to design and execute better studies in the future which will hopefully lead to an improved understanding (i.e., larger effect sizes) of critical outcomes such as firm performance.

Our conclusion can be summarized in a simple phrase: “It’s the measurement, stupid!” Threats to construct validity such as less ideal operationalization of constructs and confounding constructs and levels of constructs seem to be the culprits, at least in part, for why observed effect sizes are so small. How did we arrive at this conclusion? We implemented a five-step protocol through which we (1) established the base rate for the phenomenon in question, (2) evaluated the extent to which the dependent variables are germane, (3) evaluated the extent to which the independent variables are germane, (4) determined whether explanatory power is improved as a consequence of improved measurement, and (5) concluded whether previously established estimates should be revised. Due to the implementation of our proposed five-step protocol, we were able to improve variance explained in outcome variables many times over—in some cases, a tenfold increase.

So, what are the main take-aways? First, remember the motto “It’s the measurement, stupid!” In most empirical research in macro studies, small effect sizes can be expected even before data are collected due to construct validity problems. Second, implementing our proposed protocol in other research domains is likely to lead to more accurate, and larger, effect estimates of relationships among constructs.

Dan R. Dalton is the founding director of the Institute for Corporate Governance, Dean Emeritus, and the Harold A. Poling Chair of Strategic Management in the Kelley School of Business, Indiana University. A Fellow of the Academy of Management, his research focuses on corporate governance and research methods and analysis. He is the recipient of the 2011 Academy of Management Research Methods Division Distinguished Career Award and a former editor-in-chief of the Journal of Management.

Herman Aguinis is the Dean’s Research Professor, a professor of organizational behavior and human resources, and the founding director of the Institute for Global Organizational Effectiveness in the Kelley School of Business, Indiana University. His research interests span several human resource management, organizational behavior, and research methods and analysis topics. He has published five books and more than 100 articles in refereed journals. He is the recipient of the 2012 Academy of Management Research Methods Division Distinguished Career Award and a former editor-in-chief of Organizational Research Methods.

The raison d’être of management research is to prove that management instruments and management methods, such as strategic planning, zero based budgeting, or the balanced scorecard, are able to enhance organizational performance. In addition, major theories in management research, for instance all contingency theories, include organizational performance as an important dependent variable in their conceptual arguments. But what is organizational performance? How can it be defined and measured in a reliable and valid manner? The Organizational Research Methods article “Exploring the Dimensions of Organizational Performance: A Construct Validity Study” provides answers to these questions.

Every time we review existing literature on the effect of management methods on organizational performance, we find it hard to compare results across studies. The contradictions between studies are mostly caused by different concepts and measurement approaches of organizational performance. If, due to completely different concepts and measurement systems, we are not able to combine study results, how can we as researchers even pretend to contribute to management research by the newest study applying a new construct measurement approach on organizational performance? Consequently, the interest into measurement approaches, construct validation and conceptual nature of organizational performance was triggered in our research team. After reviewing previous literature on this subject we recognized that no construct validation study addressing jointly the conceptual level of organizational performance and the construct validity of a comprehensive set of indicators at the operational level had been published before. This was the gap we wanted to close with our study.

Following Combs, Crook, and Shook (2005)1 we distinguish between operational and organizational performance. In this framework operational performance combines all non-financial outcomes of organizations. Furthermore, the conceptual domain of organizational performance is limited to economic outcomes. On this basis, we identify four organizational performance dimensions: profitability, liquidity, growth, and stock market performance. For each of these dimensions, we propose and test a set of construct valid indicators on a large panel data set with 37,262 firm-years for 4,868 listed US-organizations.

Interestingly, the growth dimension is troublesome under conditions of high environmental instability (e.g., in 2002 after the dotcom bubble or at the beginning of the financial crises in 2008). We perceive two possible explanations for this finding. First, growth is examined based on three aspects of size: sales, employees, and assets. These aspects differ in their reactivity with regard to increasing environmental instability (e.g., although sales might decrease immediately, investments already under way will be finished, thus increasing an organization’s assets base). Second, Higgins (1977)2 introduced the concept of a sustainable growth rate that must be in alignment with overall organizational performance, the financial policy, and the dividend payout ratio. If an organization grows at a rate above its sustainable growth rate, the other aspects (e.g., other dimensions of organizational performance) will eventually decrease. Fully developing these two arguments was beyond the scope of our article. However, they pose interesting research questions for future research on the growth dimension of organizational performance.

In summary, we propose a validated set of measurement indicators for the organizational performance construct for future management research. Furthermore, we highlight situations, in which construct validity is hampered.