Justifying Aggregation with Consensus-Based Constructs: A Review and Examination of Cutoff Values for Common Aggregation Indices

Authors

• David Jonathan Woehr, University of North Carolina Charlotte
• Andrew Caleb Loignon, University of North Carolina Charlotte
• Paul B. Schmidt, University of North Carolina Charlotte
• Misty L. Loughry, Georgia Southern UniversityFollow
• Matthew W. Ohland, Purdue University

Document Type

Article

Publication Date

3-2015

Publication Title

Organizational Research Methods

DOI

10.1177/1094428115582090

Abstract

Management researchers often use consensus-based composition models to examine the antecedents and effects of higher-level constructs. Typically, researchers present three indices, r_wg, ICC(1), and ICC(2), to demonstrate agreement and consistency among lower-level units when justifying aggregation. Nevertheless, researchers debate what values for these indices are sufficient. This study examines the distributional characteristics of ICCs and r_wg values from three sources: the multilevel literature, a large multinational sample of student teams, and a large sample of randomly generated “pseudo teams.” Our results support existing cutoff criteria for ICCs but suggest that generally accepted values for r_wg may, under certain circumstances, reflect pseudo-agreement (i.e., agreement observed among two raters not attributable to the same target). Thus, when there is minimal between-group variance (i.e., low ICCs), it is difficult to determine whether high r_wg values reflect agreement or pseudo-agreement. Based on these findings, we provide recommendations to help researchers interpret aggregation indices.

Recommended Citation

Woehr, David Jonathan, Andrew Caleb Loignon, Paul B. Schmidt, Misty L. Loughry, Matthew W. Ohland. 2015. "Justifying Aggregation with Consensus-Based Constructs: A Review and Examination of Cutoff Values for Common Aggregation Indices." Organizational Research Methods: Sage Publishing. doi: 10.1177/1094428115582090
https://digitalcommons.georgiasouthern.edu/management-facpubs/9

Copyright

This work is archived and distributed under the repository's Standard Copyright and Reuse License (opens in new tab). End users may copy, store, and distribute this work without restriction. For all other uses, permission must be obtained from the copyright owners or their authorized agents.