Justifying Aggregation with Consensus-Based Constructs: A Review and Examination of Cutoff Values for Common Aggregation Indices
Organizational Research Methods
Management researchers often use consensus-based composition models to examine the antecedents and effects of higher-level constructs. Typically, researchers present three indices, rwg, ICC(1), and ICC(2), to demonstrate agreement and consistency among lower-level units when justifying aggregation. Nevertheless, researchers debate what values for these indices are sufficient. This study examines the distributional characteristics of ICCs and rwg values from three sources: the multilevel literature, a large multinational sample of student teams, and a large sample of randomly generated “pseudo teams.” Our results support existing cutoff criteria for ICCs but suggest that generally accepted values for rwg may, under certain circumstances, reflect pseudo-agreement (i.e., agreement observed among two raters not attributable to the same target). Thus, when there is minimal between-group variance (i.e., low ICCs), it is difficult to determine whether high rwg values reflect agreement or pseudo-agreement. Based on these findings, we provide recommendations to help researchers interpret aggregation indices.
Woehr, David Jonathan, Andrew Caleb Loignon, Paul B. Schmidt, Misty L. Loughry, Matthew W. Ohland.
"Justifying Aggregation with Consensus-Based Constructs: A Review and Examination of Cutoff Values for Common Aggregation Indices."
Organizational Research Methods.