A comparison of methods for estimating internal consistency reliability of tests with dichotomously-scored items

Abstract

The use of Kuder and Richardson's formula 20 (KR20) (1937) to estimate reliability has been controversial. The purpose of this investigation was to review internal consistency reliability estimation techniques for unidimensional tests with dichotomously-scored items. Eleven methods were compared using a series of 98 simulated item-by-person response patterns with positive off-diagonal covariances, including patterns known to reflect perfect reliability by Loevinger's index of homogeneity (1947) and KR20. The upper limit of 1.0 was achieved in both perfect patterns only using methods described by Cliff (1984), Horst (1953), Loevinger, and Raju (1982). Lower limits of reliability were projected through linear regression. The ratio of off-diagonal covariance to test variance was used as the independent variable. Zero was included in the 95% confidence interval for Y-intercepts with Cliff's, Horst's, and Kuder-Richardson's techniques. Negative Y-intercepts were computed for the techniques of Cliff, Huck (1978), Loevinger, and Winer (1971). Positive Y-intercepts were computed for the techniques of Ayabe (1994), Guttman (L1 and L 2) (1945), Raju, and ten Berge and Zegers (1978). Between the upper and lower limits, reliability estimates generally increased as the ratio of off-diagonal covariance to total variance increased. It was concluded that the majority of estimation techniques do not meet minimum criteria for interpretation. Only the methods of Cliff, Horst, and Raju generally met the requirements for •reliability estimation techniques. Compared to KR20, the mean increases in reliability estimated with these three methods were .12 with Raju's ratio of actual to maximal KR20, .04 with Horst's method, and .00 with Cliff's γ-reliability technique.