Multiple views of L1 writing score reliability

Abstract

This paper provides an in-depth investigation of the reliability of scores on the Manoa Writing Placement Examination (MWPE) based on a reanalysis of the scores of 6875 students who took MWPE during a four year period. Classical test theory analyses indicated that the MWPE scores were reasonably consistent across semesters (with a slight overall rise in scores over the four years), and that they produced traditional reliability estimates ranging from .51 to .74. The standard error of measurement was also examined in relationship to placement decision making. Generalizability theory was used to examine the relative importance of the numbers of topics and ratings to the consistency of scores. The results indicate that consistency would be increased more effectively by increasing the number of topics than by increasing the number of ratings. In addition, phi(lambda) indexes and signal-to-noise ratios were calculated to estimate score dependability at various decision points. The discussion focuses on the relative usefulness of these various approaches to reliability in practical test use, development, and interpretations.