Rating Writing Samples: An Interdepartmental Perspective

Abstract

This study investigated the degree to which significant differences existed between the mean writing scores of native speakers and international students at the end of their ESL 100 and ENG 100 freshman composition courses, respectively. Eight English Language Institute teachers (from the ESL Department) and eight English Department teachers were paid to rate 112 randomly assigned compositions without knowing which type of students had written each. As a side issue, the degree to which raters differed in the scores they assigned (both between and within departments) was also investigated. A holistic six-point rating scale initially devised by the composition teachers in the English Department was used by all raters. A three-way analysis of variance with repeated measures on two of the three factors was conducted to determine whether there were significant differences for main effects due to the type of student (ESL 100 versus ENG 100), the raters (ESL or English Departments), the order of reading within each department, or any interaction of these three factors. Raters were also asked to choose the best and worst features (from a list of six possibilities: cohesion, content, mechanics, organization, syntax, and vocabulary) of each composition as they rated it. The frequencies of these responses were analyzed using chi-square statistics for overall statistical differences followed by more detai1ed analyses for differences between and within the two departments. Raters were also asked to choose the best and worst features (from a list of six possibilities: cohesion, content, mechanics, organization, syntax, and vocabulary) of each composition as they rated it. The frequencies of these responses were analyzed using chi-square statistics for overall statistical differences followed by more detai1ed analyses for differences between and within the two departments. All results are discussed in terms of how ESL testing and decision making were affected.