Rater group bias in the speaking assessment of four L1 Japanese ESL students

Abstract

The purpose of this study, modeled after Kobayashi’s (1982) investigation of writing evaluations, is to determine whether the factors like language background and educational training affect raters’ assessments of four Japanese medical students in a controlled oral interview. Rater groups include ESL-trained L1 English and L1 Japanese speakers, a peer group, and native speakers with no ESL background and negligible contact with Japanese L1 speakers. The four interviewees are first rated according to seven categories: grammar, fluency, content, pronunciation, pragmatics, compensation techniques, and overall intelligibility, and are then ranked from “most able” to “least able”. Group means per category are calculated and a FACETS (Linacre, 1996) analysis is used to look at the interaction between rater groups and rating categories. The findings show variation, though not seemingly as a result of the factors being investigated. Some general conclusions about the rater groups are made from the bias analysis. The limitations of the study are presented, and suggestions for further research are provided.