Multiple-choice discourse completion tasks in Japanese English language assessment

Abstract

A new class of multiple-choice discourse completion tasks (MDCTs) is beginning to gain popularity in the Japan English as a Foreign Language (JEFL) assessment context. In this study, an experimental MDCT test was administered to a sample of Japanese university students. An item format analysis was conducted focusing on the construct validity and discrimination of MDCTs in measuring the English listening proficiency of JEFL speakers. Using a combination of classical test theory and Rasch analysis, test performance was analyzed in regard to two research questions: (a) whether a pragmatic proficiency construct is related to item difficulty, and (b) whether the use of different distractor types has an effect on item discrimination characteristics. The results suggest that a pragmatic proficiency construct plays a role in determining item difficulty on MDCTs, bringing into question the construct validity of MDCTs as a listening language proficiency measurement. Additionally, MDCT item discrimination might be affected by the type of distractors being used, hinting at possible ways to optimize discrimination of MDCTs in norm-referenced testing (NRT). Given the high probability of continued use of MDCTs in the JEFL context and the need for more investigation into these items, this study is hopefully an early step towards better and more informed MDCT test design and use.