Rater bias in assessing the pragmatics of KFL learners using facets analysis

Abstract

As interest in research on second language pragmatics increases, some pragmatics research has been done on Korean as a foreign language (KFL) learners. This research has focused on pedagogical aspects of Korean pragmatics and interlanguage pragmatics. However, very little research has been done on the pragmatics assessment of KFL learners, in terms of discussing appropriate test types and whether, certain speech acts and test types affect raters' assessment of KFL learners' pragmatics performance. The focus of this study is on investigating (a) whether factors such as test types and speech acts affect raters' assessments of the pragmatics of KFL learners and (b) which test types are most appropriate for assessing pragmatics of KFL learners. For these purposes, this study analyzes three interactions from the test results using the computer program FACETS (Linacre, 1996): the interactions between rater bias and test types, rater bias and speech acts, and examinees and test types. This study uses three different pragmatics tests adapted from Hudson, Detmer, and Brown's (1995) pragmatics prototype tests: Open-written' Discourse Completion Task, Language Lab, and Role-play. Within each of these three test types are three speech acts: refusal, apology, and request. The results of this research indicate that all three raters showed different degrees of severity in their ratings', depending on the type of speech act. Additionally, each examinee showed different degrees of proficiency depending on the test type. I will discuss which test types are most appropriate for assessing KFL learners' pragmatics performance, how certain speech acts and test types affect raters' assessments, and how what these research findings mean for KFL classrooms.