Are we testing what we think we are? A multi-site investigation of typed and handwritten L2 Chinese writing assessments

Abstract

The extent to which writing modality (i.e., hand-writing vs. keyboarding) impacts second-language (L2) writing assessment scores remains unclear. For alphabetic languages like English, research shows mixed results, documenting both equivalent and divergent scores between typed and handwritten tests (e.g., Barkaoui & Knouzi, 2018). However, for non-alphabetic languages like Chinese, typed responses consistently outscore handwritten ones across proficiency levels (e.g., Bourgerie et al., 2023; Liao, 2023; Zhang & Min, 2019; Zhu et al., 2016), and advanced proficiency groups. This study examined whether this scoring differential exists at the intermediate level. Writing samples were collected from 60 intermediate-level university L2 Chinese learners (N = 120), with counter-balanced prompts and modalities. Two certified raters used a three-criterion analytic rubric for scoring. A linear mixed-effects model showed that regardless of modality or prompt order, test takers produced more content and achieved higher scores on typed responses, even when controlling for language proficiency, multilingual experiences, and Chinese writing/typing habits. These findings suggest that handwritten and typed L2 Chinese test tasks may assess different constructs, carrying implications for various assessment contexts (classroom assessment, placement testing, proficiency tests) and assessment research.