A Validation Argument for Cloze Test Item Function in Second Language Assessment

Abstract

This study presents a validation argument for cloze item function. While cloze tests have been said to measure reading ability in L1 contexts, most L2 research has linked their function to measures of general language proficiency. This study explores this issue more closely through examining evidence to construct a warranted interpretation/use argument (Kane, 2013) for cloze item function. Performance on fifteen 30-item cloze passages was examined for both native users of English and second language learners from Japan and Russia. Passages were examined in relation to their composition, syntactic structure, and cohesive features based on a number of text analysis methods. Item-level features for each of the 450 cloze items were also classified according to their composition and their interaction with surrounding context. For the latter, native user data were collected to gather information about the kinds of contextual information utilized by the individual items through a series of sentence completion tasks. These tasks, as well as L1 cloze test data, were gathered using Amazon Mechanical Turk. Variables for each passage composition, syntactic structure, cohesion, and item characteristics were explored in relation to Rasch logit measures of item difficulty for both the L1 and L2 cloze datasets, and used to create a structural equation model explaining the influence of each on item function. The results indicated that cloze items performed very similarly for both L1 and L2 users of English. Items were found to access context on multiple levels, and different classes of items functioned well for both groups of examinees. Other similarities were also observed through test analyses, item analyses, and correlations between item difficulty and passage-level variables. In addition, a single structural model was able to explain item function for both groups. Differences were found for L1 and L2 examinees in how factors for passage composition, structure, cohesion, and items related to item function, however, even these results seemed more in line with differences in the ability of the examinees rather than differences in the construct being measured for each. Overall, the evidence pointed to cloze items measuring the same construct for L1 and L2 test takers.