Japanese placement tests at the University of Hawai‘i: Applying item response theory

Abstract

First-rate placement procedures are important for effective teaching and learning in any language program because they help create classes that are relatively homogeneous in terms of the language proficiency of the students. The main purpose of this study was to investigate how effectively and efficiently the current norm-referenced Japanese Placement Test (JPT) battery for the Japanese language program (three multiple-choice tests and essay test) at UHM separates the incoming students of Japanese into different course levels. The XCalibretm computer software program (Assessment Systems, 1997) was used to estimate the discrimination, difficulty, and guessing parameters for each item on each multiple-choice test. Based on these IRT analyses, we were able to suggest that all three multiple-choice tests be reduced in length while maintaining the same, if not better, level of reliability. Additional analyses of the interrater reliability of the essay tests using the Spearman-Brown prophecy formula led us to suggest that the essay test might be made more efficient by using two raters instead of three. The pattern of correlation coefficients between the tests indicates a certain degree of convergent validity for all the tests in this study, especially the subscales within the essay test. At the same time, when factor analysis was applied, support for divergent validity was also found based on language skills and testing method.