Please use this identifier to cite or link to this item: http://hdl.handle.net/10125/68977

The Usefulness of the Computer-Based Speaking Tasks of the AP Japanese Exam

File Size Format  
Suzumura hawii 0085A 10398.pdf 3.31 MB Adobe PDF View/Open

Item Summary

Title:The Usefulness of the Computer-Based Speaking Tasks of the AP Japanese Exam
Authors:Suzumura, Nana
Contributors:Kondo-Brown, Kimi (advisor)
East Asian Language & Literature (department)
Keywords:Language
Educational tests & measurements
Linguistics
AP exam
discourse analysis
show 4 moregeneralizability theory
many-facet Rasch measurement
mixed methods research
performance assessment
show less
Date Issued:2020
Publisher:University of Hawai'i at Manoa
Abstract:The Advanced Placement (AP) Japanese Language and Culture exam (the AP Japanese exam) is a large-scale high-stakes test targeting high school students in the United States. It is a computer-based test that consists of speaking, listening, reading, and writing sections. The present study focuses on the speaking section which aims to measure examinees’ interpersonal communication skills and presentational skills. The usefulness of the speaking tasks was investigated from a variety of perspectives. Drawing upon Bachman and Palmer’s (1996) model of test usefulness, the present study scrutinized reliability, construct validity, and authenticity of the AP Japanese speaking test tasks.
In order to strengthen arguments regarding the usefulness of the AP Japanese speaking test tasks, the present study employed a mixed methods research design. The primary design was a concurrent design (R. B. Johnson & Onwuegbuzie, 2004; Teddlie & Tashakkori, 2006, 2009). The present study also employed a partial sequential design to integrate quantitative and qualitative methods at the analysis stage. Evidence for the usefulness of the test was collected by examining the speaking tasks used for the AP Japanese exam in the past, examinees’ language samples and test scores from the simulation of the AP Japanese exam, and examinee and rater survey responses. Four sets of conversation tasks and two sets of cultural perspective presentation tasks were selected from past test items and used in the simulation test. A total of 111 high school students, from two U.S. states, who were planning to take the AP Japanese exam participated in the present study. Their performance was rated by a total of six raters who participated in the present study: three raters for the conversation task and three raters for the presentation task. From the quantitative strand, generalizability theory and many-facet Rasch measurement (MFRM) (Linacre, 1989) were employed to analyze the test scores and to collect evidence primarily for reliability and construct validity. From the qualitative strand, discourse analysis was employed to analyze test task characteristics and examinees’ performance and to collect evidence primarily for authenticity and construct validity.
The present study found that overall the AP Japanese speaking section has a reasonable level of usefulness in that it showed a reasonable level of reliability and it aims to represent more dimensions of speaking ability than other computer-based language tests. Yet it also has some issues related to construct validity and authenticity, especially for the conversation task. In particular, the present study recommends reviewing the amount of contextual information, the proportion of information-seeking prompts and non-information-seeking prompts, and the appropriateness of the conversation scoring criteria in relation to the nature of the conversation prompts. The present study illustrated how intricately different facets of test usefulness can interact. This study exemplified how useful a mixed methods approach can be in collecting more comprehensive evidence for test validation and how that in turn can allow for more meaningful and practical suggestions to improve the overall usefulness of a language test.
Pages/Duration:460 pages
URI:http://hdl.handle.net/10125/68977
Rights:All UHM dissertations and theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission from the copyright owner.
Appears in Collections: Ph.D. - East Asian Languages and Literatures (Japanese)


Please email libraryada-l@lists.hawaii.edu if you need this content in ADA-compliant format.

Items in ScholarSpace are protected by copyright, with all rights reserved, unless otherwise indicated.