Please use this identifier to cite or link to this item: http://hdl.handle.net/10125/51493

English Proficiency and Entrance to Doctoral Studies at One Colombian University: A Standard Setting Study

File Description SizeFormat 
2016-08-phd-janssen_r.pdfVersion for non-UH users. Copying/Printing is not permitted3.2 MBAdobe PDFView/Open
2016-08-phd-janssen_uh.pdfFor UH users only3.85 MBAdobe PDFView/Open

Item Summary

Title: English Proficiency and Entrance to Doctoral Studies at One Colombian University: A Standard Setting Study
Authors: Janssen, Gerriet
Keywords: standard setting
Body of Work Method
Bookmark Method
cut score
Colombia
Issue Date: Aug 2016
Publisher: [Honolulu] : [University of Hawaii at Manoa], [August 2016]
Abstract: Critically important to test and program development is the enterprise of standard setting. This activity establishes descriptions of different performance levels and then uses a principled methodology to construct a cut score on a test: a point on a score scale that represents the separation of these different performance levels. To ensure that the creation of this cut score has been done in a way that is “systematic, reproducible, objective, and defensible” (Cizek & Bunch, 2007, p. 8), standard setting draws upon a great body of theoretical and empirical resources, protecting test users from the negative consequences of arbitrary decision making.
Recently, one Colombian university developed coursework and a placement exam for an EAP program supporting PhD student publication and presentation skills. During this program’s development, a classification exam was created to place the PhD students into different course levels. Though not part of this exam’s uses, university leaders soon established that certain course levels would now be an entrance requirement for different PhD programs. This decision was arbitrary in that it was made without understanding (a) what decision makers wanted their PhD students to be able to do in English or (b) how these skills intersected with the different course levels that were being offered. In short, this decision was made without taking advantage of the technology of standard setting.
Responding to this situation, this study—using the Body of Work Method for the performance writing and speaking sections of the exam (Kingston, Kahl, Sweeney, & Bay, 2001; Kingston & Tiemann, 2012) and the Bookmark Method for the multiple-choice grammar-vocabulary-reading comprehension section (Mitzel, Lewis, Patz, & Green, 2001; Lewis, Mitzel, Mercado, & Schulz, 2012)—elicited from 23 panelists—all stakeholders in the university’s PhD programs—their conceptualization of the minimum English proficiencies for PhD study at this university. From these individual recommendations, one cut score was initially calculated; however, the different procedural, external, and internal validity evidence indicated that two cut scores—one more lenient, one more severe—would be a better solution for this context. However, in light of other evidence from the validity argument, this dissertation suggests as its final recommendation that the university prioritize which tools of learning it would like to develop, instead of worrying about which limitations it should impose.
Description: Ph.D. University of Hawaii at Manoa 2016.
Includes bibliographical references.
URI/DOI: http://hdl.handle.net/10125/51493
Appears in Collections:Ph.D. - Second Language Studies


Please contact sspace@hawaii.edu if you need this content in an ADA compliant alternative format.

Items in ScholarSpace are protected by copyright, with all rights reserved, unless otherwise indicated.