Please use this identifier to cite or link to this item:
Assessing interactional competence in a multiparty roleplay task: A mixed-methods study
|Title:||Assessing interactional competence in a multiparty roleplay task: A mixed-methods study|
|Contributors:||Brown, James D. (advisor)|
Second Language Studies (department)
|Keywords:||English as a second language|
Educational tests & measurements
Mixed methods research
show 2 moreRasch
|Date Issued:||Dec 2018|
|Publisher:||University of Hawaiʻi at Mānoa|
|Abstract:||In an effort to develop an assessment instrument in measuring interactional competence (IC) with a method that is congruent with the current research findings on IC and IC development (e.g., Hall, Hellermann, & Pekarek Doehler, 2011; Pekarek Doehler & Pochon-Berger, 2015), the present study investigated students’ performances on a multiparty roleplay on a task called Socializing. Using the sequential mixed methods design (Greene, 2007; Tashakkori & Teddlie, 2003), the study explored empirical evidence garnered through qualitative and quantitative research methods to test if the proposed rubric can provide a valid and reliable measurement of IC on this performance assessment task. |
The participants of this study were 180 undergraduate engineering students at a university in Thailand who were taking an EFL course that targets social communication skills in professional contexts. Students were randomly grouped together and were asked to have a conversation for 10 minutes, in a roleplay task which they must introduce themselves as their character and try establishing business contacts for their hypothetical companies. The data for this study included 34 video-recordings of the group roleplay performances.
Conversation analysis (Clift, 2016; Sacks, 1992; Schegloff, 2007; Sidnell & Stivers, 2013) was employed to identify comparable interactional activities and determine the interactional methods students utilized in carrying out those activities. The productive activities are self-introduction, work talk, business contact exchange, post-conference arrangement talk, and an interaction to bring about the termination of the roleplay performance. Three recipient actions include students’ management and display of their understanding, students’ management of alignment, and finally, their display of affiliative stance.
Six raters from various teaching and training backgrounds were recruited to apply the proposed rubric in evaluating the students’ IC on the eight items, combining both productive and recipient actions. The Many-Faceted Rasch Measurement (Linacre, 1989) with the Partial Credit Scoring model (Masters, 1982) provided integrated measurement reports of the rating practice. The findings revealed that students’ ability on this IC construct mostly exceed the difficulty of the socializing task. Self-introduction and understanding display had been identified as the two easiest items, followed by alignment display, work talk, affiliation display, activity termination, making post-conference arrangements, and bringing up contact exchange, respectively. The analysis also suggested that most raters were reliable in applying the rating scale, though they demonstrated a higher degree of uniformity in evaluating productive activities compared to their ratings of recipient actions. Overall, the mixed methods research design is seen to have provided a much-needed framework in this process of exploring the validity evidence of the proposed rubric in assessing IC for the multiparty roleplay performances on the socializing task.
|Description:||Ph.D. Thesis. University of Hawaiʻi at Mānoa 2018.|
|Rights:||All UHM dissertations and theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission from the copyright owner.|
|Appears in Collections:||
Ph.D. - Second Language Studies|
Please email firstname.lastname@example.org if you need this content in ADA-compliant format.
Items in ScholarSpace are protected by copyright, with all rights reserved, unless otherwise indicated.