Assessing interactional competence in a multiparty roleplay task: A mixed-methods study
Date
2018-12
Authors
Contributor
Advisor
Department
Instructor
Depositor
Speaker
Researcher
Consultant
Interviewer
Narrator
Transcriber
Annotator
Journal Title
Journal ISSN
Volume Title
Publisher
Volume
Number/Issue
Starting Page
Ending Page
Alternative Title
Abstract
In an effort to develop an assessment instrument in measuring interactional competence (IC) with a method that is congruent with the current research findings on IC and IC development (e.g., Hall, Hellermann, & Pekarek Doehler, 2011; Pekarek Doehler & Pochon-Berger, 2015), the present study investigated students’ performances on a multiparty roleplay on a task called Socializing. Using the sequential mixed methods design (Greene, 2007; Tashakkori & Teddlie, 2003), the study explored empirical evidence garnered through qualitative and quantitative research methods to test if the proposed rubric can provide a valid and reliable measurement of IC on this performance assessment task.
The participants of this study were 180 undergraduate engineering students at a university in Thailand who were taking an EFL course that targets social communication skills in professional contexts. Students were randomly grouped together and were asked to have a conversation for 10 minutes, in a roleplay task which they must introduce themselves as their character and try establishing business contacts for their hypothetical companies. The data for this study included 34 video-recordings of the group roleplay performances.
Conversation analysis (Clift, 2016; Sacks, 1992; Schegloff, 2007; Sidnell & Stivers, 2013) was employed to identify comparable interactional activities and determine the interactional methods students utilized in carrying out those activities. The productive activities are self-introduction, work talk, business contact exchange, post-conference arrangement talk, and an interaction to bring about the termination of the roleplay performance. Three recipient actions include students’ management and display of their understanding, students’ management of alignment, and finally, their display of affiliative stance.
Six raters from various teaching and training backgrounds were recruited to apply the proposed rubric in evaluating the students’ IC on the eight items, combining both productive and recipient actions. The Many-Faceted Rasch Measurement (Linacre, 1989) with the Partial Credit Scoring model (Masters, 1982) provided integrated measurement reports of the rating practice. The findings revealed that students’ ability on this IC construct mostly exceed the difficulty of the socializing task. Self-introduction and understanding display had been identified as the two easiest items, followed by alignment display, work talk, affiliation display, activity termination, making post-conference arrangements, and bringing up contact exchange, respectively. The analysis also suggested that most raters were reliable in applying the rating scale, though they demonstrated a higher degree of uniformity in evaluating productive activities compared to their ratings of recipient actions. Overall, the mixed methods research design is seen to have provided a much-needed framework in this process of exploring the validity evidence of the proposed rubric in assessing IC for the multiparty roleplay performances on the socializing task.
Description
Keywords
English as a second language, Educational tests & measurements, FACETS, Interactional competence, Mixed methods research, Rasch, Rubric development
Citation
Extent
218 pages
Format
Geographic Location
Time Period
Related To
Related To (URI)
Table of Contents
Rights
All UHM dissertations and theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission from the copyright owner.
Rights Holder
Local Contexts
Collections
Email libraryada-l@lists.hawaii.edu if you need this content in ADA-compliant format.