Volume 29 Number 3, Special Issue: Multimodality in CALL

Permanent URI for this collectionhttps://hdl.handle.net/10125/111242

Browse

Now showing 1 - 12 of 12

Call for papers for a special issue on Robot-assisted language learning: Recent trends and future directions
(University of Hawaii National Foreign Language Resource Center, 2025-10-01) Derakhshan, Ali; Xie, Haoran
Investigating pragmatic learning opportunities and outcomes in different SCMC modes
(University of Hawaii National Foreign Language Resource Center, 2025-10-01) Peng, Yuchen; Lei, Yuhong
Multimodal synchronous-computer-mediated communication (SCMC) provides many possibilities for improving language learners’ pragmatic competence (Maa & Taguchi, 2022). However, the research exploring how SCMC mode affects the pragmatic learning process and its outcome is limited. This study investigates the effects of SCMC modes on pragmatic learning opportunities in interactions and the subsequent outcome of request-making. Forty-eight Chinese EFL learners, divided into video-chat and text-chat groups, conducted four interactional role-play tasks in which they engaged in dyadic discussions on the writing of four requests to a third party. Results indicate that the learners more frequently discussed pragmalinguistics in video rather than text mode. Regarding the outcome of request-making, emojis occurred more frequently after the text chat when the learners were making requests to friends. It was also found that the video-based interactions provided more opportunities for discussing the pragmalinguistic features, which led to increased use of internal modifications to mitigate requests made to high-status requestees; text-based interactions provide opportunities for discussing situational and social features and were related to increased use of upgraders to intensify requests made to friends.
Effects of captioned video on L2 speech segmentation in intermediate learners of Spanish
(University of Hawaii National Foreign Language Resource Center, 2025-10-01) Montero Perez, Maribel; Pattemore, Anastasia
Studies into the role of audiovisual input and captions (i.e., subtitles in the same language as the audio) for L2 learning have shown that such input can positively influence L2 learners’ acquisition of lexical items. Few studies have explored the effects of captioning on learners’ capacity to segment the speech stream. This study investigates the effects of watching an episode of a Spanish TV-series (72 minutes) on speech segmentation among 37 (low-)intermediate L2 learners of Spanish in a captioned (n = 12), uncaptioned (n = 14), and control (n = 11) group. One week before the treatment, participants completed the LexTALE-ESP vocabulary size test. They also took a pretest-posttest shadowing task, which measures learners’ ability to repeat back sentences. It included 20 treatment, 20 related (i.e., from the same TV-series as the treatment), and 20 unrelated (i.e., another TV-series) sentences presented in auditory or audiovisual input mode. Linear mixed-effects models revealed that watching video contributed to learners’ speech segmentation, irrespective of captioning presence. Shadowing task performance improved significantly on treatment and related but not on unrelated sentences. In addition, shadowing task performance was positively influenced by learners’ vocabulary size. Input mode of the shadowing sentences did not influence posttest performance.
Does digital multimodal composing help improve EFL students' metacognitive knowledge?
(University of Hawaii National Foreign Language Resource Center, 2025-10-01) Xu, Wei; Tan, Xiao; Wang, Chaoran; Fu, Dan
Metacognitive knowledge, closely correlated with one’s writing skills, has received increasing research and pedagogical attention. Applying Flavell’s metacognition framework, this study adopts a quasi-experimental design to investigate whether and how a DMC assignment as a pedagogical intervention influences Chinese English as a Foreign Language (EFL) writers’ metacognition knowledge development. A total of 49 EFL college students were divided into the control (n = 25) and the experiment DMC (n = 24) groups. Both groups took pre- and post-intervention surveys on metacognitive knowledge. To understand how the development took place, we also collected and analyzed data on focal students’ learning logs and post-assignment interviews, where students discussed their writing processes in detail. The results suggest that although both groups showed noticeable development in person and strategy knowledge, the DMC group exceeded the traditional writing group in the latter category, which could be attributed to the fact that students in the DMC group were engaged in more decision-making opportunities. This study provides valuable implications on designing DMC assignments in EFL writing classrooms to engage students in frequent reflection, conscious decision-making, and creative expression.
Exploring multimodality in technology-mediated collaborative writing: An activity theory
(University of Hawaii National Foreign Language Resource Center, 2025-10-01) Stell, Annita
Technology-mediated collaborative writing (CW) is a popular pedagogical approach in second language (L2) education. Research shows the potential benefits of co-authoring in pairs using digital devices in various areas. Given its reliance on various modes of communication, further research on the layers of modality could provide further insights into L2 learning opportunities. This study uses a qualitatively driven mixed methods design to explore the CW processes of 12 international undergraduate students as they co-author two source-based argumentative essays in pairs using laptops. Through adopting an Activity Theory framework and concept of multimodality, this study investigates the affordances and constraints of using various mediating tools (language and technology) during the co-composition processes of two essays. Data were collected from audio-recorded collaborative dialogues and keystroke logging files from the Inputlog program generated during the CW activities. Findings reveal how pairs employed different mediating tools according to their pair dynamics and individual writing strategies. Notably, some pairs relied heavily on external tools such as Google search, while others used internal tools such as language and prior knowledge, creating different multimodal L2 learning opportunities.
Language proficiency over nonverbal sound effects in children's eBook incidental word learning
(University of Hawaii National Foreign Language Resource Center, 2025-10-01) Sun, He; Roberts, Adam; Tan, Jessica; Leh, Jieying; Moh, Yvonne Cui Yun
While multimodal glossing is supported by theories like Mayer’s Cognitive Theory of Multimedia Learning, its effectiveness remains debated due to potential cognitive demands. This study examined how repeated exposures, working memory (WM), and gloss lookup behaviors influence vocabulary retention. Ninety-six Chinese high school students were randomly assigned to four conditions (L2 textual gloss, pictorial gloss, pictorial plus L2 textual gloss, and no gloss). Results showed that (a) repeated exposures to target words enhanced vocabulary learning; (b) repeated exposures to glosses increased processing efficiency (measured by gloss lookup duration); (c) Reading Span was associated with processing efficiency and vocabulary learning; (d) Non-word Repetition’s impact on processing efficiency was reversed with repeated gloss exposures, while its impact on vocabulary learning outcomes was complex; (e) pictorial glosses were associated with the highest processing efficiency, while the Multimodal Gloss Condition showed a significant increase in processing efficiency with repeated exposures, and (f) metacognitive strategy use interacted complexly with cognitive engagement with input. These findings suggest that repeated exposure to glosses can mitigate the cognitive demands of multimodal inputs, offering valuable pedagogical insights for the design of effective glossing strategies in multimedia learning contexts.
Multimodal glosses revisited: Interplay of individual differences and repeated exposures
(University of Hawaii National Foreign Language Resource Center, 2025-10-01) Wu, Ying; Liu, Xiaoxue; Zhao, Yuanyuan; Chen, Xinyan; Gao, Jianwu
While multimodal glossing is supported by theories like Mayer’s Cognitive Theory of Multimedia Learning, its effectiveness remains debated due to potential cognitive demands. This study examined how repeated exposures, working memory (WM), and gloss lookup behaviors influence vocabulary retention. Ninety-six Chinese high school students were randomly assigned to four conditions (L2 textual gloss, pictorial gloss, pictorial plus L2 textual gloss, and no gloss). Results showed that (a) repeated exposures to target words enhanced vocabulary learning; (b) repeated exposures to glosses increased processing efficiency (measured by gloss lookup duration); (c) Reading Span was associated with processing efficiency and vocabulary learning; (d) Non-word Repetition’s impact on processing efficiency was reversed with repeated gloss exposures, while its impact on vocabulary learning outcomes was complex; (e) pictorial glosses were associated with the highest processing efficiency, while the Multimodal Gloss Condition showed a significant increase in processing efficiency with repeated exposures, and (f) metacognitive strategy use interacted complexly with cognitive engagement with input. These findings suggest that repeated exposure to glosses can mitigate the cognitive demands of multimodal inputs, offering valuable pedagogical insights for the design of effective glossing strategies in multimedia learning contexts.
The integration of auditory and textual input in vocabulary learning from subtitled viewing: An eye-tracking study
(University of Hawaii National Foreign Language Resource Center, 2025-10-01) Wang, Andi
Numerous studies have documented the benefits of watching audio-visual materials with on-screen text for L2 vocabulary learning (Montero Perez, 2022). The provision of both auditory and textual input allows learners to link auditory and written forms (or L1 meanings) of unknown words during viewing, which could potentially facilitate vocabulary learning. However, little is known about the dynamics of text-audio synchrony in subtitled viewing and how the processing of written words in relation to the audio may lead to vocabulary learning. Eighty-one intermediate-to-advanced Chinese learners of English watched an English documentary with one of three on-screen texts (i.e., captions, L1 subtitles, and bilingual subtitles), while their eye movements were monitored. Participants’ awareness of 17 unknown words and vocabulary learning gains were assessed via stimulated recalls and three vocabulary tests. Results revealed that captions facilitated text-audio synchronisation, whereas L1 subtitles generally led to reading ahead and skipping. Bilingual subtitles enabled synchronisation of L1 translations with L2 audio but often resulted in skipping L2 forms. Most text-audio processing behaviours led to moderate predicted probabilities of vocabulary learning and participants’ reported awareness, with no significant within-group difference, except for the processing of L2 unknown words in bilingual subtitles.
Multimodal effects of processing and learning contextualized L2 vocabulary
(University of Hawaii National Foreign Language Resource Center, 2025-10-01) Malone, Jonathan
This study examined how unimodal and bimodal learning conditions impact L2 vocabulary processing and learning, comparing reading only (RO) and reading while listening (RWL) groups. 119 high intermediate-advanced English learners read or read-while-listening a short story embedded with 25 target pseudowords, repeated 10 times each, and had their eye movements tracked. Processing was defined through two behavioral measures of individual exposure frequency: visit count (VC) to target interest areas and total reading time per visit count (TRT per visit). Three learning posttests measured form and meaning recognition and meaning recall. Processing analysis revealed (a) significantly more visits to targets in RO than RWL, and (b) significantly higher reading time per visit in RWL than RO. Learning gain scores indicated equal or superior outcome scores for RWL compared with RO on all measures. Analyses also revealed that summed VC was a significant predictor of form recognition, while summed TRT was a significant predictor of meaning recall. Results are discussed within theoretical discussions of multimodality research regarding vocabulary processing and learning, particularly in how bimodal conditions in computer-assisted language learning contexts can impact both the quantity and length of exposures, and positively influence word learning.
The effects of input modality on novel L2 vocabulary processing during reading: A closer look into attention and awareness
(University of Hawaii National Foreign Language Resource Center, 2025-10-01) Tuzcu, Aysen
Digital tools such as audiobooks enable language learners to read and listen to a text simultaneously. Access to such bimodal input has been argued to enhance L2 vocabulary learning from reading by directing learners’ attention to novel words and facilitating the development of form-meaning links (Long, 2017). However, research on the cognitive processes underlying vocabulary learning from bimodal input remains limited. This study compared L2 readers’ attention and awareness levels of novel words under reading-only (RO) and reading-while-listening (RWL) conditions. Sixty-three L2 English speakers were randomly assigned to RO and RWL groups and read a 9500-word text containing 24 target pseudowords over two days while their eye movements were recorded. Attention was assessed through eye-tracking data, while awareness levels and learning of the target pseudowords were assessed using retrospective verbal reports and a form recognition test. Results indicated similar levels of attention and awareness across both groups. However, a positive relationship between attention and awareness emerged only in the RO group. These results suggest that while bimodal input may not significantly affect L2 readers’ attention or awareness levels, it influences the relationship between these constructs. Additionally, both attention and awareness predicted word form learning, indicating their importance in vocabulary learning.
Ecological semiotics: Multimodality, multilingualism, and situated language learning in the AI era
(University of Hawaii National Foreign Language Resource Center, 2025-10-01) Godwin-Jones, Robert; Robert Godwin-Jones
Using a variety of media beyond text—audio, video, visual—has long been a common practice in second language learning. Digital media has dramatically increased that opportunity. Multimedia-friendly mobile devices provide access to all media formats, leading to the frequent practice of combining modes. At the same time, social media has increasingly seen a mixing of languages along with the use of images, voice, and video. Studies of multimodal enhancements in second language acquisition (SLA) have generally shown positive learning outcomes, although results vary widely depending on conditions. A variety of theories have been developed in support of the mediating and motivating effects of integrating multimedia into language instruction. Those include different theories of semiotics, the science of signs, as well as the use of ecological theories such as sociomaterialism. The use of learners’ first language (L1) in instructed SLA has been shown to be a valuable scaffolding tool (often in combination with multimodality), leading to the practice of pedagogical translanguaging. Since 2022, the availability of generative artificial intelligence (AI) has been shifting the media landscape. As AI systems steadily improve in multimedia capabilities, existing methods and theories on multimedia integration into SLA may need to be revisited. An approach based on ecological semiotics may prove helpful.
Connecting multimodal learning processes and outcomes in CALL: Introduction to the special issue
(University of Hawaii National Foreign Language Resource Center, 2025-10-01) Hui, Bronson; Kessler, Matt

Browse

Recent Submissions