Language proficiency over nonverbal sound effects in children's eBook incidental word learning

Abstract

While multimodal glossing is supported by theories like Mayer’s Cognitive Theory of Multimedia Learning, its effectiveness remains debated due to potential cognitive demands. This study examined how repeated exposures, working memory (WM), and gloss lookup behaviors influence vocabulary retention. Ninety-six Chinese high school students were randomly assigned to four conditions (L2 textual gloss, pictorial gloss, pictorial plus L2 textual gloss, and no gloss). Results showed that (a) repeated exposures to target words enhanced vocabulary learning; (b) repeated exposures to glosses increased processing efficiency (measured by gloss lookup duration); (c) Reading Span was associated with processing efficiency and vocabulary learning; (d) Non-word Repetition’s impact on processing efficiency was reversed with repeated gloss exposures, while its impact on vocabulary learning outcomes was complex; (e) pictorial glosses were associated with the highest processing efficiency, while the Multimodal Gloss Condition showed a significant increase in processing efficiency with repeated exposures, and (f) metacognitive strategy use interacted complexly with cognitive engagement with input. These findings suggest that repeated exposure to glosses can mitigate the cognitive demands of multimodal inputs, offering valuable pedagogical insights for the design of effective glossing strategies in multimedia learning contexts.