The integration of auditory and textual input in vocabulary learning from subtitled viewing: An eye-tracking study

Abstract

Numerous studies have documented the benefits of watching audio-visual materials with on-screen text for L2 vocabulary learning (Montero Perez, 2022). The provision of both auditory and textual input allows learners to link auditory and written forms (or L1 meanings) of unknown words during viewing, which could potentially facilitate vocabulary learning. However, little is known about the dynamics of text-audio synchrony in subtitled viewing and how the processing of written words in relation to the audio may lead to vocabulary learning. Eighty-one intermediate-to-advanced Chinese learners of English watched an English documentary with one of three on-screen texts (i.e., captions, L1 subtitles, and bilingual subtitles), while their eye movements were monitored. Participants’ awareness of 17 unknown words and vocabulary learning gains were assessed via stimulated recalls and three vocabulary tests. Results revealed that captions facilitated text-audio synchronisation, whereas L1 subtitles generally led to reading ahead and skipping. Bilingual subtitles enabled synchronisation of L1 translations with L2 audio but often resulted in skipping L2 forms. Most text-audio processing behaviours led to moderate predicted probabilities of vocabulary learning and participants’ reported awareness, with no significant within-group difference, except for the processing of L2 unknown words in bilingual subtitles.