6 files

Visual effect on tone identification (Wei et al., 2022)

Download all (1.7 MB)
posted on 24.10.2022, 22:32 authored by Yanjun Wei, Lin Jia, Fei Gao, Jianqin Wang

Purpose: Previous studies have demonstrated that tone identification can be facilitated when auditory tones are integrated with visual information that depicts the pitch contours of the auditory tones (hereafter, visual effect). This study investigates this visual effect in combined visual–auditory integration with high- and low-variability speech and examines whether one’s prior tonal-language learning experience shapes the strength of this visual effect.

Method: Thirty Mandarin-naïve listeners, 25 Mandarin second language learners, and 30 native Mandarin listeners participated in a tone identification task in which participants judged whether an auditory tone was rising or falling in pitch. Moving arrows depicted the pitch contours of the auditory tones. A priming paradigm was used with the target auditory tones primed by four multimodal conditions: no stimuli (A−V−), visual-only stimuli (A−V+), auditory-only stimuli (A+V−), and both auditory and visual stimuli (A+V+).

Results: For Mandarin naïve listeners, the visual effect in accuracy produced under the cross-modal integration (A+V+ vs. A+V−) was superior to a unimodal approach (A−V+ vs. A−V−), as evidenced by a higher d prime of A+V+ as opposed to A+V−. However, this was not the case in response time. Additionally, the visual effect in accuracy and response time under the unimodal approach only occurred for high-variability speech, not for low-variability speech. Across the three groups of listeners, we found that the less tonal-language learning experience one had, the stronger the visual effect.

Conclusion: Our study revealed the visual–auditory advantage and disadvantage of the visual effect and the joint contribution of visual–auditory integration and high-variability speech on facilitating tone perception via the process of speech symbolization and categorization.

Supplemental Material S1. Intensity, pitch fundamental frequency (fo) of the T2 and T4 starting points and ending points, and tone duration averaged across the four syllables (“ba,” “pa,” “da,” and “ta”) used in the experiment for the high-variability (H+) stimuli and the low-variability (H–) stimuli. 

Supplemental Material S2. A description of the stimuli presented in each of the three incongruent conditions.

Supplemental Material S3. Means, standard deviations (in parentheses), and ANOVA statistics of the accuracy of the Mandarin learners.

Supplemental Material S4. Means, standard deviations (in parentheses), and ANOVA statistics of the RT (ms) of the Mandarin learners. 

Supplemental Material S5. Means and standard deviations (in parentheses) of the accuracy of the native Mandarin listeners.

Supplemental Material S6. Means and standard deviations (in parentheses) of the RT (ms) of the native Mandarin listeners.

Wei, Y., Jia, L., Gao, F., & Wang, J. (2022). Visual–auditory integration and high-variability speech can facilitate Mandarin Chinese tone identification. Journal of Speech, Language, and Hearing Research. Advance online publication. https://doi.org/10.1044/2022_JSLHR-21-00691


This research was supported by Beijing Social Science Foundation for Young Scholars awarded to the first author (20YYC023).