Visual cues in adaptation to noise-vocoded speech (Trotter et al., 2021)

Purpose: This study first aimed to establish whether viewing specific parts of the speaker’s face (eyes or mouth), compared to viewing the whole face, affected adaptation to distorted noise-vocoded sentences. Second, this study also aimed to replicate results on processing of distorted speech from lab-based experiments in an online setup.

Method: We monitored recognition accuracy online while participants were listening to noise-vocoded sentences. We first established if participants were able to perceive and adapt to audiovisual four-band noise-vocoded sentences when the entire moving face was visible (AV Full). Four further groups were then tested: a group in which participants viewed the moving lower part of the speaker’s face (AV Mouth), a group in which participants only see the moving upper part of the face (AV Eyes), a group in which participants could not see the moving lower or upper face (AV Blocked), and a group in which participants saw an image of a still face (AV Still).

Results: Participants repeated around 40% of the key words correctly and adapted during the experiment, but only when the moving mouth was visible. In contrast, performance was at floor level, and no adaptation took place, in conditions when the moving mouth was occluded.

Conclusions: The results show the importance of being able to observe relevant visual speech information from the speaker’s mouth region, but not the eyes/upper face region, when listening and adapting to distorted sentences online. Second, the results also demonstrated that it is feasible to run speech perception and adaptation studies online, but that not all findings reported for lab studies replicate.

Supplemental Material S1. LMER summary table for the main effect of Trial in the AV Full condition, reporting the model estimate, standard error of the estimate, degrees of freedom for the fixed factor, and the t- and p-values.

Supplemental Material S2. LMER summary table, reporting the reference level of the factor block, the model estimate, standard error of the estimate, degrees of freedom for the fixed factor, and the t- and p-values.

Supplemental Material S3. Full LMER summary table, reporting the reference level of the factors block and Condition, the model estimate, standard error of the estimate, degrees of freedom for the fixed factor, and the t- and p-values.

Supplemental Material S4. Full LMER summary table, reporting the reference level of the factors trial and Condition, the model estimate, standard error of the estimate, degrees of freedom for the fixed factor, and the t- and p-values.

Trotter, A. S., Banks, B., & Adank, P. (2021). The relevance of the availability of visual speech cues during adaptation to noise-vocoded speech. Journal of Speech, Language, and Hearing Research. Advance online publication. https://doi.org/10.1044/2021_JSLHR-20-00575