Hand gestures as visual prosody: BOLD responses to audio–visual alignment are modulated by the communicative nature of the stimuli

Biau, Emmanuel; Morís Fernández, Luis; Holle, Henning; Avila, César; Soto-Faraco, Salvador

Abstract

During public addresses, speakers accompany their discourse with spontaneous hand gestures (beats) that are tightly synchronized with the prosodic contour of the discourse. It has been proposed that speech and beat gestures originate from a common underlying linguistic process whereby both speech prosody and beats serve to emphasize relevant information. We hypothesized that breaking the consistency between beats and prosody by temporal desynchronization, would modulate activity of brain areas sensitive to speech-gesture integration. To this aim, we measured BOLD responses as participants watched a natural discourse where the speaker used beat gestures. In order to identify brain areas specifically involved in processing hand gestures with communicative intention, beat synchrony was evaluated against arbitrary visual cues bearing equivalent rhythmic and spatial properties as the gestures. Our results revealed that left MTG and IFG were specifically sensitive to speech synchronized with beats, compared to the arbitrary vision-speech pairing. Our results suggest that listeners confer beats a function of visual prosody, complementary to the prosodic structure of speech. We conclude that the emphasizing function of beat gestures in speech perception is instantiated through a specialized brain network sensitive to the communicative intent conveyed by a speaker with his/her hands.

Citation

Biau, E., Morís Fernández, L., Holle, H., Avila, C., & Soto-Faraco, S. (2016). Hand gestures as visual prosody: BOLD responses to audio–visual alignment are modulated by the communicative nature of the stimuli. NeuroImage, 132, 129-137. https://doi.org/10.1016/j.neuroimage.2016.02.018

• Journal Article Type: Article
• Acceptance Date: Feb 9, 2016
• Online Publication Date: Feb 15, 2016
• Publication Date: May 15, 2016
• Deposit Date: Feb 17, 2016
• Publicly Available Date: Feb 17, 2016
• Journal: NeuroImage
• Print ISSN: 1053-8119
• Electronic ISSN: 1095-9572
• Publisher: Elsevier
• Peer Reviewed: Peer Reviewed
• Volume: 132
• Pages: 129-137
• DOI: https://doi.org/10.1016/j.neuroimage.2016.02.018
• Keywords: Speech perception, Gestures, Audiovisual speech, Multisensory integration, MTG, fMRI
• Public URL: https://hull-repository.worktribe.com/output/385528
• Publisher URL: http://www.sciencedirect.com/science/article/pii/S1053811916001257
• Additional Information: This article is maintained by: Elsevier; Article Title: Hand gestures as visual prosody: BOLD responses to audio–visual alignment are modulated by the communicative nature of the stimuli; Journal Title: NeuroImage; CrossRef DOI link to publisher maintained version: http://dx.doi.org/10.1016/j.neuroimage.2016.02.018; Content Type: article; Copyright: © 2016 Elsevier Inc. All rights reserved.