University of Hull logo

Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions

Holle, Henning; Obleser, Jonas; Rueschemeyer, Shirley-Ann; Gunter, Thomas C.

Authors

Dr Henning Holle H.Holle@hull.ac.uk
Senior Lecturer in Psychology / Leader of Cognitive and Clinical Neuroscience group ( www.hull.ac.uk/neuroscience )

Jonas Obleser

Shirley-Ann Rueschemeyer

Thomas C. Gunter

Abstract

Iconic gestures are spontaneous hand movements that illustrate certain contents of speech and, as such, are an important part of face-to-face communication. This experiment targets the brain bases of how iconic gestures and speech are integrated during comprehension. Areas of integration were identified on the basis of two classic properties of multimodal integration, bimodal enhancement and inverse effectiveness (i.e., greater enhancement for unimodally least effective stimuli). Participants underwent fMRI while being presented with videos of gesture-supported sentences as well as their unimodal components, which allowed us to identify areas showing bimodal enhancement. Additionally, we manipulated the signal-to-noise ratio of speech (either moderate or good) to probe for integration areas exhibiting the inverse effectiveness property. Bimodal enhancement was found at the posterior end of the superior temporal sulcus and adjacent superior temporal gyrus (pSTS/STG) in both hemispheres, indicating that the integration of iconic gestures and speech takes place in these areas. Furthermore, we found that the left pSTS/STG specifically showed a pattern of inverse effectiveness, i.e., the neural enhancement for bimodal stimulation was greater under adverse listening conditions. This indicates that activity in this area is boosted when an iconic gesture accompanies an utterance that is otherwise difficult to comprehend. The neural response paralleled the behavioral data observed. The present data extends results from previous gesture-speech integration studies in showing that pSTS/STG plays a key role in the facilitation of speech comprehension through simultaneous gestural input.

Journal Article Type Article
Publication Date 2010-01
Journal Neuroimage
Print ISSN 1053-8119
Publisher Elsevier
Peer Reviewed Peer Reviewed
Volume 49
Issue 1
Pages 875-884
Institution Citation Holle, H., Obleser, J., Rueschemeyer, S., & Gunter, T. C. (2010). Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. NeuroImage, 49(1), (875-884). doi:10.1016/j.neuroimage.2009.08.058. ISSN 1053-8119
DOI https://doi.org/10.1016/j.neuroimage.2009.08.058
Keywords Cognitive Neuroscience; Neurology