Skip to main content

Research Repository

Advanced Search

What iconic gesture fragments reveal about gesture-speech integration: when synchrony is lost, memory can help.

Obermeier, Christian; Gunter, Thomas C.; Holle, Henning

Authors

Christian Obermeier

Thomas C. Gunter

Profile Image

Dr Henning Holle H.Holle@hull.ac.uk
Reader in Psychology / Leader of Cognitive and Clinical Neuroscience group (https://www.hull.ac.uk/neuroscience)



Abstract

The present series of experiments explores several issues related to gesture-speech integration and synchrony during sentence processing. To be able to more precisely manipulate gesture-speech synchrony, we used gesture fragments instead of complete gestures, thereby avoiding the usual long temporal overlap of gestures with their coexpressive speech. In a pretest, the minimal duration of an iconic gesture fragment needed to disambiguate a homonym (i.e., disambiguation point) was therefore identified. In three subsequent ERP experiments, we then investigated whether the gesture information available at the disambiguation point has immediate as well as delayed consequences on the processing of a temporarily ambiguous spoken sentence, and whether these gesture-speech integration processes are susceptible to temporal synchrony. Experiment 1, which used asynchronous stimuli as well as an explicit task, showed clear N400 effects at the homonym as well as at the target word presented further downstream, suggesting that asynchrony does not prevent integration under explicit task conditions. No such effects were found when asynchronous stimuli were presented using a more shallow task (Experiment 2). Finally, when gesture fragment and homonym were synchronous, similar results as in Experiment 1 were found, even under shallow task conditions (Experiment 3). We conclude that when iconic gesture fragments and speech are in synchrony, their interaction is more or less automatic. When they are not, more controlled, active memory processes are necessary to be able to combine the gesture fragment and speech context in such a way that the homonym is disambiguated correctly. © 2011 Massachusetts Institute of Technology.

Citation

Obermeier, C., Gunter, T. C., & Holle, H. (2011). What iconic gesture fragments reveal about gesture-speech integration: when synchrony is lost, memory can help. Journal of cognitive neuroscience, 23(7), 1648-1663. https://doi.org/10.1162/jocn.2010.21498

Journal Article Type Article
Acceptance Date Jan 1, 2011
Online Publication Date May 10, 2011
Publication Date Jul 1, 2011
Publicly Available Date Mar 29, 2024
Journal Journal of cognitive neuroscience
Print ISSN 0898-929X
Electronic ISSN 1530-8898
Publisher Massachusetts Institute of Technology Press
Peer Reviewed Peer Reviewed
Volume 23
Issue 7
Pages 1648-1663
DOI https://doi.org/10.1162/jocn.2010.21498
Keywords gesture-speech integration, disambiguation, synchronization, memory, iconic co-speech gestures, N400
Public URL https://hull-repository.worktribe.com/output/409726
Publisher URL https://www.mitpressjournals.org/doi/10.1162/jocn.2010.21498
PMID 20350188