Event processing in the visual world: Projected motion paths during spoken sentence comprehension

Lindsay, Shane; Kamide, Yuki; Kukona, Anuenue; Scheepers, Christoph

Abstract

Motion events in language describe the movement of an entity to another location along a path. In two eye-tracking experiments we found that comprehension of motion events involves the online construction of a spatial mental model that integrates language with the visual world. In the first experiment, participants listened to sentences describing the movement of an agent to a goal location with verbs suggesting a more upwards (e.g., “jump”) or more downwards oriented path (e.g., “crawl”) while concurrently viewing a visual scene depicting the agent, the goal, and some ‘empty space’ in between. We found that in the rare event of fixating the empty space region between agent and goal, visual attention was biased upwards or downwards depending on the kind of verb. In Experiment 2, the sentences were presented concurrently with scenes featuring a central ‘obstruction’ which would not only impose further constraints on verb-related motion paths, but also increase the likelihood of fixating the area in-between the agent and the goal. The results from this experiment corroborated and refined the previous findings. Specifically, eye-movement effects started immediately after hearing the verb and were in line with data from an additional mouse tracking task which encouraged a more explicit spatial re-enactment of the motion event. In revealing how event comprehension operates in the visual world, these findings suggest a mental simulation process whereby spatial details of motion events are mapped onto the world through visual attention. The strength and detectability of such effects in overt eye-movements is constrained by the visual world and the fact that perceivers rarely fixate regions of empty space.

Citation

Lindsay, S., Kamide, Y., Kukona, A., & Scheepers, C. (2016). Event processing in the visual world: Projected motion paths during spoken sentence comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition, 42(5), 804-812. https://doi.org/10.1037/xlm0000199

• Journal Article Type: Article
• Acceptance Date: Aug 19, 2015
• Publication Date: May 1, 2016
• Deposit Date: Sep 15, 2015
• Publicly Available Date: Nov 23, 2017
• Journal: Journal of experimental psychology : learning, memory, and cognition
• Print ISSN: 0278-7393
• Publisher: American Psychological Association
• Peer Reviewed: Peer Reviewed
• Volume: 42
• Issue: 5
• Pages: 804-812
• DOI: https://doi.org/10.1037/xlm0000199
• Keywords: Motion event processing, Sentence comprehension, Spatial processing, Verb semantics, Eye movements
• Public URL: https://hull-repository.worktribe.com/output/379041
• Publisher URL: http://psycnet.apa.org/doiLanding?doi=10.1037%2Fxlm0000199
• Additional Information: Author's accepted manuscript of article which has been accepted for future publication in: Journal of experimental psychology : learning, memory, and cognition. This article may not exactly replicate the final version published in the APA journal. It is not the copy of record. http://www.apa.org/pubs/journals/xlm/
• Contract Date: Sep 15, 2015