Week 3
Spoken Language Processing in a Visual Context
The visual world paradigm
Participants hear/produce an utterance while looking at a visual display
While their eye movements are recorded.
Allows to study real-time language comprehension/production in natural tasks.
The visual world paradigm
The visual display
The visual world paradigm
The task
Action-based: "Pick up the..."
Non action-based: "Anne went to Starbucks and bought coffee."
The visual world paradigm
The linking hypothesis
How to link gaze position with language processing?
- Activation of linguistic representation →
- Probability of attention shifted to the corresponding picture →
- Probability of fixation
The visual world paradigm
The linking hypothesis
Word recognition in the visual world
Word recognition in the visual world
Parallel activation during word recognition: Allopenna et al. (1998)
Sentence processing in the visual world
Sentence processing in the visual world
Cooper (1974)
Rapid eye movements to pictures that were named in stories.
Visual attention highly correlated with spoken sentence processing.
Sentence processing in the visual world
Effect of the visual context: Tanenhause et al. (1995)
Significantly more early looks to the empty towel in the one-referent than in the two-referent condition.
Listeners rapidly use visual context to disambiguate linguistic input.
Sentence processing in the visual world
Syntactic ambiguities: Snedeker and Trueswell (2004)
"Tickle/Feel/Choose the frog with the feather."
Sentence processing in the visual world
Syntactic ambiguities: Snedeker and Trueswell (2004)
Sentence processing in the visual world
Incrementality: Altmann and Kamide (1999)
"The boy will eat/move the cake."
Sentence processing in the visual world
Incrementality in sentence processing
A wide array of information including real-world knowledge as well as linguistic knowledge has been shown to contribute to predictive processing.
including linguistic markers of tense, gender, case, etc.; some phonological patterns; and sentential and/or discourse context.
Sentence processing in the visual world
Time course of pragmatic inference: Huang and Snedeker (2009, 2011)
"Point to the girl that has some of the socks"
Speech production in the visual world
Speech production in the visual world
Gleitman et al. (2007)
- Participants were more likely to mention the cued character first.
- And subsequently alter their choice of verb or their syntactic structure (active vs. passive).
- "The guy on the left loses to the guy on the right."
Summary
The visual world paradigm continues to be a popular tool to investigate spoken language processing.
It has its limitations (e.g. the effect of visual context on language processing).
But it's known for the ability to assess time courses of processing
And its ability to address the interplay of language and vision.
Summary
