The influence of semantics on the visual processing of natural scenes

A long standing question in cognitive science has been: is visual processing completely encapsulated and separate from semantics or can visual processing be influenced by semantics? We address this question in two ways: 1) Do pictures and words share similar representations and 2) Does semantics modulate visual processing. Using multi-voxel pattern analysis (MVPA) and fMRI decoding we examined the similarity of neural activity across pictures and words that describe natural scenes. A whole brain MVPA searchlight revealed multiple brain regions in the occipitotemporal, posterior parietal and frontal cortices that showed transfer from pictures to words and from words to pictures. In addition to sharing similar representations across pictures and words, can words dynamically influence the processing of visual stimuli? Using Event Related Potentials (ERPs) and good and bad exemplars of natural scenes, we show that top-down expectation, initiated via a category cue (e.g. the word ‘Beach’), dynamically influences the processing of natural scenes. Good and bad exemplars first evoked differential ERPs in the time-window 250-350 ms from stimulus onset, with the bad exemplars showing greater negativity over frontal electrode sites, when the cue matched the image. Interestingly, this good/bad effect disappeared when the images were mismatched to the cue. Overall, these studies taken together, provide evidence for the influence of semantics on the visual processing of natural scenes.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Neuroscience

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2017

Author dc:creator

• Kumar, Manoj

Contributors dc:contributor

• Beck, Diane M.
• Federmeier, Kara D.
• Gratton, Gabriele
• Koyejo, Oluwasanmi

• Natural scenes
• Semantics
• Statistical regularities
• Multi-voxel pattern analysis (MVPA)
• Era

Statement dc:rights

• Copyright 2017 Manoj Kumar

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/98123

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/98123