When the Brain Plays a Game: Neural Responses to Visual Dynamics during Naturalistic Visual Tasks
type: CreativeWork
description: Many day-to-day tasks involve processing of complex visual information in a continuous stream. While much of our knowledge on visual processing has been established from reductionist approaches in lab-controlled settings, very little is known about the processing of complex dynamic stimuli experienced in everyday scenarios. Traditional investigations employ event-related paradigms that involve presentation of simple stimuli at select locations in visual space and discrete moments in time. In contrast, visual stimuli in real-life are highly dynamic, spatially-heterogeneous, and semantically rich. Moreover, traditional experiments impose unnatural task constraints (e.g., inhibited saccades), thus, it is unclear whether theories developed under the reductionist approach apply in naturalistic settings. Given these limitations, alternative experimental paradigms and analysis methods are necessary. Here, we introduce a new approach for investigating visual processing, applying the system identification (SI) framework. We investigate the modulation of stimulus-evoked responses during a naturalistic task (i.e., kart race game) using non-invasive scalp recordings. In recent years, multivariate modeling approaches have become increasingly popular for assessing neural response to naturalistic stimuli. Encoding models use stimulus patterns to predict brain responses and decoding models use patterns of brain responses to predict stimulus that drove these responses. In this dissertation, we employ a hybrid method that “encodes” the stimulus to predict “decoded” brain responses. Using this approach, we measure the stimulus-response correlation (SRC), i.e. temporal correlation of neural response and dynamic stimulus. This SRC can be used to assess the strength of stimulus-evoked activity to uniquely experienced naturalistic stimulus. To demonstrate this, we measured the SRC during a kart race videogame. We find that SRC increased with active play of the game, suggesting that stimul…
abstract: Many day-to-day tasks involve processing of complex visual information in a continuous stream. While much of our knowledge on visual processing has been established from reductionist approaches in lab-controlled settings, very little is known about the processing of complex dynamic stimuli experienced in everyday scenarios. Traditional investigations employ event-related paradigms that involve presentation of simple stimuli at select locations in visual space and discrete moments in time. In contrast, visual stimuli in real-life are highly dynamic, spatially-heterogeneous, and semantically rich. Moreover, traditional experiments impose unnatural task constraints (e.g., inhibited saccades), thus, it is unclear whether theories developed under the reductionist approach apply in naturalistic settings. Given these limitations, alternative experimental paradigms and analysis methods are necessary. Here, we introduce a new approach for investigating visual processing, applying the system identification (SI) framework. We investigate the modulation of stimulus-evoked responses during a naturalistic task (i.e., kart race game) using non-invasive scalp recordings. In recent years, multivariate modeling approaches have become increasingly popular for assessing neural response to naturalistic stimuli. Encoding models use stimulus patterns to predict brain responses and decoding models use patterns of brain responses to predict stimulus that drove these responses. In this dissertation, we employ a hybrid method that “encodes” the stimulus to predict “decoded” brain responses. Using this approach, we measure the stimulus-response correlation (SRC), i.e. temporal correlation of neural response and dynamic stimulus. This SRC can be used to assess the strength of stimulus-evoked activity to uniquely experienced naturalistic stimulus. To demonstrate this, we measured the SRC during a kart race videogame. We find that SRC increased with active play of the game, suggesting that stimul…
author: Ki, Jason
keywords: Neuroscience, Naturalistic Vision, Visual Attention, EEG, Bioelectrical and Neuroengineering, Bioimaging and Biomedical Optics, Cognitive Neuroscience, Computational Neuroscience, Vision Science
date published: 2021-01-01T08:00:00Z
date modified: 2026-07-24T01:57:28Z
provider: City University of New York - City College
publisher: Networked Digital Library of Theses and Dissertations
type: WebPage
source organization: City University of New York - City College