Spatiotemporal Neural Dynamics of Audiovisual Perception and Cross-modal Mental Imagery

In everyday life, we constantly perceive multisensory events that contain both visual and auditory information. However, how the neural activity and representations unfold during the perception of naturalistic audiovisual stimuli remains less understood. Beyond perception, we often recall past experiences and reconstruct them through mental imagery. The extent to which perception and recall share neural substrates, and how their spatiotemporal dynamics and spectral mechanisms differ, remains unclear. This thesis addresses this gap by leveraging naturalistic audiovisual stimuli and combining the high spatial resolution of fMRI with the high temporal resolution of EEG to examine how the brain processes real-world audiovisual events and recalls them by cross-modal information. First, we characterized where and when different levels of stimulus information are processed in the brain. Our results reveal early asymmetrical cross-modal interaction, with acoustic information represented in both early visual and auditory regions, while visual information only identified in visual cortices. Low-level features were processed with similar onset but distinct temporal dynamics, whereas higher-level categorical and semantic information emerged later in multisensory association areas. Comparing neural representations to a two-branch deep neural network model highlighted the necessity of early cross-modal connections to build a biologically plausible model of audiovisual perception. With EEG-fMRI fusion, we provided a spatiotemporally resolved account of neural activity during the processing of naturalistic audiovisual stimuli. Next, we investigated cross-modal memory recall and imagery using time-resolved fMRI multivariate pattern analyses. When participants recalled visual contents from sounds or recalled sounds from silent videos, we observed reversed spatiotemporal hierarchies in corresponding sensory areas. In visual recall and imagery, the activity in the primary visual cortex was delayed relative to higher-level visual regions; in auditory recall, the typical bottom-up progression during perception was absent and the primary auditory cortex was not engaged. In addition, we identified the engagement and temporal dynamics of hippocampal, parahippocampal, retrosplenial, and precuneus regions. Finally, we exploited the fine temporal resolution of EEG to examine the temporal dynamics and spectral signatures of audiovisual perception and cross-modal mental imagery. We found perceptual processes produced rapid but transient patterns, while memory reconstruction processes evoked more sustained and temporally generalized patterns. Importantly, modality-specific differences emerged, with auditory imagery showing broad generalization earlier in time compared to visual imagery. Time-frequency analyses revealed frequency-specific contributions: theta-band activity supported early perceptual encoding, whereas alpha-band oscillations underpinned the sustained maintenance of both externally perceived and internally generated representations. Together, this work provides an integrated account of how the brain encodes real-world audiovisual events and reconstructs them during memory recall and imagery, advancing our understanding of multisensory perception and memory.

Name thesis:degree_name

Ph D

Discipline thesis:degree_discipline

Neuroscience

Grantor dc:publisher

The University of Western Ontario

Year dc:date.issued

2026

Author dc:creator

• Hu, Yu

Advisors dc:contributor.advisor

• Mohsenzadeh, Yalda
• Diedrichsen, Jörn

• Multisensory Perception
• Memory Recall
• Mental Imagery
• Neural Dynamics
• fMRI
• EEG

Language dc:language.iso

en

Handle dc:identifier.uri

https://hdl.handle.net/20.500.14721/39588

OAI identifier oai:identifier

oai:uwo.scholaris.ca:20.500.14721/39588