Efficient audio-visual representations for reasoning and synthesis tasks

Events that occur in the real world often leave acoustic and visual imprints. Highly evolved organisms, such as human beings, have capabilities to perceive such events jointly across multiple modalities (such as audio and video). This allows for faster and more accurate perception with the possibility of supplementing information lacunae in one modality by another. As we seek to enable artificial intelligence (AI) systems to complement humans in their endeavors, it is therefore critical that they be equipped with such mulitmodal reasoning capabilities as well and be able to undertake tasks that humans usually solve effortlessly. Towards this end, this dissertation explores reasoning and synthesis tasks in the audio-visual space, particularly, the tasks of: (a) disambiguating mixed/noisy audio by leveraging video, and (b) being able to predict a video from audio. This dissertation proposes methods to tackle these challenges while striving to ensure that the proposed solutions be deployable in resource-constrained environments where there might be an inadequacy of high-performance computing resources or a paucity of training data. Concretely this dissertation makes the following four contributions: (i) a novel geometry-aware sparse scene graph based representation is proposed to undertake audio-source separation given videos of the audio sources in their natural settings; (ii) a multimodal, variational encoder-decoder model, called Sound2Sight, is introduced to synthesize the frames of a video given the audio coherently; (iii) an improved regime for training, unimodal, and audio-conditioned frame prediction systems, which factors in the predictive uncertainty of the model, is put forth. This adaptation results in such models requiring lesser data and fewer epochs for training; (iv) finally, a method to compress the dominant image representation tool in multimodal deep neural networks, Convolutional Neural Network (CNN), is presented so that they might operate in environments with lower computation capacity by exploiting filter activation patterns and inter-filter weight dependencies.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2022

Author dc:creator

• Chatterjee, Moitreya

Contributors dc:contributor

• Ahuja, Narendra
• Hasegawa-Johnson, Mark A
• Do, Minh N
• Gupta, Saurabh
• Wang, Yuxiong
• Owens, Andrew
• Harwath, David

• Audio-visual
• Scene Understanding
• Multimodal
• Frame Generation
• Audio Source Separation
• Machine Learning
• Computer Vision

Statement dc:rights

• Copyright 2022 Moitreya Chatterjee

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/117723

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F117723