Unsupervised sound separation

In this thesis, we tackle the problem of training a machine to perceive, disentangle and reconstruct independent source waveforms from a given mixture audio recording without the need of explicit supervision. The problem becomes even more apparent with current state-of-the-art approaches which are largely dependent on the existence of vast amounts of carefully curated data. In essence, this thesis presents a holistic approach on how people can develop sound separation algorithms based on neural networks which are able to scale up to multiple users, modalities and datasets without the need of annotated data. To that end, the contributions of this thesis is threefold. The first part of the thesis describes novel unsupervised and self-supervised algorithms for sound source separation problems under a wide spectrum of environmental setups. The second part aims to expand the applicability of sound separation systems using external condition information (e.g. video, text and other semantic discriminative concepts) which consists of the multi-modal aspect of this work. Finally, the last chapter presents potential obstacles towards the deployment of the aforementioned algorithms (e.g. scarcity of labels, lack of data on the same device during training, limited computational resources, reluctance of the users to share their private data, erroneous predictions, etc.) as well as proposes novel solutions which can be seamlessly integrated into their real-world implementations.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Tzinis, Efthymios

Contributors dc:contributor

• Smaragdis, Paris
• Hasegawa-Johnson, Mark
• Misailovic, Sasa
• Hershey, John R

• Sound Separation
• Audio-visual Perception
• Unsupervised Learning
• Self-supervised Learning
• Speech Enhancement
• Efficient Neural Networks
• Federated Learning

Statement dc:rights

• Copyright 2023 Efthymios Tzinis

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/120294

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F120294