Multimodal content analysis and streaming in multiview environment

A multiview environment where cameras are deployed to capture the same scene is essential in countless applications. However, the complexity of its content analysis and delivery requires intricate design. The complexities could be summarized into two folds: 1) what to deliver at every timestamp, and 2) how to deliver under the constraint of bandwidth. This thesis attempts to leverage learning-based algorithms to analyze the multimodal cues, including visual and audio cues, in the multiview environment to help multiview content delivery that satisfies audience preferences and needs. Based on the nature of multiview content, we divide the research scope into on-demand and real-time content analysis and delivery. We start from on-demand multiview content analysis, study multi-camera view recommendation, and move on to on-demand content delivery of 360◦ videos. Then, considering the capricious characteristics of real-time multiview content, we investigate how continual learning on pre-trained models could aid in its analysis. Finally, we investigate speaking intent detection in 360◦ video streaming, which could be leveraged to distribute limited bandwidth over 360◦ video frames and deliver a more immersive experience in virtual reality meetings. Different aspects investigated in this thesis could provide insights into how various types of multi-view content could be analyzed and delivered to satisfy users’ potential multi-faceted needs.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Lee, Kuan-Ying

Contributors dc:contributor

• Nahrstedt, Klara
• Wang, Yu-Xiong
• Do, Minh
• Cesar, Pablo

• Learning-based Methods
• Multiview
• Multimodal
• Streaming

Statement dc:rights

• Copyright 2025 Kuan-Ying Lee

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/129862

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F129862