Enabling Immersive Experience in 360 Video Streaming with Deep Learning

Virtual Reality (VR) has become increasingly popular in recent years, promoting numerous applications in fields such as education, entertainment, and the military. Among these, 360 video streaming is a prominent VR application. Despite its potential, many challenges exist in the 360 video streaming pipeline that must be addressed to ensure immersive user experiences. This dissertation proposes three research endeavors to address three significant technical challenges and consequently improve user experiences in 360 video streaming. Firstly, the research reveals the uniqueness of human attention in VR, which affects head movement prediction accuracy and, consequently, the efficiency of 360 video viewing. This led to the development of a new panoramic saliency detection approach tailored for 360 videos. Enhanced head movement prediction, augmented with this new saliency detection, results in more bandwidth-efficient 360 video streaming, delivering high-quality content to users. Secondly, the research identifies a privacy risk in VR, the potential leakage of sensitive information via visual side-channels from Head-Mounted Display devices. This exploration led to the development of INTRUDE, a highly accurate attack framework that leverages the unique correlation between a user's head movements and video content to infer video titles. Thirdly, the research sheds light on the issue of disconnected design between video downscaling and enhancement, leading to unsatisfactory 360 video streaming performance. The investigation resulted in the development of JOIN, a joint video downscaling and enhancement framework for neural-enhanced streaming. This end-to-end approach replaces the disconnected design, offering a more cohesive and efficient solution for streaming high-quality 360 videos in low bandwidth conditions. Collectively, these projects advance the state of VR technology, enhancing immersive experiences by safeguarding user privacy and improving video quality.

Author

• Nguyen, Anh Phan

• 360 Video Streaming
• Head Movement Prediction
• Neural Enhancement
• Saliency Modeling
• Tile Adaptation
• Video Identification

Identifier

hdl:1920/13816

OAI identifier oai:identifier

oai:MARS:1920/13816