Towards efficient and powerful machine learning vision systems for mobile: unifying interactive segmentation and matting

Recent advancements in hardware have significantly enhanced the capabilities of smartphones, tablets and head-mounted devices, transforming these mobile devices from mere communication tools into powerful multimedia and creative platforms. This transformation has created unprecedented demand for advanced image editing functionalities running behind mobile applications. Today, users no longer like waiting until they return home to their desktop computers and launch software like Photoshop for sophisticated image manipulation. Instead, they expect equally powerful editing features that allow immediate modifications at the moment they capture or discover content worth sharing on their mobile devices. These features empower users across various contexts, from social media creators to professionals, profoundly influencing the way visual content is produced, consumed, and shared, and more importantly, the way how people interact, connect, and share their interests and cherished moments with those they love. These image editing functionalities are largely backed with recent innovations in efficient deep learning in computer vision areas. However, deploying heavy machine learning algorithms efficiently on mobile platforms remains challenging due to inherent constraints such as limited computation power, battery capacity, and the need for real-time respond. These resource limitations directly affect user experience, making it essential to develop highly optimized image editing algorithms specifically for the mobile environment. These algorithms must deliver high accuracy, high computational efficiency, low latency, real-time user interactivity, and robust generalization across diverse visual contents. In this thesis, we focus on segmentation and matting, which serve as foundational editing techniques for background replacement, portrait, and precise foreground object extraction, etc. We develop efficient neural network based algorithms designed for the resource-constrained mobile environment. Moreover, we tailor our editing functionalities to practical user scenarios, adding interactivity in segmentation tasks and reducing input requirements for matting processes, for better user experience and compatibility across wider range of mobile devices. Specifically, TraceNet ackles efficient and interactive instance segmentation by explicitly locating the user-selected instance via receptive field tracing, and thus significantly reduce computations in neural networks. Meanwhile, I-Matting addresses trimap-free image matting via a hierarchical adversarial training mechanism and a patch-rank component, and improves the matting accuracy while reducing computations. Collectively, these methods constitute a meaningful step towards efficient mobile-based image editing, demonstrating their effectiveness through accuracy and efficiency improvements in extensive datasets.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Wu, Mingyuan

Contributors dc:contributor

• Nahrstedt, Klara

• Machine Learning
• Segmentation
• Matting

Statement dc:rights

• Copyright 2025 Mingyuan Wu

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/129245

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F129245