Patronus: multi-modal sensing, analytics, and localization assistance for heterogeneous working environments

Today’s “Jarvis-like” AI assistants excel at generic, consumer-oriented tasks, yet they remain ill-suited for heterogeneous working environments—dynamic laboratories and industrial facilities where hands-on professionals must juggle safety-critical processes, rapidly changing context, and a deluge of multimodal data. This thesis argues that effective assistance in these settings hinges on the tight co-design of three pillars: (i) a low-cost, scalable, and evolvable sensing infrastructure, (ii) a trustworthy real-time analytics pipeline, and (iii) an accurate, practical localization layer that enables context-aware humandata interaction. In this thesis, we introduce Patronus, a modular framework composed of five interoperable systems that collectively satisfy these requirements. SENSELET++ deploys a plug-and-play sensor network and anomaly analytics for scalable environmental monitoring. MachineStethoscope enables on-device, unsupervised health monitoring for legacy rotating machinery. WeldMon fuses heterogeneous signals and introduces synthetic fault augmentation to improve failure prediction in ultrasonic welding. GaugeTracker digitizes analog gauges entirely on low-cost IoT hardware, leveraging multiple vision and vision language models for robust transcription. Finally, AnyLoc provides energy-efficient visual localization that operates under low-resolution and low-light conditions in cluttered indoor scenes. Together, these systems power MAINTGlasses, a hands-free smart-glasses interface that delivers spatially relevant insights to professionals in real time. Deployments across cleanrooms, nanofabrication labs, and server rooms demonstrate that Patronus fosters safer, faster, and more efficient workflows. By unifying sensing, analytics, and localization, this work charts a practical path toward intelligent environments that actively understand and support complex human work.

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Tian, Beitong

Contributors dc:contributor

• Nahrstedt, Klara
• Caesar, Matthew
• Soltanaghai, Elahe
• Shenoy, Prashant

• Internet Of Things
• Smart Glasses
• Multimodal Sensing
• Condition Monitoring
• Environmental Monitoring
• Visual Localization
• Intelligent Environments

Statement dc:rights

• Copyright © 2025 Beitong Tian. All rights reserved.

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/129844

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F129844