Towards micro foundation models for robust multimodal IoT sensing

The thesis argues and advocates for the feasibility and utility of micro foundation models (µFMs), a key direction for future smart IoT/CPS systems that exploits advances in self-supervised pre-training to support robust intelligent inference tasks. We demonstrate key beneficial properties such as latent representation independence from the downstream task, robustness to domain shifts, ability to learn from unlabeled data, and enhanced structural resiliency of edge systems. Importantly, we demonstrate the emergence of these properties after pre-training with only moderate amounts of unlabeled data, earning the name µFMs. To make the argument, evaluate model efficacy, and surface some of the underlying challenges, this thesis describes a vibration-based µFM, called VibroFM, pre-trained with moderate amounts of unlabeled acoustic and seismic sensing data, to support target classification and tracking applications. VibroFM is pre-trained in an environment-agnostic fashion using unlabeled sensor data. It can then be fine-tuned to a given deployment using a small amount of in-situ labeled data. The paper shows that VibroFM (i) improves the robustness of several downstream tasks, (ii) efficiently adapts to different environmental conditions (using only small amounts of fine-tuning), (iii) allows few-shot generalization to unseen targets, and (iv) generalizable to multiple downstream tasks each at a minimal labeling and system cost. We further show that VibroFM can execute in real-time on embedded sensor nodes. We compare the robustness and performance of VibroFM to conventional supervised deep neural networks, showing the advantages of the former. Combined with the feasibility of executing µFMs in resource-limited settings and the sufficiency of only moderate amounts of data for their pre-training, we conclude the importance of micro foundation models as a promising research direction for the IoT/CPS community.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Kimura, Tomoyoshi

Contributors dc:contributor

• Abdelzaher, Tarek

• Foundation Models
• Internet Of Things
• Self-supervised Learning

Statement dc:rights

• Copyright 2024 Tomoyoshi Kimura

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/127289

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F127289