A survey of IMU based cross-modal transfer learning in human activity recognition

Despite living in a multi-sensory world, most AI models are limited to textual and visual understanding of human motion and behavior. Inertial measurement units (IMUs) provide a salient signal to understand human motion; however, in practice, they have been understudied due to numerous difficulties, including the uniterpretability and lack of data. In fact, full situational awareness of human motion could best be understood through a combination of sensors. In this survey, we investigate how knowledge can be transferred and utilized amongst modalities for Human Activity or Action Recognition (HAR), i.e. cross-modality transfer learning. We motivate the importance and potential of IMU data and its applicability in cross-modality learning as well as the importance of studying the HAR problem. We categorize HAR related tasks by time and abstractness and then compare various types of multimodal HAR datasets. We also distinguish and expound on many related but inconsistently used terms in the literature, such as transfer learning, domain adaptation, representation learning, sensor fusion, and multimodal learning, and describe how cross-modal learning fits with all these concepts. Then, we review the literature on IMU-based cross-modal transfer for HAR. The two main approaches for cross-modal transfer are instance-based transfer, where instances of one modality are mapped to another (e.g. knowledge is transferred in the input space), or feature-based transfer, where the model relates the modalities in an intermediate latent space (e.g. knowledge is transferred in the feature space). Finally, we discuss future research directions and applications in cross-modal HAR.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Kamboj, Abhi

Contributors dc:contributor

• Do, Minh

• Inertial Measurement Units
• Human Action Recognition
• Transfer Learning
• Cross-modal Learning
• Multimodal Learning
• Sensor Fusion

Statement dc:rights

• Copyright 2024 Abhi Kamboj

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/124581

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F124581