Using wearable IMUs for multi-modal denoising and tracking

Modern earphones are equipped with microphones and inertial measurement units (IMUs). IMUs are motion sensors used to deduce human activities, such as jogging, falling, and device rotation. These sensors are integrated into many wearable devices, including smartphones, watches, and earphones. While IMUs are well-established sensors, they reveal new potential when utilized on earphone platforms, particularly on the user’s face. Earphone IMUs offer two primary advantages: (a) they can capture jaw vibrations during speech, and (b) they provide smoother and cleaner head motion data compared to lower-body movements. Exploring these opportunities, we propose two applications for earphones. First, by detecting jaw vibrations, IMUs can support the microphone in performing multimodal self-supervised speech denoising. During activities such as phone calls or voice assistant interactions, vibrations from the throat travel through the jawbone and skull, inducing a voltage in the IMU. Although this IMU data is lower resolution and more distorted than microphone recordings, it is unaffected by ambient sounds, providing a unique advantage for multi-modal speech enhancement. Specifically, we explore whether the uninterfered, yet distorted, IMU signal can aid in enhancing speech when the microphone’s signal is compromised by non-stationary ambient noise. Secondly, using the clean acceleration data from the IMU, we propose an infrastructure-free indoor dead reckoning algorithm. By leveraging the inertial sensors in both earphones and smartphones, we can estimate a user’s indoor location and gazing orientation. Additionally, by playing 3D sounds through the earphones and monitoring the user’s responses, we can recalibrate errors in location and orientation estimation. We believe this innovative combination of IMU and acoustics could mark a significant advancement towards indoor Acoustic Augmented Reality (AAR).

Name thesis:degree_name

Ph.D.

Level thesis:degree_level

Dissertation

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Wei, Yu-Lin

Contributors dc:contributor

• Roy Choudhury, Romit
• Al-Hassanieh, Haitham
• Smaragdis, Paris
• Srikant, Rayadurgam
• Sabharwal, Ashutosh

• Multimodal Denoising
• Imu
• Indoor Localization
• Wearable

Statement dc:rights

• Copyright 2024 Yu-Lin Wei

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/125698

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F125698