Enhancing touch interactions with passive finger acoustics

In this report, we introduce a unique method of interacting with mobile devices through passive finger acoustics. Phone users typically use one of two fingers when interacting with a touchscreen device, leaving the other fingers idle. The motivation is that users can make use of their idle fingers to enhance the experience of an application. By wearing minimally obtrusive rings on the thumb and index finger, users can make distinguishable clicking sounds to quickly perform actions, without having to interact directly with the screen. Our system leverages the microphone embedded in the mobile device to capture sound and recognize sound in real-time without requiring Internet connection. The different sounds introduce new ways for users to interact with their devices, without cluttering the screen. We evaluated our system on an Android drawing application, which allows the user to switch tools based on clicks. We optimized our system based on accuracy, classification speed, and ease of use, in order to create a comfortable user experience

Name thesis:degree_name

Master of Science in Information Studies

Level thesis:degree_level

Masters

Discipline thesis:degree_discipline

Information Studies

Grantor

The University of Texas at Austin

Year dc:date.issued

2019

Author dc:creator

• Gong, Yifan

Advisors dc:contributor.advisor

• Thomaz, Edison
• Gurari, Danna

• Machine learning
• Finger acoustics
• Passive finger acoustics
• Mobile device interaction
• Mobile device user experience

Identifier URI

http://dx.doi.org/10.26153/tsw/8222

OAI identifier oai:identifier

oai:repositories.lib.utexas.edu:2152/81209