GestureTips: Enhancing Gesture Discoverability in Virtual Reality Applications through LLM-supported Tips

In immersive virtual reality (VR), hand gestures enable natural and intuitive interaction, but they are inherently invisible and not self-revealing. Unlike traditional 2D GUI-based interfaces, where menus or buttons visually guide the user, VR opens up a vastly greater degree of freedom, allowing gestures to occur across three-dimensional space, including along depth and not just across a flat plane. While this freedom unlocks richer and more expressive interactions, it also introduces new challenges: novice or new users often struggle to discover the range of available gestures, leading to frustration or moments of being stuck. Without clear guidance, the fluid, exploratory experience that VR promises can quickly become confusing or overwhelming. To tackle this challenge, we developed GestureTips, a multimodal in-application guidance system that helps users discover what gestures they need to perform to complete the desired interaction. GestureTips allows a user to request help from within VR, for example by asking a question aloud, and then see an animated demonstration of the needed gesture overlaid in the virtual environment. This design provides immediate, context-aware gesture guidance without forcing the user to leave the immersive experience or consult an external manual. We implemented GestureTips in a prototype VR music application and explored two variants of the help interface: a global GestureTips mode with a single floating help icon accessible anywhere, and a local GestureTips mode with multiple help hotspots attached to specific objects to provide context-specific tips. To evaluate GestureTips and the extent to which they support gesture discoverability, we conducted a user study with 24 participants using a conventional manual-based help approach as the baseline. Using a within-subject design, each participant completed a set of gesture-based tasks under three conditions: a baseline with no in-app help beyond a static gesture manual, the global GestureTips, and the local GestureTips. We measured how quickly participants could discover and perform each required gesture in each condition and gathered qualitative feedback on workload, usability, and user preferences. The study revealed that participants’ time to discover and perform gestures on their initial attempts did not significantly differ among the three conditions. GestureTips, whether global or local, provided comparable efficiency to a static reference manual. Importantly, GestureTips did not introduce additional cognitive overhead or delay task completion, demonstrating that its on-demand guidance was at least as effective as traditional manual methods. However, participants consistently expressed higher satisfaction and perceived greater support when GestureTips was available, despite the lack of significant performance differences. Users reported that animated in-app guidance reduced perceived effort and frustration by clearly visualizing the required gesture, and participants appreciated GestureTips primarily for quick gesture recall, noting the convenience of voice-based querying over manually scanning an extensive list of gestures. Between the two GestureTips modes, participants expressed mixed preferences: while many appreciated the single, easily accessible global help icon in the global condition, others valued the convenience of localized, context-specific triggers in the local condition. Some confusion arose when localized access points did not respond adequately, indicating that future systems might benefit from combining multiple localized triggers with a unified gesture knowledge base. These findings suggest that GestureTips is most beneficial as an on-demand assistance tool following initial gesture training rather than as a standalone onboarding solution. Users indicated a preference for a hybrid model, beginning with structured guidance such as brief tutorials or a visual cheat sheet, followed by GestureTips for occasional, quick reference. Participants who transitioned from manual guidance to GestureTips found it convenient for reinforcing learned gestures without interrupting their immersive experience. An important implication is the need for flexible and efficient help accessibility: participants recommended multiple redundant mechanisms, including voice queries and easily reachable help triggers, to minimize interruption during tasks. Feedback also highlighted the importance of clearly positioning animated gesture cues within users’ primary fields of view, reducing potential confusion or oversight. Providing a global fallback mechanism alongside context-sensitive triggers may ensure users reliably receive assistance whenever needed. Looking ahead, future research should focus on scaling GestureTips to accommodate larger gesture sets and more complex VR environments. Improvements are needed in refining the visibility and reliability of gesture feedback, addressing issues with localized cue placement, and enhancing gesture recognition robustness. Additionally, exploring scenarios that encourage spontaneous gesture discovery could further validate GestureTips’ potential benefits and clarify its role in enhancing gesture discoverability.

Author dc:creator

• Vedant Nandoskar (24399047)

• Computer Science
• Virtual Reality (VR)
• Human-Computer Interaction (HCI)
• Immersive Computing
• Natural User Interfaces
• Interaction Design
• User Interface Systems
• Assistive Interaction Techniques
• Applied User Studies
• Usability and Learnability

Statement dc:rights

• In Copyright

DOI dc:identifier

https://doi.org/10.25417/uic.32991932.v1

OAI identifier oai:identifier

oai:figshare.com:article/32991932