Evaluating and Improving Hands-Free Interaction for Microsoft HoloLens

Abstract

This thesis investigates the potential of HoloLens, a mixed reality device, as a hands-free assistive technology (AT) solution for users and hand fatigue using head mounted devices. Existing AT options, like voice-controlled assistants, offer some assistance but struggle with complex tasks requiring navigation and manipulation. Our research addresses this gap by exploring how HoloLens, with its combination of voice commands and eye gaze, can empower users with greater independence in tasks like web browsing and email access. The study involved user testing with a focus on understanding the strengths and limitations of HoloLens for hands-free interaction. The study focused on executing basic browser navigation tasks using separately only hand gestures and then voice commands. While exploring these interaction methods, limitations became apparent, particularly with voice commands for actions like scrolling. To address this, an attempt was made to develop a browser application incorporating scrolling commands. However, due to technical constraints and limited documentation, this was unsuccessful. To further investigate the effectiveness of each interaction method, a test involving users was conducted comparing hand gestures and voice commands for scrolling tasks within a browser and Microsoft Store. Our findings highlight the limitations of current AT solutions and emphasize the need for more versatile and user-centric options. While the research doesn't definitively crown HoloLens as the ultimate answer, it positions it as a promising contender due to its ability to combine voice and eye tracking for interaction. Additionally, it offers potential to address user fatigue, a common issue with virtual and augmented reality devices, by reducing reliance on hand gestures. Overall, this research contributes to the ongoing quest for a seamless and inclusive hands-free AT experience. By identifying limitations and exploring the potential of HoloLens, we've provided valuable data to be used as a stepping stone to help develop assistive technology and possibly empower users with disabilities and users facing situational disabilities.