It's Not Always What the Eye Can See – Challenges in the Evaluation of Augmented Reality
Kurtis Goad, Kevin C. Mangold, Susanne M. Furman
Augmented reality (AR) technology is developing at a fast pace. Usability evaluation methodologies for AR need to be updated to accommodate the increasing complexity of how people interact with AR technology. Eye tracking metrics, which have been historically successful in traditional, often computer-based technology, have been underutilized when evaluating AR devices. Literature review shows documentation of methods used to evaluate AR since 2000. But these reviews do not address the reasons why evaluators have chosen or excluded certain methods, like eye tracking, and did not address evaluations performed that were not published. This exploratory study addressed these gaps by deploying semi-structured interviews to gain an understanding of the experiences professionals have when evaluating AR technology. Our results show that participants rely most heavily on user feedback and questionnaires/surveys during evaluation. Most participants did not use eye tracking methods. They cited a number of challenges evaluating AR technology and the use of eye tracking including difficulties in data collection and a lack of consistency and standards across devices.
, Mangold, K.
and Furman, S.
It's Not Always What the Eye Can See – Challenges in the Evaluation of Augmented Reality, NIST Interagency/Internal Report (NISTIR), National Institute of Standards and Technology, Gaithersburg, MD, [online], https://doi.org/10.6028/NIST.IR.8489, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=956488
(Accessed December 3, 2023)