It's Not Always What the Eye Can See – Challenges in the Evaluation of Augmented Reality

Published

September 19, 2023

Author(s)

Kurtis Goad, Kevin C. Mangold, Susanne M. Furman

Abstract

Augmented reality (AR) technology is developing at a fast pace. Usability evaluation methodologies for AR need to be updated to accommodate the increasing complexity of how people interact with AR technology. Eye tracking metrics, which have been historically successful in traditional, often computer-based technology, have been underutilized when evaluating AR devices. Literature review shows documentation of methods used to evaluate AR since 2000. But these reviews do not address the reasons why evaluators have chosen or excluded certain methods, like eye tracking, and did not address evaluations performed that were not published. This exploratory study addressed these gaps by deploying semi-structured interviews to gain an understanding of the experiences professionals have when evaluating AR technology. Our results show that participants rely most heavily on user feedback and questionnaires/surveys during evaluation. Most participants did not use eye tracking methods. They cited a number of challenges evaluating AR technology and the use of eye tracking including difficulties in data collection and a lack of consistency and standards across devices.

Citation

NIST Interagency/Internal Report (NISTIR) - 8489

Report Number

8489

NIST Pub Series

NIST Interagency/Internal Report (NISTIR)

Pub Type

NIST Pubs

Download Paper

https://doi.org/10.6028/NIST.IR.8489

Local Download

Citation

Goad, K. , Mangold, K. and Furman, S. (2023), It's Not Always What the Eye Can See – Challenges in the Evaluation of Augmented Reality, NIST Interagency/Internal Report (NISTIR), National Institute of Standards and Technology, Gaithersburg, MD, [online], https://doi.org/10.6028/NIST.IR.8489, https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=956488 (Accessed October 17, 2025)