Industrial XR: Fulfilling Human Potential in Smart Factories

Step into the factory of the future. Alicia, an operations manager, sits at her workstation viewing a digitally enhanced video feed of the facility using cameras installed in strategic locations. Wearing safety gear, a maintenance engineer named Bob checks his tablet for the next machine to fix. Equipped with a headset and controllers, Dave, a software engineer at HQ, serves as a virtual tour guide for Carrie, the company’s lead executive. Wearing an augmented reality (AR) headset, Carrie surveys her machines as she walks through the facility. With Dave’s guidance, she sees digital information, such as a machine’s status, appearing in her view.

Each able to experience a virtual overlay onto a physical environment that provides more context relevant for their jobs, these co-workers can realize their potential as a team through industrial extended reality (XR), an umbrella term that encompasses a spectrum of technologies from partially immersive AR to completely immersive virtual reality.

This factory might be hard to imagine, but each technology already exists. What’s missing are standard formats, protocols and guidelines for them to work seamlessly with one another. In other words, the communication channels among these technologies remain shut.

The National Institute of Standards and Technology (NIST) is helping U.S. factories realize this vision through advanced information technology. Vital to this transformation are standards. Standards enable efficient organization of data and create a common language for machines and people to communicate. The factory of the future will require people to work closely and collaboratively with complex machines, but many unanswered questions remain. How do we pin down the precise locations of people and machines? How do we present the right information at the right time about machines to people? And, how can we do it all safely?

Leveraging industrial XR, manufacturers can present data in a spatially relevant way that takes advantage of people’s senses and natural ability to instantly recognize objects. For example, our most fond childhood memories often start with a specific time and place. For instance, one of us (Bill) will never forget sitting on his father’s shoulders during an afternoon at Disney World watching his hero Mickey Mouse lead the parade. Similarly, we can easily describe the spatial context of our workplace.

To support the pragmatic use of industrial XR, data standards play a key role. Our Product Lifecycle Data Exploration and Visualization project explores new approaches to bridge the communication gap between human and machine. Our project uses XR as a platform for presenting standard data representations to workers on the floor. Using existing standards from both manufacturing and IT communities, we develop more precise, interoperable and efficient approaches for industrial XR experiences.

Currently, our research focuses on delivering contextual views of machines for human workers by correlating their location and performance data. In doing so, we have already achieved many aspects of the activities of the stars of our story: Alicia, Bob, Carrie and Dave.

We recently published an article that demonstrates the use of standard geospatial representations to calculate the best path for a worker to travel — just as Carrie is being led through the facility by Dave.

Central to our approach is the use of standards for digitally representing geospatial layouts. Our project builds on standards published by the Open Geospatial Consortium (OGC) that provide guidelines for location-based services. An OGC standard called IndoorGML makes it possible for multiple devices to access the same digital content relative to a shared spatial context. Our approach enables the integration of a variety of technologies, including hardware such as webcams, tablets and headsets, and software such as computer vision toolkits and game engines. Note that our approach requires up-front time to create a spatial representation of a facility. Much like blueprints to a construction project, well-defined plans are critical to robust communication.

Manufacturing is just one domain that benefits from XR technologies. At NIST, researchers are leveraging XR to help solve other important problems, including understanding the spread of wildfires, enabling AR for first responders using smart helmets, and easing the training of industrial robots. Those NISTers interested in XR belong to a recently publicly announced group, called the NIST XR Community of Interest, where we share our cross-disciplinary experiences and skills. We encourage you to check out our website to learn more.

We anticipate that XR technologies will soon become a staple in modern daily life. As evidence, smartphones are already incorporating embedded hardware to make AR ubiquitous. To support the efficient and safe use of such technologies, standards are critical. XR technologies, like AR, are inherently designed to augment people’s awareness of what’s around them. Fulfilling the technological potential requires proper placement of things (both virtual and physical) and efficient communication between them. The technology will let humans fulfill their potential to do more things better and more safely.