Testing Facilities

Navigation

• Testing of Indoor Localization Systems 
• Challenges of Testing Indoor Localization Systems 
• Testing Facilities 
• ISO/IEC 18305:2016 
• Back to Location-Based Services portfolio page

Indoor Localization Facilities at NIST

Credit: ©Robert Rathe

There are two facilities for testing indoor localization systems at NIST:

• “Buildings with Dots” on the NIST Campus in Gaithersburg, MD
• “Public Safety Immersive Test Center (PSITC)” at the First Responder Network Authority facility in Boulder, CO (jointly operated by NIST)

These facilities are complementary, and they play different roles. The facility in Gaithersburg allows testing in large spaces and different types of buildings, as well as possibility of comprehensive testing in non-line-of-sight (NLOS) radio frequency (RF) signal propagation scenarios. The facility in Boulder provides high-rate (as high as 120 Hz), ground-truth location and pose information for the entity to be localized or tracked (ELT). This allows development of better localization and pose estimation algorithms for systems that use inertial measurement units (IMUs).

Descriptions of the facilities follow.

Buildings with Dots

Around 2010, NIST set up an indoor localization system testing facility in its Gaithersburg, Maryland campus. The test bed consists of 1300+ testing points randomly laid out in six buildings. Each test point, also called a “dot” at NIST, is a marker on the floor of a building along with a corresponding ID number on a close by wall. To the best of our knowledge, the NIST test bed is the largest of its kind in the world.

The buildings were chosen based on guidance from the international standard ISO/IEC 18305, Test and evaluation of localization and tracking systems. Except for the Net Zero Energy Residential Test Facility (NZERTF), which is a single-family house with a detached garage, all other buildings are large. One is a 13-story building with a sprawling first floor and a cavernous basement. Another is a building that is totally underground with two subterranean floors and very challenging radio communications connectivity with the outside. A third building is a 3-story office and laboratory building. The fourth building represents industrial environments like a factory floor. The fifth building is again a very large building that is a combination of various workshops (carpentry, metal works, etc.), a warehouse, and office space. The total area in these buildings exceed 60,000 square meters.

The dots are made of durable plastic material glued to the building floors and in the stairwells. They were surveyed by a contractor. Therefore, NIST is in possession of the precise 3D ground-truth coordinates [(northing, easting, elevation over sea level) and (latitude, longitude, elevation over sea level)] of the dots.

The facility makes it possible to rigorously test indoor localization systems according to the procedures specified in ISO/IEC 18305. A given system is tested in all six buildings. There is a total of 14 testing scenarios that could be used in each building, except for NZERTF where fewer scenarios are used due to its small size. Each scenario involves one or two people wearing/carrying localization devices associated with the system under test following a pre-determined path in the building. The path passes through many dots. When each person reaches a dot on the path, s/he would press a button on the localization device to prompt the system under test to generate a location estimate along with a timestamp. Given that the path is pre-planned and  hence the sequence of dots visited in the scenario are known a priori, the localization error, which is the difference between the location estimate for the dot and its ground-truth coordinates, can be readily computed. This allows the computation of a large number of aggregated performance metrics specified in ISO/IEC 18305. There are scenarios for various modes of mobility for the person wearing/carrying the localization device. The person may walk normally, stop at certain places, crawl on the floor, sidestep, walk backwards, etc.

 These modes of mobility model the behavior of emergency responders visiting a building. It is important to consider these mobility modes because they may affect the performance and localization accuracy of the system under test. There are also scenarios corresponding to the use of an elevator and placing the localization device on a pushcart corresponding to transporting a sizeable object in a building. Each scenario in a large building may last 20-30 minutes. It is vital to test indoor localization systems in large buildings, because they may use accelerometers and gyroscopes that have an inherent drift causing the localization error to grow with time. The localization system may also use range and angle estimation methods based on radio frequency (RF) technology. It makes a difference whether there is a line-of-sight signal propagation path between a pair of devices, or the signal has to go through a few walls or other obstacles. Therefore, it is important to test in large buildings made with different construction materials and multipath propagation characteristics. Testing in one or two small buildings may not provide meaningful and reliable performance results.

Last, but not the least, ISO/IEC 18305 takes a “black box” approach to system testing. It assumes that the people carrying out the tests know nothing about the inner workings of the system under test or what technologies it uses. The testing procedures and performance metrics are rich enough to be used for any system and yet yield meaningful performance results.

Public Safety Immersive Test Center (PSITC)

The Public Safety Immersive Test Center (PSITC) is a state-of-the-art facility designed and built in the spring of 2022 by the National Institute of Standards and Technology’s (NIST) Public Safety Communications Research (PSCR) Division. The approximately 1,076 square feet center is located in the First Responder Network Authority’s Technical Facility in Boulder, Colorado. The PSITC serves as a research and development facility for purposes related to user interface/user experience (UI/UX) and indoor location-based services (LBS) for PSCR researchers and innovators. NIST PSCR and the FirstNet Authority have been partnering agencies for a decade, focusing on furthering the development of public safety communications. The PSITC is available to first responders and private and academic organizations [partnered with PSCR and the FirstNet Authority] that focus on public safety response. These organizations and individuals may visit the center to receive hands-on experience with the technology and conduct related research. The center will allow for researchers and first responders to leverage immersive virtual simulations and evaluate technologies that can support a variety of situations, such as a smoke-filled building, search and rescue scenarios, or even terrorist attacks. No controllers are required for the simulation to enable users to organically interact with and pick up props, touch walls, furniture, crawl on their hands and knees, and more.

The facility includes:

• A modular layout 
• A motion capture system with 62 high-speed optical tracking cameras capable of centimeter accuracy 
• A variety of augmented & virtual reality headsets 
• A mobile staircase and other equipment for z-axis motion 
• Physical furniture and gear to add a tactile component to simulations