IREX 10: Performance Metrics

OVERVIEW

This page provides a high-level introduction to common performance metrics presented in IREX X reports. IREX X is a technology evaluation that does not assess performance for a specific deployment. The metrics are generic so as to convey a broad sense of algorithmic capabilities. It should be noted that biometric decision making is often analogous to the problem of binary classification, and as such no single numerical value can comprehensively describe the accuracy of a biometric system.

Identification Systems

Due to its scalability, iris recognition is well-suited for large deployments ranging from border security, prison management, expedited processing, duplicate detection, and distribution of resources. Most of these applications utilize iris recognition in  identification mode. Biometric identification systems search users against a database of enrolled identifies and return their proper identities when found.

Accuracy for the identification task cannot be fully characterized by a single numerical value. Two distinct types of error are possible. The first, a false negative, is roughly analagous to a "miss". In an access control scenario, a false negative would deny access to an authorized user. The other type of error, a false positive, occurs when a searched individual is falsely associated with one of the enrolled identities. This would grant access to an unauthorized user for the access control scenario. The decision criteria of the system can be adjusted to reduce the rate at which one type of error occurs, but at the expense of increasing the other. In the leaderboard, algorithms are compared by fixing the rate of false positives (FPIR) and reporting the rate of false negatives (FNIR). The full relationship between the two error rates is characterized by a DET plot.

Some publications report a matcher's rank one "hit rate". While this is a valid performance metric, it characterizes accuracy for closed-set applications (where it can be guaranteed that every searched individual is enrolled) which is rarely ever the operational use case. Equal Error Rates are also deprecated in IREX X because they report accuracy at operating thresholds that are typically far from what would be used in practice.

Contact Information

Inquiries and comments may be submitted to irex@nist.gov.  Subscribe to the IREX mailing list to receive emails when announcements or updates are made.

Created May 31, 2019, Updated August 26, 2019