Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Sensitivity Analysis for Biometric Systems: A Methodology Based on Orthogonal Experiment Designs



Yooyoung Lee, James J. Filliben, Ross J. Micheals, P J. Phillips


The purpose of this paper is to introduce an effective and structured methodology for carrying out a biometric system sensitivity analysis. The goal of sensitivity analysis is to provide the researcher/developer with the insight and understanding of the key factors—algorithmic, subject-based, camera-based, procedural, image quality, environmental, among others—that affect the matching performance of the biometric system under study. This proposed methodology consists of two steps: 1) the design and execution of orthogonal fractional factorial experiment designs which allows the scientist to efficiently investigate the effect of a large number of factors—and interactions—simultaneously, and 2) the use of a select set of statistical data analysis graphical procedures which are fine-tuned to unambiguously highlight important factors, highlight important interactions, and to also show locally-optimal factor settings. We illustrate this methodology by application to a study of VASIR—the high-performance, open-source, NIST iris-based biometric system, which has just come under recent release. In particular, we investigated k = 8 algorithmic factors from the VASIR system by constructing a (2^(6-1)) x (3^1) x (4^1) orthogonal fractional factorial design, generated the corresponding performance data, and applied an appropriate set of analysis graphics to determine the relative importance of the eight factors, the relative importance of the 28 2-term interactions, and the local best-settings of the eight algorithms. The results showed that VASIR’s performance was primarily driven by six factors out of the eight, along with four two-term interactions. A virtue of the two-step methodology is that it is systematic and general, and hence may be applied with equal rigor and effectiveness to other biometric systems.
NIST Interagency/Internal Report (NISTIR) - 7855
Report Number


sensitivity analysis, iris recognition, biometrics, experiment design, VASIR, video-based biometric system, orthogonal, fractional factorial, characterization, uncertainty, factor effects, interaction effects, optimization, main effects plots, interaction plots


Lee, Y. , Filliben, J. , Micheals, R. and Phillips, P. (2012), Sensitivity Analysis for Biometric Systems: A Methodology Based on Orthogonal Experiment Designs, NIST Interagency/Internal Report (NISTIR), National Institute of Standards and Technology, Gaithersburg, MD, [online], (Accessed May 25, 2024)


If you have any questions about this publication or are having problems accessing it, please contact

Created May 17, 2012, Updated November 10, 2018