NIST logo

Publication Citation: Measures, Uncertainties, and Significance Test in Operational ROC Analysis

NIST Authors in Bold

Author(s): Jin Chu Wu; Alvin F. Martin; Raghu N. Kacker;
Title: Measures, Uncertainties, and Significance Test in Operational ROC Analysis
Published: January 31, 2011
Abstract: In operational ROC (receiver operating characteristic) analysis of fingerprint-image matching algorithms on large datasets, the measures and their accuracies are investigated in the three scenarios: 1) the true accept rate (TAR) of genuine scores at a specified false accept rate (FAR) of impostor scores, 2) the TAR and FAR at a given threshold, and 3) the equal error rate (EER). The key issue is how to compute the accuracy. The accuracy is calculated using the nonparametric two-sample bootstrap based on our extensive studies of bootstrap variability on large datasets. The ultimate goal is to perform the comparison. Using the standard errors computed, the significance test is carried out to determine whether the difference between the performance of one algorithm and a hypothesized value, or the difference between the performances of two algorithms where the correlation is taken into account is statistically significant. In the case that the alternative hypothesis is accepted, the sign of the difference is employed to determine which is better than the other. Examples are provided.
Citation: Journal of Research (NIST JRES) -
Volume: 116
Issue: 1
Pages: pp. 517 - 537
Keywords: Receiver operating characteristic (ROC) analysis, Fingerprint, Biometrics, Nonparametric bootstrap, Standard error, Confidence interval, Significance test
Research Areas: Biometrics
PDF version: PDF Document Click here to retrieve PDF version of paper (233KB)