Measures, Uncertainties, and Significance Test in Operational ROC Analysis

Published

January 31, 2011

Author(s)

Jin Chu Wu, Alvin F. Martin, Raghu N. Kacker

Abstract

In operational ROC (receiver operating characteristic) analysis of fingerprint-image matching algorithms on large datasets, the measures and their accuracies are investigated in the three scenarios: 1) the true accept rate (TAR) of genuine scores at a specified false accept rate (FAR) of impostor scores, 2) the TAR and FAR at a given threshold, and 3) the equal error rate (EER). The key issue is how to compute the accuracy. The accuracy is calculated using the nonparametric two-sample bootstrap based on our extensive studies of bootstrap variability on large datasets. The ultimate goal is to perform the comparison. Using the standard errors computed, the significance test is carried out to determine whether the difference between the performance of one algorithm and a hypothesized value, or the difference between the performances of two algorithms where the correlation is taken into account is statistically significant. In the case that the alternative hypothesis is accepted, the sign of the difference is employed to determine which is better than the other. Examples are provided.

Citation

Journal of Research (NIST JRES) -

Volume

116

Issue

1

NIST Pub Series

Journal of Research (NIST JRES)

Pub Type

NIST Pubs

Download Paper

Local Download

Keywords

Receiver operating characteristic (ROC) analysis, Fingerprint, Biometrics, Nonparametric bootstrap, Standard error, Confidence interval, Significance test

Citation

, J. , Martin, A. and Kacker, R. (2011), Measures, Uncertainties, and Significance Test in Operational ROC Analysis, Journal of Research (NIST JRES), National Institute of Standards and Technology, Gaithersburg, MD, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=903816 (Accessed April 25, 2024)