Bootstrap method versus analytical approach for estimating uncertainty of measure in ROC analysis on large datasets

Published

July 17, 2018

Author(s)

Jin Chu Wu, Alvin F. Martin, Gregory A. Sanders, Raghu N. Kacker

Abstract

The nonparametric two-sample bootstrap is employed to estimate uncertainties of statistics of interest in receiver operating characteristic (ROC) analysis on large datasets with/without data dependency due to multiple use of the same subjects in many disciplines, based on our studies of bootstrap variability. On the other hand, it would seem that the analytical method might be used for the same purpose. However, comparing these two methods, the differences are noteworthy. (1) The bootstrap method takes account of how genuine scores and impostor scores are distributed, which is associated with how the matching system works; but the analytical method does not. (2) If datasets involve data dependency, the bootstrap method can take it into account; but the analytical method cannot. (3) The covariance term that occurred in the speaker recognition evaluations while estimating the standard error of the measure can be taken into account intrinsically by the bootstrap method; but it is very hard to estimate analytically. (4) The analytical method generally underestimates the uncertainties of statistics of interest; but the bootstrap method estimates them more conservatively. To demonstrate these observations, the data used in this article were generated from a variety of sources: speaker recognition evaluations, biometrics evaluations, simulated data with normal distributions, and simulated data with nonparametric distributions.

Citation

NIST Interagency/Internal Report (NISTIR) - 8218

Report Number

8218

NIST Pub Series

NIST Interagency/Internal Report (NISTIR)

Pub Type

NIST Pubs

Download Paper

DOI Link

Keywords

Metrology, measurement uncertainty, ROC analysis, large datasets, bootstrap, data dependency, analytical method, biometrics, speaker recognition

Citation

, J. , Martin, A. , Sanders, G. and Kacker, R. (2018), Bootstrap method versus analytical approach for estimating uncertainty of measure in ROC analysis on large datasets, NIST Interagency/Internal Report (NISTIR), National Institute of Standards and Technology, Gaithersburg, MD, [online], https://doi.org/10.6028/NIST.IR.8218 (Accessed April 19, 2024)