Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Bootstrap method versus analytical approach for estimating uncertainty of measure in ROC analysis on large datasets



Jin Chu Wu, Alvin F. Martin, Gregory A. Sanders, Raghu N. Kacker


The nonparametric two-sample bootstrap is employed to estimate uncertainties of statistics of interest in receiver operating characteristic (ROC) analysis on large datasets with/without data dependency due to multiple use of the same subjects in many disciplines, based on our studies of bootstrap variability. On the other hand, it would seem that the analytical method might be used for the same purpose. However, comparing these two methods, the differences are noteworthy. (1) The bootstrap method takes account of how genuine scores and impostor scores are distributed, which is associated with how the matching system works; but the analytical method does not. (2) If datasets involve data dependency, the bootstrap method can take it into account; but the analytical method cannot. (3) The covariance term that occurred in the speaker recognition evaluations while estimating the standard error of the measure can be taken into account intrinsically by the bootstrap method; but it is very hard to estimate analytically. (4) The analytical method generally underestimates the uncertainties of statistics of interest; but the bootstrap method estimates them more conservatively. To demonstrate these observations, the data used in this article were generated from a variety of sources: speaker recognition evaluations, biometrics evaluations, simulated data with normal distributions, and simulated data with nonparametric distributions.
NIST Interagency/Internal Report (NISTIR) - 8218
Report Number


Metrology, measurement uncertainty, ROC analysis, large datasets, bootstrap, data dependency, analytical method, biometrics, speaker recognition


, J. , Martin, A. , Sanders, G. and Kacker, R. (2018), Bootstrap method versus analytical approach for estimating uncertainty of measure in ROC analysis on large datasets, NIST Interagency/Internal Report (NISTIR), National Institute of Standards and Technology, Gaithersburg, MD, [online], (Accessed May 20, 2024)


If you have any questions about this publication or are having problems accessing it, please contact

Created July 17, 2018, Updated November 10, 2018