Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Validation of Two-Sample Bootstrap in ROC Analysis on Large Datasets Using AURC



Jin Chu Wu, Alvin F. Martin, Raghu N. Kacker


Sampling variability can result in uncertainties of measures. The nonparametric two-sample bootstrap method has been used to compute uncertainties of measures in receiver operating characteristic (ROC) analysis on large datasets, such as the standard error (SE) of the equal error rate in biometrics, the SE of a detection cost function in speaker recognition evaluation, etc. It is hard to calculate uncertainties of these statistics of interest without using bootstrap methods. The SE of the area under ROC curve (AURC) can be computed analytically using the Mann-Whitney statistic. It can also be calculated using the nonparametric two-sample bootstrap method. The analytical result could be treated as a ground truth. The relative errors of bootstrap-method results with respect to the analytical-method results using different matching algorithms were examined. It turned out that they were quite small. Hence, this validates the nonparametric two-sample bootstrap method applied in ROC analysis on large datasets.
NIST Interagency/Internal Report (NISTIR) - 7733
Report Number


ROC analysis, bootstrap, area under ROC curve, uncertainty, standard error, biometrics, speaker recognition


, J. , Martin, A. and Kacker, R. (2010), Validation of Two-Sample Bootstrap in ROC Analysis on Large Datasets Using AURC, NIST Interagency/Internal Report (NISTIR), National Institute of Standards and Technology, Gaithersburg, MD, [online], (Accessed May 17, 2024)


If you have any questions about this publication or are having problems accessing it, please contact

Created October 11, 2010, Updated February 19, 2017