Unreliable evidence in binary classification problems

Published: May 07, 2019

Author(s)

David W. Flater

Abstract

Binary classification problems include such things as classifying email messages as spam or non-spam and screening for the presence of disease (which can be seen as classifying a subject as disease-positive or disease- negative). Both Bayesian and frequentist approaches have been applied to these problems. Both kinds of approaches provide poor estimates of the predictive value of tests for which the number of positive results in the sample is either very small or very large. A classifier that does not account for the uncertainty of these estimates is vulnerable to making inferences from unreliable evidence. This report explains the problem and explores options for accounting for the often-neglected uncertainty. A neat solution that does no harm to less uncertain cases remains elusive.
Citation: Technical Note (NIST TN) - 2044
Report Number:
2044
NIST Pub Series: Technical Note (NIST TN)
Pub Type: NIST Pubs
Created May 07, 2019, Updated May 07, 2019