NIST logo

Publication Citation: Uncertainty Due to Finite Resolution Measurements

NIST Authors in Bold

Author(s): Steven D. Phillips; B Tolman; William T. Estler;
Title: Uncertainty Due to Finite Resolution Measurements
Published: January 01, 2004
Abstract: We investigate the influence of finite resolution on measurement uncertainty from a perspective of the Guide to the Expression of Uncertainty in Measurement (GUM). Finite resolution in the presence of Gaussian noise yields a distribution of results that strongly depends on the relative location of the true value to the resolution increment. We show that although there is no simple expression relating the standard deviation of the distribution to the uncertainty at a specified level of confidence, there is an analytic relation between the mean value and the standard deviation. We further investigate the conflict between the GUM and ISO 14253-2 regarding the method of evaluating the standard uncertainty due to finite resolution and show that, on average, the GUM method is superior, but still approximate.
Conference: ASPE Spring Topical Meeting on Uncertainty Analysis in Measurement and Design
Proceedings: Proceedings of ASPE Spring Topical Meeting on Uncertainty Analysis in Measurement and Design
Pages: pp. 14 - 27
Location: State College, PA
Dates: June 30-July 1, 2004
Keywords: A/D;analog to digital conversion;error;resolution;uncertainty
Research Areas: Metrology, Manufacturing