Measurement is difficult due to the complexity of the underlying technologies being addressed. In addition, measuring conformance in web applications requires large test sets covering a wide a range of standards.
Currently, W3C WEB Applications working group members are responsible for developing the necessary sets of standards for web applications. Until recently, no testing effort had begun.
The long term goal of this project is to establish a metrology infrastructure that automatically generates conformance tests directly from a specification, thus improving relevant standards and interoperability among implementations of the respective standard.
It has become well-known that software is not adequately tested and is rushed to market full of bugs. A recent NIST study indicated that the annual cost to the country for inadequate testing is as high as 59 billion dollars. NIST has vast experience in technology evaluation applicable to data exchange and is uniquely positioned to provide the metrology, test and measurement infrastructure required to advance the state of the art. Conformance test development requires significant resources -- beyond the means of most organizations. Common data and metrics permit direct comparisons of approaches and NIST is viewed as the neutral objective arbiter of evaluation design and implementation.