Measurement and Certification in Information Technology

Published

October 7, 1999

Author(s)

D Wallace

Abstract

Achieving software certification may seem an unreasonable goal. When it comes to determining whether a software product is dependable, safe, and effective, consumers of many types of software are largely on their own. Yet, today's methods provide many opportunities for assurance of at least some components of software systems.The open availability of credible measurement and test methods is an important step toward assuring the quality of software-based systems and promoting competitiveness in the information technology (IT) market. Many national and international organizations are now working toward this goal, including national metrology institutes in the European Community and Japan, and industry groups such as Open Group (X/Open) and Underwriters Laboratories.At the US National Institute of Standards and Technology's Information Technology Laboratory, our work focuses on establishing comprehensive certification capability for the IT industry. The NIST ITL approach uses the principles of measurement science, adapting them to measuring a software product's conformance to particular standards, as well as its performance and dependability. ITL also works with industry to establish credible, cost-effective test suites to demonstrate software conformance to particular standards. ITL then issues these suites to accredited test laboratories, certified by either the NIST-administered National Voluntary Laboratory Accreditation Program (NVLAP) or by the private sector.This presentation will describe NIST's work, focusing on principles of measurement science and how they can be adapted for software. Software measurement science should use the same basic principles as physical measurement science, which requires a reference, a measurement method, and an uncertainty statement. At NIST, we have identified different types of references, measurement methods, and uncertainties depending on the type of software being tested and the attributes being measured.We describe the use of such principles in international accreditation of software testing laboratories and their certification programs. NIST has worked with industry and academic researchers to develop reference implementations that are defined by standards, tested by certifiable test methods, and traceable to standards. These implementations are available to organizations to assess their own measurement methods or assign test-method values.The methods we discuss rely on standards, reference materials, or experience. These approaches can be applied to evolving technology so that the standards, reference materials and data are available as the technology matures. These methods may not yet provide 100 percent certification, but we believe they are a necessary route to that goal.Public testing technology gives vendors criteria so they can self-certify their products as compliant with a known measurement technology. Third-party commercial testing laboratories can also use this public technology to meet user-group requirements. Certification is not a guarantee against failure, only a statement about risk. For the certification process to work, there must be credible and cost-effective tests available, clearly defined testing methods, and standardized reporting formats. Further, user organizations must promote and require product certification.To this end, NIST administers the National Voluntary Laboratory Accreditation Program, or NVLAP, a series of laboratory accreditation programs. Each LAP includes specific calibration and test standards, as well as methods and protocols to satisfy accreditation needs in a particular area.Dependability testing is a major concern for certification. One way to improve dependability is to evaluate a system's formal specification against a set of formally defined requirements, producing a formal or semi-formal proof. Source code and tests are derived from the formal specification, providing traceability to the requirements. A second method exa

Citation

NASA Workshop on Risk Management

Pub Type

Journals

Keywords

certification, computer software, information technology, software, software certification, software measurement science, standards

Citation

Wallace, D. (1999), Measurement and Certification in Information Technology, NASA Workshop on Risk Management (Accessed April 18, 2024)