NIST Administrative Manual, Subchapter 4.09
Transmittal Date - 12/21/93

Background
 
 

The publication of information on machine-readable media, particularly computer software, has become an essential means for disseminating the results of an increasing number of NIST technical programs. The need for editorial review as a vital part of quality management for information on machine-readable media has long been recognized. This need was the focus of a special study by an ad hoc Committee on Software Documentation that reported to WERB on September 30, 1983, and recommended:
 
 

"Computer programs can only be sent to NTIS when program documentation exists and when the program documentation has been cleared by WERB."[sic]
 
 

The Committee also recognized a need for technical review of both the algorithms used and the software program itself. It stated that such reviews are best handled in the technical organization that produced the software and should be performed before the program documentation is submitted to the Editorial Review Board for approval and release. These conclusions provided the basis for the clarification of related policy in a memorandum from the Chairperson of WERB to Operating Unit Directors and division chiefs in 1984. In June 1987, the Director encouraged WERB to establish procedures to ensure that the review of publications on computer-readable media is as vigorously carried out as the review of publications in print media. In January 1990, the Director required that software review and approval be clearly defined for the technical staff as well as the Editorial Review Boards.
 
 

Since the ad hoc Committee report was issued, the need for software review has become more intensive because:
 
 

o an increasing number of NIST technical program outputs are computer software or include software; and
 
 

o many software products produced at NIST are more complex in the sense that they combine algorithms for calculations, application-specific programming, extensive databases, and various special aids for the performance of user tasks, including increasing amounts of "artificial intelligence."

Many additional pathways, other than NTIS, have also been adopted for the distribution of NIST software products to the public.
 
 

Elements of Software Quality Management
 
 

To define a practical approach to editorial review as it applies to computer software products and to avoid unrealistic expectations, it is important to recognize the many elements that enter into Total Quality Management considerations for software. These elements may include, but are not necessarily limited to, the following:
 
 

o effectiveness and efficiency of algorithms that are embedded in the program;
 
 

o effectiveness and efficiency of programmed logic;
 
 

o freedom from coding errors and logic errors;
 
 

o integrity of databases that may be embedded or callable;
 
 

o completeness and effectiveness of comments in source code available to the user, if any;
 
 

o presence and effectiveness of operating messages related to user inputs, errors, etc.;
 
 

o presence and effectiveness of error recovery provisions;
 
 

o presence and effectiveness of "user friendly" provisions such as help messages, menu screens, and installation programs;
 
 

o completeness and effectiveness of software testing programs; and
 
 

o completeness and effectiveness of documentation that makes it possible for users to install and operate the software; this documentation may include coded text files or tutorials that constitute extensions of the primary hardcopy documentation.
 
 

Editorial review can address only the last of the above elements in significant depth; other elements must be addressed in detail as part of the software engineering task, and the performance of technical functions can only be evaluated by appropriate software and application experts. However, the information communicated by the documentation should be sufficient for the intended user to operate the program without difficulties, and it should provide adequate details to support practical expectations; effective editorial review of the documentation and the evaluation of such information will contribute substantially to the quality of the entire software product.
 
 

In some cases, text will be embedded throughout program code; it may not be feasible to conduct an editorial review of all such text on any basis other than the random sampling that may be accomplished by exercising the program. However, tabulated values of experimental data should be reviewed with special care. The quality of such text as well as other elements that affect total software quality must be fully addressed by quality assurance procedures that operate during software engineering and testing.
 
 

Scope of NIST Software Products
 
 

NIST software products include, for example:
 
 

o standard conformance test suites for software standards;
 
 

o standard reference databases and related applications;
 
 

o prototypes of research being conducted at NIST;
 
 

o laboratory automation programs and data acquisition routines;
 
 

o data processing programs; and
 
 

o machine control programs and programs that support factory automation.
 
 

Software Documentation Requirements
 
 

Acceptable documentation that supports the use of NIST software depends upon the complexity of the software and the needs of the intended user. Thus, a simple programmed routine that will be transmitted to one expert user may be documented by a single page of printed text that addresses the applicable requirements. On the other hand, complex software directed toward users having limited computer-related skills must be supported by more extensive documentation.
 
 

In many cases it is necessary or desirable to distinguish between user documentation and technical documentation depending upon the needs and qualifications of the people addressed. These two classes of software documentation may be published in different forms, they are usually addressed to different communities, and they generally include different elements of information as determined by the author. Both classes of software documentation must meet the editorial review requirements of this appendix.
 
 

Total documentation that supports the use of NIST software products must include:
 
 

o a description of the purpose of the software;
 
 

o a description of algorithms, data tables, etc.;
 
 

o a statement of available software media (disk, disk configuration and capacity, tape, CD-ROM, etc.);
 
 

o hardware requirements for the use of the software:
 
 

-machine family;
-memory;
-disk drives, tape drives, CD-ROM reader, etc.;
-display (text, graphics, graphics adapter, etc.);
-printer, plotter;
-communications (network facilities or modem), if required.
 
 

o operating system requirements for executable programs;
 
 

o specification of applications programs, if required to access and use database files, spreadsheet files, word processing files, etc.;
 
 

o specification of an interpreter or compiler if one is required to use the program code that is provided;
 
 

o statement of program tests, testing levels (alpha, beta, acceptance), test results, and other validation-related information; description and results of tests of embedded algorithms, etc.;
 
 

o statement of inputs required from the user;
 
 

o statement concerning technical support available from NIST:
 
 

-level of support (if none, indicate "none");
-name(s) and telephone number(s) of qualified persons;
 
 

o software installation instructions appropriate for the intended user;
 
 

o instructions for the use of the software (if it is not menu-driven appropriately);
 
 

o statement of error messages, if applicable, and error recovery procedures;
 
 

o an appropriate disclaimer concerning the limitations of testing procedures performed; and
 
 

o a statement, approved by the NIST Deputy Chief Counsel, of the limitations of NIST liability for problems arising from the use of the software.
 
 

In some cases it may be desirable to provide essential documentation information in coded text ("read-me") files. In such cases, the primary hardcopy print documentation must include sufficient information to permit a typical user to install the software, access the appropriate files, and read the text on the screen or obtain a printout. In these cases, the author must also provide hardcopy printouts of such files with the documentation that is submitted to the Editorial Review Board for review.
 
 

Criteria for the Editorial Review of NIST Software
 
 

The benefits of editorial review go beyond the simple correctness of text and extend to the plausibility of the information that is communicated. In the review of computer software documentation, it is important to address the degree to which the purpose of the documentation is accomplished and the effectiveness of the complete package of software together with the documentation. Specific criteria include:
 
 

o completeness of the documentation;
 
 

o logical organization;
 
 

o good writing practice and clarity of expression;
 
 

o ease of use by the intended user(s);
 
 

o compliance with NIST policies for technical publications; and
 
 

o high probability that the software in combination with the documentation will achieve the intended purpose of the product.
 
 

Sometimes it may be desirable and appropriate to list a computer program as an appendix to an interagency/ internal report (NISTIR) that is written as a record of progress on a technical program or system. Such reports are not intended for outside distribution and use, and the status of the computer program development and testing must be clearly stated; persons, if other than the author, who can be contacted for additional information must be identified.
 
 

Procedures for the Review of Software
 
 

Procedures for the review of computer software are the same as procedures for the review of technical manuscripts on print media with the following additional considerations:
 
 

o Form NIST-114 must be signed and must clearly indicate that quality assurance requirements of the originating organizational unit have been satisfactorily met with respect to the functional characteristics of the software and the editorial quality of the documentation;
 
 

o if the software is available on an appropriate medium (disks, CD-ROM, etc.), it should accompany the documentation manuscript submitted to the Editorial Review Board. Otherwise, the manuscript should be accompanied by a statement of the location and availability of the software;
 
 

o hardcopy printouts of coded ("read-me") text files, if any, must accompany the primary documentation submitted for review;
 
 

o reader(s) selected must have background knowledge of the application technology and an understanding of good software engineering practice. It may be necessary to designate two readers to meet this requirement; and
 
 

o readers are encouraged to exercise the software using procedures defined in the documentation and to comment, from the viewpoint of the intended user, on the effectiveness of the documentation; to the extent possible, comments on other quality attributes of the software are also encouraged.
 
 

Editorial Review Board Approval
 
 

Editorial Review Board approval of documentation that supports the use of software is limited to the documentation. Approval of any software package and the decision to release it to the public are the responsibility of the Operating Unit Director. Comments and recommendations concerning the functional quality of software may result from the Editorial Review Board review. Such comments and recommendations are for the consideration of the Operating Unit Director, the responsible technical managers, and the staff members involved in the creative work.