A Standard Reporting Format for Summative Usability Evaluations

Published

January 10, 2007

Author(s)

Jean C. Scholtz, Emile L. Morse

Abstract

Poor usability is an uncontrolled source of overhead, caused by the need for users to correct errors and continually re-learn complex user interfaces. Software that is measurably usable reduces errors, reduces training costs, and reduces maintenance costs, while increasing user productivity and satisfaction. The business case for software usability has been clearly established. Many software vendors subject their products to usability testing during the development cycle, however, when companies or organizations make large purchase decisions for software, they currently have little visibility into the usability of the products they are buying. How can these potential software purchasers use usability as a factor in their decisions? An effort to increase the visibility of software usability was begun in October of 1997 by the U.S. National Institute of Standards and Technology (NIST: http://www.nist.gov) and resulted in the creation of the Industry USability Reporting (IUSR) Project (http://www.nist.gov/iusr). Cooperating in this effort are prominent suppliers of software, representatives from large consumer organizations, usability consultants, and academics. NIST¿s role is to facilitate and co-ordinate the activities of this group of usability advocates. Early on the group decided to concentrate its efforts on developing a common usability reporting format for sharing usability data with consumer organizations and validating the format by pilot trials. Through a series of four workshops, the IUSR team developed a document called the Common Industry Format for Usability Test Reports (CIF). Members contributed information about the formats that were being used by their companies¿ usability groups. They determined which elements were essential to a good report and which things were less important. The structure of the essential elements was developed using a consensus process. The resulting CIF is meant to be generated by a usability professional at the software supplier company and interpreted by a usability professional at the consumer company. It contains an executive summary for management and more detailed sections containing the description of the product, the objectives of the test, demographic data about the participants, the context in which the product was evaluated, the experimental design of the evaluation, the results produced using the CIF metrics of effectiveness, efficiency, and satisfaction, and the data analysis techniques used. Note that the CIF does NOT specify how a usability test is to be performed; it merely stipulates how the results of a test are to be reported. Informational Annexes (Appendices) include a Glossary of terms and a checklist and template (MS Word) to jump-start CIF preparation. In May 2001, the CIF was submitted to the National Committee for Information Technology Standards (NCITS: http://www.ncits.org/) as a Proposed Standard. NCITS approved the Proposal in November 2001. ANSI (American National Standards Institute) approval on December 12, 2001 resulted in an American national standard for usability reports. ANSI/NCITS-354-2001 contains a description of how the results of a usability test should be reported so that other usability professionals can determine whether the testing reflects their target user, tasks, and context. Now, for the first time, usability information can be factored into purchase decisions. Armed with this information, usability professionals in consumer companies can communicate their concerns to those in their companies who make critical purchasing decisions that are key to the productivity and satisfaction of the employees who will ultimately have to use the product.

Citation

Interactions

Pub Type

Others

Citation

Scholtz, J. and Morse, E. (2007), A Standard Reporting Format for Summative Usability Evaluations, Interactions (Accessed May 5, 2024)