The IUSR Project: Industry USability Report

DATE:  August 10, 1999

This document is an overview of the Industry USability Report (IUSR)) Project. The IUSR Project is designed to help potential purchasers of software obtain information about the usability of supplier products. There are two parts to the IUSR project:

1. a proposed format for sharing usability information, and
2. a pilot study which will allow both producer and consumer companies to test the effectiveness of using usability test results as procurement criteria and to verify the usefulness of the reporting format.

Introduction

Many factors affect a corporation’s decision about which software products to purchase. One key factor is the software’s usability. In simple terms, usability reflects:

1. how easy the software is to learn,
2. how easy the software is to use
3. how productively users will work
4. the amount of support users will need

Software developers employ a variety of techniques to ensure software usability. In general terms, these techniques involve studying the users to develop an understanding of their needs and iteratively refining versions of the software based on usability testing results.

Purpose and Goals of the IUSR Project

In making purchase decisions, companies and organizations have traditionally had little indication of how usable a product would be or how much training and support its users would need. The situation has made it difficult to compare products, to plan for support, or estimate total cost of ownership.

The goals of the IUSR Project are to:

• Encourage software suppliers and consumer organizations to work together to understand user needs and tasks.

• Develop a common usability reporting format for sharing usability data with consumer organizations.

• Conduct a pilot plan to determine how well the usability reporting format works and to determine the value of using this format in software procurement.

Scope of the IUSR Project

Usability testing can be valuable for a wide range of products, however the IUSR Project is initially focused only on software. We do recognize that the usability of hardware (printers, copiers, fax machines, etc.) is important and often tightly integrated with software. The initial focus on software was intended to narrow the focus of the initial project so that a pilot study could be conducted. Extending the scope of the reporting standard to include hardware and other products should be addressed later in the project.

Reporting Format

The IUSR Project is developing the initial version of a common format for reporting usability test results, referred to as the Common Industry Format (CIF). The intended reader is a usability or human factors professional. The reporting format identifies the minimum format of shared usability information to allow consumer organizations to evaluate test results or replicate the tests if desired. Organizations that participate in using the format may choose to provide more than the minimum format.

Why a Common Format is Needed

A format providing common information is needed because there are many possible ways to report usability results. The purchase team may require an evaluation of the validity and relevance of any test that it uses to support its decision-making. A common format for reporting the test and its results will facilitate evaluation of the test and the interpretation of its results. It should also reduce misinterpretation of the test results. A common format for reporting usability tests will therefore benefit both suppliers and consumers of software products.

Scope of the Reporting Format

The detailed instructions for the report format are given in the document entitled “Common Industry Format for Usability Test Reports”.
The report format covers such topics as:

The description of the product

As some products have several releases and versions, the product description should include this information. The description should describe basic functionality of the product and the intended users of the product.

The goals of the test

User testing may be performed to accomplish a variety of goals, including problem identification or diagnosis, design alternatives comparison, or to complete a summative test. The goal(s) of the reported test should be clearly stated.

The test participants

This section should include information on the number of users who participated and the criteria by which they were selected.

The tasks the users were asked to perform

This section should list the specific tasks that participants were asked to perform during the study.

The experimental design of the test

This section should explain the logical configuration of the test conditions, including independent variables, what comparisons, if any, are intended between groups, and how conditions which might contaminate the results are brought under control.

The method or process by which the test was conducted

This section should report the sequence of events that was actually employed to instruct the test users, how well they followed it, any intervention such as coaching, and materials used to give instructions or ask questions of them.

The usability measures and data collection methods

Usability measures may include objective measures of effectiveness, efficiency, and how much effort is required to learn to use the product successfully. Subjective data on user satisfaction may also be collected.

The numerical results

This section should report the data analysis procedures and summary data. The results may contain summary statistics such as the mean, range, standard deviation, and standard error of the estimate.

Who Performs the Test

The usability test may be performed by the software supplier using its own usability group or by contracting with an independent testing facility or a consumer organization. The consumer organization interested in purchasing this software may accept the test results or may replicate the test.

In many cases, the supplier organization will provide the results of the last usability test conducted in the course of software development to the consumer organization. This allows organizations that have a usability testing program in place, to participate without incurring the expense of additional usability testing. Proposed changes in the product design that may occur as a result of the usability test may be listed in the report documentation. This reduces the need of having to perform additional usability test solely for the purpose of delivering the report to the consumer organizations. However, proposed changes listed in the report can in no way be taken as a guarantee that a supplier organization will make those changes in the final shipping product.

How Test Results Will Be Used

The procurement process for acquiring software often involves an organization’s purchasing department. Few purchasing departments have the skills needed to evaluate usability test results. For this reason, it is recommended that the consumer organization’s usability group human factors group should interpret the test results. This group will be responsible for interpreting the test results and integrating their evaluation into the procurement process.

On the supplier side, the sales process is typically the responsibility of a sales and marketing organization which is not likely to have skills in usability testing. It is recommended that the sales organization will work with the supplier’s usability or human factors group to identify an appropriate study and package the results in the common format.

Because usability test data may be subject to misinterpretation if incorrectly reported or published, it is assumed that in many cases the results will be shared under a non-disclosure agreement between supplier and prospective purchaser. Supplier organizations should treat both large and small consumers equally in allowing access to this information so as not to provide larger organizations with unfair advantage in competitive situations. Specific negotiations regarding non-disclosure are left to the individual organizations.

The Pilot Study

A Pilot will be conducted to determine the value of incorporating usability results into decision-making for software purchases, and to refine the procedures and reporting format. The initial focus will be on software for upgrades, new products, or custom applications.

Pilot Duration

The Pilot will begin in 1999 and run for 2-1/2 years

Pilot Description

Suppliers and consumers who wish to participate in the formal pilot study will enter into a Cooperative Research Agreement (CRADA) with the National Institute of Standards and Technology (NIST). This mechanism will facilitate sharing results from the pilot studies to refine the common usability reporting format and the associated metrics. NIST will host periodic workshops to discuss and consider refining the Pilot program.

During the formal pilot, pairs of Consumers and Suppliers who have entered into a Cooperative Research Agreement with NIST will form the teams that will work together on agreed upon software products. Each pair of companies will commit to the following:

The Consumer Company will

1. Consult with the Supplier to select a target software product with a minimum two-month advance notice
2. Request usability test data from the Supplier on the target product.
3. Apply the reported results in making decisions for software purchases.
4. Gather internal data on overhead, productivity, and user satisfaction, and record it in a common format in the NIST database.

The Supplier Company will

1. Conduct a usability test of the targeted software product.
2. Report the test and its results in the format specified by the Workshop
3. Put a nondisclosure agreement in place for sharing the usability test results with the Consumer corporation.

Interested companies are invited to participate in the pilot. Additional information on the pilot study will be found in the document entitled “Pilot Testing of the Common Industry Format for Usability Reporting” (To be announced: approx 30 August 1999).

Companies may, of course, use the common reporting format on their own. We recommend that consumer organizations wishing to use usability test information as a factor in purchasing decisions develop a plan to determine if there are measurable benefits in doing so. We encourage companies to contact us for guidance on implementing plans in their organizations. In addition, we encourage companies to contribute data from efforts in their companies to a database maintained at NIST. This data will be periodically reviewed to determine the effects, if any, of these efforts.

Usage and Liability Issues

The IUSR Project common reporting format is intended to simplify information exchange between business organizations. There is no certification that information supplied by the supplier is correct; nor is there any certification regarding the effectiveness of any usability efforts during the software development process. The working group responsible for the creation of this document and accompanying common usability reporting format is not liable should any of the information be inaccurate. There is no liability on the part of the National Institute of Standards and Technology (NIST) who provide a repository for the common reporting format and the accompanying documentation and any companies who participate in this program. (See accompanying Letter of Intent *not available).

History of the IUSR Project

The IUSR Project is being conducted by a group of representatives from both supplier and consumer companies hosted by the National Institute for Standards and Technology (NIST).

During 1998, the IUSR Project held two workshops. The purpose of these workshops was to bring together a group of usability experts from both supplier and consumer corporations who would be able to:

• Define a minimal common specification for reporting "procurement" usability tests.
• Design a pilot study to evaluate the costs and benefits of including usability test results in software procurement.

Organizations who are producers or consumers of software products (many organizations are both) are invited to participate in future workshops, the pilot study and to comment on the project.

Relationship to Other Standards Organizations

The IUSR Project is currently an independent effort, not connected with any other standards body. We are working with representatives from other standards bodies such as ISO, IEEE, and Human Factors and Ergonomics Society to minimize overlap in our respective efforts and to leverage each other’s work. The SIGCHI liaison to Standards is a member of our workshop. The USER Project common reporting format is consistent with ISO 9241-11 and ISO 13407. We view this work as an implementation of that ISO work.

We are currently investigating approaches to turning over a later version of our work to a standards body when we have adequate validation of the usefulness of this work.

Future Directions

This first version of the IUSR Project will be expanded and refined if there is sufficient interest on the part of the business community. Participants in the pilot project will contribute results to a database maintained at NIST. This information will be used to refine the product. We encourage organizations that would like to participate or comment to contact the NIST representative at iuwinfo [at] nist.gov (iuwinfo[at]nist[dot]gov)

White Paper available for download in Word 97 format