CIFter - Letter of Invitation

George Mason University

m/s 3 f 5

Fairfax, Virginia 22030-4444

September 07, 2000

Greetings.

You are receiving this note because you told Eric Burnette, Emile Morse, Sharon Laskowski, or Wayne Gray that you were interested in participating in the NIST CIFter web usability study. Let me take a few paragraphs to remind you about CIFter.

OVERVIEW

The CIFter study is intended to be an in-depth usability evaluation of the same website by multiple usability experts. The goal of CIFter is to identify and quantify the best web usability practices and thereby start the process of defining web usability standards. NIST (the National Institute of Standards and Technology) and the personal finance website, The Motley Fool, are putting a limited version of the Motley Fool website on a CD-ROM. (I will refer to this CD as the CIFter website to avoid possible confusion with the live and always changing Motley Fool website.). As this was going on, Motley Fool and NIST were working to develop a set of usability questions. These questions are intended to be the same sorts of questions that Motley Fool would pose to an outside consultant whom they had hired to evaluate their website.

We have two goals for the CIFter study. The first goal is to identify web usability practices that are efficient, economical, and effective. To achieve this goal we will be asking 6-18 usability evaluation groups to perform evaluations of the CIFter website. I will take the lead in comparing and contrasting the evaluations and in ensuring that the results are disseminated in all of the usual professional outlets. Our second goal is to publish a second CD that will contain the outcome of CIFter. This CD could be used by students or anyone else to compare their evaluation of the Motley Fool website with the evaluations conducted during CIFter. This data set of evaluations may well become the benchmark for training the next generation of usability evaluation specialists.

DETAILS

1. CIFter is a compound acronym. The CIF is the Common Industry Format for reporting the results of usability evaluations. The CIF has been developed by a consortium of developer and user companies to standardize expectations regarding usability testing and reporting (for background, see http://www.nist.gov/iusr). The "ter" in CIFter, is the "test, evaluation, and report" of the CIF. NIST has tested the CIF on conventional (i.e., non-web) software. Part of CIFter will be a test of the CIF as a reporting instrument for usability evaluations of websites.

2. Another part of CIFter will be the incorporation into the CIFter CD of three usability analysis software tools for the quantitative evaluation of websites: WebVIP, VISVIP, and JWands. WebVIP is an event capture tool. VISVIP is a user path visualization tool. JWands is a tool developed by Andy Sears to simulate the effect of different connection speeds (e.g., 28,800 versus T1) on website use.

3. We are looking for 6 to 18 groups of web usability experts to do a usability evaluation of the CIFter website. Our intent is to find a diverse assortment of individuals and groups. In this diversity we hope to approximate the diversity of groups out there who are currently doing website evaluation. For example, we are looking for university professors who may want to incorporate the evaluation into a class project, in-house usability labs that typically do evaluation of software developed internally, in-house labs that typically evaluate software (web or non-web) before or after it is purchased from someone else, and companies or individuals that do usability evaluation for hire. If one of these descriptions does not fit you then let me know and I will amend my list. Our goal is to be inclusive not exclusive. If you do or teach usability evaluation for a living, then we want you to participate in CIFter.

EXPECTATIONS

We are asking you to do a usability evaluation for free. To keep this manageable, the level of effort that we are expecting each group to donate is approximately one person-month. Those of you who are accustomed to "billable hours" should keep count of what the billable hours would be for your involvement. Those of you who do not keep billable hours should do so for this effort.

TECHNIQUES

NIST has developed a suite of usability analysis software tools that will enable more quantitative evaluations of websites than is the current norm. NIST is giving these tools away along with detailed instructions on how to use them. NIST would be happy to answer any questions you might have regarding these tools. You are free to keep the tools and to use them in other projects.

As to what techniques you use to do your evaluation for CIFter -- that is largely up to you. You can do whatever you normally do, you can mix what you do with NIST's tools or you can invent new methods of usability evaluation just for CIFter. Our only requirement is that you document what you do in sufficient detail that another group (remember those students using your evaluation as the "gold standard") can duplicate your methodology. The CIFter project has a strong bias towards empirical usability testing. If you want to combine empirical testing with more analytic methods, that is fine. However, we do ask that you do more than a simple Heuristic Evaluation.

COMMITMENT

Recap -- We are asking you to devote approximately one person-month of your time plus other resources (e.g., usability test participants) to participate in CIFter. Your effort can be self-paced but we need to have you complete your evaluation and submit all materials to Wayne Gray by December 15th 2000. The only exceptions are university classes that must wait until Spring 2001 to begin their involvement in CIFter.

If you choose to pursue CIFter we need you to make several commitments. First, if you work for an organization, we need some sort of commitment from that organization that you will be provided with the time and resources required to do the work. Second, we need some acknowledgment from you and your organization that all CIFter related documents and data that you provide as part of the evaluation can be publicly distributed, summarized, or incorporated in any way, shape, or form that NIST sees fit. As a minimum, this information will be provided to the other evaluators, will be used by me to create my final report to NIST, and some of it will find its way into professional papers and presentations.

SUPPORT DURING THE EVALUATION

The CD is scheduled for release on September 22th. The CD will contain two copies of the CIFter website, one instrumented with NIST usability software tools and one non-instrumented. The CD will also contain instructions on how to use the tools. Also included will be the list of usability issues generated by the Motley Fool Usability Team and NIST.

Due to technical issues, the CIFter website will only support the Internet Explorer 5 Browser (software included on CD). All of us involved view this as regrettable. However, efforts to resolve this issue threatened to delay the start of this project more than we wished. We are looking forward to the next version of Netscape and hope to add it to any future evaluations.

During the evaluation you will have access to me, Wayne Gray, and through me you will have access to the Motley Fool Usability Team. Please do not attempt an end run. My role will be to filter things slightly so that the Motley Fool team is not answering the same question 6-18 times.

DO YOU WANT TO PLAY WITH US?

So, do you? If so, then please let me know as soon as you can. If you need me or NIST or Motley Fool to talk with someone in your organization then please let me know.

If you have questions, please send them to me. I will forward them to NIST or The Motley Fool as appropriate.

Note that in a very real sense, we are defining much of this as we go. If you choose to participate you will become a partner and at various points in the project, we will be looking to you for advice.

Cheers,

Wayne D. Gray
Program Director, Human Factors & Applied Cognitive
George Mason University

CONTACT INFORMATION:

Email: gray [at] gmu.edu (gray[at]gmu[dot]edu)
Phone: 703-993-1357
Website: http://hfac.gmu.edu/~gray

<Return to the CIFter Home Page>