Agreement Concerning Dissemination of TAC Results
The annual Text Analysis Conference (TAC) is an event in which
organizations with an interest in Natural Language Processing
take part in a coordinated series of experiments using the same
experimental data. The results of these individual experiments
are then presented at a workshop where tentative comparisons may be made.
In order to preserve the desired pre-competitive nature of these conferences,
the organizers have developed a set of guidelines constraining the
dissemination and publication of TAC evaluation results. These guidelines
are meant to preclude the publication of incomplete or inaccurate information
that could damage the reputation of the conference or its participants and
could discourage participation in future conferences. The guidelines apply
to all TAC participants and to organizations requesting access to the archive
of TAC results.
A signed agreement is required of each organization participating in
the TAC evaluations or accessing the results archive. Any organization
that is found to have violated the terms or spirit of the agreement may be
denied participation in future TACs.
- SCIENTIFIC OR TECHNICAL PUBLICATIONS: Scientific or technical
publications, including newsletters from universities or research
laboratories, should adhere to community standards for fairness and
objectivity and should accurately and clearly state the limitations of
the testing conditions and other factors which might influence scores.
The experimental nature of the tasks, data and evaluation procedures
should also be stated. The full conference proceedings should always
- ADVERTISEMENTS: No advertisements using the TAC evaluation results
can be placed in magazines, journals, newspapers, or other publications.
- PRESS RELEASES: Press releases about TAC results to organizations
with national/international coverage are also prohibited.
- MARKETING LITERATURE, LOCAL NEWSLETTERS: Although it is recognized
that extensive evaluation discussions are not appropriate in this type
of literature, it is expected that any claims made on the basis of
evaluation results are accurate, that the evaluation measures used to
substantiate these claims are stated, and that a reference is made to
the full conference proceedings. Where promotional material is subject
to prepublication revision by the media, the author should make every
effort to see that the revision does not cause a violation of the guidelines.
- WEB SITES: Web sites are a type of marketing literature with a global
distribution. The guidelines for marketing literature in general apply to web sites
as well. References to TAC should link to the TAC web site.
- Cross-system comparisons may not be made with other named teams for
individual tests, and may only be made when they are supported by
accepted methods of statistical significance testing. Comparisons must be
accompanied by the results of those tests and should reference the
publication of those tests in the conference proceedings or related
literature. Informal, qualitative comparisons with recognized
baselines or benchmarks, and with general levels or trends in performance,
must be clearly stated to be such and thus open to statistical reassessment.
- The TAC program manager must be notified of any publication
that quotes or contains references to the TAC evaluation
results, and must be provided with a copy of such publication
ENDORSEMENT (Please type):
I, _________________________________________________ (name), acting on
behalf of ____________________________________________ (organization),
acknowledge my understanding of the above guidelines and accept this
agreement. As a condition of our participation in the TAC evaluation,
and/or gaining access to the archive of TAC results, my organization
and its employees and representatives agree to abide
by these guidelines. I am qualified to make this commitment.
NIST is an agency of the
U.S. Department of Commerce