Skip to main content

NOTICE: Due to a lapse in annual appropriations, most of this website is not being updated. Learn more.

Form submissions will still be accepted but will not receive responses at this time. Sections of this site for programs using non-appropriated funds (such as NVLAP) or those that are excepted from the shutdown (such as CHIPS and NVD) will continue to be updated.

U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Contradictions and Justifications: Extensions to the Textual Entailment Task

Published

Author(s)

Ellen M. Voorhees

Abstract

The third PASCAL Recognizing Textual Entailment Challenge (RTE-3) contained an optional task that extended the main entailment task by requiring a system to make three-way entailment decisions (entails, contradicts, neither) and to justify its response.  Contradiction was rare in the RTE-3 test set, occurring in only about 10% of the cases, and systems found accurately detecting it difficult.  Subsequent analysis of the results shows a test set must contain many more entailment pairs for the three-way task than the traditional two-way task to have equal confidence in system comparisons.  Each of six human judges representing eventual end users rated the quality of a justification by assigning "understandability" and "correctness" scores.  Ratings of the same justification across judges differed significantly, signaling the need for a better characterization of the justification task.
Proceedings Title
Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL 2008)
Conference Dates
June 15-20, 2008
Conference Location
Columbus, OH
Conference Title
46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL 2008)

Keywords

natural language processing, textual entailment

Citation

Voorhees, E. (2008), Contradictions and Justifications: Extensions to the Textual Entailment Task, Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL 2008), Columbus, OH, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=152115 (Accessed October 14, 2025)

Issues

If you have any questions about this publication or are having problems accessing it, please contact [email protected].

Created June 16, 2008, Updated February 19, 2017
Was this page helpful?