PASCAL Recognizing Textual Entailment Challenge (RTE-7) at TAC 2011
Given two text fragments called 'Text' and 'Hypothesis', Textual
Entailment Recognition is the task of determining whether the
meaning of the Hypothesis is entailed (can be inferred) from the Text.
The goal of the first RTE Challenge was to provide the NLP community
with a benchmark to test progress in recognizing textual entailment,
and to compare the achievements of different groups. Since its
inception in 2004, the RTE Challenges have promoted research in
textual entailment recognition as a generic task that captures major
semantic inference needs across many natural language processing
applications, such as Question Answering (QA), Information Retrieval
(IR), Information Extraction (IE), and multi-document Summarization.
After the first three highly successful PASCAL RTE Challenges, RTE
became a track at the 2008 Text Analysis Conference, which brought it
together with communities working on NLP applications. The
interaction has provided the opportunity to apply RTE systems to
specific applications and to move the RTE task towards more realistic
RTE-7 pursues the direction taken in RTE-6, focusing on textual entailment in context, where the entailment decision draws on the larger context available in the targeted application settings.
The RTE-7 tasks focus on recognizing textual entailment in two application settings:
Knowledge Base Population.
- Main Task (Summarization setting): Given a corpus and a set of "candidate" sentences retrieved by Lucene from that corpus, RTE systems are required to identify all the sentences from among the candidate sentences that entail a given Hypothesis. The RTE-7 Main Task is based on the TAC Update Summarization Task. In the Update Summarization Task, each topic contains two sets of documents ("A" and "B"), where all the "A" documents chronologically precede all the "B" documents. An RTE-7 Main Task "corpus" consists of 10 "A" documents, while Hypotheses are taken from sentences in the "B" documents.
- Novelty Detection Subtask (Summarization setting): In the Novelty Detection variant of the Main Task, systems are required to judge if the information contained in each H (based on text snippets from B summaries) is novel with respect to the information contained in the A documents related to the same topic. If entailing sentences are found for a given H, it means that the content of H is not new; if no entailing sentences are detected, it means that information contained in the H is novel.
- KBP Validation Task (Knowledge Base Population setting): Based on the
TAC Knowledge Base Population (KBP) Slot-Filling task, the KBP
validation task is to determine whether a given relation
(Hypothesis) is supported in an associated document (Text). Each slot fill that is proposed by a system for the KBP Slot-Filling task would create one evaluation item for the RTE-KBP Validation Task: The Hypothesis would be a simple sentence created from the slot fill, while the Text would be the source document that was cited as supporting the slot fill.
|Proposed RTE-7 Schedule|
|April 29||Main Task: Release of Development Set|
|April 29||KBP Validation Task: Release of Development Set|
|June 10||Deadline for TAC 2011 track registration|
|August 17||KBP Validation Task: Release of Test Set|
|August 29||Main Task: Release of Test Set|
|September 8||Main Task: Deadline for task submissions|
|September 15||Main Task: Release of individual evaluated results|
|September 16||KBP Validation Task: Deadline for task submissions|
|September 23||KBP Validation Task: Release of individual evaluated results|
|September 25||Deadline for TAC 2011 workshop presentation proposals|
|September 29||Main Task: Deadline for ablation tests submissions|
|October 6||Main Task: Release of individual ablation test results|
|October 25||Deadline for system reports (workshop notebook version)|
|November 14-15||TAC 2011 Workshop|
The mailing list for the RTE Track is email@example.com. The list is used to discuss and define the task guidelines for the track, as well as for general discussion related to textual entailment and its evaluation. To subscribe, send a message to firstname.lastname@example.org such that the body consists of the line:
subscribe rte <FirstName> <LastName>
In order for your messages to get posted to the list, you must send
them from the email address used when you subscribed to the list.
To unsubscribe, send a message from the subscribed email address to email@example.com such that the body consists of the line:
For additional information on how to use mailing lists hosted at NIST,
send a message to firstname.lastname@example.org such
that the body consists of the line:
Luisa Bentivogli, CELCT and FBK, Italy
Peter Clark, Vulcan Inc., USA
Ido Dagan, Bar Ilan University, Israel
Hoa Trang Dang, NIST, USA
Danilo Giampiccolo, CELCT, Italy