Go to the NIST home page Go to the TAC Home Page TAC Banner

Return to TAC Homepage

TAC 2009 Tracks
    KBP
    RTE
    Summarization
Call for Participation
Reporting Guidelines
TAC 2009 Workshop




PASCAL Recognizing Textual Entailment Challenge (RTE-5) at TAC 2009


Introduction

Given two text fragments called 'Text' and 'Hypothesis', Textual Entailment Recognition is the task of determining whether the meaning of the Hypothesis is entailed (can be inferred) from the Text. The goal of the first RTE Challenge was to provide the NLP community with a benchmark to test progress in recognizing textual entailment, and to compare the achievements of different groups. Since its inception in 2004, the PASCAL RTE Challenges have promoted research in textual entailment recognition as a generic task that captures major semantic inference needs across many natural language processing applications, such as Question Answering (QA), Information Retrieval (IR), Information Extraction (IE), and multi-document summarization.

After the first three highly successful PASCAL RTE Challenges, RTE became a track at the 2008 Text Analysis Conference, which brought it together with communities working on NLP applications (specifically, QA and summarization). The interaction has provided the opportunity to apply RTE systems to specific applications and to move the RTE task towards more realistic application scenarios.

What's new in RTE-5

The main RTE-5 task is similar to the RTE-4 task, with the following changes:

  • The average length of the Texts will be higher.
  • Texts will come from a variety of sources and will not be edited from their source documents. Thus, systems will be asked to handle real text that may include typographical errors and ungrammatical sentences.
  • A development set will be released.
  • The textual entailment recognition task would be based on only three application settings: QA, IE, and IR.

In addition to the main task (Textual Entailment Recognition), RTE-5 will offer a new Textual Entailment Search pilot that is situated in the summarization application setting, where the task would be to find all Texts in a set of documents that entail a given Hypothesis.

Schedule

RTE-5 Schedule
April 3Release of Development Set (Search Pilot)
May 29Release of Development Set (Main Task)
May 31Deadline for TAC 2009 track registration
September 2Release of Test Set (Main Task, Search Pilot)
September 9Deadline for task submissions (Main Task, Search Pilot)
September 18Release of individual evaluated results (Main Task, Search Pilot)
September 25Deadline for TAC 2009 workshop presentation proposals
October 22Deadline for systems' reports

Mailing List

The mailing list for the RTE Track is rte@nist.gov. The list is used to discuss and define the task guidelines for the track, as well as for general discussion related to textual entailment and its evaluation. To subscribe, send a message to listproc@email.nist.gov such that the body consists of the line:
            subscribe rte <FirstName> <LastName>
In order for your messages to get posted to the list, you must send them from the email address used when you subscribed to the list. To unsubscribe, send a message from the subscribed email address to listproc@email.nist.gov such that the body consists of the line:
            unsubscribe rte
For additional information on how to use mailing lists hosted at NIST, send a message to listproc@email.nist.gov such that the body consists of the line:
            HELP

Organizing Committee

    Luisa Bentivogli, CELCT and FBK, Italy
    Ido Dagan, Bar Ilan University, Israel
    Hoa Trang Dang, NIST, USA
    Danilo Giampiccolo, CELCT, Italy
    Bernardo Magnini, FBK, Italy

BACK to Information for All TAC 2009 Track Participants

NIST is an agency of the
U.S. Department of Commerce

privacy policy / security notice / accessibility statement
disclaimer
FOIA

Last updated: Tuesday, 28-Mar-2017 11:20:39 EDT
Comments to: tac-web@nist.gov