Go to the NIST home page Go to the TAC Home Page TAC Banner

Return to TAC Homepage

TAC 2009 Tracks
    KBP
    RTE
    Summarization
Call for Participation
Reporting Guidelines
TAC 2009 Workshop




TAC 2009 Summarization Track


The goal of the Summarization Track is to foster research on systems that produce short, coherent summaries of text. The 2009 Summarization Track has two tasks:

  1. Update Summarization: The update summarization task is to write a short (~100 words) summary of a set of newswire articles, under the assumption that the user has already read a given set of earlier articles. The summaries will be evaluated for overall responsiveness and content (based on Columbia University's Pyramid Method). The update summarization task was first run on a large scale in the TAC 2008 Summarization Track.
  2. Automatically Evaluating Summaries Of Peers (AESOP): The AESOP task is to automatically score a summary for a given metric. The focus in 2009 is on metrics that reflect summary content, such as overall responsiveness and Pyramid scores. AESOP is a new task in 2009 and complements the basic summarization task by building a collection of automatic evaluation tools that support development of summarization systems.

NIST will distribute all required test data at no cost to track participants via password-protected web pages. Test documents for summarization will come from the AQUAINT-2 collection, and track participants must submit the AQUAINT-2 User Agreement to NIST.

Mailing List

The mailing list for the Summarization Track is duc_list@nist.gov. The list is used to discuss and define the task guidelines for the track, as well as for general discussion related to summarization and its evaluation. To subscribe, send a message to listproc@email.nist.gov such that the body consists of the line:
            subscribe duc_list <FirstName> <LastName>
In order for your messages to get posted to the list, you must send them from the email address used when you subscribed to the list. To unsubscribe, send a message from the subscribed email address to listproc@email.nist.gov such that the body consists of the line:
            unsubscribe duc_list
For additional information on how to use mailing lists hosted at NIST, send a message to listproc@email.nist.gov such that the body consists of the line:
            HELP

Schedule

Preliminary TAC 2009 Summarization Track Schedule
July 1Release of test data (Update task)
July 15Deadline for participants' submissions (Update task)
August 24Release of test data (AESOP)
August 30Deadline for participants' submissions (AESOP)
September 4Release of individual evaluated results (Update task, AESOP)
September 25Deadline for TAC 2009 workshop presentation proposals
October 22Deadline for systems' reports


BACK to Information for All TAC 2009 Track Participants

NIST is an agency of the
U.S. Department of Commerce

privacy policy / security notice / accessibility statement
disclaimer
FOIA

Last updated: Tuesday, 28-Mar-2017 11:20:40 EDT
Comments to: tac-web@nist.gov