TAC 2009 Summarization Track
The goal of the Summarization Track is to foster research on systems that produce short, coherent summaries of text. The 2009 Summarization Track has two tasks:
- Update Summarization: The update summarization task is to write a short (~100 words) summary of a set of newswire articles, under the assumption that the user has already read a given set of earlier articles. The summaries will be evaluated for overall responsiveness and content (based on Columbia University's Pyramid Method). The update summarization task was first run on a large scale in the TAC 2008 Summarization Track.
- Automatically Evaluating Summaries Of Peers (AESOP): The AESOP task is to automatically score a summary for a given metric. The focus in 2009 is on metrics that reflect summary content, such as overall responsiveness and Pyramid scores. AESOP is a new task in 2009 and complements the basic summarization task by building a collection of automatic evaluation tools that support development of summarization systems.
NIST will distribute all required test data at no cost to track participants via password-protected web pages. Test documents for summarization will come from the AQUAINT-2 collection, and track participants must submit the AQUAINT-2 User Agreement to NIST.
The mailing list for the Summarization Track is firstname.lastname@example.org. The list is used to discuss and define the task guidelines for the track, as well as for general discussion related to summarization and its evaluation. To subscribe, send a message to email@example.com such that the body consists of the line:
subscribe duc_list <FirstName> <LastName>
In order for your messages to get posted to the list, you must send
them from the email address used when you subscribed to the list.
To unsubscribe, send a message from the subscribed email address to firstname.lastname@example.org such that the body consists of the line:
For additional information on how to use mailing lists hosted at NIST,
send a message to email@example.com such
that the body consists of the line:
|Preliminary TAC 2009 Summarization Track Schedule|
|July 1||Release of test data (Update task)|
|July 15||Deadline for participants' submissions (Update task)|
|August 24||Release of test data (AESOP)|
|August 30||Deadline for participants' submissions (AESOP)|
|September 4||Release of individual evaluated results (Update task, AESOP)|
|September 25||Deadline for TAC 2009 workshop presentation proposals|
|October 22||Deadline for systems' reports|