TAC 2008 Summarization Track
The goal of the Summarization Track is to develop systems that produce short, coherent summaries of text. The TAC Summarization Track continues the efforts of the Document Understanding Conference; please visit the DUC website for additional information about past summarization tasks and results.
The 2008 Summarization Track will have two tasks:
- Update Task: Piloted in DUC 2007, the update summarization task is to write a short (~ 100-word) summary of a set of newswire articles, under the assumption that the user has already read a given set of earlier articles. The summaries will be evaluated for readability and content (based on Columbia University's Pyramid Method). Test documents will come from the AQUAINT-2 collection, and task participants must submit the AQUAINT-2 User Agreement to NIST.
- Opinion Pilot: The pilot task will be to write summaries of opinions from blogs. Participants will be given the questions from the TAC QA Track and the text snippets output by QA systems, and will produce short coherent summaries of the answers to the questions, either from the text snippets themselves, or from the associated documents. Summaries will be assessed for both readability and content. Evaluation of content will be based on the Nugget Pyramid Method used in the QA Track. Test documents will come from the Blog06 collection, and task participants must submit the Blog06 User Agreement to NIST.
It is anticipated that NIST will distribute all required test data (including documents for summarization) at no cost to track participants via password-protected web pages.
Participants in the Opinion Summarization Pilot should subscribe to the mailing lists for both the Question Answering and the Summarization Tracks.
The mailing list for the Summarization Track is firstname.lastname@example.org. The list is used to discuss and define the task guidelines for the track, as well as for general discussion related to summarization and its evaluation. To subscribe, send a message to email@example.com such that the body consists of the line:
subscribe duc_list <FirstName> <LastName>
In order for your messages to get posted to the list, you must send
them from the email address used when you subscribed to the list. For
additional information on how to use mailing lists hosted at NIST,
send a message to firstname.lastname@example.org such
that the body consists of the line:
|TAC 2008 Summarization Track Schedule|
|July 1||Release of test data (Update task)|
|July 14, 6:00 AM (EDT)||EXTENDED Deadline for participants' submissions (Update task)|
|late August||Release of individual evaluated results (Update task)|
|August 22||Release of test data (Opinion pilot)|
|September 2, 11:59 PM (EDT)||Deadline for participants' submissions (Opinion pilot)|
|early October||Release of individual evaluated results (Opinion pilot)|