Go to the NIST home page Go to the TAC Home Page TAC Banner

Return to TAC Homepage

About TAC
TAC KBP 2014
Past Tracks
Past Data
Publications
Contact




TAC 2008 Summarization Track

The goal of the Summarization Track is to develop systems that produce short, coherent summaries of text. The TAC Summarization Track continues the efforts of the Document Understanding Conference; please visit the DUC website for additional information about past summarization tasks and results.

The 2008 Summarization Track will have two tasks:
  1. Update Task: Piloted in DUC 2007, the update summarization task is to write a short (~ 100-word) summary of a set of newswire articles, under the assumption that the user has already read a given set of earlier articles. The summaries will be evaluated for readability and content (based on Columbia University's Pyramid Method). Test documents will come from the AQUAINT-2 collection, and task participants must submit the AQUAINT-2 User Agreement to NIST.
  2. Opinion Pilot: The pilot task will be to write summaries of opinions from blogs. Participants will be given the questions from the TAC QA Track and the text snippets output by QA systems, and will produce short coherent summaries of the answers to the questions, either from the text snippets themselves, or from the associated documents. Summaries will be assessed for both readability and content. Evaluation of content will be based on the Nugget Pyramid Method used in the QA Track. Test documents will come from the Blog06 collection, and task participants must submit the Blog06 User Agreement to NIST.

It is anticipated that NIST will distribute all required test data (including documents for summarization) at no cost to track participants via password-protected web pages. Participants in the Opinion Summarization Pilot should subscribe to the mailing lists for both the Question Answering and the Summarization Tracks.

Mailing List

The mailing list for the Summarization Track is duc_list@nist.gov. The list is used to discuss and define the task guidelines for the track, as well as for general discussion related to summarization and its evaluation. To subscribe, send a message to listproc@email.nist.gov such that the body consists of the line:
            subscribe duc_list <FirstName> <LastName>
In order for your messages to get posted to the list, you must send them from the email address used when you subscribed to the list. For additional information on how to use mailing lists hosted at NIST, send a message to listproc@email.nist.gov such that the body consists of the line:
            HELP

Schedule

TAC 2008 Summarization Track Schedule
July 1Release of test data (Update task)
July 14, 6:00 AM (EDT)EXTENDED Deadline for participants' submissions (Update task)
late AugustRelease of individual evaluated results (Update task)
August 22Release of test data (Opinion pilot)
September 2, 11:59 PM (EDT)Deadline for participants' submissions (Opinion pilot)
early OctoberRelease of individual evaluated results (Opinion pilot)

BACK to All TAC 2008 Tracks

NIST is an agency of the
U.S. Department of Commerce

privacy policy / security notice / accessibility statement
disclaimer
FOIA

Last updated: Tuesday, 19-Oct-2010 11:34:17 EDT
Comments to: tac-web@nist.gov