TAC KBP 2014
TAC 2010 Summarization Track
TAC 2010 Summarization Tasks
The goal of the Summarization Track is to foster research on systems that produce short, coherent summaries of text. The 2010 Summarization Track has two tasks:
- Guided Summarization: The goal of guided summarization is to encourage a deeper linguistic (semantic) analysis of the source documents instead of relying only on document word frequencies to select important concepts. The guided summarization task is to write a 100-word summary of a set of 10 newswire articles for a given topic, where the topic falls into a predefined category. Participants (and human summarizers) are given a list of aspects for each category, and a summary must include all aspects found for its category. Categories and their aspects might include:
- Accidents and Natural Disasters: what happened; date; location; reasons for accident/disaster; casualties; damages; rescue efforts/countermeasures
- Attacks: what happened; date; location; casualties; damages; perpetrators; rescue efforts/countermeasures
- Health and Safety: what is the issue; who is affected; how they are affected; why it happens; countermeasures
- Endangered Resources: description of resource; [importance of resource; threats to resource; countermeasures
- Trials and Investigations: who is under investigation, who is investigating/suing; why (general); specific charges; sentence/consequences; how do they plead/react to charges
Additionally, an "update" component of the guided summarization task is to write a 100-word "update" summary of a subsequent 10 newswire articles for the topic, under the assumption that the user has already read the earlier articles.
(The update summarization task was run in the Summarization track of TAC 2008 and TAC 2009.) Summaries will be evaluated for readability, content, and overall responsiveness.
- Automatically Evaluating Summaries Of Peers (AESOP): The AESOP task is to automatically score a summary for a given metric. As in 2009, the focus is on metrics that reflect summary content. AESOP complements the basic summarization task by building a collection of automatic evaluation tools that support development of summarization systems.
The mailing list for the Summarization Track is firstname.lastname@example.org. The list is used to discuss and define the task guidelines for the track, as well as for general discussion related to summarization and its evaluation. To subscribe, send a message to email@example.com such that the body consists of the line:
subscribe duc_list <FirstName> <LastName>
In order for your messages to get posted to the list, you must send
them from the email address used when you subscribed to the list.
To unsubscribe, send a message from the subscribed email address to firstname.lastname@example.org such that the body consists of the line:
For additional information on how to use mailing lists hosted at NIST,
send a message to email@example.com such
that the body consists of the line:
|TAC 2010 Summarization Track Schedule|
|June 22||Release of test data (Guided task)|
|July 6||Deadline for participants' submissions (Guided task)|
|August 23||Release of test data (AESOP)|
|August 29||Deadline for participants' submissions (AESOP)|
|September 7||Release of individual evaluated results (Guided task, AESOP)|
|September 26||Deadline for TAC 2010 workshop presentation proposals|
|October 27||Deadline for systems' reports|