TAC 2011 Summarization Track
TAC 2011 Summarization Tasks
The goal of the Summarization Track is to foster research on systems that produce short, coherent summaries of text. The 2011 Summarization Track has three tasks:
- Guided Summarization: The goal of guided summarization is to encourage a deeper linguistic (semantic) analysis of the source documents instead of relying only on document word frequencies to select important concepts. The guided summarization task is to write a 100-word summary of a set of 10 newswire articles for a given topic, where the topic falls into a predefined category. Participants (and human summarizers) are given a list of aspects for each category, and a summary must include all aspects found for its category. Additionally, an "update" component of the guided summarization task is to write a 100-word "update" summary of a subsequent 10 newswire articles for the topic, under the assumption that the user has already read the earlier articles.
Summaries will be evaluated for readability, content, and overall responsiveness. (The guided summarization task was run in the TAC 2010 Summarization Track.)
- Automatically Evaluating Summaries Of Peers (AESOP): The AESOP task is to automatically score a summary for a given metric. AESOP complements the basic summarization task by building a collection of automatic evaluation tools that support development of summarization systems. (AESOP was run in the TAC 2010 Summarization Track.)
- Multiling Pilot: The aim of the MultiLing pilot is to foster and promote the use of multi-lingual algorithms for summarization. This includes the effort of transforming an algorithm or a set of resources from a mono-lingual to a multi-lingual version.
What's NEW in 2011
- Guided Summarization: Redundancy in the update summaries will be evaluated with respect to pyramids for inital summaries. Test documents for the Guided Summarization task will come from the newswire portion of the TAC 2010 KBP Source Data, rather than AQUAINT and AQUAINT-2.
- AESOP: In addition to metrics that reflect summary content (Pyramid, Responsiveness), AESOP will target Readability in 2011. Correlations at the summary level (within each topic) will also be reported.
- Multilingual Summarization Pilot: A new muliti-lingual summarization pilot is planned. See Multiling Pilot Home Page for details.
Data and User Agreements
Test documents for the Guided Summarization and AESOP tasks will come from the newswire portion of the TAC 2010 KBP Source Data, which will be distributed by the LDC. Test documents for the Multiling Pilot will be based on WikiNews. See TAC 2011 Summarization Registration Instructions for how to submit agreement forms and obtain data.
The mailing list for the Summarization Track is firstname.lastname@example.org. The list is used to discuss and define the task guidelines for the track, as well as for general discussion related to summarization and its evaluation. To subscribe, send a message to email@example.com such that the body consists of the line:
subscribe duc_list <FirstName> <LastName>
In order for your messages to get posted to the list, you must send
them from the email address used when you subscribed to the list.
To unsubscribe, send a message from the subscribed email address to firstname.lastname@example.org such that the body consists of the line:
For additional information on how to use mailing lists hosted at NIST,
send a message to email@example.com such
that the body consists of the line:
(See Multiling Pilot Home Page for Multiling Pilot schedule)
|TAC 2011 Summarization Track Schedule|
|by May 1||TAC 2010 KBP Source Data available from the LDC|
|June 10||Deadline for TAC 2011 track registration|
|July 1||Release of test data (Guided task)|
|July 17||Deadline for participants' submissions (Guided task)|
|August 22||Release of test data (AESOP)|
|August 28||Deadline for participants' submissions (AESOP)|
|September 7||Release of individual evaluated results (Guided task, AESOP)|
|September 25||Deadline for TAC 2011 workshop presentation proposals|
|October 25||Deadline for system reports (workshop notebook version)|
|November 14-15||TAC 2011 Workshop|