TAC 2008 Question Answering Track
The goal of the QA Track is to develop systems that search large document collections and retrieve precise answers to questions (rather than entire documents). The TAC QA track evolved from the TREC QA track. Please visit the TREC website for additional information about Question Answering at TREC, including an open archive of TREC QA data and instructions on how to obtain additional detailed Past TREC Results for individual submitted runs.
The 2008 QA Track will focus on finding answers to short series of opinion questions, where each series of 2-4 questions will be about a particular target. There will be two types of questions, each with its own evaluation metric.
Answers must be retrieved from documents in the Blog06 corpus. Participants should obtain the Blog06 collection directly from the University of Glasgow. (See http://ir.dcs.gla.ac.uk/test_collections/.) As a service to those participants who do not wish to run their own search engines over the entire Blog06 corpus, NIST will distribute the top 50 documents for each target (as retrieved by a Lucene-based search engine) at not cost to track participants; participants wishing access to these top documents must submit the Blog06 User Agreement to NIST (even if the group has already submitted the agreement to the University of Glasgow to obtain the entire collection).
To improve the quality and completeness of the pool of answers in the QA track, NIST will accept manual runs in addition to fully automatic ones. A submission is considered to be a manual run if it involves any human intervention in response to the particular test data, whether it is simply to put the questions into a canonical form for an otherwise fully automatic QA system, or whether it entails a person performing the entire question answering task. At the time of submission, participants must fully describe the extent and nature of any manual intervention. Participants who submit a manual run must also submit a fully automatic run.
Joint Blog Task The TAC 2008 QA task is part of a larger information retrieval, analysis, and synthesis task involving the TREC Blog Track and the TAC Summarization Track. A subset of the targets from the QA Track will be the basis for TREC Blog Track topics, where the task in TREC will be to retrieve documents containing opinions about the target; the TREC Blog Track homepage contains additional information about the document retrieval task in the Blog Track. To support an additional level of text analysis, the answer strings and supporting documents that are returned by TAC QA participants will be given to TAC Summarization participants, who will create fluent summaries of the answers to the questions from the QA track.
The mailing list for the Question Answering Track is firstname.lastname@example.org. The list is used to discuss and define the task guidelines for the track, as well as for general discussion related to QA and its evaluation. To subscribe, send a message to email@example.com such that the body consists of the line:
NIST is an agency of the
U.S. Department of Commerce
Last updated: Tuesday, 19-Oct-2010 11:25:40 EDT
Comments to: firstname.lastname@example.org