Go to the NIST home page Go to the TAC Home Page TAC Banner

Return to TAC Homepage

About TAC
TAC KBP 2014
Past Tracks
Past Data
Publications
Contact




TAC 2008: Call for Participation

Text Analysis Conference (TAC)
http://www.nist.gov/tac/

Track Evaluations: February - October 2008
Workshop: November 17-19, 2008

Conducted by:
National Institute of Standards and Technology (NIST)

With support from:
Intelligence Advanced Research Projects Activity (IARPA)

The Text Analysis Conference (TAC) is a new series of evaluation workshops organized to encourage research in Natural Language Processing and related applications, by providing a large test collection, common evaluation procedures, and a forum for organizations to share their results. TAC grew out of the Document Understanding Conference (DUC) and the Question Answering Track of the Text REtrieval Conference (TREC). TAC comprises a set of tasks known as "tracks," each of which focuses on a particular subproblem of NLP.

You are invited to participate in TAC 2008. NIST will provide test data for each track, and track participants will run their NLP systems on the data and return their results to NIST for evaluation. Organizations may choose to participate in any or all of the tracks. The annual conference culminates in a November workshop at NIST in Gaithersburg, Maryland. Dissemination of TAC work and results other than in the (publicly available) conference proceedings is welcomed, but the conditions of participation specifically preclude any advertising claims based on TAC results. All results submitted to NIST are published in the proceedings and are archived on the TAC web site.

Task Description

TAC 2008 will have three tracks: Question Answering, Recognizing Textual Entailment, and Summarization. Each track is a continuation of an evaluation series previously organized under different frameworks, though specific tasks in a track may differ from previous years.

Below is a brief summary of the tracks. The exact definition of the tasks to be performed in each TAC 2008 track is still being formulated. Track discussion takes place on the track mailing list. To be added to a track mailing list, follow the instructions given in the track web page for contacting the mailing list. For questions about the track, send mail to the track coordinator (or post the question to the track mailing list once you join).

  • Question Answering
    The goal of the QA Track is to develop systems that search large document collections to retrieve precise answers to questions, rather than entire documents.
    Track coordinator: Hoa Trang Dang, hoa.dang@nist.gov
    Track web page: http://www.nist.gov/tac/2008/qa/
    Mailing list: trec-qa@nist.gov
  • Recognizing Textual Entailment
    The goal of the RTE Track is to develop systems that recognize when one piece of text entails another.
    Track coordinator: Danilo Giampiccolo, giampiccolo@celct.it
    Track web page: http://www.nist.gov/tac/2008/rte/
    Mailing list: rte@nist.gov
  • Summarization
    The goal of the Summarization Track is to develop systems that produce short, coherent summaries of text.
    Track coordinator: Hoa Trang Dang, hoa.dang@nist.gov
    Track web page: http://www.nist.gov/tac/2008/summarization/
    Mailing list: duc_list@nist.gov

Conference Format

TAC 2008 culminates in a November workshop that is used as a forum both for presentation of results (including failure analyses and system comparisons), and for more lengthy system presentations describing techniques used, experiments run on the data, and other issues of interest to researchers in NLP. Track participants interested in giving a presentation during the workshop should submit a 500-word abstract in September describing the experiments they performed. As there is a limited amount of time for oral presentations, the TAC advisory committee will use the abstracts to determine which participants are asked to speak and which will present in a poster session.

Data

For most tracks, training and test materials are available from NIST; some tracks may use special collections that are available from other organizations for a fee. Many of the existing TREC/DUC data (documents, topics, and judgments) are available for training purposes and may also be used in some of the tracks.

All documents are typical of those seen in a real-world situation (i.e., there will not be arcane vocabulary, but there may be missing pieces of text or typographical errors). For most tracks, the judgments against which each system's output will be scored will be made by experienced assessors based on manual analysis and the pooled output of all track participants.

Registration for TAC 2008 Tracks

Organizations wishing to participate in any of the TAC 2008 tracks should register online by March 4, 2008. Registration for a track does not commit you to participating in the track, but is helpful to know for planning. Late registration will be permitted only if resources allow. Any questions about conference participation should be sent to the TAC project manager: tac-manager@nist.gov.

Schedule

    Tentative TAC 2008 Schedule
    March 4Deadline for registration for track participation
    Beginning March 4Submit signed User Agreements to NIST
    mid June - early SeptemberDeadlines for results submission for most tracks
    September 16 (estimated)Deadline for workshop presentation proposals
    September 16 (estimated)Deadline for proposals for future tracks/tasks
    By early OctoberRelease of individual evaluated results to participants
    mid OctoberDeadline for participants' notebook papers
    November 17-19TAC 2008 workshop in Gaithersburg, Maryland, USA
    early February 2009Deadline for participants' final proceedings papers

Advisory Committee

    John Burger, MITRE
    John Conroy, IDA/CCS
    Ido Dagan, Bar Ilan University
    Hoa Trang Dang, NIST (chair)
    Maarten de Rijke, University of Amsterdam
    Bonnie Dorr, University of Maryland
    Donna Harman, NIST
    Andy Hickl, Language Computer Corporation
    Ed Hovy, ISI/USC
    Boris Katz, Massachusetts Institute of Technology
    Bernardo Magnini, ITC-irst
    Kathy McKeown, Columbia University
    Ani Nenkova, University of Pennsylvania
    Drago Radev, University of Michigan
    Lucy Vanderwende, Microsoft Research
    Ellen Voorhees, NIST
    Ralph Weischedel, BBN

NIST is an agency of the
U.S. Department of Commerce

privacy policy / security notice / accessibility statement
disclaimer
FOIA

Last updated: Monday, 18-Oct-2010 15:49:22 EDT
Comments to: tac-web@nist.gov