Go to the NIST home page Go to the TAC Home Page TAC Banner

Return to TAC Homepage

About TAC
TAC KBP 2014
Past Tracks
Past Data
Publications
Contact




Overview

There has been a growing recognition of the importance of community-wide evaluations for research in information technologies. The Text Analysis Conference is a series of workshops that provides the infrastructure for large-scale evaluation of Natural Language Processing technology.

Mission

TAC's mission is to support research within the Natural Language Processing community by providing the infrastructure necessary for large-scale evaluation of NLP methodologies. TAC's primary purpose is not competitive benchmarking; the emphasis is on advancing the state of the art through evaluation results. In particular, the TAC workshop series has the following goals:

  • to promote research in NLP based on large common test collections;
  • to improve evaluation methodologies and measures for NLP;
  • to build a series of test collections that evolve to anticipate the evaluation needs of modern NLP systems;
  • to increase communication among industry, academia, and government by creating an open forum for the exchange of research ideas;
  • to speed the transfer of technology from research labs into commercial products by demonstrating substantial improvements in NLP methodologies on real-world problems.

Organization

TAC is organized by the Retrieval Group of the Information Access Division (IAD) in the Information Technology Laboratory at the National Institute of Standards and Technology (NIST). Initiated in 2008, TAC grew out of NIST's Document Understanding Conference (DUC) for text summarization, and the Question Answering Track of the Text Retrieval Conference (TREC). TAC is sponsored by NIST and other U.S. government agencies and is overseen by an Advisory Committee consisting of representatives from government, industry, and academia.

TAC Cycle

A TAC cycle consists of a set tracks, areas of focus in which particular NLP tasks are defined. The tracks serve several purposes. First, tracks act as incubators to experiment with new research areas; the first running of a track often defines what the problem really is, and a track creates the necessary infrastructure (test collections, evaluation methodology, etc.) to support research on its tasks. The tracks also demonstrate the robustness of core NLP technology in that the same techniques are frequently appropriate for a variety of tasks. Finally, the tracks make TAC attractive to a broader community by providing tasks that match the research interests of more groups. The TAC advisory committee selects the set of tracks that will be run in a given year of TAC based on track proposals.

The TAC cycle begins at the start of the calendar year with the Call for Track Participation, and concludes in February of the following year when the conference proceedings and track data for that cycle are archived on the TAC web site and made available to the general public. For each TAC cycle, NIST distributes test data for each track; participants run their own NLP systems on the data and return their results to NIST; NIST pools the individual results, judges them for correctness, and evaluates the results. Data and evaluated results are generally not made available to non-participants until the end of the TAC cycle.

The TAC cycle culminates with a workshop that is a forum for participants to share their experiences and plan future tasks/evaluations. The workshop is usually held at NIST during the last full week before the U.S. Thanksgiving holiday. Attendance at TAC workshops is open to the public, but workshop attendees must pre-register in order to enter the NIST campus.

General TAC Schedule
MarchDocument disks available to registered track participants
February-April**Discussion/refinement of track guidelines on mailing lists
by MayTrack guidelines finalized for most tracks
late Spring**Deadline for registration for Track Participation
July-August**Deadline for participants' submissions for most tracks
August - SeptemberRelease of individual evaluated results to participants
mid September**Deadline for workshop presentation proposals
mid September**Deadline for proposals for future tracks/tasks
mid October**Deadline for workshop notebook papers
mid November**TAC Workshop
early February**Deadline for final proceedings papers
** Action items for track participants

Additional Information


NIST is an agency of the
U.S. Department of Commerce

privacy policy / security notice / accessibility statement
disclaimer
FOIA

Last updated: Wednesday, 22-May-2013 11:58:14 EDT
Comments to: tac-web@nist.gov