There has been a growing recognition of the importance of community-wide evaluations for research in information technologies. The Text Analysis Conference is a series of workshops that provides the infrastructure for large-scale evaluation of Natural Language Processing technology.
TAC's mission is to support research within the Natural Language Processing community by providing the infrastructure necessary for large-scale evaluation of NLP methodologies. TAC's primary purpose is not competitive benchmarking; the emphasis is on advancing the state of the art through evaluation results. In particular, the TAC workshop series has the following goals:
TAC is organized by the Retrieval Group of the Information Access Division (IAD) in the Information Technology Laboratory at the National Institute of Standards and Technology (NIST). Initiated in 2008, TAC grew out of NIST's Document Understanding Conference (DUC) for text summarization, and the Question Answering Track of the Text Retrieval Conference (TREC). TAC is sponsored by NIST and other U.S. government agencies and is overseen by an Advisory Committee consisting of representatives from government, industry, and academia.
A TAC cycle consists of a set tracks, areas of focus in which particular NLP tasks are defined. The tracks serve several purposes. First, tracks act as incubators to experiment with new research areas; the first running of a track often defines what the problem really is, and a track creates the necessary infrastructure (test collections, evaluation methodology, etc.) to support research on its tasks. The tracks also demonstrate the robustness of core NLP technology in that the same techniques are frequently appropriate for a variety of tasks. Finally, the tracks make TAC attractive to a broader community by providing tasks that match the research interests of more groups. The TAC advisory committee selects the set of tracks that will be run in a given year of TAC based on track proposals.The TAC cycle begins at the start of the calendar year with the Call for Track Participation, and concludes in February of the following year when the conference proceedings and track data for that cycle are archived on the TAC web site and made available to the general public. For each TAC cycle, NIST distributes test data for each track; participants run their own NLP systems on the data and return their results to NIST; NIST pools the individual results, judges them for correctness, and evaluates the results. Data and evaluated results are generally not made available to non-participants until the end of the TAC cycle.
The TAC cycle culminates with a workshop that is a forum for participants to share their experiences and plan future tasks/evaluations. The workshop is usually held at NIST during the last full week before the U.S. Thanksgiving holiday. Attendance at TAC workshops is open to the public, but workshop attendees must pre-register in order to enter the NIST campus.
NIST is an agency of the
U.S. Department of Commerce
Last updated: Wednesday, 22-May-2013 11:58:14 EDT
Comments to: email@example.com