Go to the NIST home page Go to the TAC Home Page TAC Banner

Return to TAC Homepage

TAC 2014 Tracks
  KBP Tracks
      Cold Start
      Entity Linking
      Slot Filling (SF)
      SF Validation
      Sentiment
      Event
          Guidelines
          Tools
          Schedule
    Data
    Schedule
    Organizers
    Call for Participation
    Track Registration
  BiomedSumm Track
Reporting Guidelines
TAC 2014 Workshop




TAC KBP 2014 Event Track

Overview

The goal of the TAC KBP Event track is to extract information about events such that the information would be suitable as input to a knowledge base. The Event Argument Extraction task aims to extract information about entities (and times) and the role they play in an event. Participating systems will extract tuples that include (EventType, Role, Argument). EventType and Role will be drawn from an externally specified ontology that is based on ACE 2005. Arguments will be strings from within a document representing the canonical (most-specific) name or description of the entity.

While this task does not require the reification of events (or linking the different arguments of an event), systems developed for this task will support KB-queries that link entities to participation in an event-- for example "List ORGANIZATIONs with a PURCHASER role". When combined with other KBP technologies (i.e. slot-filling and entity-linking), more complex, multi-hop queries are possible-- for example, "List ORGANIZATIONSs with MEMBERs who have served as an ATTACKER".

This is a new task in 2014 and will be evaluated in English only. In 2015, we expect to extend to additional TAC languages and add an evaluation of a system's ability to reify events and connect their arguments.

Task

Systems will be given a 500-1,000 document corpus and asked to extract instances of arguments that play a role in some event. The corpus will be a mix of newswire and discussion forum documents. A discussion forum document may contain multiple posts. The corpus will be manually and automatically filtered to ensure at least a few instances of all event-types. The discussion-forum posts will be automatically filtered to identify those posts that are not simply reposts of newswire documents. Very long discussion-forum threads will be truncated.

Systems will be evaluated on F-Score of (EventType, Role, NormalizedArgumentString, Realis) over the pooled assessments. The assessment pool for each document will include a human-participant. The human-participants submission will be assessed in the same manner as the system submissions.

Preliminary Schedule

    MarchTrack guidelines posted. ACE 2005 data released to participants as they sign up.
    April 1LDC releases sample documents (~50) to participants in preparation for pilot assessment
    April 15Participants (optionally) submit pilot output for pilot assessment. Pilot may be automatic, manual, or some combination. While participation is not required, participants are encouraged to submit output to better understand this new task (e.g. receive feedback on the assessment of inference, nominal extents, realis).
    May 15LDC releases the full set of pilot assessment to participants who submitted pilot output.
    June 15Deadline for registration for track participation
    August 11-18Event Argument Extraction evaluation window
    OctoberRelease of individual evaluated results to participants
    October 7Deadline for short system descriptions
    October 7Deadline for workshop presentation proposals
    mid OctoberNotification of acceptance of presentation proposals
    Nov 1 Nov 9Deadline for system reports (workshop notebook version)
    November 17-18TAC 2014 workshop in Gaithersburg, Maryland, USA
    February 15, 2015Deadline for system reports (final proceedings version)

References

While the ACE 2005 event annotation is being provided to all participants, this task diverges from ACE in some cases. One example of divergence is the addition of correct answers derived through inference/world knowledge. This evaluation will treat as correct some cases that were explicitly excluded in ACE 2005.

Track Coordinator

Marjorie Freedman (BBN, mfreedma@bbn.com)


NIST is an agency of the
U.S. Department of Commerce

privacy policy / security notice / accessibility statement
disclaimer
FOIA

Last updated: Tuesday, 28-Mar-2017 11:22:15 EDT
Comments to: tac-web@nist.gov