Go to the NIST home page Go to the TAC Home Page TAC Banner

Return to TAC Homepage

TAC 2014 Tracks
  KBP Tracks
      Cold Start
          Guidelines
          Tools
          Schedule
      Entity Linking
      Slot Filling (SF)
      SF Validation
      Sentiment
      Event
    Data
    Schedule
    Organizers
    Call for Participation
    Track Registration
  BiomedSumm Track
Reporting Guidelines
TAC 2014 Workshop




TAC KBP 2014 Cold Start Track

Overview

The Cold Start track combines entity linking and slot filling to populate an empty knowledge base using a predefined schema for the facts and relations that will compose the KB. The Cold Start schema is specified by the TAC KBP English slot filling task and consists of three entity types (PER, ORG, GPE) and binary relation types corresponding to the slots in the English slot filling track, plus inverse slots.

Cold Start will have two variants in 2014. The first is identical to the 2013 task. Given a collection of documents, the Cold Start system must find all PER, ORG, and GPE entities mentioned by name in the collection, create a KB node for each entity, and link each name mention to the correct entity node in the KB. All Cold Start relations that are attested in the document collection must be included in the KB, along with a confidence value and provenance consisting of a document and offsets for the subject, predicate, and object of the relation. For relations whose objects are themselves Cold Start entities (such as per:siblings or org:subsidiaries), systems must link to the KB node representing the correct entity; relations whose objects are strings (such as per:title or org:website) will use strings to represent the object. If a KB includes multiple instances of the same subject-predicate-object triple (but with different provenance information), the triple with the highest confidence value will be evaluated, and additional triples with confidence lower value may be evaluated as resources permit.

A Cold Start evaluation query consists of a named entity mention in the document collection (an "entry point"), and a sequence of one or more relation types. The entry point selects a single corresponding entity node in the KB, and the sequence of relation types is followed to arrive at a set of terminal slot fillers at the end of the sequence. The terminal slot fillers are then assessed and scored as in the English slot filling task. For example, a typical Cold Start query may ask "What are the ages of the siblings of the Bart Simpson mentioned in Document 42?" or "What schools are attended by the children of the Marge Simpson mentioned in Document 9?" Such "one-hop" queries will verify that the knowledge base is well-formed in a way that goes beyond basic entity linking and slot filling. Each Cold Start evaluation query will have multiple entry points (i.e., multiple mentions of the same entity), in order to mitigate cascaded errors caused by submitted KBs that are not able to link every name mention to a KB entity node.

The second task variant removes the requirement that an entire text collection must be processed. Instead, participants in this variant will receive the evaluation queries, and need only produce those entities and relations that would be found by the queries. A TAC slot filling system can easily be applied to this task by running initially from each evaluation query entry point, then recursively applying the slot filler to the identified entities. We hope that slot filling participants will now find significantly lower barriers to entry for this new task variant.

Preliminary Schedule

    MarchTrack guidelines posted
    June 15Deadline for registration for track participation
    August 4-20 Cold Start evaluation window
    OctoberRelease of individual evaluated results to participants
    October 7Deadline for short system descriptions
    October 7Deadline for workshop presentation proposals
    mid OctoberNotification of acceptance of presentation proposals
    Nov 1 Nov 9Deadline for system reports (workshop notebook version)
    November 17-18TAC 2014 workshop in Gaithersburg, Maryland, USA
    February 15, 2015Deadline for system reports (final proceedings version)

Track Coordinator

James Mayfield (Johns Hopkins University, jamesmayfield@gmail.com)


NIST is an agency of the
U.S. Department of Commerce

privacy policy / security notice / accessibility statement
disclaimer
FOIA

Last updated: Monday, 27-Oct-2014 13:25:08 EDT
Comments to: tac-web@nist.gov