Go to the NIST home page Go to the TAC Home Page TAC Banner

Return to TAC Homepage

TAC 2014 Tracks
  KBP Tracks
      Cold Start
      Entity Linking
      Slot Filling (SF)
      SF Validation
          Guidelines
          Tools
          Schedule
      Sentiment
      Event
    Data
    Schedule
    Organizers
    Call for Participation
    Track Registration
  BiomedSumm Track
Reporting Guidelines
TAC 2014 Workshop




TAC KBP 2014 Slot Filler Validation Track

Overview

The slot filler validation (SFV) track focuses on the refinement of output from English slot filling (SF) systems by either combining information from multiple slot filling systems, or applying more intensive linguistic processing to validate individual candidate slot fillers.

SFV is a diagnostic task that eliminates the need for a team to have a full slot filling system and concentrates on the refinement of existing system(s) output. The input to the SFV system is a set of submission files from several slot filling runs (with the run ID anonymized appropriately). The output of the SFV system is a binary classification (Correct/Incorrect) of each candidate slot filler in each SF run.

SFV is motivated by a use case in which the SFV system is used as a component of a full SF system. The evaluation measures the effect of using the SFV output to filter the contributing SF runs. Each contributing SF run will be filtered and rescored in the same way as for the full SF task, and the results compared against the scores for the unfiltered SF run. SFV tries to increase Precision of the contributing SF runs without significantly reducing Recall, and the objective function is to maximize the mean F-score over each of the filtered contributing runs.

Slot Filler Validation can be viewed as a Recognizing Textual Entailment (RTE) task, where the "text" consists of the provenance returned with each candidate slot filler, and the "hypothesis" is a natural language expression of the relation between the target entity and the candidate filler. The text entails the hypothesis if and only if the candidate slot filler is Correct according to the associated provenance.

Preliminary Schedule

    April 17Track guidelines posted
    AprilDistribution of document collections
    June 15Deadline for registration for track participation
    September 2-9 15-29Slot Filler Validation evaluation window
    By late SeptemberRelease of individual evaluated results to participants
    October 7Deadline for short system descriptions
    October 7Deadline for workshop presentation proposals
    mid OctoberNotification of acceptance of presentation proposals
    November 1Deadline for system reports (workshop notebook version)
    November 17-18TAC 2014 workshop in Gaithersburg, Maryland, USA
    February 15, 2015Deadline for system reports (final proceedings version)

Track Coordinator

Hoa Dang (NIST, hoa.dang@nist.gov)
Yasaman Haghpanah (NIST, yasaman.haghpanah@nist.gov)


NIST is an agency of the
U.S. Department of Commerce

privacy policy / security notice / accessibility statement
disclaimer
FOIA

Last updated: Tuesday, 02-Sep-2014 11:20:08 EDT
Comments to: tac-web@nist.gov