Go to the NIST home page Go to the TAC Home Page TAC Banner

Return to TAC Homepage

Track Home
Schedule
Guidelines
KBP Data
Tools

KBP 2013 Home




TAC KBP 2013 Slot Filler Validation Track

Overview

The slot filler validation (SFV) track focuses on the refinement of output from English slot filling (SF) systems by either combining information from multiple slot filling systems, or applying more intensive linguistic processing to validate individual candidate slot fillers.

SFV is a diagnostic task that eliminates the need for a team to have a full slot filling system and concentrates on the refinement of existing system(s) output. The input to the SFV system is a set of submission files from several slot filling runs (with the run ID anonymized appropriately). The output of the SFV system is a binary classification (Correct/Incorrect) of each candidate slot filler in each SF run.

SFV is motivated by a use case in which the SFV system is used as a component of a full SF system. The evaluation measures the effect of using the SFV output to filter the contributing SF runs. Each contributing SF run will be filtered and rescored in the same way as for the full SF task, and the results compared against the scores for the unfiltered SF run. SFV tries to increase Precision of the contributing SF runs without significantly reducing Recall, and the objective function is to maximize the mean F-score over each of the filtered contributing runs.

Slot Filler Validation can be viewed as a Recognizing Textual Entailment (RTE) task, where the "text" consists of the provenance returned with each candidate slot filler, and the "hypothesis" is a natural language expression of the relation between the target entity and the candidate filler. The text entails the hypothesis if and only if the candidate slot filler is Correct according to the associated provenance.

Schedule

    JuneTrack guidelines posted
    By early JuneDistribution of document collections
    June 15Deadline for registration for track participation
    September 2-22Extended Slot Filler Validation evaluation window
    By late SeptemberRelease of individual evaluated results to participants
    September 30Deadline for short system descriptions
    September 30Deadline for workshop presentation proposals
    mid OctoberNotification of acceptance of presentation proposals
    November 8Deadline for system reports (workshop notebook version)
    November 18-19TAC KBP 2013 workshop in Gaithersburg, Maryland, USA
    February 15, 2014Deadline for system reports (final proceedings version)

Track Coordinator

Hoa Trang Dang (NIST, hoa.dang@nist.gov)


NIST is an agency of the
U.S. Department of Commerce

privacy policy / security notice / accessibility statement
disclaimer
FOIA

Last updated: Friday, 24-Jan-2014 14:01:55 EST
Comments to: tac-web@nist.gov