TAC KBP 2014 Slot Filler Validation Track

Overview

The slot filler validation (SFV) track focuses on the refinement of output from English slot filling (SF) systems by either combining information from multiple slot filling systems, or applying more intensive linguistic processing to validate individual candidate slot fillers.

SFV is a diagnostic task that eliminates the need for a team to have a full slot filling system and concentrates on the refinement of existing system(s) output. The input to the SFV system is a set of submission files from several slot filling runs (with the run ID anonymized appropriately). The output of the SFV system is a binary classification (Correct/Incorrect) of each candidate slot filler in each SF run.

SFV is motivated by a use case in which the SFV system is used as a component of a full SF system. The evaluation measures the effect of using the SFV output to filter the contributing SF runs. Each contributing SF run will be filtered and rescored in the same way as for the full SF task, and the results compared against the scores for the unfiltered SF run. SFV tries to increase Precision of the contributing SF runs without significantly reducing Recall, and the objective function is to maximize the mean F-score over each of the filtered contributing runs.

Slot Filler Validation can be viewed as a Recognizing Textual Entailment (RTE) task, where the "text" consists of the provenance returned with each candidate slot filler, and the "hypothesis" is a natural language expression of the relation between the target entity and the candidate filler. The text entails the hypothesis if and only if the candidate slot filler is Correct according to the associated provenance.

Preliminary Schedule

April 17	Track guidelines posted
April	Distribution of document collections
June 15	Deadline for registration for track participation
September 2-9 15-29	Slot Filler Validation evaluation window
By late September	Release of individual evaluated results to participants
October 7	Deadline for short system descriptions
October 7	Deadline for workshop presentation proposals
mid October	Notification of acceptance of presentation proposals
Nov 1 Nov 9	Deadline for system reports (workshop notebook version)
November 17-18	TAC 2014 workshop in Gaithersburg, Maryland, USA
February 15, 2015	Deadline for system reports (final proceedings version)

Track Coordinator