Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
OpenKWS14 Evaluation
The OpenKWS14 is currently accepting signups for the evaluation. To join the evaluation or inquire about the evaluation, send an email to openkws-poc [at] nist.go (openkws-poc[at]nist[dot]gov) to be added to the mailing lists. Developers are expected to build systems according to the Draft KWS14 Evaluation Plan linked below.
Signup procedure
- Read the evaluation plan to become familiar with the evaluation.
- Sign and return the OpenKWS14 registration form to openkws-poc [at] nist.gov (openkws-poc[at]nist[dot]gov).
- Sign and return the OpenKWS14 Babel data licensing agreement V2 to openkws-poc [at] nist.gov (openkws-poc[at]nist[dot]gov).
- Complete a Dry Run Evaluation.
-
- The dry run is an opportunity for developers to make sure they are able to generate valid system output that can be scored with the NIST scoring tools. The actual performance of the system is not of interest during the dry run so developers may feel free to use any method to generate their system output, e.g., a random system, training on the dry run data, etc. The Evaluation Infrastructure Setup Instructions enumerates the steps to complete a dry run.
- NIST highly encourages new teams build a Vietnamese system to familiarize themselves with the Babel resources in advance of the surprise language evaluation as part of their dry run.
Schedule
| January 31 2014 | OpenKWS registration closes |
| March 24 2014 | Surprise language build pack download begins |
| April 2 2014, 2:00 PM EDT | NIST sends password for surprise language build pack |
| April 14 2014 | Surprise language evaluation pack download begins |
| April 23 2014, 2:00 pm EDT | NIST sends password for surprise language evaluation pack and keywords |
| April 30 2014, 2:00pm EDT | Sites submit surprise language system outputs to NIST |
| May 7 2014 | NIST reports surprise language results |
| July 2014 | OpenKWS14 meeting in Washington DC metro area |
Documentation
- The OpenKWS14 Evaluation Plan, v11: This document describes the data inputs and outputs for the evaluation.
- Babel Data Specification, August 26, 2013: This document describes the structure of the Babel data supplied to the participants.
- Language Specific Peculiarities Documents:
-
- Vietnamese: The 2012 Surprise Language LSP 03282013
Data Resources
OpenKWS participants will receive three types of resources: build packs, evaluation packs, and IndusDB releases. See Section 2 of the evaluation plan for a description of the resources. Teams must complete both the registration form and data license in order receive data. Resources will be distributed as follows:
- Build packs: Build packs will be distributed to teams that signup for OpenKWS14.
-
- The Vietnamese Build pack will be provided at signup time.
- The 2014 Surprise Language build pack will be distributed according to the schedule above
- Evaluation packs: Evaluation packs will be distributed to teams that signup for OpenKWS.
-
- The Vietnamese Evaluation pack will be provided at signup time. The reference transcripts will not be released to the team until they successfully submit results on the Vietnamese eval pack.
- The 2014 Surprise Language build pack will be distributed according to the schedule above.
- IndusDB: IndusDB releases will be issued as appropriate for the team.
Evaluation tools
NIST maintains the Framework for Detection Evaluations (F4DE) Toolkit that supports the evaluation. The package is available on the MIG Tools page.
Contact
openkws-poc [at] nist.gov (openkws-poc[at]nist[dot]gov)