Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

OpenKWS13 Evaluation

The OpenKWS13 is currently accepting signups for the evaluation. To join the evaluation or inquire about the evaluation, send an email to openkws-poc [at] nist.go (openkws-poc[at]nist[dot]gov) to be added to the mailing lists. Developers are expected to build systems according to the OpenKWS13 Evaluation Plan.

Signup procedure

  1. Sign and return the OpenKWS13 registration form to openkws-poc [at] (openkws-poc[at]nist[dot]gov).
  2. Sign and return the OpenKWS13 Babel data licensing agreement to openkws-poc [at] (openkws-poc[at]nist[dot]gov).
  3. Optional: NIST has prepared a Babel-structured version of the STD 2006 Development and Evaluation Conversational Telephone speech data. Participants who do not have access to the STD resources through Linguistic Data Consortium membership can obtain this data after signing the LDC OpenKWS13 Data Agreement and returning it to LDC, Attention: Ilya Ahtaridis, fax 215-573-2175.
  4. Complete the Evaluation Infrastructure Setup Instructions.


February 28 2013 OpenKWS13 registration closes
March 25 2013 MITRE sends password-protected surprise language build pack for 107
March 27 2013, 2:00 PM EDT NIST sends password for surprise language build pack
April 22 2013 MITRE sends password-protected surprise language evaluation pack

April 24 2013, 2:00 pm EDT

NIST sends password for surprise language evaluation pack and keywords
May 1 2013, 2:00pm EDT Sites submit surprise language system outputs to NIST
May 3 2013 NIST reports surprise language results
July 17 2013 OpenKWS13 meeting in Washington DC metro area


Teleconference notes

Evaluation tools

NIST maintains the Framework for Detection Evaluations (F4DE) Toolkit that supports the evaluation. The package is available on the MIG Tools page.


openkws-poc [at] (openkws-poc[at]nist[dot]gov)

Created January 15, 2013, Updated April 9, 2014