*
Bookmark and Share

NIST Open Keyword Search 2013 Evaluation (OpenKWS13)

The OpenKWS13 is currently accepting signups for the evaluation. To join the evaluation or inquire about the evaluation, send an email to openkws-poc@nist.gov to be added to the mailing lists. Developers are expected to build systems according to the OpenKWS13 Evaluation Plan.

Signup procedure

  1. Sign and return the OpenKWS13 registration form to openkws-poc@nist.gov.
  2. Sign and return the OpenKWS13 Babel data licensing agreement to openkws-poc@nist.gov.
  3. Optional: NIST has prepared a Babel-structured version of the STD 2006 Development and Evaluation Conversational Telephone speech data. Participants who do not have access to the STD resources through Linguistic Data Consortium membership can obtain this data after signing the LDC OpenKWS13 Data Agreement and returning it to LDC, Attention: Ilya Ahtaridis, fax 215-573-2175.
  4. Complete the Evaluation Infrastructure Setup Instructions.

Schedule

February 28 2013 OpenKWS13 registration closes
March 25 2013 MITRE sends password-protected surprise language build pack for 107
March 27 2013, 2:00 PM EDT NIST sends password for surprise language build pack
April 22 2013 MITRE sends password-protected surprise language evaluation pack

April 24 2013, 2:00 pm EDT

 NIST sends password for surprise language evaluation pack and keywords
May 1 2013, 2:00pm EDT Sites submit surprise language system outputs to NIST
May 3 2013 NIST reports surprise language results
July 17 2013 OpenKWS13 meeting in Washington DC metro area


Documentation

Teleconference notes

Evaluation tools

NIST maintains the Framework for Detection Evaluations (F4DE) Toolkit that supports the evaluation. The package is available on the MIG Tools page.

Contact

openkws-poc@nist.gov