Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
OpenKWS13 Evaluation
The OpenKWS13 is currently accepting signups for the evaluation. To join the evaluation or inquire about the evaluation, send an email to openkws-poc [at] nist.go (openkws-poc[at]nist[dot]gov) to be added to the mailing lists. Developers are expected to build systems according to the OpenKWS13 Evaluation Plan.
Signup procedure
- Sign and return the OpenKWS13 registration form to openkws-poc [at] nist.gov (openkws-poc[at]nist[dot]gov).
- Sign and return the OpenKWS13 Babel data licensing agreement to openkws-poc [at] nist.gov (openkws-poc[at]nist[dot]gov).
- Optional: NIST has prepared a Babel-structured version of the STD 2006 Development and Evaluation Conversational Telephone speech data. Participants who do not have access to the STD resources through Linguistic Data Consortium membership can obtain this data after signing the LDC OpenKWS13 Data Agreement and returning it to LDC, Attention: Ilya Ahtaridis, fax 215-573-2175.
- Complete the Evaluation Infrastructure Setup Instructions.
Schedule
| February 28 2013 | OpenKWS13 registration closes |
| March 25 2013 | MITRE sends password-protected surprise language build pack for 107 |
| March 27 2013, 2:00 PM EDT | NIST sends password for surprise language build pack |
| April 22 2013 | MITRE sends password-protected surprise language evaluation pack |
| April 24 2013, 2:00 pm EDT | NIST sends password for surprise language evaluation pack and keywords |
| May 1 2013, 2:00pm EDT | Sites submit surprise language system outputs to NIST |
| May 3 2013 | NIST reports surprise language results |
| July 17 2013 | OpenKWS13 meeting in Washington DC metro area |
Documentation
- The OpenKWS13 Evaluation Plan, v4: This document describes the data inputs and outputs for the evaluation. To highlight changes, the V4 compared V3 is available.)
- Babel Data Specification, February 6 2013: This document describes the structure of the Babel data supplied to the participants.
- OpenKWS13 FAQ, March 12 2013
- The Surprise Language LSP 03282013
Teleconference notes
Evaluation tools
NIST maintains the Framework for Detection Evaluations (F4DE) Toolkit that supports the evaluation. The package is available on the MIG Tools page.
Contact
openkws-poc [at] nist.gov (openkws-poc[at]nist[dot]gov)