Skip to main content
U.S. flag

An official website of the United States government

Dot gov

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Https

Secure .gov websites use HTTPS
A lock ( ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

OpenKWS16 Evaluation

The OpenKWS16 is currently accepting signups for the evaluation. To join the evaluation or inquire about the evaluation, send an email to openkws-poc@nist.gov to be added to the mailing lists. Developers are expected to build systems according to the published evaluation plan. The significant changes between OpenKWS15 and OpenKWS16 are:

  • OpenKWS teams will be supported in multilingual KWS by the release of 24 Babel Build Packs.
  • Additional resources from the LDC will be provided to teams.
  • Evaluation changes:
    • The Full Language Pack (FullLP) will be the primary evaluation condition. Training will include the released training material, the Surprise Language Build pack (80 hrs of training audio, 50% transcribed) but no phonetic lexicon.
    • The Very Limited Language Pack training condition will be supported as a contrastive training condition
    • The phonetic lexicons will be released only after the evaluation to support analysis.
  • Evaluation schedule changes:
    • Reduction of KWS system build time and testing time.
      • 1 week to build a FullLP system
      • 1 week to process the evaluation data with the FullLP system
      • 1 week to submit additional contrastive system runs.

Signup procedure

  1. Read the evaluation plan to become familiar with the evaluation.
  2. Sign and return the 2016 Open Keyword Search Evaluation Participation Agreement to openkws-poc@nist.gov.
  3. Sign and return the 2016 Open Keyword Search Evaluation Babel Data Agreement to openkws-poc@nist.gov.
  4. Sign and return the 2016 Open Keyword Search Evaluation LDC Data Agreement to openkws-poc@nist.gov.
  5. Complete a Dry Run Evaluation.  
    • The dry run is an opportunity for developers to make sure they are able to generate valid system output that can be scored with the NIST scoring tools. The actual performance of the system is not of interest during the dry run so developers may feel free to use any method to generate their system output, e.g., a random system, training on the dry run data, etc. The Evaluation Infrastructure Setup Instructions enumerate the steps to complete a dry run. 
    • NIST highly encourages new teams to build a Vietnamese, Tamil, or Swahili system to familiarize themselves with the Babel resources in advance of the surprise language evaluation as part of their dry run.

Schedule

February 28, 2016 OpenKWS16 registration closes
July 18, 2016
Surprise language build pack download begins
July 20, 2016, 2:00 PM EDT NIST sends password for surprise language build pack
July 21, 2016 2:00 PM EDT
Scholar Seeded Web Sites emailed to the OpenKWS community
July 25, 2016 Surprise language evaluation pack download begins

July 27, 2016, 2:00 pm EDT

NIST sends password for surprise language evaluation pack and keywords
Aug 3, 2016, 2:00pm EDT Sites submit FullLP surprise language system outputs to NIST
Aug 5, 2016
NIST posts FullLP results and phonetic lexicon
Aug 10, 2016 Sites submit contrastive runs.
NIST releases EvalPart1 transcripts.
Aug 17, 2016 NIST releases scoring reports and data files.
September, 2016 OpenKWS16 meeting in Washington DC metro area


Documentation

Data Resources

The OpenKWS16 Data Resources page contains four types of resources:  build packs, evaluation packs, IndusDB releases, and Language Pack Resources. Teams must complete both the registration form and data license in order receive data.  Resources will be distributed as follows:

  • Build packs:
    • Build packs for 24 Babel Languages will be distributed to teams that signup for OpenKWS16.
    • The 2016 Surprise Language build pack will be distributed according to the schedule above
  • Evaluation packs:
    • Evaluation packs for Vietnamese, Tamil, and Swahili will be distributed to teams that signup for OpenKWS16.
    • The 2016 Surprise Language evaluation pack will be distributed according to the schedule above.
  • LDC resources will be distributed via their download service.
  • IndusDB releases will be issued as appropriate for the team.
  • Language Pack Resources will be provided as appropriate for the team.

Evaluation tools

NIST provided tools are described in the Evaluation Infrastructure Setup Instructions.

Contact

openkws-poc@nist.gov

Created January 29, 2016, Updated August 25, 2016