Skip to main content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

OpenSAT

OpenSAT Evaluation Series
 

The next evaluation will be the OpenSAT 2019 Evaluation and will include the same tasks as the OpenSAT Pilot: Automatic Speech Recognition (ASR), Speech Activity Detection (SAD), and Keyword Search (KWS), and three data domains including a newly collected simulated public safety communications dataset in support of the public safety community. The simulated public safety communications dataset is intended to include the Lombard effect in speech resulting from first responder type background noise, and some speech with expression of urgency. 

The NIST Speech Analytic Technologies evaluation series (OpenSAT) goal is to provide broad support for the advancement of speech analytic technologies by including multiple speech analytic tasks and multiple data domains. Developers can choose from one to all tasks and from one to all data domains.

Objectives of the series are: 

  • to bring together developers in different speech analytic tasks through evaluations on common datasets. 
  • to provide an opportunity for sharing and leveraging knowledge among developers whose primary focus is on different speech analytic tasks.
  • to provide developers an opportunity to apply their systems to multiple data domains for  performance comparison among different domains.
  • to provide an opportunity for developers to apply multiple speech analytic systems to common datasets and compare performance with the pool of developers participating in each task and data domain. 

Send email to opensat_poc [at] nist.gov with request to be added to the mailing list, to receive updates, or to ask questions or leave comments.


OpenSAT19 (schedule updated 2/06/19)
 

OpenSAT19 Evaluation Plan (Updated 3/28/2019)
03/29/2018 - 06-14-2019     Development data release (updated dates)
06/17/2019 - 07/01/2019     Evaluation data release (updated date)
08/20/2019 - 08/21/2019     Post Evaluation Workshop

Tasks
Speech Activity Detection (SAD)
Automatic Speech Recognition (ASR)
Key Word Search (KWS)

Data
For SAD, ASR, KWS tasks     Low Resource Language - (Pashto language) from the IARPA Babel collection
For SAD, KWS tasks                Audio extracted from amateur online videos - from the Video Annotation for Speech Technologies (VAST) collection (English language) 
For SAD, ASR, KWS tasks     Simulated public safety communications - from the PSC collection (English language)


OpenSAT Pilot 2017

Tasks
Speech Activity Detection (SAD)
Automatic Speech Recognition (ASR)
Key Word Search (KWS)

Data
For SAD, ASR, KWS tasks     Low Resource Language - from the IARPA Babel collection (Pashto language)
For SAD task only                   Audio extracted from YouTube videos - from the Video Annotation for Speech Technologies (VAST) collection (Arabic, Mandarin and English languages) 
For SAD, ASR, KWS tasks     First responder/dispatcher operational recordings - from the June 18th 2007, Charleston, South Carolina, Sofa Super Store Fire (English language)

Documentation
Open Speech Analytic Technologies Pilot (OpenSAT Pilot) Evaluation Plan
Open Speech Analytic Technologies Pilot (OpenSAT Pilot) Evaluation Report 


 

 

Created September 30, 2016, Updated April 2, 2019