Open Speech Analytic Technologies (OpenSAT) Evaluation Series

Summary

The NIST Open Speech Analytic Technologies (OpenSAT) Evaluation Series was established to convene researchers working on speech analytics in challenging acoustic environments. By implementing objective, large-scale common evaluations, OpenSAT aims to advance the state-of-the-art across the field. Following a 2017 pilot focusing on speech activity detection (SAD), automatic speech recognition (ASR), and keyword search (KWS), the series launched its first formal evaluation in 2019. In 2020, in addition to SAD, ASR, and KWS, OpenSAT expanded its scope through three major collaborations: the DIHARD III Challenge, partnered with the Linguistic Data Consortium (LDC) to improve speaker diarization; the Fearless Steps Challenge Phase III, partnered with UTDallas-CRSS to develop SAD, ASR, speaker diarization, speaker identification, and conversational analysis; and the OpenASR Challenge, partnered with IARPA to target ASR for low-resource languages.

Description

The objectives of OpenSAT are as follows:

• To evaluate state-of-the-art speech analytics technologies specifically within the public safety communications domain.
• To provide a standardized forum where the speech analytics community could develop and test multiple technologies in parallel using a common dataset.
• To create opportunities for cross-pollination of expertise by connecting developers focusing on the same analytic tasks as well as those working on complementary technologies.

Contact

For questions or comments, email opensat_poc [at] nist.gov (opensat_poc[at]nist[dot]gov).

---

OpenSAT20

OpenSAT20 will follow the organizational framework of OpenSAT19, with the following modifications:

1. The scope will be narrowed to a single data domain (public safety communications), compared to the three domains used previously.
2. The data package will be comprised of a new, unexposed "Test set" and the OpenSAT19 evaluation data, designated as the "Progress set."
3. Public leaderboard will display scores for the Progress set. Participants will be given the option to anonymize their team names.
4. System descriptions will be made available to all participants via the OpenSAT20 website.
5. Remote attendance options will be provided for those unable to attend the workshop in person.

Tasks

• Speech Activity Detection (SAD)
• Automatic Speech Recognition (ASR)
• Key Word Search (KWS)

Data

The data domain for the OpenSAT20 Evaluation will be simulated public safety communications spoken in English. The evaluation data will be extracted from unexposed portions of the SAFE-T corpus that was collected by the Linguistic Data Consortium (LDC) and initially made available for the OpenSAT19 Evaluation. The audio recordings in the SAFE-T corpus contain speech potentially with increased vocal effort induced by first-responder type background noise conditions and is expected to be challenging for systems to process with a high degree of accuracy.

Evaluation Plan

• 2020 OpenSAT Evaluation Plan V1.6 (pdf) -  updated July 1, 2020

Schedule

• July 31, 2020: Deadline for participant registration.
• July 31, 2020: Final date to access Training, Development, and Evaluation data.
• July 31, 2020: Final date to upload system outputs to NIST for scoring; Progress set scores will be posted to the leaderboards until this date.
• Post-July 31: Test set scores will be released (contingent upon the submission of a system description).
• September 4, 2020: Deadline for workshop registration. [Click HERE to register]
• September 16–17, 2020: Virtual Workshop. 

---

OpenSAT19

Tasks

• Speech Activity Detection (SAD)
• Automatic Speech Recognition (ASR)
• Key Word Search (KWS)

Data

The evaluation utilized the following datasets across the SAD, ASR, and KWS tasks:

Dataset	Language	Tasks	Description
IARPA Babel	Pashto	SAD, ASR, KWS	Low-resource language
Video Annotation for Speech Technologies (VAST)	English	SAD, KWS	Amateur online videos
Public Safety Communications (PSC)	English	SAD,  ASR, KWS	Simulated public safety communications

Evaluation Plan

OpenSAT19 Evaluation Plan (pdf) - updated 3/28/2019

Schedule

• Development data release 03/29/2018 - 06-14-2019 
• Evaluation data release 06/17/2019 - 07/01/2019
• Post evaluation workshop 08/20/2019 - 08/21/2019 

---

OpenSAT Pilot 2017

Tasks

• Speech Activity Detection (SAD)
• Automatic Speech Recognition (ASR)
• Key Word Search (KWS)

Data

The evaluation utilized the following datasets across the SAD, ASR, and KWS tasks:

Dataset	Language	Tasks	Description
IARPA Babel	Pashto	SAD, ASR, KWS	Low-resource language
Video Annotation for Speech Technologies (VAST)	English, Arabic, Mandarin	SAD	Amateur online videos
Sofa Super Store Fire	English	SAD, ASR, KWS	First responder/dispatcher operational recording from the June 18th 2007, Charleston, South Carolina, Sofa Super Store Fire

Documentation

• Open Speech Analytic Technologies Pilot (OpenSAT Pilot) Evaluation Plan
• Open Speech Analytic Technologies Pilot (OpenSAT Pilot) Evaluation Report

Created July 30, 2026, Updated August 6, 2026