Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
Open Speech Analytic Technologies (OpenSAT) Evaluation Series
Summary
The NIST Open Speech Analytic Technologies (OpenSAT) Evaluation Series was established to convene researchers working on speech analytics in challenging acoustic environments. By implementing objective, large-scale common evaluations, OpenSAT aims to advance the state-of-the-art across the field. Following a 2017 pilot focusing on speech activity detection (SAD), automatic speech recognition (ASR), and keyword search (KWS), the series launched its first formal evaluation in 2019. In 2020, in addition to SAD, ASR, and KWS, OpenSAT expanded its scope through three major collaborations: the DIHARD III Challenge, partnered with the Linguistic Data Consortium (LDC) to improve speaker diarization; the Fearless Steps Challenge Phase III, partnered with UTDallas-CRSS to develop SAD, ASR, speaker diarization, speaker identification, and conversational analysis; and the OpenASR Challenge, partnered with IARPA to target ASR for low-resource languages.
Description
The objectives of OpenSAT are as follows:
- To evaluate state-of-the-art speech analytics technologies specifically within the public safety communications domain.
- To provide a standardized forum where the speech analytics community could develop and test multiple technologies in parallel using a common dataset.
- To create opportunities for cross-pollination of expertise by connecting developers focusing on the same analytic tasks as well as those working on complementary technologies.
Contact
For questions or comments, email opensat_poc [at] nist.gov (opensat_poc[at]nist[dot]gov).
OpenSAT20
OpenSAT20 will follow the organizational framework of OpenSAT19, with the following modifications:
- The scope will be narrowed to a single data domain (public safety communications), compared to the three domains used previously.
- The data package will be comprised of a new, unexposed "Test set" and the OpenSAT19 evaluation data, designated as the "Progress set."
- Public leaderboard will display scores for the Progress set. Participants will be given the option to anonymize their team names.
- System descriptions will be made available to all participants via the OpenSAT20 website.
- Remote attendance options will be provided for those unable to attend the workshop in person.
Tasks
- Speech Activity Detection (SAD)
- Automatic Speech Recognition (ASR)
- Key Word Search (KWS)
Data
The data domain for the OpenSAT20 Evaluation will be simulated public safety communications spoken in English. The evaluation data will be extracted from unexposed portions of the SAFE-T corpus that was collected by the Linguistic Data Consortium (LDC) and initially made available for the OpenSAT19 Evaluation. The audio recordings in the SAFE-T corpus contain speech potentially with increased vocal effort induced by first-responder type background noise conditions and is expected to be challenging for systems to process with a high degree of accuracy.
Evaluation Plan
- 2020 OpenSAT Evaluation Plan V1.6 (pdf) - updated July 1, 2020
Schedule
- July 31, 2020: Deadline for participant registration.
- July 31, 2020: Final date to access Training, Development, and Evaluation data.
- July 31, 2020: Final date to upload system outputs to NIST for scoring; Progress set scores will be posted to the leaderboards until this date.
- Post-July 31: Test set scores will be released (contingent upon the submission of a system description).
- September 4, 2020: Deadline for workshop registration. [Click HERE to register]
- September 16–17, 2020: Virtual Workshop.
OpenSAT19
Tasks
- Speech Activity Detection (SAD)
- Automatic Speech Recognition (ASR)
- Key Word Search (KWS)
Data
The evaluation utilized the following datasets across the SAD, ASR, and KWS tasks:
| Dataset | Language | Tasks | Description |
| IARPA Babel | Pashto | SAD, ASR, KWS | Low-resource language |
| Video Annotation for Speech Technologies (VAST) | English | SAD, KWS | Amateur online videos |
| Public Safety Communications (PSC) | English | SAD, ASR, KWS | Simulated public safety communications |
Evaluation Plan
OpenSAT19 Evaluation Plan (pdf) - updated 3/28/2019
Schedule
- Development data release 03/29/2018 - 06-14-2019
- Evaluation data release 06/17/2019 - 07/01/2019
- Post evaluation workshop 08/20/2019 - 08/21/2019
OpenSAT Pilot 2017
Tasks
- Speech Activity Detection (SAD)
- Automatic Speech Recognition (ASR)
- Key Word Search (KWS)
Data
The evaluation utilized the following datasets across the SAD, ASR, and KWS tasks:
| Dataset | Language | Tasks | Description |
| IARPA Babel | Pashto | SAD, ASR, KWS | Low-resource language |
| Video Annotation for Speech Technologies (VAST) | English, Arabic, Mandarin | SAD | Amateur online videos |
| Sofa Super Store Fire | English | SAD, ASR, KWS | First responder/dispatcher operational recording from the June 18th 2007, Charleston, South Carolina, Sofa Super Store Fire |