OpenASR20 Challenge Results

The OpenASR (Open Automatic Speech Recognition) 2020 Challenge was the second open challenge associated with the IARPA MATERIAL program, after the OpenCLIR (Open Cross-Language Information Retrieval) 2019 Challenge. Capabilities tested in these open challenges are expected to ultimately support the MATERIAL task of effective triage and analysis of large volumes of text and audio content in a variety of less-studied languages. OpenASR20 was implemented as a track of NIST’s OpenSAT (Speech Analytic Technologies) evaluation series.

The goal of the OpenASR20 challenge was to assess the state of the art of automatic speech recognition (ASR) technologies for low-resource languages. ASR was performed on speech datasets, and written text output had to be produced.

Please refer to the OpenASR20 Challenge Evaluation Plan for a full description of the challenge and its rules and procedures.

Languages

OpenASR20 was offered for the following ten low-resource languages, of which participants could attempt as many as they wished:

• Amharic (AMH)
• Cantonese (CAN)
• Guarani (GUA)
• Javanese (JAV)
• Kurmanji Kurdish (KUR)
• Mongolian (MON)
• Pashto (PAS)
• Somali (SOM)
• Tamil (TAM)
• Vietnamese (VIE)

Data

The data for the challenge consisted of conversational telephone speech stemming from the IARPA Babel program, with the exception of Somali which stemmed from MATERIAL. More details regarding technical data details can be found in section 3 of the IARPA Babel Data Specifications for Performers. For each language, separate training, development, and evaluation datasets were provided.

Training Conditions

The challenge offered two training conditions: 

• Constrained training (mandatory):
  • Speech data: Limited to a 10-hour subset designated for Constrained training in provided training dataset for the language in question
  • Non-speech data: Any publicly available data
• Unconstrained training (optional): Any publicly available data

Metrics

• Primary metric: Word Error Rate (WER)
• Additional metrics:
  • Character Error Rate (CER)
  • Time and memory resources used (self-reported)

Schedule

The most important milestones of the schedule of the challenge were as follows:

• Registration: August - October, 2020
• Development period: August 2020 - June 15, 2021
• Evaluation period: November 3-10, 2020

Participation

28 teams from 12 countries registered to participate, out of which nine fully completed the challenge (i.e. submitted valid output for at least one language under the Constrained training condition). Table 1 lists the fully participating teams.

Table 1: OpenASR20 Participants

Results

Table 2 lists the best WER result achieved by each team, ordered by language, training condition (Unconstrained submissions in italics), and WER score. Late submissions are marked as such and listed at the bottom of the table. Self-reported time and memory resources are not included in this overview of results.

Table 2: OpenASR20 Results. † = late submission.

System Descriptions

As part of the evaluation submission, participants were required to include a paper to describe their systems. Participants were also encouraged to submit their work to be included in the OpenASR and Low-Resource ASR Development Special Session at INTERSPEECH 2021. The system descriptions with consent to be released publicly are provided below:

• Catskills OpenASR20 System Description
• NSYSU-MITLab OpenASR20 System Description
• TalTech OpenASR20 System Description
• THUEE OpenASR20 System Description
• TNT+MMT OpenASR20 System Description

Disclaimer

NIST serves to coordinate the evaluations in order to support research and to help advance the state- of-the-art. NIST evaluations are not viewed as a competition, and such results reported by NIST are not to be construed, or represented, as endorsements of any participant’s system, or as official findings on the part of NIST or the U.S. Government.

Contact

Please email openasr_poc [at] nist.gov (openasr_poc[at]nist[dot]gov) for any questions or comments regarding the OpenASR Challenge.