OpenASR21 Challenge Results

The OpenASR (Open Automatic Speech Recognition) 2021 Challenge was the third open challenge and the second OpenASR challenge associated with the IARPA MATERIAL program, after the OpenCLIR (Open Cross-Language Information Retrieval) 2019 Challenge and the OpenASR 2020 Challenge Capabilities tested in these open challenges are expected to ultimately support the MATERIAL task of effective triage and analysis of large volumes of text and audio content in a variety of less-studied languages. OpenASR21 was implemented as a track of NIST’s OpenSAT (Speech Analytic Technologies) evaluation series.

The goal of the OpenASR21 challenge was to assess the state of the art of automatic speech recognition (ASR) technologies for low-resource languages. ASR was performed on speech datasets, and written text output had to be produced.

Please refer to the OpenASR21 Challenge Evaluation Plan for a full description of the challenge and its rules and procedures.

Languages and Casing

OpenASR21 was offered for 15 low-resource languages, of which participating teams could attempt as many as they wished. Ten of these languages were repeated from OpenASR20 with unchanged datasets; five languages were new. All languages were offered with a case-insensitive scoring dataset. Of the five new languages, three were offered with an additional case-sensitive scoring dataset. The case-sensitive scoring was to serve as a proxy for assessment of ASR performance on proper nouns. The new languages are marked as such below:

• Amharic (AMH)
• Cantonese (CAN)
• New: Georgian (GEO)
• New: Farsi (FAR)
• Guarani (GUA)
• Javanese (JAV)
• New: Kazakh (KAZ) (including additional case-sensitive scoring dataset)
• Kurmanji Kurdish (KUR)
• Mongolian (MON)
• Pashto (PAS)
• Somali (SOM)
• New: Swahili (SWA) (including additional case-sensitive scoring dataset)
• New: Tagalog (TAG) (including additional case-sensitive scoring dataset)
• Tamil (TAM)
• Vietnamese (VIE)

Data

The data for the challenge consisted of speech data stemming from the IARPA Babel program, with the exception of Somali. which stemmed from the IARPA MATERIAL program. More details regarding technical data details can be found in section 3 of the IARPA Babel Data Specifications for Performers. For each language (and case scoring condition, where applicable), separate training, development, and evaluation datasets were provided.

For CIS, the datasets for most of the languages stemmed from the IARPA Babel program. The Somali and Farsi datasets stemmed from the IARPA MATERIAL program. For the case-sensitive scoring datasets, all data stemmed from the IARPA MATERIAL program.

The data consisted of conversational speech for the case-insensitive scoring datasets. For the case-sensitive scoring datasets, the data consisted of a mix of three genres: conversational speech, news broadcast, and topical broadcast.

Training Conditions

The challenge offered three training conditions. The Constrained-plus training was new for OpenASR21.

• Constrained training (mandatory):
  • Speech data: Limited to a 10-hour subset designated for Constrained training in provided training dataset for the language in question
  • Non-speech data: Any publicly available data
• New: Constrained-plus training (optional):
  • Same training data restrictions as Constrained Training, but additionally allowed publicly available and previously existing speech pretrained models as follows:
    • Pretrained models created from unlabeled speech data in any language
    • Pretrained models created from labeled speech data in any language except the language being processed
• Unconstrained training (optional):
  • Any publicly available data

Metrics

• Primary metric: Word Error Rate (WER)
• Additional metrics:
  • Character Error Rate (CER)
  • Time and memory resources used (self-reported)

Schedule

The most important milestones of the schedule of the challenge were as follows:

• Registration: August - October, 2021
• Evaluation period: November 3-10, 2021

Participation

Table 1 lists the teams that submitted valid output, and for which languages they submitted valid output.

26 teams from 13 countries originally registered to participate. Fifteen teams submitted valid output for at least one language under the Constrained training condition.

The table marks the following issues in the pertinent table cells:

• The team only made late submissions (up to seven days after the official evaluation period ended) for the language in question.
• The team did not make a required Constrained training condition submission for the language in question.
• The team did not submit the required system description.

Table 1: OpenASR21 Participating teams and languages.

Results

Table 2 lists the best WER result achieved by each team, ordered by language, scoring, training condition, and WER score. The CER score for the same submission is also provided.

The table marks the following issues in the Notes column:

• The submission was late (up to seven days after the official evaluation period ended). For some cases, this results in two best WER scores shown for a given team, language, and conditions - their best score, which was for a late submission, as well as their best on-time score. Late submissions are listed at the bottom of the table.
• The team did not make a required Constrained training condition submission for the language in question.
• The team did not submit the required system description.

Self-reported time and memory resources are not included in this overview of results.