Skip to main content
U.S. flag

An official website of the United States government

Dot gov

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Https

Secure .gov websites use HTTPS
A lock ( ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Performance Analysis of the 2017 NIST Language Recognition Evaluation

Published

Author(s)

Seyed Omid Sadjadi, Timothée N. Kheyrkhah, Craig S. Greenberg, Douglas A. Reynolds, Elliot Singer, Lisa Mason, Jaime Hernandez-Cordero

Abstract

The 2017 NIST language recognition evaluation (LRE) was held in the autumn of 2017. Similar to the past LRE’s, the basic task in LRE17 was language detection, with an emphasis on discriminating closely related languages (14 in total) selected from 5 language clusters, namely Arabic, Chinese, English, Iberian, and Slavic. Compared to previous LRE’s, LRE17 featured several new aspects including i) audio data extracted from online videos (AfV), ii) a small, yet representative, Dev set for system training and development, iii) system outputs in form of log-likelihood scores (as opposed to log-likelihood ratios), and iv) a normalized cross-entropy performance measure as an alternative metric. A total of 18 teams from 25 research and industrial organizations participated in the evaluation and submitted 79 valid system outputs under fixed and open training scenarios that were first introduced in LRE15. In this paper, we report a deeper analysis of system performance broken down by multiple factors such as data source and gender per language, as well as a cross-year (i.e., LRE15 versus LRE17) performance comparison of leading systems to measure progress over the 2-year period. In addition, we present a comparison of primary versus ``single'' best submissions to understand the importance of fusion on overall performance.
Proceedings Title
Interspeech 2018
Conference Dates
September 2-6, 2018
Conference Location
Hyderabad, -1

Keywords

language detection, language identification, language recognition, NIST evaluation, NIST LRE
Created September 2, 2018, Updated February 27, 2020