Performance Analysis of the 2017 NIST Language Recognition Evaluation

Published

September 2, 2018

Author(s)

Omid Sadjadi, Timothée N. Kheyrkhah, Craig Greenberg, Douglas A. Reynolds, Elliot Singer, Lisa Mason, Jaime Hernandez-Cordero

Abstract

The 2017 NIST language recognition evaluation (LRE) was held in the autumn of 2017. Similar to the past LRE's, the basic task in LRE17 was language detection, with an emphasis on discriminating closely related languages (14 in total) selected from 5 language clusters, namely Arabic, Chinese, English, Iberian, and Slavic. Compared to previous LRE's, LRE17 featured several new aspects including i) audio data extracted from online videos (AfV), ii) a small, yet representative, Dev set for system training and development, iii) system outputs in form of log-likelihood scores (as opposed to log-likelihood ratios), and iv) a normalized cross-entropy performance measure as an alternative metric. A total of 18 teams from 25 research and industrial organizations participated in the evaluation and submitted 79 valid system outputs under fixed and open training scenarios that were first introduced in LRE15. In this paper, we report a deeper analysis of system performance broken down by multiple factors such as data source and gender per language, as well as a cross-year (i.e., LRE15 versus LRE17) performance comparison of leading systems to measure progress over the 2-year period. In addition, we present a comparison of primary versus ''single'' best submissions to understand the importance of fusion on overall performance.

Proceedings Title

Interspeech 2018

Conference Dates

September 2-6, 2018

Conference Location

Hyderabad, IN

Pub Type

Conferences

Download Paper

Local Download

Keywords

language detection, language identification, language recognition, NIST evaluation, NIST LRE

Citation

Sadjadi, O. , Kheyrkhah, T. , Greenberg, C. , Reynolds, D. , Singer, E. , Mason, L. and Hernandez-Cordero, J. (2018), Performance Analysis of the 2017 NIST Language Recognition Evaluation, Interspeech 2018, Hyderabad, IN, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=925490 (Accessed April 24, 2024)