Sadjadi, O.
, Kheyrkhah, T.
, Greenberg, C.
, Reynolds, D.
, Singer, E.
, Mason, L.
and Hernandez-Cordero, J.
(2018),
Performance Analysis of the 2017 NIST Language Recognition Evaluation, Interspeech 2018, Hyderabad, IN, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=925490
(Accessed December 3, 2024)
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
Performance Analysis of the 2017 NIST Language Recognition Evaluation
Published
Author(s)
, , , Douglas A. Reynolds, Elliot Singer, Lisa Mason, Jaime Hernandez-Cordero
Abstract
The 2017 NIST language recognition evaluation (LRE) was held in the autumn of 2017. Similar to the past LRE's, the basic task in LRE17 was language detection, with an emphasis on discriminating closely related languages (14 in total) selected from 5 language clusters, namely Arabic, Chinese, English, Iberian, and Slavic. Compared to previous LRE's, LRE17 featured several new aspects including i) audio data extracted from online videos (AfV), ii) a small, yet representative, Dev set for system training and development, iii) system outputs in form of log-likelihood scores (as opposed to log-likelihood ratios), and iv) a normalized cross-entropy performance measure as an alternative metric. A total of 18 teams from 25 research and industrial organizations participated in the evaluation and submitted 79 valid system outputs under fixed and open training scenarios that were first introduced in LRE15. In this paper, we report a deeper analysis of system performance broken down by multiple factors such as data source and gender per language, as well as a cross-year (i.e., LRE15 versus LRE17) performance comparison of leading systems to measure progress over the 2-year period. In addition, we present a comparison of primary versus ''single'' best submissions to understand the importance of fusion on overall performance.Proceedings Title
Interspeech 2018
Conference Dates
September 2-6, 2018
Conference Location
Hyderabad, IN
Pub Type
Conferences
Download Paper
Keywords
language detection, language identification, language recognition, NIST evaluation, NIST LRE
Citation
Issues
If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.
Created September 1, 2018, Updated October 12, 2021