The 2007 NIST Language Recognition evaluation (LRE-07) is part of an ongoing series of evaluations of language recognition technology. LRE-07 will be held in the second half of calendar year 2007. It is planned for October of 2007, with the evaluation workshop to be held in December in Orlando, Florida. The evaluation will be similar to previous NIST language evaluations held in 1996, 2003, and 2005. A detailed evaluation plan describing the evaluation task and the evaluation protocols is now available (LRE07EvalPlan-v8b.pdf, updated July 20, 2007).

There will be two data sets and a supplement data set available for training and development:

• The first set consists of the data used in each of the three preceeding evaluations and the conversations of the 12 language CallFriend Corpus published by the LDC. This set is now available directly from the Linguistic Data Consortium (LDC) to all prospective evaluation participants who request it.

• The second set consists of newly collected conversations for languages without training data from other sources. This set is available also directly from the LDC in March to all registered evaluation participants who request it. For these conversations, NIST has designated two segments of each of the three durations in each of the conversation sides for development purposes if sites wish to use them. The timestamps for these segments are now available (uploaded May 14, 2007).

• The supplement set which contains the full conversations from which the primary LRE05 evaluation segments were drawn from is also available to registered evaluation participants. This set is available from NIST.

Sites planning to participate in the evaluation (including sites that are LDC members) must complete and sign the LDC license agreement and an addendum to the license and fax them to the LDC. Upon receiving the signed licenses, the LDC will send the sites the requested training data sets.

To register for the evaluation, fill out the registration form and fax it to NIST. Last day to register is September 1, 2007.

A beta version of the scoring software implementing the performance measurement as described in the LRE07 evaluation plan is now available (score_lre07.v01b.tgz, updated July 30, 2007).

The schedule for the evaluation is given below:

Mar 1, 2007 - Additional training data available
Sep 1, 2007 - Committment from sites to participate
Oct 1, 2007 - Data arrives at sites
Oct 17, 2007 - Submissions due to NIST
Oct 26, 2007 - Preliminary results released to participants
Dec 11-12, 2007 - Evaluation workshop
May 1, 2008 - Public results released

[ LRE Home ]