The goal of the OpenCLIR (Open Cross Language Information Retrieval) evaluation is to develop methods to locate text and speech content in “documents” (speech or text) in low-resource languages, using English queries. This capability is one of several expected to ultimately support effective triage and analysis of large volumes of data, in a variety of less studied languages. Successful systems will be able to adapt to new languages and new genres.
The OpenCLIR evaluation was created out of the IARPA MATERIAL program that encompasses more tasks, including domain classification and summarization, and more languages and query types. The purpose of OpenCLIR is to provide a simplified, smaller scale evaluation open to all. Please see IARPA's MATERIAL website for more information on the MATERIAL program.
We are pleased to announce the first OpenCLIR evaluation, OpenCLIR19. Registration will open in July 2018, with the official test taking place in January/February 2019. Details can be found in the evaluation plan linked under Documentation below.
The first OpenCLIR evaluation will declare a winner in two separate categories, text and audio data. The winners will receive a monetary award (USD 10,000 in the text category, USD 20,000 in the auditory category) and have the opportunity to attend the MATERIAL PI meeting in May 2019 to receive their prize and present their work. Please see the evaluation plan for more details and rules regarding the prizes.
Documentation and Resources
- OpenCLIR19 Evaluation Plan v1.19 (updated April 2, 2019)
- OpenCLIR19 IARPA Prize Challenge Rules and Regulations
- OpenCLIR19 Query Relevance Guidelines
- MATERIAL Query Overview
- OpenCLIR Evaluation Query Language Specification
- OpenCLIR19 system description template
Schedule (updated March 18, 2019)
|Release of evaluation plan||July 2018|
|Registration period||Mid-July, 2018 - November 30, 2018|
Release of Build Packs (training data)
Release of ANALYSIS, DEV, QUERY-DEV (encrypted data, decryption keys)
August 21, 2018 - May 31, 2019
August 21, 2018
August 21, 2018
|Release of EVAL, QUERY-EVAL (encrypted data)||March 4, 2019|
Release of EVAL, QUERY-EVAL (decryption keys)
System output due to NIST
March 11 - May 31, 2019
March 11, 2019
May 31, 2019
|System description due to NIST||June 21, 2019|
|Winners prize ceremony||TBD|
- To participate register via the OpenCLIR19 registration website: https://openclir.nist.gov.
- Tools, software, and additional resources will be posted here.
- Please email email@example.com for any questions or comments regarding the OpenCLIR19 evaluation.