IREX 10: Frequently Asked Questions

Please read all of this FAQ before sending any questions to irex [at] nist.gov (irex[at]nist[dot]gov). We update this FAQ as new questions are posed, so always check here first.  

General Questions

Who can participate?

In general, any company, academic institution, organization, or individual with the ability to implement the API. Applications cannot be accepted from some countries or entities based on US government prohibitions.

Can I participate anonymously?

No. In general, we do not accept submissions from email accounts that are not verifiably associated with a corporate entity of some form; e.g. a company, university or organization. Submissions from a personal hotmail account (or similar) will likely be rejected.

How do I participate in IREX 10?

The first step is to sign and mail the Participation Agreement to

     Gaithersburg, MD 20899-8940

Beyond this, we recommend you download and carefully read these additional IREX application materials:

• API & Concept of operations document
• Encryption and Data Transfer Procedure

What image datasets does IREX use in its evaluations?

See IREX test datasets.

Can we view the iris test images?

No. Unfortunately, the test datasets are sequestered and cannot be distributed to participants. However, some basic properties of the test images are available.

Are participants provided with a representative sample of iris test data?

Participants are not provided with a subset of representative images. The primary test dataset is Controlled Unclassified Information (CUI) and NIST does not have permissions to distribute the images.

Why do you sometimes horizontally flip the iris images used for testing?

We horizontally flip (i.e. mirror image) the iris images used for nonmated searches. The purpose is to prevent mistakes in the ground truth from inflating FPIR. If the searched subject is unknowingly enrolled in the database under another subject identifier, a correct identification by the matcher would be erroneously labeled a false positive. Horizontally flipping the images avoids this problem. See Section 6.4 of the IREX III final report for a detailed explanation and analysis of flipping.

Routine Correspondence

You can expect an email acknowledging the submission of an algorithm and whether it successfully validates within a week of submission. You can expect a second email when the results are posted to the IREX 10 website. See also the turnaround times below.

Turnaround time for submissions

Submissions are processed in the order in which they are received. We try to post results as soon as possible. The turnaround time depends on the number of algorithms currently in the queue, the availability of staff to conduct the analyses, the availability of processing hardware, and the runtime speed of the submissions.

Staff and hardware are shared across multiple projects (e.g. IREX, MINEX, FRVT) so the workload in other projects is a factor. The typical turnaround time for an IREX 10 submission is 2 - 6 weeks, though times can be longer during vacation / holiday seasons.  The turnaround time will also be longer if there is a problem with your submission. If your submission fails to run to completion and requires correction, your resubmission goes to the back of the queue.

Retracting submission results

Once an algorithm is submitted and successfully run over the test data, the results will be posted to the IREX 10 website. There is no provision for retracting algorithms. You may submit a revised algorithm, but the results for that algorithm will be posted for the revision and will not replace the previous results.

How often can I submit:

You can submit one algorithm every four calendar months. This may change if the submission frequency places too much of a burden on our computational resources.

Debugging

We cannot debug submissions beyond providing some diagnostic output.

• If your algorithm fails to pass validation at our end, we can provide you with error codes and the output we get on our end.
• If your algorithm fails during evaluation, we may be able to provide you with some diagnostics (e.g. Valgrind output) depending on the nature of the failure. Our validation scripts provide more diagnostic information than our evaluation scripts do -- in the interests of best execution times during evaluations.
• The evaluation data is sequestered. We cannot provide you with examples of the images on which your code failed during evaluation or any information about the sequestered data other than what is posted at IREX Datasets.

Can I submit an algorithm in MatLab?

No. We have no plans to support MatLab submissions.

Can I submit an algorithm in Python?

No. We are considering supporting Python submissions at some time in the future; timing will depend on the availability of resources.

Can I submit a docker container?

No.

Where can I find the passcode for the validation dataset?

The passcode to decrypt the validation images will be provided once you have submitted a complete application to participate.

Can I get a preview of the results on our algorithm before they are posted to the IREX 10 webpage?

No.

Can I submit using Ubuntu 20.04.x?

Yes. If it fails, we will tell you but cannot debug.

If I cannot find an answer here, who should I contact?

Send an email to irex [at] nist.gov (irex[at]nist[dot]gov).

Questions about API

Can I recommend changes to the API?

Yes, if you find something useful or required is missing from the API  send your request to irex [at] nist.gov (irex[at]nist[dot]gov).

If our matcher cannot find enough candidates to fill the candidate list, is it OK to return fewer candidates?

Yes. The second argument passed to identifyTemplate() specifies how many candidates should be returned, but you may return fewer candidates if you wish.

Contact Information

Inquiries and comments may be submitted to irex [at] nist.gov (irex[at]nist[dot]gov).  Subscribe to the IREX mailing list to receive emails when announcements or updates are made.