2014-08-15 Draft API for comment
The first draft API for FIVE algorithms is now online. It is very similar to that used in the class V track of the last FRVT evaluation. Developers are specifically asked to comment– particularly on whether it supports measurement of the full capability of algorithms to detect, track and recognize faces in video. Comments should be email to five AT nist DOTgov by September 1. Two further comment periods will follow.
2014-07-16 Program Announcement
Scope: The Face in Video Evaluation (FIVE) is being conducted to assess the capability of face recognition algorithms to correctly identify or ignore persons appearing in video sequences – i.e. the open-set identification problem. Both comparative and absolute accuracy measures are of interest, given the goals to determine which algorithms are most effective and whether any are viable for the following primary operational use-cases: 1. High volume screening of persons in the crowded spaces (e.g. an airport); 2. Low volume forensic examination of footage from a crime scene (e.g. a convenience store); 3. Persons in business meetings (e.g. for video-conferencing); and 4. Persons appearing in television footage. These applications differ in their tolerance of false positives, whether a human examiner will review outputs, the prior probabilities of mate vs. non-mate presence, and the cost of recognition errors.
Out of scope: Gait, iris and voice recognition; Recognition across multiple views (e.g. via stereoscopic techniques); Tracking across sequential cameras (re-identification); anomaly detection; detection of evasion.
Relationship to FRVT: The Face Recognition Vendor Tests of 2000, 2002, 2006, 2010, and 2013 gave quantitative statements of accuracy and speed of mostly still-image face recognition algorithms. The last test included a video track (FRVT class V) – results from that work are being provided to participants. Our new FIVE program supersedes the FRVT work but proceeds in an almost identical manner.
Test progression: Software submitted to NIST will be evaluated on sequestered sets to quantify accuracy and speed. Algorithms must be implemented behind the formal C++ API to be published by NIST. This will be very similar to the API used in the prior FRVT evaluation. The test will be conducted over at least three iterative cooperative test-report-test phases engaging algorithm developers.This process will culminate in the publication of reports on this website and in the open literature.
Test data: This program will leverage several archival video corpora that are sequestered at NIST. Each includes subjects who are generally neither cooperative nor actively uncooperative. The datasets have in-common that several subjects are usually present in any given sequence, that only one fixed camera observes them, and that frontal views are the exception rather than the norm. The datasets vary in terms of camera quality, video quality, compression and pedestrian motion. Video imagery will primarily be compared with enrolled still-image datasets (video-to-still) of varying size and quality, and for which one or more views of a subject will be available. In addition, still-to-video and video-to-video tests will be executed.
None of the test data can be provided to participants. Instead prospective participants should leverage public domain and proprietary datasets as available. For the surveillance application NIST is aware of a very suitable video corpus that has been made available to qualified developers – please contact five AT nist DOT gov for details.
Standardization: The FIVE activity is expected to give quantitative support to the development of the ISO/IEC 30137 multipart standard that has recently been initiated in the SC37 Biometrics Subcommittee. Particularly, Working Group 5 there is developing a performance testing and reporting standard for video-surveillance systems. Working Group 4 is formulating recommendations for the design and specification of such systems. Finally Working Group 3 is considering biometric data needs layered on top of existing video interchange standards e.g. ISO/IEC 22311:2012.
August 15 – October 30: API publication and public comment periods
November 17 – January 8: Phase 1 algorithm submission period
Spring 2015 – Phase 2 submission period
Summer 2015 – Phase 3 submission period
For further information please contact five AT nist DOT gov .
This work is funded in part by the Department of Homeland Security's Science and Technology Directorate with PM Patricia Wolfhope.