The Nimble Challenge 2018 (NC2018) Evaluation is the second annual evaluation to support research and help advance the state of the art for image and video forensics technologies – technologies that determine the region and type of manipulations in imagery (image/video data) and the phylogenic process that modified the imagery. The NC2018 is currently being designed building of experience from the NC2017 Evaluation. We expect to continue support for the following tasks:
Prospective Nimble participants can subscribe to the Nimble mailing list for announcements by sending a request to the contact below and can take part in the evaluation by completing the registration and license agreements below.
There are many exciting changes being planned for the evaluation cycle: more development resources, a scoring server with a leaderboard style evaluation, manipulation operation-specific system evaluations which allows research to focus on one specific form of manipulations, and detailed, automatic diagnostics of system success/failures.
The planning for NC '18 has just begun. Please share the MFC’18 Call for Participation Flyer with your colleagues. The anticipated changes between NC '17 and NC '18 to be minimal. Please see the NC '17 Evaluation Plan to begin understanding the structure of the evaluation tasks, data, and metrics.
By signing up for the evaluation, you’ll be getting a wealth of data resources to conduct your media forensics research. The data will be designated as either: (1) a development resource that will be accessible at signup, (2) past year evaluation resources provided after completing a dry run evaluation, and (3) the NC’18 evaluation resources during the formal evaluation in April ’18. Here’s a quick summary of the resources:
Data Set Type
Data Set Name
Number of Forensic Probes (true manipulations and non-manipulations)
World Data Set Size
NC2016 – Both Nimble Science and Nimble Web
NC’17 Development Image Data
NC’17 Development Video Data
NC’18 Development Image and Video Data
NC’17 Evaluation Images
Full for 1/3 subset
NC’17 Evaluation Videos
Full for 1/3 Subset
NC ‘18 Evaluation
NC’18 Evaluation Images
NC’18 Evaluation Videos
NIST provided tools are described in the Evaluation Infrastructure Setup Instructions.
nimble_poc [at] nist.gov (nimble_poc[at]nist[dot]gov)