Skip to main content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Nimble Challenge 2017 Evaluation

The inaugural Nimble Challenge Evaluation is the first in a series of annual evaluations to support research and help advance the state of the art for image and video forensics technologies - technologies that determine the region and type of manipulation in imagery. The NC2017 will support the following five evaluation tasks:

  • Image Manipulation Detection and Localization - Given a single probe image, detect if the probe was manipulated and provide mask(s) indicating the "type" of manipulation(s) found and the type(s) of manipulations the system was designed to detect.
  • Splice Detection and Localization - Given two images, detect if a region of a donor image has been spliced into a probe image and, if so, provide two masks indicating the region(s) of the donor image that were spliced into the probe and the region(s) of the probe image that were spliced from the donor [same as NC2016 definition]
  • Provenance Filtering - Given a probe image and a set of images representing the world, return the top N images from the world data set which contributed to creating a probe image.
  • Provenance Graph Building - Produce a phylogeny graph for a probe image
    • Variation 1: End-to-End Provenance - Provenance output produced by processing the N images returned by provenance filtering.
    • Variation 2: Oracle Filter Provenance - Provenance output produced from a NIST-provided small collection of images.
  • Video Manipulation Detection - Detect if the probe video was manipulated.

Prospective Nimble participants can subscribe to the Nimble mailing list for announcements by sending a request to the contact below and can take part in the evaluation by completing the registration and license agreements below.

Schedule

Dates Development Resources
December 2, 2016
  • Registration opens
December 9, 2016
  • Evaluation Plan posted
December 9, 2016
  • NC2017 Dev1 Beta4 available
December 16, 2016
  • Scoring software update
February 27, 2017
  • NC2017 Dev2 available
February 28, 2017
  • Manipulation Detection and Localization and Splice Detection and Localization Dry Run submissions due.
  • Provenance Scorer released.
Wed. March 29, 2017
  • NIST Dry Run scores returned.
  • NC2017 Dev3 released.
  • NIST sends World Data
Mon. April 10, 2017
  • Encrypted Evaluation Probe Data distributed

Wed. April 12, 2017

(2:00 pm EDT)

  • NIST sends Probe Data decryption Key
  • Evaluation Period begins

Wed. April 26, 2017

(2:00 pm EDT)

  • Team Submissions due for:
    • Manipulation Detection and Localization
    • Splice Detection and Localization
    • Provenance Filtering
    • End-to-End Provenance Graph

Thur. April 27, 2017

(2:00 pm EDT)

  • Oracle Provenance Graph Data Distributed

Thur. May 4, 2017

(2:00 pm EDT)

  • Team submissions due for:
    • Oracle Provenance Graph
Wed. May 10, 2017
  • Scores released to participants
July 2017
  • DARPA PI Meeting

Documentation

The evaluation plan can be found by accessing the following link: nc2017evaluationplan_20170804.pdf.

The slides for the 6 month PI meeting can be found by accessing the following link: nist_6monthpimeeting_final-performer.pdf.

The slides for the 1 year PI meeting can be found by accessing the following link: nist_medifor_july17pimeeting-merge_v12.pdf

The slides from the “The 2017 Nimble Challenge Evaluation: Results and Future Directions” talk given at the CVPR Workshop on Media Forensics at CVPR 2017.

Data Resources

The user name and password for the Nimble17 Data Resources page will be provided after successfully sending the completed signup documents per above. During the evaluation, you will receive the following data:

  • NC2016 - A data set of manipulations from 2016
  • NC2017 Development resources
  • NC2017 Evaluation resources

Evaluation Tools

NIST provided tools are described in the Evaluation Infrastructure Setup Instructions.

Contact

mfc_poc [at] nist.gov

Created September 28, 2016, Updated February 21, 2018