Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

TRECVID 2017 Evaluation for Surveillance Event Detection

TRECVID 2017 Evaluation for Surveillance Event Detection

The Surveillance Event Detection (SED) evaluation is a track within the TRECVID Evaluation. The goal of the 2017 SED evaluation track is to support the development of technologies to detect visual events (people engaged in particular activities) in a large collection of streaming video data. The main evaluation will be implemented using a 10 hour subset of the multi-camera airport surveillance domain evaluation data collected by the Home Office Scientific Development Branch (HOSDB).

The 2017 SED evaluation is the 10th evaluation of event detection technologies preceded by the 2008, 2009, 2010, 2011, 2012, 2013, 20142015 and 2016 evaluations. The task is to detect observations of events based on the event definition. Systems may process the full corpus using multiple passes prior to outputting a list of putative events observations. The primary condition for this task will be single-camera input (i.e., the camera views are processed independently). Multiple-camera input may optionally be run as an additional contrastive condition.

We have a mailing list for announcements/discussion -- contact sed_poc [at] nist.gov (sed_poc[at]nist[dot]gov) to subscribe.

Evaluation Planning

The following resources are available:

Software

Evaluation submissions will be scored using the Framework for Detection Evaluations (F4DE) toolkit downloadable from the the NIST F4DE Github page at https://github.com/usnistgov/F4DE (submission checker tool requires version 3.4.0 or after)

Dry Run (DEV17)

The dry run period for the 2017 TRECVID Surveillance Event Detection Evaluation will run through June 2017. Dry run submissions will be accepted at any time during that period. The procedure for participating is as follows:

  • Sign up as a TRECVID Participant as specified on the TRECVID web site.
  • Obtain the data sets by completing the licensing agreement as specified below. The dry run will use part of the development corpus.
  • Download the Experiment Control File (ECF): expt_2017_allED_DEV17_ENG_s-camera_NIST_1.xml
  • Run experimental system(s) on the video intervals specified in the ECF.
  • Package system outputs in ViPER XML format and make sure they validate against the validation script (that can be found in F4DE -- see evaluation plan for details)
  • Prepare a tar.bz2 of all outputs as described in the evaluation plan and submit the archive via FTP. Make sure submissions validate using the TV17SED-SubmissionChecker tool.
  • To assist teams generating such archives for the first time, a testTEAM submission is available: SED17_testTEAM_retroED_DEV17_1.tar.bz2, that pass the TV17SED-SubmissionChecker tool.
  • Notify sed_poc [at] nist.gov (sed_poc[at]nist[dot]gov). Make sure to include a contact point's information (name, email address, location, site long and short name, and possibly a phone number for us to contact you in case of problems), as well as a possible alternate contact person.
  • Submissions are due 11:59PM Eastern Time (UTC-4), June 30th, 2017.
  • Results of the dry run will be returned by July 7th, 2017 (approx.).

Formal Evaluation (EVAL17)

Submissions for the formal evaluation are due July 21rd, 2017. The procedure for participating is as follows:

  • [Recommended] Complete the dry run as specified above.
  • Download the Experiment Control Files (ECF): There is only one ECF file defined in the eval plan for the retrospective event detection: expt_2017_allED_EVAL17_ENG_s-camera_NIST_1.xml
  • Run experimental system(s) on the video intervals specified in the ECF.
  • Package system outputs in ViPER XML format and make sure they validate against the validation script (that can be found in F4DE -- see evaluation plan for details).
  • Prepare a tar.bz2 of all outputs as described in the evaluation plan and submit the archive via FTP. Make sure submissions validate using the TV17SED-SubmissionChecker tool.
  • To assist teams generating such archives for the first time, a testTEAM submission is available: SED17_testTEAM_retroED_EVAL17_1.tar.bz2, that pass the TV17SED-SubmissionChecker tool.
  • Notify sed_poc [at] nist.gov (sed_poc[at]nist[dot]gov.) Make sure to include a contact point's information (name, email address, location, site long and short name, and possibly a phone number for us to contact you in case of problems), as well as a possible alternate contact person.
  • Submissions are due 11:59AM Eastern Daylight Time (UTC-4), July 21st, 2017.
  • Results of the formal evaluation will be returned by August 7th, 2017 (approx.).

Group Dynamic Subset (SUB17)

Submissions for the Group Dynamic Subset are due July 21st, 2017. The procedure for participating is as follows:

  • [Recommended] Complete the dry run as specified above.
  • Download the Experiment Control Files (ECF): There is only one ECF file defined in the eval plan for the retrospective event detection: expt_2017_allED_SUB17_ENG_s-camera_NIST_1.xml
  • Run experimental system(s) on the video intervals specified in the ECF.
  • Package system outputs in ViPER XML format and make sure they validate against the validation script (that can be found in F4DE -- see evaluation plan for details).
  • Prepare a tar.bz2 of all outputs as described in the evaluation plan and submit the archive via FTP. Make sure submissions validate using the TV17SED-SubmissionChecker tool.
  • To assist teams generating such archives for the first time, a testTeam submission is available: SED17_testTEAM_retroED_SUB17_1.tar.bz2, that pass the TV17SED-SubmissionChecker tool.
  • Notify sed_poc [at] nist.gov (sed_poc[at]nist[dot]gov). Make sure to include a contact point's information (name, email address, location, site long and short name, and possibly a phone number for us to contact you in case of problems), as well as a possible alternate contact person.
  • Submissions are due 11:59AM Eastern Daylight Time (UTC-4), July 21st, 2017.
  • Results of the Group Dynamic Subset will be returned by August 7th, 2017 (approx.).

 

Data Resources

 

Created May 18, 2017, Updated June 14, 2017