TRECVID 2018 Evaluation for Activities in Extended Video

ActEV: Activities in Extended Video

Task coordinator: ActEV NIST team<ActEV-nist [at] nist.gov (ActEV-nist[at]nist[dot]gov)>

Currently the ActEV leaderboard evaluation is running (link)

CFP for 1st International  Workshop on Human Activity Detection in multi-camera, Continuous, long-duration Video (HADCV)(link)

Summary:

The ActEV evaluation seeks to evaluate robust automatic activity detection algorithms for a multi-camera streaming video environment. ActEV is an extension of the annual TRECVID Surveillance Event Detection (SED) evaluation by adding a large collection of multi-camera video data, both of simple and complex activities. ActEV will address activity detection for both forensic applications and for real-time alerting. The ActEV evaluation be run both as a blind and then as a leaderboard evaluation.

At a high level, activity detection algorithms can be classified as either: unconstrained and constrained. The first evaluation will focus on the unconstrained condition only.

Unconstrained: For the latency, unconstrained condition, a system processes the whole video in the full corpus prior to returning a list of detected activity instances. The primary application of the evaluation condition includes a forensic investigation/analysis tool.

Constrained: For the latency, constrained condition, a system detects the presence of the target activity instance within a predefined duration (e.g., 2 seconds) from the time the target activity begins in a video streaming environment. A system on this evaluation condition supports a real-time alert application.

What is Activity Detection in Videos?

By activity detection, we mean to detect visual events (people engaged in particular activities) in a large collection of streaming video data.

What: The ActEV evaluation is being conducted to assess the robustness of automatic activity detection for a multi-camera streaming video environment.

Who: NIST invites all organizations, particularly universities and corporations, to submit their results using their technologies to the ActEV evaluation server.  The evaluation is open worldwide. Participation is free. NIST does not provide funds to participants. 

How: To take part in the ActEV evaluation you need to register on the TRECVID website.  We will then send you information on how to create an account on the ActEV scoring server (see website for details). Using a valid registered account, you will be able to upload your JSON format results (see website for: Evaluation Plan) to the ActEV scoring server and participate in the ActEV evaluation. Each team is limited to few submissions (see website for: Evaluation Plan). The ActEV evaluation be run both as a blind and then as a leaderboard evaluation.

Why: The primary driver of the evaluation is to support investigations of crime by automatic activity detection in streaming video. These videos are of interest to NIST's partner agencies that seek to employ automatic activity detection in investigating crime. ActEV evaluation seeks to develop robust automatic activity detection for a multi-camera streaming video environment. Activities will be enriched by person and object detection. ACtEV will address activity detection for both forensic applications and for real-time alerting.

Data: The ActEV data used for this evaluation will be a unreleased portion of the VIRAT dataset. The  data will be provided in MPEG-4 format. (see website for: Licensing and Participation Agreements, and Evaluation Plan).

Metrics: The main scoring metrics will be based on temporal localization and spatio-temporal localization. We will use standard evaluation measures (e.g., probability of missed detection, rate of false alarm, precision, recall, average precision) and self-reported speed (see website for: Evaluation Plan)

Tentative Schedule:

ActEV’18 Blind Evaluation Schedule

• Registration deadline: April 1, 2018
• March 15, 2018: NIST Dry Run Evaluation Begins on Scoring Server
• April 10, 2018: NIST Dry Run Evaluation Ends
• April 25, 2018: 2:00pm NIST releases VIRAT-V1-Test Index files
• May 16, 2018 2:00pm EST: AD and AOD submissions due
• May 17, 2018: NIST releases VIRAT-V1-Test “Reference segmentation” IFF team submits AD or AOD
• May 23, 2018 2:00pm EST: Reference segmentation submissions due: AD and AOD

Tentative TRECVID ActEV Open Leaderboard Evaluation Schedule 

• June 11, 2018: NIST start of ActEV Leaderboard Evaluation Scoring Server
• September 22, 2018 : NIST end of ActEV Leaderboard Evaluation
• November 15, 2018:  NIST presents leaderboard evaluation results for ActEV at TRECVID’18

NIST Role: NIST will be conducting the ActEV evaluation which replaces the annual TRECVID Surveillance Event Detection (SED) evaluation.

Evaluation Plan: Please click!

Task coordinator: ActEV NIST team<ActEV-nist [at] nist.gov (ActEV-nist[at]nist[dot]gov)>