The TRECVID 2007 BBC Rushes Summarization Evaluation Pilot

Published

November 14, 2007

Author(s)

Paul D. Over, Alan Smeaton, Phillip Kelly

Abstract

This paper provides an overview of a pilot evaluation of video summaries produced automatically from unedited BBC dramatic series video. It was carried out under the auspices of the TREC Video Retrieval Evaluations (TRECVID.) Twenty-two research teams submitted video summaries compressed to at least 4% of the original video's duration to remove redundant and insignificant material. The output of two baseline systems was contributed by Carnegie Mellon University. Ground truth lists of important segments from each video were created by Dublin City University. At NIST each summary was judged by three humans with respect to how much of the ground truth was included, how easy the summary was to understand, and how much repeated material the summary contained. Additional objective measures included: how long it took the assessor to judge the included ground truth and how much system effort was expended creating each summary. Pairwise agreement on judging the inclusion of ground truth averaged 78%. Assessor agreement on amount of repeated video was less than on the ease of understanding the summary. While it appears it was difficult to exceed the two baselines, a few systems did.

Proceedings Title

2007 ACM Multimedia Conference: TRECVID Video Summarization Workshop Proceedings

Conference Title

2007 ACM Multimedia Conference: TRECVID Video Summarization Workshop

Pub Type

Conferences

Download Paper

Local Download

Keywords

BBC rushes, evaluation, TRECVID, video summarization

Citation

Over, P. , Smeaton, A. and Kelly, P. (2007), The TRECVID 2007 BBC Rushes Summarization Evaluation Pilot, 2007 ACM Multimedia Conference: TRECVID Video Summarization Workshop Proceedings, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=51190 (Accessed May 10, 2024)