The TRECVID 2007 BBC Rushes Summarization Evaluation Pilot
Paul D. Over, Alan Smeaton, Phillip Kelly
This paper provides an overview of a pilot evaluation of video summaries produced automatically from unedited BBC dramatic series video. It was carried out under the auspices of the TREC Video Retrieval Evaluations (TRECVID.) Twenty-two research teams submitted video summaries compressed to at least 4% of the original video's duration to remove redundant and insignificant material. The output of two baseline systems was contributed by Carnegie Mellon University. Ground truth lists of important segments from each video were created by Dublin City University. At NIST each summary was judged by three humans with respect to how much of the ground truth was included, how easy the summary was to understand, and how much repeated material the summary contained. Additional objective measures included: how long it took the assessor to judge the included ground truth and how much system effort was expended creating each summary. Pairwise agreement on judging the inclusion of ground truth averaged 78%. Assessor agreement on amount of repeated video was less than on the ease of understanding the summary. While it appears it was difficult to exceed the two baselines, a few systems did.
2007 ACM Multimedia Conference: TRECVID Video Summarization Workshop Proceedings
2007 ACM Multimedia Conference: TRECVID Video Summarization Workshop
BBC rushes, evaluation, TRECVID, video summarization
, Smeaton, A.
and Kelly, P.
The TRECVID 2007 BBC Rushes Summarization Evaluation Pilot, 2007 ACM Multimedia Conference: TRECVID Video Summarization Workshop Proceedings, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=51190
(Accessed December 1, 2023)