Dang, H.
and Conroy, J.
(2008),
Mind the Gap: Dangers of Divorcing Evaluations of Summary Content from Linguistic Quality, COLING 2008: Proceedings of the 22nd International Conference on Computational Linguistics, Manchester, GB, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=152186
(Accessed September 18, 2024)
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
Secure .gov websites use HTTPS
A lock (
) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.
Mind the Gap: Dangers of Divorcing Evaluations of Summary Content from Linguistic Quality
Published
Author(s)
, John M. Conroy
Abstract
In this paper, we analyze the state of current human and automatic evaluation of topic-focused summarization in the Document Understanding Conference main task for 2005-2007. The analyses show that while ROUGE has very strong correlation with responsiveness for both human and automatic summaries, there is a significant gap in responsiveness between humans and systems which is not accounted for by the ROUGE metrics. In addition to teasing out gaps in the current automatic evaluation, we propose a method to maximize the strength of current automatic evaluations by using the method of canonical correlation. We apply this new evaluation method, which we call ROSE (ROUGE Optimal Summarization Evaluation), to find the optimal linear combination of ROUGE scores to maximize correlation with human responsiveness.Proceedings Title
COLING 2008: Proceedings of the 22nd International Conference on Computational Linguistics
Conference Dates
August 18-22, 2008
Conference Location
Manchester, GB
Conference Title
Coling 2008
Pub Type
Conferences
Download Paper
Citation
Issues
If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.
Created August 18, 2008, Updated February 19, 2017