Skip to main content
U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

A Comparative Analysis of Cascade Measures for Novelty and Diversity

Published

Author(s)

Ian M. Soboroff, Charles L. Clarke, Nick Craswell, Azin Ashkan

Abstract

Traditional editorial effectiveness measures, such as nDCG, remain standard for Web search evaluation. Unfortunately, these traditional measures can inappropriately reward re- dundant information and can fail to reflect the broad range of user needs that can underlie a Web query. To address these deficiencies, several researchers have recently proposed effectiveness measures for novelty and diversity. Many of these measures are based on simple cascade models of user behavior, which operate by considering the relationship be- tween successive elements of a result list. The properties of these measures are still poorly understood, and it is not clear from prior research that they work as intended. In this paper we examine the properties and performance of cascade measures with the goal of validating them as tools for measuring effectiveness. We explore their commonalities and differences, placing them in a unified framework; we dis- cuss their theoretical difficulties and limitations, and com- pare the measures experimentally, contrasting them against traditional measures and against other approaches to mea- suring novelty. Data collected by the TREC 2009 Web Track is used as the basis for our experimental comparison. Our results indicate that these measures reward systems that achieve an balance between novelty and overall precision in their result lists, as intended. Nonetheless, other measures provide insights not captured by the cascade measures, and we suggest that future evaluation efforts continue to report a variety of measures.
Proceedings Title
Proceedings of the Fourth ACM International Conference on Web Search and Data Mining (WSDM 2011)
Conference Dates
February 9-12, 2011
Conference Location
Hong Kong
Conference Title
The Fourth ACM International Conference on Web Search and Data Mining (WSDM 2011)

Keywords

information retrieval, search evaluation, effectiveness measures, diversity ranking

Citation

Soboroff, I. , Clarke, C. , Craswell, N. and Ashkan, A. (2011), A Comparative Analysis of Cascade Measures for Novelty and Diversity, Proceedings of the Fourth ACM International Conference on Web Search and Data Mining (WSDM 2011), Hong Kong, -1, [online], https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=907308 (Accessed December 3, 2024)

Issues

If you have any questions about this publication or are having problems accessing it, please contact reflib@nist.gov.

Created February 10, 2011, Updated February 19, 2017