An axiomatic analysis of diversity evaluation metrics: Introducing the rank-biased utility metric

Amigó, E, Spina, D and Carrillo-de-Albornoz, J 2018, 'An axiomatic analysis of diversity evaluation metrics: Introducing the rank-biased utility metric', in Proceedings of SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, Ann Arbor, MI, USA, 8-12 July 2018, pp. 625-634.


Document type: Conference Paper
Collection: Conference Papers

Title An axiomatic analysis of diversity evaluation metrics: Introducing the rank-biased utility metric
Author(s) Amigó, E
Spina, D
Carrillo-de-Albornoz, J
Year 2018
Conference name SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval
Conference location Ann Arbor, MI, USA
Conference dates 8-12 July 2018
Proceedings title Proceedings of SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval
Publisher Association for Computing Machinery
Place of publication New York, United States
Start page 625
End page 634
Total pages 10
Abstract Many evaluation metrics have been defined to evaluate the effectiveness ad-hoc retrieval and search result diversification systems. However, it is often unclear which evaluation metric should be used to analyze the performance of retrieval systems given a specific task. Axiomatic analysis is an informative mechanism to understand the fundamentals of metrics and their suitability for particular scenarios. In this paper, we define a constraint-based axiomatic framework to study the suitability of existing metrics in search result diversification scenarios. The analysis informed the definition of Rank-Biased Utility (RBU) - an adaptation of the well-known Rank-Biased Precision metric - that takes into account redundancy and the user effort associated to the inspection of documents in the ranking. Our experiments over standard diversity evaluation campaigns show that the proposed metric captures quality criteria reflected by different metrics, being suitable in the absence of knowledge about particular features of the scenario under study.
Subjects Information Systems Theory
Information Retrieval and Web Search
Keyword(s) Evaluation
Search result diversification
Axiomatic analysis
DOI - identifier 10.1145/3209978.3210024
Copyright notice © 2018 The Author(s)
ISBN 9781450356572
Versions
Version Filter Type
Citation counts: TR Web of Science Citation Count  Cited 1 times in Thomson Reuters Web of Science Article | Citations
Altmetric details:
Access Statistics: 7 Abstract Views  -  Detailed Statistics
Created: Wed, 19 Sep 2018, 13:35:00 EST by Catalyst Administrator
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us