Desirable Properties for Diversity and Truncated Effectiveness Metrics

Albahem, A, Spina, D, Scholer, F, Moffat, A and Cavedon, L 2018, 'Desirable Properties for Diversity and Truncated Effectiveness Metrics', in Proceedings of the 23rd Australasian Document Computing Symposium 2018, Dunedin, New Zealand, 11-12 December 2018, pp. 1-7.


Document type: Conference Paper
Collection: Conference Papers

Title Desirable Properties for Diversity and Truncated Effectiveness Metrics
Author(s) Albahem, A
Spina, D
Scholer, F
Moffat, A
Cavedon, L
Year 2018
Conference name Australasian Document Computing Symposium (ADCS 2018)
Conference location Dunedin, New Zealand
Conference dates 11-12 December 2018
Proceedings title Proceedings of the 23rd Australasian Document Computing Symposium 2018
Publisher ACM
Place of publication New York, United States
Start page 1
End page 7
Total pages 7
Abstract A wide range of evaluation metrics have been proposed to measure the quality of search results, including in the presence of diversification. Some of these metrics have been adapted for use in search tasks with different complexities, such as where the search system returns lists of different lengths. Given the range of requirements, it can be difficult to compare the behavior of these metrics. In this work, we examine effectiveness metrics using a simple property-based approach. In particular, we present a case-analysis framework to define and study fundamental properties that seem integral to any evaluation metric. An example of a simple property is that a ranking with only one non-relevant document should never score lower than a ranking with two non-relevant documents. The framework facilitates quantifying the ability of metrics to satisfy properties, both separately and simultaneously, and to identify those cases where properties are violated. Our analysis shows that the Average Cube Test and Intent-Aware Average Precision are two metrics which fail to satisfy the desirable properties, and hence should be used with caution.
Subjects Information Retrieval and Web Search
Copyright notice © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ISBN 9781450365499
Versions
Version Filter Type
Access Statistics: 10 Abstract Views  -  Detailed Statistics
Created: Tue, 26 Mar 2019, 09:36:00 EST by Catalyst Administrator
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us