A Formal Account of Effectiveness Evaluation and Ranking Fusion

Amigó, E, Giner, F, Mizzaro, S and Spina, D 2018, 'A Formal Account of Effectiveness Evaluation and Ranking Fusion', in Proceedings of the 8th ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR 2018), Tianjin, China, 14-17 September 2018, pp. 123-130.


Document type: Conference Paper
Collection: Conference Papers

Title A Formal Account of Effectiveness Evaluation and Ranking Fusion
Author(s) Amigó, E
Giner, F
Mizzaro, S
Spina, D
Year 2018
Conference name ICTIR 2018
Conference location Tianjin, China
Conference dates 14-17 September 2018
Proceedings title Proceedings of the 8th ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR 2018)
Publisher Association for Computing Machinery
Place of publication New York, United States
Start page 123
End page 130
Total pages 8
Abstract This paper proposes a theoretical framework which models the information provided by retrieval systems in terms of Information Theory. The proposed framework allows to formalize: (i) system effectiveness as an information theoretic similarity between system outputs and human assessments, and (ii) ranking fusion as an information quantity measure. As a result, the proposed effectiveness metric improves popular metrics in terms of formal constraints. In addition, our empirical experiments suggest that it captures quality aspects from traditional metrics, while the reverse is not true. Our work also advances the understanding of theoretical foundations of the empirically known phenomenon of effectiveness increase when combining retrieval system outputs in an unsupervised manner.
Subjects Information Retrieval and Web Search
Keyword(s) information theory
evaluation
ranking fusion
DOI - identifier 10.1145/3234944.3234958
Copyright notice © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ISBN 9781450356565
Versions
Version Filter Type
Altmetric details:
Access Statistics: 16 Abstract Views  -  Detailed Statistics
Created: Tue, 23 Oct 2018, 16:00:00 EST by Catalyst Administrator
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us