Assessing efficiency-effectiveness tradeoffs in multi-stage retrieval systems without using relevance judgments

Clarke, C, Culpepper, J and Moffat, A 2016, 'Assessing efficiency-effectiveness tradeoffs in multi-stage retrieval systems without using relevance judgments', Information Retrieval Journal, vol. 19, no. 4, pp. 351-377.


Document type: Journal Article
Collection: Journal Articles

Title Assessing efficiency-effectiveness tradeoffs in multi-stage retrieval systems without using relevance judgments
Author(s) Clarke, C
Culpepper, J
Moffat, A
Year 2016
Journal name Information Retrieval Journal
Volume number 19
Issue number 4
Start page 351
End page 377
Total pages 27
Publisher Springer Science+Business Media
Abstract Large-scale retrieval systems are often implemented as a cascading sequence of phases-a first filtering step, in which a large set of candidate documents are extracted using a simple technique such as Boolean matching and/or static document scores; and then one or more ranking steps, in which the pool of documents retrieved by the filter is scored more precisely using dozens or perhaps hundreds of different features. The documents returned to the user are then taken from the head of the final ranked list. Here we examine methods for measuring the quality of filtering and preliminary ranking stages, and show how to use these measurements to tune the overall performance of the system. Standard top-weighted metrics used for overall system evaluation are not appropriate for assessing filtering stages, since the output is a set of documents, rather than an ordered sequence of documents. Instead, we use an approach in which a quality score is computed based on the discrepancy between filtered and full evaluation. Unlike previous approaches, our methods do not require relevance judgments, and thus can be used with virtually any query set. We show that this quality score directly correlates with actual differences in measured effectiveness when relevance judgments are available. Since the quality score does not require relevance judgments, it can be used to identify queries that perform particularly poorly for a given filter. Using these methods, we explore a wide range of filtering options using thousands of queries, categorize the relative merits of the different approaches, and identify useful parameter combinations.
Subject Data Format not elsewhere classified
Keyword(s) Effectiveness
Efficiency
Experimentation
Measurement
Search
DOI - identifier 10.1007/s10791-016-9279-1
ISSN 1386-4564
Versions
Version Filter Type
Citation counts: Scopus Citation Count Cited 0 times in Scopus Article
Altmetric details:
Access Statistics: 93 Abstract Views  -  Detailed Statistics
Created: Thu, 10 Aug 2017, 12:52:00 EST by Catalyst Administrator
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us