A case for improved evaluation of query difficulty prediction

Scholer, F and Garcia, S 2009, 'A case for improved evaluation of query difficulty prediction', in Mark Sanderson, Cheng Xiang Zhai, Justin Zobel, James Allan, Javed A. Aslam (ed.) Proceedings of 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Boston, USA, 19 - 23 July 2009, pp. 640-641.


Document type: Conference Paper
Collection: Conference Papers

Title A case for improved evaluation of query difficulty prediction
Author(s) Scholer, F
Garcia, S
Year 2009
Conference name 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
Conference location Boston, USA
Conference dates 19 - 23 July 2009
Proceedings title Proceedings of 32nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval
Editor(s) Mark Sanderson, Cheng Xiang Zhai, Justin Zobel, James Allan, Javed A. Aslam
Publisher Association for Computing Machinery (ACM)
Place of publication New York, USA
Start page 640
End page 641
Total pages 2
Abstract Query difficulty prediction aims to identify, in advance, how well an information retrieval system will perform when faced with a particular search request. The current standard evaluation methodology involves calculating a correlation coefficient, to indicate how strongly the predicted query difficulty is related with an actual system performance measure, usually Average Precision. We run a series of experiments based on predictors that have been shown to perform well in the literature, comparing these across different TREe runs. Our results demonstrate that the current evaluation methodology is severely limited. Although it can be used to demonstrate the performance of a predictor for a single system, such performance is not consistent over a variety of retrieval systems. We conclude that published results in the query difficulty area are generally not comparable, and recommend that prediction be evaluated against a spectrum of underlying search systems.
Subjects Performance Evaluation; Testing and Simulation of Reliability
Keyword(s) Experimentation
Measurement
Copyright notice Copyright 2009 the Authors
ISBN 9781605584836
Versions
Version Filter Type
Citation counts: Scopus Citation Count Cited 10 times in Scopus Article | Citations
Access Statistics: 327 Abstract Views  -  Detailed Statistics
Created: Thu, 02 Jun 2011, 15:22:00 EST by Catalyst Administrator
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us