Using clicks as implicit judgements: Expectations versus observations

Scholer, F, Shokouhi, M, Billerbeck, B and Turpin, A 2008, 'Using clicks as implicit judgements: Expectations versus observations', in Advances in Information Retrieval, Glasgow, United Kingdom, 30 March - 03 April 2008.


Document type: Conference Paper
Collection: Conference Papers

Title Using clicks as implicit judgements: Expectations versus observations
Author(s) Scholer, F
Shokouhi, M
Billerbeck, B
Turpin, A
Year 2008
Conference name 30th European Conference on IR Research ECIR 2008
Conference location Glasgow, United Kingdom
Conference dates 30 March - 03 April 2008
Proceedings title Advances in Information Retrieval
Publisher Springer Verlag
Place of publication Berlin, Germany
Abstract Clickthrough data has been the subject of increasing popularity as an implicit indicator of user feedback. Previous analysis has suggested that user click behaviour is subject to a quality bias¿that is, users click at different rank positions when viewing effective search results than when viewing less effective search results. Based on this observation, it should be possible to use click data to infer the quality of the underlying search system. In this paper we carry out a user study to systematically investigate how click behaviour changes for different levels of search system effectiveness as measured by information retrieval performance metrics. Our results show that click behaviour does not vary systematically with the quality of search results. However, click behaviour does vary significantly between individual users, and between search topics. This suggests that using direct click behaviour¿click rank and click frequency¿to infer the quality of the underlying search system is problematic. Further analysis of our user click data indicates that the correspondence between clicks in a search result list and subsequent confirmation that the clicked resource is actually relevant is low. Using clicks as an implicit indication of relevance should therefore be done with caution.
Subjects Information Systems Organisation
DOI - identifier 10.1007/978-3-540-78646-7_6
Copyright notice © 2008 Springer Verlag
Versions
Version Filter Type
Citation counts: TR Web of Science Citation Count  Cited 11 times in Thomson Reuters Web of Science Article | Citations
Scopus Citation Count Cited 13 times in Scopus Article | Citations
Altmetric details:
Access Statistics: 179 Abstract Views  -  Detailed Statistics
Created: Fri, 09 Oct 2009, 08:09:01 EST by Catalyst Administrator
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us