Users versus models: what observation tells us about effectiveness metrics

Moffat, A, Thomas, P and Scholer, F 2013, 'Users versus models: what observation tells us about effectiveness metrics', in Qi He, Arun Iyengar (ed.) Proceedings of the 22nd ACM International Conference on Information and Knowledge Management, San Francisco, United Sates, 27 October - 1 November 2013, pp. 659-668.


Document type: Conference Paper
Collection: Conference Papers

Title Users versus models: what observation tells us about effectiveness metrics
Author(s) Moffat, A
Thomas, P
Scholer, F
Year 2013
Conference name CIKM '13
Conference location San Francisco, United Sates
Conference dates 27 October - 1 November 2013
Proceedings title Proceedings of the 22nd ACM International Conference on Information and Knowledge Management
Editor(s) Qi He, Arun Iyengar
Publisher ACM
Place of publication New York, United Sates
Start page 659
End page 668
Total pages 10
Abstract Retrieval system effectiveness can be measured in two quite different ways: by monitoring the behavior of users and gathering data about the ease and accuracy with which they accomplish certain specified information-seeking tasks; or by using numeric effectiveness metrics to score system runs in reference to a set of relevance judgments. In the second approach, the effectiveness metric is chosen in the belief that user task performance, if it were to be measured by the first approach, should be linked to the score provided by the metric. This work explores that link, by analyzing the assumptions and implications of a number of effectiveness metrics, and exploring how these relate to observable user behaviors. Data recorded as part of a user study included user self-assessment of search task difficulty; gaze position; and click activity. Our results show that user behavior is influenced by a blend of many factors, including the extent to which relevant documents are encountered, the stage of the search process, and task difficulty. These insights can be used to guide development of batch effectiveness metrics.
Subjects Information Retrieval and Web Search
Computer-Human Interaction
Keyword(s) Retrieval experiment
evaluation
system measurement.
DOI - identifier 10.1145/2505515.2507665
Copyright notice © 2013 ACM
ISBN 9781450322638
Versions
Version Filter Type
Citation counts: Scopus Citation Count Cited 29 times in Scopus Article | Citations
Altmetric details:
Access Statistics: 143 Abstract Views  -  Detailed Statistics
Created: Tue, 21 Jan 2014, 08:24:00 EST by Catalyst Administrator
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us