User performance versus precision measures for simple search tasks

Turpin, A and Scholer, F 2006, 'User performance versus precision measures for simple search tasks', in Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retreival, Seattle, Washington, 6-11 August 2006.


Document type: Conference Paper
Collection: Conference Papers

Attached Files
Name Description MIMEType Size
n2006001961.pdf Published Version application/pdf 143.89KB
Title User performance versus precision measures for simple search tasks
Author(s) Turpin, A
Scholer, F
Year 2006
Conference name Conference on Research and Development in Information Retreival
Conference location Seattle, Washington
Conference dates 6-11 August 2006
Proceedings title Proceedings of the 29th annual international ACM SIGIR conference on research and development in information retreival
Publisher Association for Computing Machinery
Place of publication USA
Abstract Several recent studies have demonstrated that the type of improvements in information retrieval system effectiveness reported in forums such as SIGIR and TREC do not translate into a benefit for users. Two of the studies used an instance recall task, and a third used a question answering task, so perhaps it is unsurprising that the precision based measures of IR system effectiveness on one-shot query evaluation do not correlate with user performance on these tasks. In this study, we evaluate two different information retrieval tasks on TREC Web-track data: a precision-based user task, measured by the length of time that users need to find a single document that is relevant to a TREC topic; and, a simple recall-based task, represented by the total number of relevant documents that users can identify within five minutes. Users employ search engines with controlled mean average precision (MAP) of between 55% and 95%. Our results show that there is no significant relationship between system effectiveness measured by MAP and the precision-based task. A significant, but weak relationship is present for the precision at one document returned metric. A weak relationship is present between MAP and the simple recall-based task.
Subjects Information and Computing Sciences not elsewhere classified
DOI - identifier 10.1145/1148170.1148176
Copyright notice © 2006 ACM
Versions
Version Filter Type
Citation counts: Scopus Citation Count Cited 159 times in Scopus Article | Citations
Altmetric details:
Access Statistics: 179 Abstract Views, 1633 File Downloads  -  Detailed Statistics
Created: Fri, 09 Oct 2009, 08:09:01 EST by Catalyst Administrator
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us