A comparison of evaluation measures given how users perform on search tasks

Thom, J and Scholer, F 2007, 'A comparison of evaluation measures given how users perform on search tasks', in Proceedings of the Twelfth Australasian Document Computing Symposium, Melbourne, Australia, 10 December 2007.


Document type: Conference Paper
Collection: Conference Papers

Attached Files
Name Description MIMEType Size
n2006006532.pdf Published version application/pdf 87.14KB
Title A comparison of evaluation measures given how users perform on search tasks
Author(s) Thom, J
Scholer, F
Year 2007
Conference name ADCS2007 Australasian Document Computing Symposium
Conference location Melbourne, Australia
Conference dates 10 December 2007
Proceedings title Proceedings of the Twelfth Australasian Document Computing Symposium
Publisher RMIT University, School of Computer Science and Information Technology
Place of publication Melbourne, Australia
Abstract Information retrieval has a strong foundation of empirical investigation: based on the position of relevant resources in a ranked answer list, a variety of system performance metrics can be calculated. One of the most widely reported measures, mean average precision (MAP), provides a single numerical value that aims to capture the overall performance of a retrieval system. However, recent work has suggested that broad measures such as MAP do not relate to actual user performance on a number of search tasks. In this paper, we investigate the relationship between various retrieval metrics, and consider how these reflect user search performance. Our results suggest that there are two distinct categories of measures: those that focus on high precision in an answer list, and those that attempt to capture a broader summary, for example by including a recall component. Analysis of runs submitted to the TREC terabyte track in 2006 suggests that the relative performance of systems can differ significantly depending on which group of measures is being used.
Subjects Business Information Management (incl. Records, Knowledge and Information Management, and Intelligence)
Copyright notice © 2007 The Authors
Versions
Version Filter Type
Access Statistics: 353 Abstract Views, 605 File Downloads  -  Detailed Statistics
Created: Fri, 09 Oct 2009, 08:09:01 EST by Catalyst Administrator
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us