Assessor error in stratified evaluation

Webber, W, Oard, D, Scholer, F and Hedin, B 2010, 'Assessor error in stratified evaluation', in Xiangji Jimmy Huang, Gareth Jones, Nick Koudas, Xindong Wu, & Kevyn Collins-Thompson (ed.) ACM International Conference on Information and Knowledge Management, Toronto, Canada, 26-30 October 2010, pp. 539-548.


Document type: Conference Paper
Collection: Conference Papers

Title Assessor error in stratified evaluation
Author(s) Webber, W
Oard, D
Scholer, F
Hedin, B
Year 2010
Conference name 19th ACM International Conference on Information and Knowledge Management
Conference location Toronto, Canada
Conference dates 26-30 October 2010
Proceedings title ACM International Conference on Information and Knowledge Management
Editor(s) Xiangji Jimmy Huang, Gareth Jones, Nick Koudas, Xindong Wu, & Kevyn Collins-Thompson
Publisher ACM
Place of publication USA
Start page 539
End page 548
Total pages 10
Abstract Several important information retrieval tasks, including those in medicine, law, and patent review, have an authoritative standard of relevance, and are concerned about retrieval completeness. During the evaluation of retrieval effectiveness in these domains, assessors make errors in applying the standard of relevance, and the impact of these errors, particularly on estimates of recall, is of crucial concern. Using data from the interactive task of the TREC Legal Track, this paper investigates how reliably the yield of relevant documents can be estimated from sampled assessments in the presence of assessor error, particularly where sampling is stratified based upon the results of participating retrieval systems. We show that assessor error is in general a greater source of inaccuracy than sampling error. A process of appeal and adjudication, such as used in the interactive task, is found to be effective at locating many assessment errors; but the process is expensive if complete, and biased if incomplete. An unbiased double-sampling method for resolving assessment error is proposed, and shown on representative data to be more efficient and accurate than appeal-based adjudication.
Subjects Information Retrieval and Web Search
Information Systems not elsewhere classified
Keyword(s) Estimation theory
e-discovery
recall.
DOI - identifier 10.1145/1871437.1871508
Copyright notice Copyright 2010 ACM
ISBN 9781450300995
Versions
Version Filter Type
Altmetric details:
Access Statistics: 176 Abstract Views  -  Detailed Statistics
Created: Wed, 09 Nov 2011, 08:09:00 EST by Catalyst Administrator
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us