The effect of document order and topic difficulty on assessor agreement

Damessie, T, Scholer, F, Jarvelin, K and Culpepper, J 2016, 'The effect of document order and topic difficulty on assessor agreement', in Proceedings of the 6th ACM International Conference on the Theory of Information Retrieval (ICTIR 2016), Newark, United States, 12-16 September 2016, pp. 73-76.


Document type: Conference Paper
Collection: Conference Papers

Title The effect of document order and topic difficulty on assessor agreement
Author(s) Damessie, T
Scholer, F
Jarvelin, K
Culpepper, J
Year 2016
Conference name ICTIR 2016
Conference location Newark, United States
Conference dates 12-16 September 2016
Proceedings title Proceedings of the 6th ACM International Conference on the Theory of Information Retrieval (ICTIR 2016)
Publisher Association for Computing Machinery
Place of publication New York, United States
Start page 73
End page 76
Total pages 4
Abstract Human relevance judgments are a key component for measuring the effectiveness of information retrieval systems using test collections. Since relevance is not an absolute concept, human assessors can disagree on particular topic-document pairs for a variety of reasons. In this work we investigate the effect that document presentation order has on inter-rater agreement, comparing two presentation ordering approaches similar to those used in IR evaluation campaigns: decreasing relevance order and document identifier order. We make a further distinction between "easy" topics and "hard" topics in order to explore system effects on inter-rater agreement. The results of our pilot user study indicate that assessor agreement is higher when documents are judged in document identifier order. In addition, there is higher overall agreement on easy topics than on hard topics.
Subjects Information Retrieval and Web Search
Keyword(s) Experimentation
measurement
relevance
assessor agreement
ordering effects
DOI - identifier 10.1145/2970398.2970431
Copyright notice Copyright is held by the owner/author(s). Publication rights licensed to Association for Computing Machinery (ACM)
ISBN 9781450344975
Versions
Version Filter Type
Citation counts: Scopus Citation Count Cited 3 times in Scopus Article | Citations
Altmetric details:
Access Statistics: 44 Abstract Views  -  Detailed Statistics
Created: Tue, 20 Dec 2016, 10:05:00 EST by Catalyst Administrator
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us