Presentation Ordering Effects On Assessor Agreement

Damessie, T, Culpepper, J, Kim, J and Scholer, F 2018, 'Presentation Ordering Effects On Assessor Agreement', in Alfredo Cuzzocrea,James Allan, Norman Paton, Divesh Srivastava, Rakesh Agrawal, Andrei Broder, Mohammed Zaki, Selcuk Candan, Alexandros Labrinidis, Assaf Schuster and Haixun Wang (ed.) Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM 2018), Torino, Italy, 22 - 26 October 2018, pp. 723-732.


Document type: Conference Paper
Collection: Conference Papers

Title Presentation Ordering Effects On Assessor Agreement
Author(s) Damessie, T
Culpepper, J
Kim, J
Scholer, F
Year 2018
Conference name CIKM 2018
Conference location Torino, Italy
Conference dates 22 - 26 October 2018
Proceedings title Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM 2018)
Editor(s) Alfredo Cuzzocrea,James Allan, Norman Paton, Divesh Srivastava, Rakesh Agrawal, Andrei Broder, Mohammed Zaki, Selcuk Candan, Alexandros Labrinidis, Assaf Schuster and Haixun Wang
Publisher Association for Computing Machinery
Place of publication New York, United States
Start page 723
End page 732
Total pages 10
Abstract Consistency of relevance judgments is a vital issue for the construction of test collections in information retrieval. As human relevance assessments are costly, and large collections can contain many documents of varying relevance, collecting reliable judgments is a critical component to building reusable test collections. We explore the impact of document presentation order on human relevance assessments. Our primary goal is to determine if assessor disagreement can be minimized through the order in which documents are presented to assessors. To achieve this goal, we compare two commonly used presentation orderings with a new ordering designed to aid assessors to more easily discriminate between relevant and non-relevant documents. By carefully controlling the presentation ordering, assessors can more quickly converge on a consistent notion of relevance during the assessment exercise, leading to higher overall judging agreement. In addition, important interactions between presentation ordering and topic difficulty on assessor agreement are highlighted. Our findings suggest that document presentation order does indeed have a substantial impact on assessor agreement, and that our new ordering is more robust than previous approaches across a variety of different topic types.
Subjects Information Retrieval and Web Search
DOI - identifier 10.1145/3269206.3271750
Copyright notice © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM
ISBN 9781450360142
Versions
Version Filter Type
Citation counts: TR Web of Science Citation Count  Cited 0 times in Thomson Reuters Web of Science Article
Scopus Citation Count Cited 0 times in Scopus Article
Altmetric details:
Access Statistics: 16 Abstract Views  -  Detailed Statistics
Created: Thu, 21 Feb 2019, 12:10:00 EST by Catalyst Administrator
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us