The influence of topic difficulty, relevance level, and document ordering on relevance judging

Damessie, T, Scholer, F and Culpepper, J 2016, 'The influence of topic difficulty, relevance level, and document ordering on relevance judging', in Sarvnaz Karimi and Mark Carman (ed.) Proceedings of the 21st Australasian Document Computing Symposium (ADCS 2016), Caulfield, Australia, 6-7 December 2016, pp. 41-48.


Document type: Conference Paper
Collection: Conference Papers

Title The influence of topic difficulty, relevance level, and document ordering on relevance judging
Author(s) Damessie, T
Scholer, F
Culpepper, J
Year 2016
Conference name ADCS 2016
Conference location Caulfield, Australia
Conference dates 6-7 December 2016
Proceedings title Proceedings of the 21st Australasian Document Computing Symposium (ADCS 2016)
Editor(s) Sarvnaz Karimi and Mark Carman
Publisher Association for Computing Machinery
Place of publication New York, United States
Start page 41
End page 48
Total pages 8
Abstract In this study we investigate the relationship between how long it takes an assessor to judge document relevance, and three key factors that may influence the judging scenario: the difficulty of the search topic for which relevance is being assessed; the degree to which the documents are relevant to the search topic; and, the order in which the documents are presented for judging. Two potential confounding influences on judgment speed are differences in individual reading ability, and the length of documents that are being assessed. We therefore propose two measures to investigate the above factors: normalized processing speed (NPS), which adjusts the number of words that were processed per minute by taking into account differences in reading speed between judges, and normalized dwell time (NDT), which adjusts the duration that a judge spent reading a document relative to document length. Note that these two measures have different relationships with overall judgment speed: a direct relationship for NPS, and an inverse relationship for NDT.
Subjects Information Retrieval and Web Search
Keyword(s) user studies
dwell time
order effects
query difficulty
DOI - identifier 10.1145/3015022.3015033
Copyright notice © 2016 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ISBN 9781450348652
Versions
Version Filter Type
Altmetric details:
Access Statistics: 116 Abstract Views  -  Detailed Statistics
Created: Tue, 20 Dec 2016, 10:05:00 EST by Catalyst Administrator
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us