Creating a test collection to evaluate diversity in image retrieval

Arni, T, Tang, J, Sanderson, M and Clough, P 2008, 'Creating a test collection to evaluate diversity in image retrieval', in Proceedings of the ACM SIGIR workshop on Beyond Binary Relevance: Preferences, Diversity, and Set-Level Judgments, Singapore, 20-24 July 2008, pp. 15-21.


Document type: Conference Paper
Collection: Conference Papers

Title Creating a test collection to evaluate diversity in image retrieval
Author(s) Arni, T
Tang, J
Sanderson, M
Clough, P
Year 2008
Conference name ACM SIGIR workshop on Beyond Binary Relevance: Preferences, Diversity, and Set-Level Judgments
Conference location Singapore
Conference dates 20-24 July 2008
Proceedings title Proceedings of the ACM SIGIR workshop on Beyond Binary Relevance: Preferences, Diversity, and Set-Level Judgments
Publisher ACM
Place of publication New York, United States
Start page 15
End page 21
Total pages 7
Abstract This paper describes the adaptation of an existing test collection for image retrieval to enable diversity in the results set to be measured. Previous research has shown that a more diverse set of results often satisfies the needs of more users better than standard document rankings. To enable diversity to be quantified, it is necessary to classify images relevant to a given theme to one or more sub-topics or clusters. We describe the challenges in building (as far as we are aware) the first test collection for evaluating diversity in image retrieval. This includes selecting appropriate topics, creating sub-topics, and quantifying the overall effectiveness of a retrieval system. A total of 39 topics were augmented for cluster-based relevance and we also provide an initial analysis of assessor agreement for grouping relevant images into sub-topics or clusters.
Subjects Information Systems not elsewhere classified
ISBN 9781605581644
Versions
Version Filter Type
Access Statistics: 87 Abstract Views  -  Detailed Statistics
Created: Fri, 04 Nov 2011, 08:38:00 EST by Catalyst Administrator
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us