Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias

Barbera, D, Roitero, K, Demartini, G, Mizzaro, S and Spina, D 2020, 'Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias', in Joemon M. Jose, Emine Yilmaz, João Magalhães, Pablo Castells, Nicola Ferro, Mário J. Silva, Flávio Martins (ed.) Proceedings of the 42nd European Conference on IR Research (ECIR 2020), Lisbon, Portugal, 14-17 April 2020, pp. 207-214.


Document type: Conference Paper
Collection: Conference Papers

Title Crowdsourcing Truthfulness: The Impact of Judgment Scale and Assessor Bias
Author(s) Barbera, D
Roitero, K
Demartini, G
Mizzaro, S
Spina, D
Year 2020
Conference name ECIR 2020: Part II- Lecture Notes in Computer Science 12036
Conference location Lisbon, Portugal
Conference dates 14-17 April 2020
Proceedings title Proceedings of the 42nd European Conference on IR Research (ECIR 2020)
Editor(s) Joemon M. Jose, Emine Yilmaz, João Magalhães, Pablo Castells, Nicola Ferro, Mário J. Silva, Flávio Martins
Publisher Springer Nature
Place of publication Switzerland
Start page 207
End page 214
Total pages 8
Abstract News content can sometimes be misleading and influence users decision making processes (e.g., voting decisions). Quantitatively assessing the truthfulness of content becomes key, but it is often challenging and thus done by experts. In this work we look at how experts and non-expert assess truthfulness of content by focusing on the effect of the adopted judgment scale and of assessors own bias on the judgments they perform. Our results indicate a clear effect of the assessors political background on their judgments where they tend to trust content which is aligned to their own belief, even if experts have marked it as false. Crowd assessors also seem to have a preference towards coarse-grained scales, as they tend to use a few extreme values rather than the full breadth of fine-grained scales.
Subjects Information Retrieval and Web Search
DOI - identifier 10.1007/978-3-030-45442-5_26
Copyright notice © Springer Nature Switzerland AG 2020
ISBN 9783030454418
Versions
Version Filter Type
Altmetric details:
Access Statistics: 141 Abstract Views  -  Detailed Statistics
Created: Tue, 12 May 2020, 10:21:00 EST by Catalyst Administrator
© 2014 RMIT Research Repository • Powered by Fez SoftwareContact us