Peer consistency evaluation in human computation

Peer consistency evaluation is often used in games with a purpose (GWAP) to evaluate workers using outputs of other workers without using gold standard answers. Despite its popularity, the reliability of peer consistency evaluation has never been systematically tested to show how it can be used as a general evaluation method in human computation systems. We present experimental results that show that human computation systems using peer consistency evaluation can lead to outcomes that are even better than those that evaluate workers using gold standard answers. We also show that even without evaluation, simply telling the workers that their answers will be used as future evaluation standards can significantly enhance the workers' performance. Results have important implication for methods that improve the reliability of human computation systems.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2013

Author dc:creator

• Huang, Shih-Wen

Contributors dc:contributor

• Fu, Wai-Tat

• crowdsourcing
• human computation

Statement dc:rights

• Copyright 2013 Shih-Wen Huang

Language dc:language

en

Handle dc:identifier

http://hdl.handle.net/2142/45484

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/45484