Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks.
R. Snow, B. O'Connor, D. Jurafsky, and A. Ng. EMNLP '08: Proceedings of the Conference on Empirical Methods in Natural Language Processing, page 254--263. Morristown, NJ, USA, Association for Computational Linguistics, (2008)
Abstract
Human linguistic annotation is crucial for many natural language processing tasks but can be expensive and time-consuming. We explore the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web. We investigate five tasks: affect recognition, word similarity, recognizing textual entailment, event temporal ordering, and word sense disambiguation. For all five, we show high agreement between Mechanical Turk non-expert annotations and existing gold standard labels provided by expert labelers. For the task of affect recognition, we also show that using non-expert labels for training machine learning algorithms can be as effective as using gold standard annotations from experts. We propose a technique for bias correction that significantly improves annotation quality on two tasks. We conclude that many large labeling tasks can be effectively designed and carried out in this method at a fraction of the usual expense.
%0 Conference Paper
%1 1613751
%A Snow, Rion
%A O'Connor, Brendan
%A Jurafsky, Daniel
%A Ng, Andrew Y.
%B EMNLP '08: Proceedings of the Conference on Empirical Methods in Natural Language Processing
%C Morristown, NJ, USA
%D 2008
%I Association for Computational Linguistics
%K affective-computing crowdsourcing dataset emotion emotion-recognition inter-rater-agreement nlp annotation-bias
%P 254--263
%T Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks.
%U https://www.aclweb.org/anthology/D08-1027/
%X Human linguistic annotation is crucial for many natural language processing tasks but can be expensive and time-consuming. We explore the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web. We investigate five tasks: affect recognition, word similarity, recognizing textual entailment, event temporal ordering, and word sense disambiguation. For all five, we show high agreement between Mechanical Turk non-expert annotations and existing gold standard labels provided by expert labelers. For the task of affect recognition, we also show that using non-expert labels for training machine learning algorithms can be as effective as using gold standard annotations from experts. We propose a technique for bias correction that significantly improves annotation quality on two tasks. We conclude that many large labeling tasks can be effectively designed and carried out in this method at a fraction of the usual expense.
@inproceedings{1613751,
abstract = {Human linguistic annotation is crucial for many natural language processing tasks but can be expensive and time-consuming. We explore the use of Amazon's Mechanical Turk system, a significantly cheaper and faster method for collecting annotations from a broad base of paid non-expert contributors over the Web. We investigate five tasks: affect recognition, word similarity, recognizing textual entailment, event temporal ordering, and word sense disambiguation. For all five, we show high agreement between Mechanical Turk non-expert annotations and existing gold standard labels provided by expert labelers. For the task of affect recognition, we also show that using non-expert labels for training machine learning algorithms can be as effective as using gold standard annotations from experts. We propose a technique for bias correction that significantly improves annotation quality on two tasks. We conclude that many large labeling tasks can be effectively designed and carried out in this method at a fraction of the usual expense.},
added-at = {2020-09-24T18:27:06.000+0200},
address = {Morristown, NJ, USA},
author = {Snow, Rion and O'Connor, Brendan and Jurafsky, Daniel and Ng, Andrew Y.},
biburl = {https://www.bibsonomy.org/bibtex/29883bc0ea13900c3c95fe82874dae6e7/ghagerer},
booktitle = {EMNLP '08: Proceedings of the Conference on Empirical Methods in Natural Language Processing},
interhash = {fa24c8db32accb61e8bf025b23574986},
intrahash = {9883bc0ea13900c3c95fe82874dae6e7},
keywords = {affective-computing crowdsourcing dataset emotion emotion-recognition inter-rater-agreement nlp annotation-bias},
location = {Honolulu, Hawaii},
pages = {254--263},
publisher = {Association for Computational Linguistics},
timestamp = {2023-05-06T19:02:37.000+0200},
title = {Cheap and Fast - But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks.},
url = {https://www.aclweb.org/anthology/D08-1027/},
year = 2008
}