Detecting Adversaries among Heterogeneous Annotators

Abstract

Supervised learning from multiple annotators has become an increasingly important problem in machine learning and data mining. We developed a probabilistic approach to this problem when annotators may be unreliable (labels are noisy) and also took into account the fact that their expertise can vary depending on the data they observe (annotators may have knowledge about different parts of the input space). The classification and annotator models obtained using the approach allows us to provide estimates of the true labels and annotator variable expertise. Further, we want to make the current approach robust to adversarial annotators. Adversarial annotators are malicious and competent enough to lead to poor classification results and hence it is important to model them.