Recent work in security and systems has embraced the use of machine learning
(ML) techniques for identifying misbehavior, e.g. email spam and fake
(Sybil) users in social networks. However, ML models are typically derived from
fixed datasets, and must be periodically retrained. In adversarial
environments, attackers can adapt by modifying their behavior or even sabotaging
ML models by polluting training data.

In this paper, we perform an empirical study of adversarial attacks against
machine learning models in the context of detecting malicious crowdsourcing
systems, where sites connect paying users with workers willing to carry out
malicious campaigns. By using human workers, these systems can easily
circumvent deployed security mechanisms, e.g. CAPTCHAs. We collect a
dataset of malicious workers actively performing tasks on Weibo, China's
Twitter, and use it to develop ML-based detectors. We show that traditional ML
techniques are accurate (95%-99%) in detection but can be highly
vulnerable to adversarial attacks, including simple evasion attacks
(workers modify their behavior) and more powerful poisoning attacks (where
administrators tamper with the training set). We quantify the robustness of ML
classifiers by evaluating them in a range of practical adversarial models using
ground truth data. Our analysis provides a detailed look at practical
adversarial attacks on ML models, and helps defenders make informed decisions in
the design and configuration of ML detectors.