Abstract : The Sybil attack is one of the most aggressive and evasive attacks in sensor networks that can affect on many aspects of network functioning. Thus, its efficient detection is of highest importance. In order to resolve this issue, in this work we propose to couple reputation systems with agents based on self-organizing map algorithm trained for detecting outliers in data. The response of the system consists in assigning low reputation values to the compromised node rendering them isolated from the rest of the network. The main improvement of this work consists in the way of calculating reputation, which is more flexible and discriminative in distinguishing attacks from normal behavior. Self-organizing map algorithm deploys feature space based on sequences of sensor outputs. Our solution offers many benefits: scalable solution, fast response to adversarial activities, ability to detect unknown attacks, high adaptability and low consumption. The testing results demonstrate its high ability in detecting and confining Sybil attack.