According to the wiki, "Having an Ender Pearl in your inventory will keep Endermen from challenging you (Due to staring). However, they will still fight back if you attack them."
Hope that clears it up for you folks.

The first Enderman I killed dropped one... guess that's them dealt with!