I'd go with the Chitauri. They were NEVER advertised as being the main villains, Whedon even said repeatedly that they were merely "Loki's army" and yet they still came across as more threatening than the Dark Elves, who WERE supposed to be the main villains.