You don't have to resort to violence, but being passive is no way to go about it. Atleast assert your anger and frustration, otherwise, the little douchebag will do as he pleases to you and your possessions.