I just bought an iPhone 4S and configured my mail-accounts. But for some reason, the new mail is only pushed when I open my Mail-app. I set the settings for incoming messages to "Push". Any idea how this is possible?

Under each individual account, make sure that it is set to Push. If push is not an option, then your email provider does not support push. The only ones that I am sure of are iCloud, Microsoft Exchange, and Gmail (if setup as an Exchange account).