High Profile will be a good option here. If you want to tweak the quality/file size tradeoff you can tweak the quality target on the Video tab. 20 is the default and is a good place to start. You could go down to 18 for higher quality and a bigger file but
it would probably be lost as Youtube recompresses it. You may be able to get away with 22 or higher.

Leave the advanced options box empty (unless you're getting problems like you're mentioning)
One 160 kbps stereo AAC track. Downmix with Dolby Pro Logic II if it's a multi-channel source.
Experiment with the Quality slider on the Video tab and find a quality level that satisfies you. x264 will do its best to keep the file size as low as possible for that amount of quality. In most cases people will see a re-encoded video so a lower quality in
the source might not be noticeable.