I've been messing with it for a while and I find MPEG-4 in the .mp4 container works best for me. If you use h.264 with .mp4 you will get an error, and I believe h.263 and .mp4 gives only gray video (e.g. the video is gray and only gray).

In my testing I've only used AAC, but I don't think it really matters unless you are recording sound only (if that works; I haven't tried it.)