Try to convert the file with a lower bitrate (HandBrake allows you to do this).
I used to run videofiles on an underclocked T3 (from 400 to 200Mhz) without any problems using 24 fps and using 480x230 format, but bitrate set to 350.

BTW are the dropped frames noticeable ? Otherwise I would not bother about it ...

Different format than .m4v maybe?
You could definitely use a lower frame rate (down to 15), although my 650 plays mpeg4 with 44,100kHz audio and 320x240 25fps video without error. (Bechmark fps = 46), so your 680 should manage your file. Must be something else loading up the system.