Well, it looks like a network issue. The file that plays choppy has 1.3Mbps bitrate, and the one that plays fine has around 1Mbps bitrate. To rule out any network issues, try to play back the choppy file from a USB drive and see if it is still choppy.

The difference might be in the WDTV base versions (WDLXTV is currently based on 1.05, but latest stock is 1.06), which could translate into differences in samba versions. The long-term solution is to ditch samba and migrate to NFS, which offers better performance (there are guides on the wiki).

Well, you'll need to check what mediainfo reports as total bitrate, but my guess is that the shows that play fine have a lower bitrate than the ones that don't play fine. Since you get your media from a network share (over Samba), you are limited to Samba's throughput, which is the worst throughput possible (Samba < NFS < DLNA).

Since it works for you with stock firmware, it means that stock firmware has some tweaks/a different version of samba which improves performance. WDLXTV has gone through a lot of changes in the samba code through the time (believe it or not, performance used to be worse), but it seems now the official samba version is better.