Solution: In case of MP4 files, try to create them "optimized for streaming".

Many video creators store MP4 files using flat layout, where the media tracks are stored complete and in sequence (e.g. entire audio track first, then entire video track). Also the media tracks are stored in the beginning of the file, while the meta data (the so-called "MooV atom") is stored at the end of the file. This is the easiest way to create the files, but it leads to additional work to play them. First of all it leads to a lot of disk seeks within the file as the corresponding data blocks from each track must be read to playback the video at a given time instant. But the main problem is that HTTP streaming (pseudo-streaming/progressive download) and similar will not work since the playback cannot start before the meta-data is read - and since it is at the end of the file, the entire file needs to be downloaded for the playback to start! Therefore many utilities supports storing the MP4 file using a different layout, optimized for streaming. There are a lot of different names for it; "Quicktime fast-start" (qt-faststart), "Web Optimize" (Handbrake uses this term), "optimize mp4 layout", "stream optimize", "HTTP FastStart" and more. Most of them do the same thing: First of all they move the meta data to the beginning of the file, so that the playback can start immediately when streaming the file. Secondly they usually interleaves the media data tracks by storing small chunks from each track in sequence so that the information for a particular instant in time are colocated within the file. This eliminates disk seeks during playback of the file, which results in better performance. As of version 0.2.4, MP4Box always stores the file with 0.5 second interleaving and meta-data at the begining, "making it suitable for HTTP streaming".

How this is handled in PS3 Media Server i don't know, but I got problems with serious stutter when streaming a MP4 file created from a BluRay using DVDFab. After a lot of investigation I came across information about the different MP4 layouts (described above), and I also found evidence that DVDFab creates the files with flat layout (http://forum.dvdfab.com/showthread.php?t=3476, but I also later confirmed it using MP4Muxer to analyze the file, which shows the Atoms tree structure)! When I used MP4Box to re-create the MP4 file with optimized layout (argument "-inter 500", which is actually the option that MP4Box uses by default) the stutter disappeared! Remember I just changed the container, there is no changes in actual media streams - no transcoding or anything.

Exactly why optimizing the MP4 file removed the stutter when streaming with PMS I don't know for sure. I was able to stream the file even before with the meta data at the end of the file (but with stuttering), so this layout must be handled by either PMS or the UPnP protocol or something... But maybe it is the disk seeking that lead to the stuttering? Maybe it's the performance improvements as a result of the interleaving that did the trick for me (and my probably a bit slow disk)?

I'm not an expert on this, maybe some of you guys can correct me, explain it better, or confirm my theory?

EDIT: I streamed the MP4 without transcoding. If using tsMuxer (selecting in the #transcode# folder) then it actually played without stuttering. This results in the MP4 file being transformed by PMS (demuxed, AAC audio transcoded to AC3, then remuxed with the video to a MPEG-PS container). The downside of this method is that it leads to a bit CPU usage on the streaming server (not much since it does not transcode the H.264 video, only the audio), but more importantly this method has sometimes lead to audio and video getting out of sync during the movie. Therefore I definetely prefer streaming the MP4 file directly without any processing (transcoding, remuxing or anything), and from now on I am making sure my MP4 files are "optimized for streaming" - and if not then I transform them using "MP4Box.exe -inter 500 file.mp4"!