I also got a m2ts who make mencoder_mt freeze, same logs than yours... But I made a new build of mencoder_mt who seems to work better... I will post it here later for those wanting to test it...maybe for an inclusion in 1.04, planned for the middle of the week

I would also like to know why a 1080p non-transcoded streams flawlessly at 30-40 mbit and the same file transcoded peaks to 150-200mbit?

"keyint=1" is not adapted at all for transcoding high bitrate blu rays... it's more a setting made to have the fastest encoding time with no real accurate bitrate control: that's why I choosed this setting by default since the beginning, for people who want to transcode 1080p with a not so powerful CPU, but over wire...

try "keyint=50:vqscale=1:vqmin=2", with your maximum bandwidth value, bitrate variations should be reduced I hope (CPU will be more stressed, but with a Quad core, that should be OK)

After replacing errors still exist. The mencoder_mt is more stable (no mencoder_mt win error), but the ps3 connections drops or is extremely stutterly. I wonder if mencoder has problems with trancoding m2t files, since all 1080p files that cause problems are m2t/m2ts. A few 1080p mkv I have are all working OK. Maybe its just a coincidence...

first, you can check if all your 4 cores are used (maybe the problem lies here)...

a more subtle possibility is maybe that "-lavdopts fast" switch I put by default with the MT build (known to badly work with some codecs, but H264 seems to behave right)... to remove it, it's a bit tricky: disable the "enhanced mencoder multicore" checkbox; go in expert settings and there, just add the following line in custom parameters