Description

Anyway, we're seeing a hang attempting to decode H264 video using DXVA2 (aka pixel format PIX_FMT_DXVA2_VLD). This is in the recently released Plex Media Center (​http://www.plexapp.com). The only workaround we've found is to use a single thread when decoding.

It would be great if you could help us diagnose the issue. We suspect an ffmpeg bug, but it's possible that we're doing something wrong as well.

In our repro case, we call avcodec_thread_init with 8 threads, then avcodec_open, then obviously avcodec_decode_video. If we don't call avcodec_thread_init and we leave the context's default of 1 thread, then no hang occurs and decoding proceeds smoothly.

The hang itself occurs on the second or third call to avcodec_decode_video. Unfortunately, because of the difficulty of debugging ffmpeg code on Windows, I don't have good stack traces to provide. All I can see is avcodec waiting on what appear to be pthreads condition variables. None of our related threads are hung, and avcodec_decode_video is not returning, so my guess would be a deadlock in ffmpeg.

It is a bit unclear what exactly you consider the problem.
Multithreaded decoding makes no sense to combine with hardware decode, so it is not supported.
So the answer to you problem would be "just don't do that then".
Now, I know that e.g. VideoLan? has issues with this because they really want to decide on whether to use hardware decode or not after opening the decoder.
This is supposed to work by the decoder automatically disabling threads when a hardware PIX_FMT is chosen.
However that seems to be buggy, patches very much welcome. Since this kind of thing needs hardware _and_ an application other that FFmpeg to test it is rather a pain and the probability of any of the FFmpeg core developers fixing it on their own is a bit low I think.

I agree that multithreaded decoding and DXVA don't really make sense together. Our problem however is similar to VideoLan?, in that we decide on the pixel format well after opening the decoder. So by the time we know that we really wanted one thread, it's already too late.

It would be fantastic if ffmpeg would gracefully revert to using a single thread under these scenarios. Hanging is clearly suboptimal. :-) So I think we agree on that as well.

Unfortunately, I find debugging ffmpeg on Windows pretty challenging, given that mingw builds don't have symbolic information as per what the standard platform tools expect. Do you have any tips or hints on how best to debug this?

If any of the ffmpeg core developers were to attack this problem, I've be overjoyed to help in terms of providing help with repro cases, builds of Plex and so forth. I'm sure we're not the only user of ffmpeg hitting this issue, so it would be great to get it fixed for everyone.

Unfortunately, I find debugging ffmpeg on Windows pretty challenging, given that mingw builds don't have symbolic information as per what the standard platform tools expect. Do you have any tips or hints on how best to debug this?

gdb works fine on Windows.

If any of the ffmpeg core developers were to attack this problem, I've be overjoyed to help in terms of providing help with repro cases, builds of Plex and so forth. I'm sure we're not the only user of ffmpeg hitting this issue, so it would be great to get it fixed for everyone.

I am still surprised that this problem does not hit vlc users.
(mplayer -vc ffh264vdpau -lavdopts threads=8 also works fine.)

VLC developers have experienced the problem, I talked about it at the VLC dev conference.
I think some fixes were made, but I am not sure it fixed everything (I assume you are using latest git to test?).

mplayer -vc ffh264vdpau -lavdopts threads=8 also works fine.

VDPAU uses a separate codec that ignores threads from the start. The problem is disabling threading once the PIX_FMT is selected.
In addition I expect it is only an issue with frame multithreading, otherwise you might be able to test with mplayer XvMC...