Seems (almost) identical to the ones I've seen. Iirc the debug build does assert out a bit earlier.
If interested you can take a look at adding the missing functionality (to drivers/nouveau) but as-is I'm not sure what nouveau can work with st/va :-\

Unlike public API's, gallium API's are fluid and not extremely well-defined. They're primarily defined by the existing users, and secondarily by documentation when those users disagree. Video API's are especially poorly defined, as there are only 2 real driver implementations (radeon + nouveau), and until recently, only st/xvmc and st/vdpau (and everyone except me hates st/xvmc).
I believe that st/va had to make some changes to how functions were called and what parameters they were allowed to have, but the updates were only made to the radeon drivers. Unfortunately I don't know what those updates were. Emil mentioned chunked decoding, but I don't really know what that is. Perhaps you can look at the st/va enablement patches for radeon and see what all they had to change.

Chunked decoding means that between begin picture and end picture the state tracker calls the decode function multiple times for the same picture.
E.g. you don't get the bitstream all at once, but rather in small chunks.
Nouveau can't deal with that cause Maarten didn't wanted that interface, radeon on the other hand never used anything else.
You just need to fix nouveau to implement bitstream buffer resizing in the same way radeon does it (or move that stuff into the state tracker like I always suggested).

(In reply to Ilia Mirkin from comment #10)
> Wouldn't it be simpler to just accumulate user buffers and then do all the
> work in end_frame instead? And like Christian said, it'd be easy enough to
> get st/va to do this.
Accumulating all the user buffers in the state tracker is what I initially suggested as well and I wouldn't mind if it goes into this direction in the long term.
You would need to add a new type for the bitstream buffer (we currently already abuse vertext buffers in the OMX encoder for this).
For the short term your patch looks like it's going into the right direction.

Created attachment 117358[details][review]
WIP - nouveau: add support for chunk decoding
I updated patch since I miss understood some parts last time.
Preserving data while resizing is commented out for testing since the initial buffer is big enough for the stream I tried.
The patch keeps st/vdpau working but st/va still does not work.
Indeed it shows a green frame. I do not see any shape but I see some pixels that let me think it does something.
How can I be sure it is not a problem from st/va ? I assume it works with radeon (only radeon for now ? according to src/gallium/targets/va/Makefile.am I was expecting intel too)
For nouveau, maybe it requires to do more work on st/va side.
I can try to go for the other solution to compare.
Which internal or external unit tests can I use ? Also if you have an idea how I can debug this ?
Maybe I can compare what contains the nouveau_bo buffer between one begin/end, between vdpau and va ? Not sure if they should contain the exact same thing at the time _end is called.
Thx

You can compare what's going on in the two cases by using valgrind-mmt (http://nouveau.freedesktop.org/wiki/Valgrind-mmt/) in combination with the demmt tool in envytools. However it will be quite tough to tell what's going on since the interesting stuff passed around in buffers rather than sending commands in the fifo. You should still see the buffers being written though.

Created attachment 117910[details][review]
WIP - nouveau: add support for chunk decoding
It now works on mpeg12 using vlc+vaapi+mesa/st/va/nouveau. The problem from the last patch was that the "desc" was written to the nouveau_bo before being actually updated.
vdpau backend still works, and vaapi backend using vdpau works too.
But for now the video is not showing correctly using gstreamer-vaapi so I need to investigate this.

Comment on attachment 117910[details][review]
WIP - nouveau: add support for chunk decoding
Actually it works with GStreamer too using vaapisink. So I am going to submit patches to mailing list. I splited it in 8 patches to make it more clear.

(In reply to Ilia Mirkin from comment #21)
> Please post patches to list so that they may be reviewed.
I cannot send them using git send-email. I am behind a proxy so I think that's why I cannot.
That's why I attached them here. And home computer is broken for now.

Patches have been sent to mailing list for review.
vc1, mpeg12, mpeg4 are working but not h264 (visible but lot of blockiness)
Need someone to try existing st/va with a radeon card and using:
LIBVA_DRIVER_NAME=gallium mpv --hwdec=vaapi anyH264Video.mp4