H.264 decoder is left in broken state after a glitch and switch in input

Description

Summary of the bug:
After what seems to be a glitch/switch in the input stream (trace_headers notes that there's 12 H.264 packets for which the BSF couldn't be applied to), the H.264 decoder still keeps decoding, but ultimately is left in a state where it seems to drop references even though it wouldn't have to. Additionally, it seems like the stream switches from MBAFF to PAFF after the glitch, although I'm not 100% sure if I'm parsing things correctly to note that.

How to reproduce:
You can decode or play the sample with pretty much anything lavc-using. It will be OK at first for a moment, and then have a glitch. Then after that glitch there's seemingly OK content (if you initialize decoding after the glitch, the decoding is OK), but the result is borked for the rest of the clip.

Yes, this is not a regression (at least not a recent one) since I've poked at this with various versions from the last year+ or so.

The problem currently is that this glitch or whatever it is happens relatively often in certain streams (although I was only recently able to capture it as it happened), and the decoder doesn't die hard enough, or re-initialize its state properly - thus leading to artifacts in long-running decoders for the rest of the session. Let's just say that's not optimal. That's the primary reason why I created this ticket.

I tested with VAAPI in mpv, and it bails out at the glitch point and falls back to swdec. Of course the next time the glitch happens, swdec is left in a state that isn't optimal.