My "educated" guess is that practically all modern video streams have "A" and "B" frames (some formats might calls this I & B etc but the name is irrelevant).

"A" frames are a single frame that store everything required to view that single frame. This is costly (in space/bandwidth) to reproduce all the data of the image for every frame. So we have...

"B" frames, which store just the differential data between the previous frame and the current frame (e.g. the sky in the background has remained the same, don't include that info, just the info required to show the changes in the frame of the person and the dog in the foreground having sex).

Therefore, if you try to seek to the 250th B-frame in a row, it must rewind to the nearest previous A-frame in order to calculate how that frame will appear after all those changed B-frames.

A lot of players consider this too expensive in resources to seek back then forward, so lots will just start replaying at the nearest A-frame. If you're online this is especially noticeable because players want to display a stream ASAP rather than make you wait while all those frames are downloaded and processed.

Some players will actually immediately start playing even if it lands on a B-frame without collecting the full frame info, those are the ones where you see bleeding colors on a black background that represent just the changing data.

Other players just choose to avoid the conundrum and don't implement seek at all (perhaps they think this is better than dropping a user into the wrong spot or making them wait).

It's mainly this along with the fact that certain files have corrupted indexes or don't have indexes at all so the player has no clue where the key frames even are so the only option for it is to rapidly search through all the frames from the start.