Auto insert fps filter as adapter for overlay filter inputs with different framerates

Description

Now that the fps filter has been added (you're my heroes for adding it), it would be great if an fps filter would be auto-inserted before the overlay filter, where framerate conversion is required if the framerates of the two input vary (the way a scale filter is often auto-inserted wherever it's required).

To do this, whenever an overlay filter is used in a filter chain, the framerate of each of its inputs should be detected (using the same function used by ffprobe to populate the avg_frame_rate property), then insert an fps filter to upsample the input with the lower framerate to match the framerate of the other one.

Currently I do this manually: run ffprobe on each video file before performing an overlay, then insert an fps filter between the lower-fps file and the overlay filter. This means 3 procedure calls. It would be great if it could all be done in 1.

I'm not sure I know what you mean by "whenever any of its inputs change" but I don't think that's the current behavior anyway.

Here is the behavior I observed:

The overlay filter takes every frame of the first (main) input in turn and copies it to the output with the same timestamp while overlaying the next frame from the second input (ignoring the timestamps / framerate of the second input).

If it runs out of frames for the second input before the first one, it simply overlays the last frame of the second input on each remaining frame of the first input.

If it runs out of frames for the first input before the second one, the rest of the frames of the second input are ignored.

Therefore, the framerate and duration of the output is always equal to the framerate and duration of the first input. This can lead to a situation where the framerate of the second input (the overlaid video) is not the same as the framerate of the output file, so it appears to be running in slow/fast motion.

Why wouldn't it be a good general solution to equalize the frame rates? Do you think a user might want one of the input streams to appear slowed down / sped up in the output just because it's been overlaid on top of another stream?

At the very least, I think this would be useful to many users if it were included as an optional behavior (i.e. triggered by a config parameter).

The most obvious reason why it is not a general solution is that not all inputs have a (fixed) framerate or something that is a suitable substitute.
Even in a very simple case of overlaying a 24 fps video with a 25 fps one, "equalizing" the fps will not lead to a good result. If you change the 24 fps one to 25 fps that will result in noticeable stutter once a second, for exact results you'd have to go to an insane 600 fps.
If the overlay filter were taking both timestamps into account you could get an _exact_ (though variable fps) output with at most 49 frames per second (though those frames would be at very variable intervals and a lot of players would have problems displaying it accurately I admit, so the win may be questionable in real-world use).

Fair enough, but as I said even now the overlay filter isn't taking both streams' timestamps into account, so what I'm suggesting is the most easily obtainable "fix" for this use case, and I've seen many postings by people using the overlay filter the way I am.

The current default behavior of the overlay filter (ending up with a fast-motion/slow-motion overlay) is, to me, not as good as what I'm suggesting, and can be very confusing.

I think Reimar explained pretty well why your suggestion is not a good idea and is, in fact, not properly defined in the first place. Please remember that filters must be designed for general use, not for any specific use case.

Currently, the overlay filters produces one output frame for each input frame on its "main" input. It uses the timestamps to determiner which overlay frame must go on the main frame.

Adding an option to output a frame also when the overlay changes, like Reimar suggests, is of course possible.

As far as I can tell your claim that it ignores the timestamps on the overlay input is just not true, and I can confirm that two video at different frame rates with visible synchronization stay on sync. The worst that happens is an off-by-one bug that is already underway of being fixed.

Please, be more specific about the actual problems you experience and are trying to solve.

1) the "main" video fps always wins for the output.
2) according to what I can see in ​https://github.com/FFmpeg/FFmpeg/blob/master/libavfilter/vf_overlay.c what it does is, for each "main" frame coming in, it looks for the overlay frame that "overlaps" that one in PTS (or matches it). So it basically outputs the original video, with a synchronized overlay on it. So order really matters here. Maybe the documentation could be updated a bit to make this more clear? It seems like reasonable functionally...unless you have 2 inputs with variable frame rates, so couldn't decide which one is the "main" one but I'd imagine that's pretty rare...

What happens if a higher frame-rate was manually specified? Does ffmpeg just repeat the same combination of main and overlay or does it also look for every repeated main frame which overlay frame is used?

What happens if a higher frame-rate was manually specified? Does ffmpeg just repeat the same combination of main and overlay or does it also look for every repeated main frame which overlay frame is used?

AFAIK, the effect of -r comes completely after the effects of filters, so depending on the value of -vsync it will result on the frame being duplicated as you describe.

IMHO, there are too many possible circumstances to be able to "Do What I Mean" always, so it is best to keep everything simple so accurate results can be achieved with the correct combination of filters without bad surprises.

I can (apparently) confirm that if the "overlay" is 30 fps and you use -r 30 for the output, it doesn't show all of the overlay frames (it basically has a "master slave" relationship, with the master--the under frame--controlling frequency of output frames. However I suppose one could get around this by (as the OP said) determining the frame rate, and increasing/decreasing it first before applying the filter, so the current behavior might be all right. Either that or insert an fps filter automatically, as mentioned, but I'd almost prefer the current way (more explicit)...