I know that x264 expects YV12 input and produces YV12 output. I also understand that there will always be a loss of color detail when converting RGB to YV12 and am not asking how to preserve that lost detail.

So, should I:
1) Continue piping raw RGB24 frames to x264, letting the encoder treat them as YV12;
2) Use a filter to convert the frames to YV12 before piping them;
3) Capture using YUV420 (YV12) BT.709 instead of RGB24;
4) Do something else entirely?

I have tried methods (1) and (2) and found that they produce slightly different colors. Obviously, my untrained eye does not know which output (in theory) has the more "correct" colors for YV12. Furthermore, if I should utilize method (2) by using, for example, VirtualDub's "convert format" filter, should I choose limited or full range?

Not true, you can encode RGB with x264 (it's stored as YUV, but decoded as RGB). But if it's for youtube, that is pointless

If you want the colors to look the same on youtube, use ConvertToYV12(matrix="rec709") in an avisynth script. It is possible now to do it with ffmpeg as well, without avisynth, using -vf colormatrix=bt601:bt709. But, for some reason, the quality is lower than the avisynth conversion even though it was ported from avisynth code

EDIT: Actually the newer versions of vdub have various matrices - it might be possible to do that conversion straight in vdub as well now, I haven't tested it yet

Thanks for the reply. I know that x264 is capable of encoding RGB using the High444 profile, but as you said doing so is pointless in my case. Let me pose my question another way for the sake of clarity: Is piping the raw RGB24 frames directly to x264 with the settings in the OP a fundamentally wrong way of doing things? Thanks.

TIs piping the raw RGB24 frames directly to x264 with the settings in the OP a fundamentally wrong way of doing things?

Yes, because if you leave x264 to do the RGB=>YV12 conversion, it will use Rec601 for the matrix. When viewed on YT (or any media player for that matter) , the colors will be slightly shifted

The parameters --colorprim bt709 --transfer bt709 --colormatrix bt709 are VUI parameters - they are essentially metatdata. They don't actually change the YUV data. And they are ignored by most players and YT. What you need to do is to change the actual YUV data, as if it had used a Rec709 matrix for the RGB=>YUV conversion

I experimented with both the convert filter, and the input/output colorspace options in vdub 1.10.4 - it looks like they don't work properly yet. Or they might be working properly in one of the beta versions

I can confirm that x264 will convert RGB to YV12 using a rec.601 matrix, even with high def video. And the --colormatrix=bt709 command just adds metadata telling the decoder how to convert back to RGB. And regarding VirtualDub all the YUV formats use rec.601. The rec.709 options on the "other" dialog don't work at all.

But it looks promising, vdub has laid the groundwork for better colorspace manipulation. Just looks to be work-in-progress. There is also the "convert" filter which has many of the same options, but that doesn't seem to work properly right now either.

So your options for the external encoder are to use ffmpeg with the colormatrix filter

Or use the traditional vdub frameserver and .avs script do the 709 conversion in avisynth

Or use a lossless intermediate with avisynth to do the 709 + x264 (or some GUI for x264)

Or use something else like aviutl if you need to edit, which does the 709 conversion correctly, and has a adv x264 GUI export plugin which basically does audio/video/muxing similar to the external encoder, but with a GUI . It's basically a free multitrack NLE

Yes, because if you leave x264 to do the RGB=>YV12 conversion, it will use Rec601 for the matrix. When viewed on YT (or any media player for that matter) , the colors will be slightly shifted

So, if I convert the video using AviSynth as you have suggested, will x264 then automatically process my video correctly using the Rec. 709 matrix? Do I have to enable any additional options in x264? Thanks.

Yes, because if you leave x264 to do the RGB=>YV12 conversion, it will use Rec601 for the matrix. When viewed on YT (or any media player for that matter) , the colors will be slightly shifted

So, if I convert the video using AviSynth as you have suggested, will x264 then automatically process my video correctly using the Rec. 709 matrix? Do I have to enable any additional options in x264? Thanks.

Once you use ConvertToYV12(matrix="rec709") , you are feeding x264 YUV data already (it's no longer RGB). x264 doesn't have to do any additional processing, no other switches are need - the color conversion is done by avisynth. You can keep the same flags eg. --colormatrix bt709 (in fact, you should, because now it's actually correct), but remember they don't actually do anything

If it's too much of a pain to do that extra step, you can still use ffmpeg with libx264 with the vdub external encoder

Obviously, VirtualDub is either not filtering correctly or refusing to pipe anything other than RGB24 frames. Unfiltered RGB24 piped to x264 causes color distortion due to the use of Rec. 601 (possible red push?). The AviSynth filter, FFmpeg filter, and x264vfw conversion, however, all look pretty close to the original to me. What do you think? I've also attached a 7z file containing all the videos as I encoded them.

You have to remember that when you post a screenshot, that's an RGB representation of your YUV video. So the method used can affect what you see. For example vdub always uses Rec601 for the RGB conversion

That's interesting, because the screenshots look correct (as in 709 matrix was used to convert back to RGB) when I examine the videos, but in the past ffmpeg would use Rec601 for YUV=>RGB (or RGB=>YUV) for everything too, similar to vdub

Yes, the only renderer that consistently reads flags is madvr, it can do full range flags, 709/601 even other obscure flags

Note - fraps is a different story, because it's stored as YUV, but decoded as RGB by the official fraps decoder. But libav implementations decode it as full range YUV, so unless you display it with a full range RGB matrix, the levels on brights & darks get clipped, gamma shifted