The main problem with real-time capture is precisely the static GOP structure. If a scene change happens to "land" on a non-I-picture, you'll almost inevitably get some blocky frames.

Some real-time encoders do have an option to detect scene changes (or, to be more precise, to detect big changes between consecutive frames), and will insert an I-picture when they "need" to. If your encoder supports this, I would suggest using a long IP GOP (ex., I=1, P=14), with the highest bitrate possible. This is assuming the encoder has good motion search.

If your encoder doesn't have the option to detect scene changes, then you need I-pictures to be as close together as possible.

And you really can't get much closer than an I-picture-only stream. At 25 Mb/s, I-picture MPEG-2 is more or less equivalent to DV, and it's not very sensitive to encoder quality (i.e., even a bad encoder will produce decent results), since I-pictures don't need any motion search, which is the "tricky" part of MPEG encoding. In fact, if your encoder does support 25 Mb/s (or more), and if you're working in SD (standard definition), don't bother with P-pictures at all, just go straight for I-only.

Even at 20 Mb/s, you can probably get very good (near-DV) quality from I-picture-only MPEG-2 (and you're completely skipping the encoder's motion search).

Alternatively, try I=1, P=1. Don't use B-pictures; in most encoders, B-pictures are given a very low "bit budget" (usually not enough to preserve all the detail, unless the image is perfectly noise-free), and, although the bits saved there are used to improve the I-pictures, you end up with a slight "shimmering", as you go from an I-picture to a B, back to an I, etc..

I assume the last one - Motion Estimation Quality - is what you refer to as scene change detection. So if I interpret your advice correctly, I should try a GOP structure of I=1 P=14 B=0 and set the Mostion Estimation Quality to 100. Does that make sense?

Also, could you shed some light on what "Target Bit Rate" might mean? It is only available with VBR. (With CBR, all you can set is Maximum Bit Rate; Target Bit Rate is grayed out.) If I want maximum image quality and don't care too much about file size, should I set both Max and Target to 20 Mb/sec. And would that be the same as CBR of 20 Mb/sec.

No, motion estimation is the motion search algorithm. It's the part that tries to determine how each "block" moved between frames. If there's no option labelled "scene detection" or "automatic I-picture insertion", or something like that, the compressor probably creates consistent GOPs (i.e., always with the same structure).

Target bitrate is the intended average. In CBR, the average is the same as the maximum, so you only get one slider. If you're going to set both to the same value, use CBR (no sense in going through the added VBR calculations when the end result is going to be CBR anyway).

VBR only really makes sense when you're trying to save space and your encoder can do multiple passes. Single-pass VBR involves a lot of "guessing" and will only produce good results if the footage is of similar complexity throughout the whole movie.

Theoretically, I=1, P=14 could produce, for the same bitrate, better quality than I-picture-only. But there's the scene change issue. Without proper I-picture insertion, you can get seriously blocky frames if there's a scene change at a P-picture.

And also, I have no idea how good ATI's motion search is. If it' not very good, the quality of P-pictures might not be as great as it should.

Do some tests to determine if ATI's motion search does a good job (and pay attention to the frames immediately after a scene change; be sure to check five or six cuts, just to make sure they didn't land on an I-picture by chance).

I suspect you'll get better quality by using only I-pictures, and setting the bitrate to 20 Mb/s (CBR).

In this situation, the motion estimation quality is irrelevant (I-pictures don't need motion estimation, they don't use temporal compression), and you never need to worry about scene changes.

I think Encore uses the MainConcept encoder, which is pretty good, and fast, but not quite up to the quality of TMPGEnc or CinemaCraft SP.

Most people I know using Macs have FCP but use a 3rd party encoder, so I'm guessing the default one isn't very good (I've never tried it myself, though).

I don't know about the other two.

Most encoders that come built into other applications seem to be oversimplified and optimised for speed, so if you're after the best quality, it's probably still a good idea to use a dedicated encoder, that will give you control over all the settings.

Of course, with so many people shooting on MicroMV and single-CCD MiniDV cameras, a slightly worse (but faster and simpler) encoder is not necessarily a bad thing.