My most recent AMV uses Millennium Actress, which I removed a ton of grain from before I started editing. The grain was especially terrible in dark areas.

In Huffyuv (and uncompressed) it looks quite good. When I run it through Zarx264 to make an MP4 it becomes problematic.

It looks much worse in motion as the banded color boundaries move. (At lower bit rates the bands also look very blocky.) This scene is definitely the worst, but there are other problematic scenes.

This image comes from a version using the following settings:Mode: Bitrate (2 Pass)Video Bitrate: 5000 KbpsPreset: PlaceboTune: Animation

For a 4:30 long video (768, 432) the file size is 168 MB, which seems rather excessive especially considering that the quality in these scenes is merely acceptable. The majority of the video looks fine in much smaller file sizes (using the default video settings).

I tried turning the bitrate up to 6000 with a previous copy of the video, but it didn't gain much quality for its 30 MB jump in size.

I don't know if I should play with the Tune, or if there is anything I could do using the Advanced settings or Command Line.... Or if there's anything I could do in Avisynth (though obviously I'd be working blind since the problem comes when the video is compressed).

I have a hard time understanding that screenshot. The artifacts I can see are all very likely introduced by the jpg compression of the screenshot.

Behind that, I can see things that look like possible overfiltering? Please do post the whole filterchain you used, and also a lossless screenshot of the original source (straight from the vob) in the scene.I'll reiterate, but grain is good. Noise is bad. If it's noise, you filter it out. If it's grain, you check how much there is, and if it's too much, you calm it a bit, but don't outright remove it. Sometimes a lack of grain is actually bad and you should be adding it (static grain, of course). This actually helps preventing banding, and with appropriate settings will result in a better quality with smaller filesize, since you help the quantizer out by tricking it. Of course, the process of adding grain is rendered pretty much useless, if not for aesthetics, by higher bitdepths, since they have more precision with quantizer and thus don't need the grain to understand gradients and so on.

Also, 2pass encoding is bad, do use crf encoding. You should be looking at crf 17 (should be low enough), preset very slow, tune animation, and then go to the advanced tab and increase the aq-strength value to 0.8 from the 0.6.

I changed the AMV after my first attempts at compressing it-- so I deleted those files-- but this was the first video I used the two-pass encoding for and it did look better than the constant quality. I'm not sure if I went as low as 13 or 15 when I tried CQ before, but it it didn't look good in these scenes. (I'll try it again with the VAQ change.)

Warlike Swans wrote:If the difference between noise and grain is whether it changes between frames then the original problem was definitely noise.

No, that is not the difference. Grain can be in motion, but generally when digitally added, it is static (you'll notice, for example, that during pans you'll see the image move behind the grain). The difference is in size and displacement. Grain tends to be quite small and evenly distributed among the whole picture, whereas noise has varying size and can be uneven and concentrated in some parts more than in others.In your case, I think you have noise, but again, the jpg compression of the image does not help. Since jpg is lossy, I do not know if some alterations are due to it or not.As for filters, as I thought that looks incredibly overfiltered. I would personally just use ttempsmooth I think. It's probably the better choice in this case.

That is not bleeding, that is macroblocking. I am acting under the assumption that the jpg compression is causing it and is not present in the original, because quite frankly I doubt the DVD is that starved.

Ok so time for my new year Grain is Not a Defect talk, but before we get there, stop using JPEG screenshots as they mask any actual problems. You've run into a few issues here and they stem from two things. The first is your oversmoothing and producing subtle gradients. The second is DCT quantization accuracy at a given bit depth. This post is going to be a bit wordy but please read all of it as I'll explain what your problem was and what caused it, as well as how to fix and avoid this in future. It's something a lot of amv encoders seem to mess up.

Let's start with the first thing you complain about, detail going to shit in x264 but being fine in Huffy. You seem to be blaming H.264 itself for some reason but that doesn't really make sense. Huffy is a lossless codec, it encodes everything exactly as it was input, providing 100% accuracy to your original content when decoding. H.264 (AVC) is lossy and only estimates the original content. Depending on the encoder, various tricks or algorithms are used to make this as visually accurate as possible as well as to provide better compression, x264's RDO is a good example of this. The way AVC works is to break your picture into large blocks called macroblocks. These are then broken down into their component chunks of data and quantized, essentially creating a new set of data that closely resembles the original, but in less detail. Think of it as averaging curves and colour variations in a way. Lossy codecs however are not very good at compressing detailed content without either being relatively large, or by dropping more detail than you'd prefer. Depending on the number of bits of accuracy the codec has, you will get more or less detail in your content for a given bitrate. This is where 8bit or 10bit image accuracy comes in. 8bit gives you 256 values per channel per pixel while 10bit gives you 1024; 4 times as much for little change in filesize. Grain itself is complex but can be handled relatively well by various patches depending on the type of grain so I don't think it's much of a problem to leave it in. Things like static grain maps and p/b frames and RDO make a huge difference here.

You've sort of gone the opposite and smoothed everything. This might have seemed like a good idea except for two things. Firstly, grain while sometimes excessive, also tricks the eye into thinking there is more detail than there really is. It also hides large imperfections. Secondly, large chunks of grain are far easier to compress than very subtle changes in gradient which is what happens when you smooth things badly. Now, depending on the amount of grain, you may indeed want to reduce it, but the way you've done it here at least is not the right way. Besides for massively distorting your image (seriously, those lanters in your above screenshot, what the hell), you've also gone and created micro gradients that you probably can't even see that well. x264 however notices them, and in 8bit quantizing, it tries to average them too much, which is where the banding is coming from. You can use 10bit here for greater accuracy and most likely eradicate this banding altogether, or you can use light grain which is what I would recommend.

I've said it a lot before but grain is not a defect. It has many uses and should generally be left alone. That said, some companies abuse the fuck out of it. You've obviously gone full bore in removing it as much as possible with pretty much the worst filters possible. You've also then gone and given bad filters much smaller sample sizes to work with, making the effect even more noticeable. What you SHOULD be doing is filtering at full resolution and then dropping down to your export res with spline36resize. I would recommend either ttempsmooth if you're in a rush and/or your grain isn't REALLY that bad (post an unedited screenshot for advice here) or otherwise hit it up with a well tuned MCTD followed by a slight AA sharpen of some sort, probably EE() would be a good choice here. You need to be really careful to only remove grain and not detail, which is what these filters are good at. Otherwise you'll introduce gradients which as I mentioned above will cause x264's quantizer to shit all over it. Depending on how much grain you have, I would also recommend GrainFactory or something after resizing so as to trick the quantizer a bit more, but it really depends on individual levels of grain here as well as the type.

Summary: you're overfiltering and this is requiring MORE bitrate than just leaving it alone. Black gradients are hard to see on many monitors so it looks to you like there is no detail to die on. Leave some grain in or remove it properly and then add it later. Fix your filter order.