x265 wrote:Which of the x264 settings should i change in order to prevent banding?

You cannot prevent banding, it's a side effect of 8-bit YUV encoding. You can reduce it from happening quite so bad through tweaking your footage through filters and adjusting the colors, but there's no way to prevent it via encoder settings.

If you provided more information we might be able to help you tweak your settings and maybe suggesting some filters for your footage to reduce the banding, but we do need more information to be able to do that. You existing settings and screenshots of before & after for the current problem would be very helpful.

First off let me say that Death Note is a tough source considering the color scheme and will be tough to keep the banding down. Increasing your aq-strength closer to 1 would help maintain detail in low detail areas and prevent more banding from occuring, but it's also going to require more bitrate if you want to maintain the same quality and avoid noise around the edges. I'd probably recommend raising psy-rd a little as well.

It's really hard to give you exact numbers, the best thing to do is take some of the scenes that were banding the worst mixed with ones that aren't showing banding so bad, and do some short encode tests. I would say try with it set at 1 for both and tweak based on some sample tests. That should provide a good balanced middle ground to work from. Raise aq for more banding reduction but reduce psy-rd to help with the noise.

RDO has little to nothing to do with what you're describing. The settings to look at in x264 are aq-strength and qcomp. Also note that dithering in avs, or any other type of noise introduction, will reduce banding in small amounts but make it WORSE if you go too far.

Mister Hatt wrote:RDO has little to nothing to do with what you're describing. The settings to look at in x264 are aq-strength and qcomp. Also note that dithering in avs, or any other type of noise introduction, will reduce banding in small amounts but make it WORSE if you go too far.

The reason I suggested RDO was to help keep the overall picture complexity and give it more of a overall appearance of better quality. I feel like his qcomp setting is fine, it's rather balanced and I don't really see the need to tweak that. Although let's be honest, just tweaking aq-strength and qcomp is not a fix all for issues like banding, going over ALL of his setting and tweaking things for this particular source would be more ideal considering the complexity of the source material. Of the few tweaks he described above, there are more tweaks that could be done to improve the overall quality of the encode.

Additionally, if you feel like changing his qcomp setting could be helpful, it would be nice if you actually gave him a recommendation instead of just saying "qcomp is what you are looking for" as he may not know which way to tweak that setting to help. It's generally a good idea when helping to at least give some kind of recommendation for him to work with

x265 wrote:Should i raise psy-rd to 0.9?

I'd recommend --psy-rdo 1.0:0.0 for initial testing on clips and tweak it down from there.

Reducing qcomp (0.7, maybe 0.6) is a good option for dither compression issues, which is what causes the banding. RDO relies on there being detail in the image to begin with, not fine detail like dither or film grain but larger more visible grains and patterns. It is for complex pattern optimisation. I think qcomp 0.7 is fine in this case but AQ is far too low, in the event of banding you want to go up, not down. Something like aq-strength 1.1 to start with, 1.2 if it doesn't help enough. The other thing to do is reduce the strength of GradFun3 in avs as it's likely making it far worse.

Mister Hatt wrote:Reducing qcomp (0.7, maybe 0.6) is a good option for dither compression issues, which is what causes the banding.

While true for particular scenes, knowing the source and judging from the fact he's encoding the entire episode, I think it would hurt the overall quality dropping it as low as 0.6...

Mister Hatt wrote:RDO relies on there being detail in the image to begin with, not fine detail like dither or film grain but larger more visible grains and patterns. It is for complex pattern optimisation.

RDO relies on the image "complexity" not necessarily image detail (although detail can affect the complexity). Animation can sometimes bit a bit tricky to encoding software when it comes to this. To the human eye a scene may LOOK rather plain because of a static background, but with some sharply outlined character moving around in front of it, sometimes it makes the image seem a bit more complex for the encoder.

I remembered researching about this stuff when I was learning about x264 settings and how to properly use them way back in the day...referring to my old bookmarks I found this concerning RDO...but hey, maybe one of the developers could be wrong

Mister Hatt wrote:I think qcomp 0.7 is fine in this case but AQ is far too low, in the event of banding you want to go up, not down. Something like aq-strength 1.1 to start with, 1.2 if it doesn't help enough. The other thing to do is reduce the strength of GradFun3 in avs as it's likely making it far worse.

Ah, actual advice now, and good solid advice, you should have started off with that