I believe it's just because you are trying to stretch the 8bit information too thin and results in some funky artifacts. Sometimes I've gotten lucky by tweaking the matte a million different ways, but mostly I avoid using that qualifier with 8bit footage.

Yea, it's lack of data. You're selecting a color to qualify, but there isn't enough differentiation between it and another color (luminance) within the scene. It may not be a bit depth issue, it maybe only coincidence. I see this all the time when two colors are so closely matched.

To fix this, I generally start by softening the qualifier and adjusting the white/blacks of it. If that doesn't work, you need to make a power window matte around the door frame that travels with it and then isolate the color within that matte, vs the rest of the shot.