Hey there MM,
I have a question thats been bugging me for quite a while.

When I make a solid selection (by solid I mean there is no feathering at all) of a solid object and invert that selection and make a mask it leaves a sort of fringe where the 'marching ants' were. Even when the marching ants arent there but I invert the mask its still the same.

Is this happening only to me?
Apparently its caused by the anti aliasing but still...

There are a few things which are likely contributing to this, and AA is certainly one of them. It may be easiest to diagnose if you can describe how you created the image above? (Step
by step so we can directly replicate).

- new document, white, 300 dpi (16 bit I think)
- solid color adj layer with bright green (no mask yet)
- circular selection AA checked
- on a new layer filled the selection with dark blue
- kept the selection and made a mask for solid color adj layer
- inverted the mask ( I can already see the whites from the bg)
- merged visible on top. set to difference
- disabled the mask for the solid color adj and the halo/fringe became visible as seen.

(I tried inverting the selection and then making the mask but still the same)

After you make your selecting, invert the selection (cnt+shit+I), then select>modify>contract, then choose 1px. This should correct the innate contracting from inverting the selection and get rid of the "shaved off" soft edge.

Found CS2 on my old laptop and its still the same...which means I'm already used to this

Normally I make paths a few pixels inside the edges so I can feather 1 px and compensate for the invert plus the contract mentioned above or use the masking panel etc. but lately I've been a little lazy and all

I just wanted to make sure its not only me and/or there is a clickable little box somewhere that can fix this

I've always come across this and usually end up doing the expand selection as a fix. I was curious about the opacity and it looks like it caps out at 25%. Maybe it is an issue with rounding and splitting the difference?

Above, the white mask is actually the combined layers beneath with a mask based on transparency.

PANZERWOLF wrote: i don't see what this setup has to do with how a mask works, or with the initial post
since you have the black layer in each group, you're just adding color values, not transparencies!

if you mask a shape with its own inverted opacity (or add a shape with the inverted mask) you end up with a fringe, ranging from opacity 0% to a maximum of exactly 25% (or 100% to 75%), regardless of your gamma curves

that's because the effect i described above is strongest where the edge is 50%:
1/x=1/0.5+1/0.5 -->
x=1/4 = 0,25 = 25%

Well, yes. Masking an already masked thing is a recipe for matte lines. I wrote about it extensively in those decade-old links above, but it's been known since at least the 80s, and much earlier than that in film opticals.

Although I used that amplified 25% 'fringe' effect extensively in the 1993 film 'Demolition Man' to capture, amplify, and color the fringe at the border of the frozen/unfrozen mask in the scenes where we wipe from an unfrozen to a frozen set.

PANZERWOLF wrote: i don't see what this setup has to do with how a mask works, or with the initial post
since you have the black layer in each group, you're just adding color values, not transparencies!

if you mask a shape with its own inverted opacity (or add a shape with the inverted mask) you end up with a fringe, ranging from opacity 0% to a maximum of exactly 25% (or 100% to 75%), regardless of your gamma curves

that's because the effect i described above is strongest where the edge is 50%:
1/x=1/0.5+1/0.5 -->
x=1/4 = 0,25 = 25%

And there are no 'transparencies.' There are only color values and and extra alpha channel, and what gets done with them generally falls into two categories.

Alpha as a clipping mask (as Photoshop does), and

Alpha as holdback matte, the way CG artists rendering on black generally have worked over the years.

(and sometimes a conversion from the latter to the former using 'unmult' or a divide blend node or what have you to divide all the color channels by the alpha and turn it into an overshot shape that the Photoshop or After Effects mask can clip back again)

and I think to think of this as 'adding transparencies' adds confusion, but what do I know. Although I have been diagnosing people's matte lines for nearly 30 years now -- so long a time that I still call them "matte lines"