Layer 1 - Regular bitmapped (Raster type) image (it will be even better if it could be a vector masked layer, but it's ok if it can't be).

Layer 2 - A vector masked layer (with fill 0%) with a stroke or outer glow, whatever. The effect of this layer must fade out (make transparent) the pixels of Layer 2, BUT WITHOUT THE NEED OF "RASTER TYPE" IT.

Idea example:

Then, if I change the color of background...

What I'm really trying to do is a PNG Icon, that I made with Shapes (vector masked layers) of a envelope, where the lines are made with strokes. So, the stroke may turn the layer below it transparent. I don't wanna use the bitmapped mask because I lose the vector benefit (I can't change the size without doing the whole process again) and I lose quality (because the final result is not the same. The transparency created by the bitmapped mask is not perfectly equivalent of the image used).

Maybe this give some idea:

As you can see, the icon is really tiny, and I thought pixel by pixel, then there is why I care so much about the perfect equivalence of transparency (in this icon I used white stroke to make the envelope lines. I wanna the white become transparent when I export as .png, but as I said, don't wanna use a bitmapped mask (create a layer mask, raster type my image, and then use it as a mask of the image itself) because I care too much about quality).

Sure it does! If it's not, will not got two up votes...
– VitoxJul 26 '14 at 7:17

Please @Komental, pay attention on my comment in the last picture (I just edited it). I'll be glad if you could help me...
– VitoxJul 26 '14 at 7:27

From your screenshots, you are using an older version of Photoshop. I don't think you can add Layer Effects to groups in your version unfortunately. The best answers below rely on this.
– daviestarJul 29 '14 at 23:17

@daviestar - Yes, I'm using Cs5.5, but it just makes HARDER because I can't apply effects directly, so I MUST to create a Smart Object to make the accepted answer works. I thought so much in choose the really best answer, that I thought what WOULD be the best way for whom has CS6+. Even I using a lot of Smart Objects to reach same result, I thought: "If I had CS6, this one would be a lot easier than any other...". The accepted best answer is not the easier for me, but the best for everybody else here (that have a up-to-date software).
– VitoxJul 30 '14 at 2:21

I really like this approach. If nothing else, it would mean you can use the technique for many icons and use the group to colour them all.
– Marc EdwardsJul 29 '14 at 1:10

I loved it man! This does exactly what I want to! Unfortunately I'm using Cs5.5 and cannot apply effects on groups, but instead groups I used Smart Objects. First I grouped all my shapes with effects (strokes) in one SmartObject, applied your trick in this object, and pixels get their opacity perfectly equivalent, BUT the color on the transitions (between the white stroke and black shape) get GRAY (but with correct opacity) and the thing look weird. Then I created a Empty layer, grouped them both together in a new SmartObject and applied a color overlay. BAM! IT WORKED LIKE A CHARM!
– VitoxJul 29 '14 at 21:50

For anyone that can apply styles in Groups (CS6 or higher), this approach is damn simple and clean! Just perfect! Thank you!
– VitoxJul 29 '14 at 21:51

Oh yeah, I only really started using this when layer styles on groups became possible. I've always found the double smart object way a bit too convoluted, but I'm glad it works for you!
– Philipp AntoniJul 30 '14 at 19:19

Being Photoshop, there’s probably quite a few approaches to this problem, but only one I can think of that maintains full vector edited and scaling.

There’s a few things going on here.

Square is just a shape layer for the square. Nothing tricky there.

The Blurry Circle group has a circle as a vector mask. The vector mask is set to subtract and also has mask feathering of 6px.

Circle is just a shape layer for the filled portion of the circle.

The Icon group is just there to neaten things up.

A color overlay layer style can be applied to the Icon group to color the entire icon.

Changes to the background colour will be shown through the icon.

A word of warning: This entire document is constructed using vector shapes (yay!). It also uses mask feathering (yay!). Mask feathering doesn’t scale when the document (boo!). However, there are scripts that can scale mask feathering (yay!).

I really liked your approach, but Mask Feathering is a tricky one. It's a little limited, and I still have to worry about duplicate the shapes I wanna show "inside" the transparent part. I'm seeking the easiness of using effects, and that your approach doesn't give to me. But it still reach great results in other situations, and because that I upvoted it! Thank you
– VitoxJul 29 '14 at 20:48

First, I will answer the question about the envelope, as I think this is what you really want to know.

Your logic is a little bit off here.

If you want to retain transparency, ie. not use white lines for the envelope, you should instead create a rectangle, then on the same layer change the shape tool to 'line', then change the Path Operations to Subtract, and subtract the 2 lines from your rectangle.

You can add multiple shapes to the same layer by holding down Shift as you click and drag to draw your shape - adding to the shape, but in your case as you want to subtract from the shape, so you hold down Alt.

EDIT - another technique to achieve the same thing:

I think you are using an old version of Photoshop, so do this: Group the black rectangle of your envelope and add the 2 (or 4) lines to a new layer. Then drag this vector mask onto the group. This will achieve the opposite of what you want, so in the group mask select all the lines, and change Path Operations to Subtract. You could actually separate all the shapes into new layers this way - by putting each shape onto the mask of a nested group.

Now for the vector 'glow' transparency.

When your circle shape layer is selected, open the Properties palette and try messing with Feather and Density. This is how you can blur the edges of a shape, however I'm not sure if it's possible to keep a hard line on the inside of the circle and blur the outside like a shadow/glow.

I understood what you mean. Your way is the "purest" way, but not the easier nor simpler. Actually, I used to create my shapes with this thing of Path subtracting Path, but this is very limited, really hard-working and if you have a bunch of paths, the thing become a mess.
– VitoxJul 29 '14 at 21:58

The reason I say your logic is off, is because the only way to remove 2 lines from a rectangle and retain transparency is to subtract those lines from the rectangle. I personally find it the easiest way, but in case you want another technique for the same thing, I have edited my answer :)
– daviestarJul 29 '14 at 23:11

Thanks for your effort, but your edited answer is the same approach that Marc Edwards already told, and I already commented why I don't selected it as the best answer in its comments section. I think You said my logic is off because you don't understood why I'm really trying to do. My idea is not simple remove lines, but create a more easy way to remove lines with smooth and even creating opacity gradients. Such thing subtracting paths don't do.
– VitoxJul 30 '14 at 1:50