What you need to do when stacking parts is to use the combine option.
This will make the software think that you are looking at 1 stl file. When
the supports
are created on the combined parts instead of the supports building through
the parts
you will now have sierras on both the top and bottom of the supports.