The element in itself should not compromise the lens compared to one with a fixed (non-IS) identical element -- but it might compromise the lens DESIGN. IS needs a small (light!) negative element at the right point to work, and this means that the design has to be worked out with that in mind. This leads to different design considerations, and might produce a less good design -- or it might not. But it is an inherent drawback with any element-driven stabilization.