2 Answers
2

You can always combine multiple layers and apply the same blending mode, but for many blending modes it won't look the same as having separate layers. That's just not mathematically possible for many of these effects, including overlay.

As you can see from the above link, overlay is not a commutative function. That is, if you change the order of the operands, the end result changes. So unlike multiply, screen, darken, etc., it does matter what order the filtered layers are placed in. And operations like these can't be compressed into a single operation on a single layer.

So you can group a bunch of multiply layers into a smart object and then set that smart object to multiply, and the results will be the same. But if you try to do that with overlay, hard light, etc., it won't work in the general case (there may be specific cases, depending on the layer contents and the order, where it could work, but these are corner cases).

Besides that link, a simple test you can do is just to group all consecutive layers with the same blend modes together. If applying the same blend mode to the group changes the way the image looks, then you can't group them together in a smart object.