1280x1024 isn't really a 'small' size by the way; it's more than 1280x720 which is a pretty common laptop monitor size. It seems there's rather issues with especially 'tall' scale factors, e.g. 5:4 screens.
Likely, if you use a less tall aspect ratio (e.g. 4:3 or 16:9), the DoTA UI will display better. Yes, you'll have some black bars or stretching.

The DoTA UI should use the screen's 'width' to scale elements, not its height. Though that may also create issues with 'wide' screens in the vertical aspect. I suppose one size fits all is just not really feasible and you would want multiple presets for multiple aspect ratios. Most commonly:
21:9, 16:10, 16:9, 5:4, 4:3.