Since I do not have Windows 7 - I need to have others test this and report findings.

pSXAuthor has some insight about this:

<pSXAuthor> best way to find out whats wrong is to switch to the debug runtime, enable full debug output, then run in debugger (or use watson, whatever)
<pSXAuthor> d3d will tell you whats wrong
<pSXAuthor> 8876086C is D3DERR_INVALIDCALL
<pSXAuthor> which usually means some problem with parameters or state

(05040)
Haze (Developer)2009-10-20 11:33

I can confirm this, it works when set to 100% / 125%, but as soon as you set it to 150% it bails out.

On a large display running without 150% is often not an option.
This workaround worked for me:
open regedit
HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers
Create a new String.
The name of the string should be the full path to mame.exe
for example: H:\Mame\mameuihiscore64.exe
The value of the string is HIGHDPIAWARE

This set the app compatibility flag for this program, for some reason it is greyed out in the GUI but this seemed to work for me.
Now I can run mame without the errors and without changing the global dpi setting.