One would expect it to work the same way, but it makes a
difference for some applications:

E.g. for all versions
of UltraEdit 14 (14.2). It crashes at startup for DEP
always on, at least on Microsoft Windows XP Professional
Edition x64 edition. (Update 2010-03-11: this problem has been
fixed with UltraEdit 15.2 and later.)

Update 2010-07-12: Mozilla Thunderbird 2.0.0.19 also crashes
at startup for DEP set to the highest level (as UltraEdit used to do).

Is there a difference between Windows XP, Windows Vista and
Windows 7?

6 Answers
6

OptIn This setting is the default configuration. On systems with processors that can implement hardware-enforced DEP, DEP is enabled by default for limited system binaries and programs that "opt-in." With this option, only Windows system binaries are covered by DEP by default.

OptOut DEP is enabled by default for all processes. You can manually create a list of specific programs that do not have DEP applied by using the System dialog box in Control Panel. Information technology (IT) professionals can use the Application Compatibility Toolkit to "opt-out" one or more programs from DEP protection. System compatibility fixes, or shims, for DEP do take effect.

AlwaysOn This setting provides full DEP coverage for the whole system. All processes always run with DEP applied. The exceptions list to exempt specific programs from DEP protection is not available. System compatibility fixes for DEP do not take effect. Programs that have been opted-out by using the Application Compatibility Toolkit run with DEP applied.

AlwaysOff This setting does not provide any DEP coverage for any part of the system, regardless of hardware DEP support. The processor does not run in PAE mode unless the /PAE option is present in the Boot.ini file.

For question 1: I think this difference is caused by the backdoors that Microsoft has put into hardware DEP for OptOut, according to Fabrice Roux (see below). In the case of IrfanView, for which Steve Gibson observed the same difference as I did for UltraEdit (see the question), the difference is caused by a non-DEP aware EXE packer (ASPack) that Microsoft coded a backdoor for.

Sources:

From
http://blog.fabriceroux.com/index.php/2007/02/26/hardware_dep_has_a_backdoor?blog=1,
"Hardware DEP has a backdoor" by Fabrice Roux.
2007-02-26.
"IrfanView was not using any trick to evade DEP ...
Microsoft just coded a backdoor used only in OPTOUT.
Bascially Microsoft checks the executable header for a
section matching one of the 3 strings. If one these
strings is found, DEP will be turned OFF for this
application by windows. ... 'aspack', 'pcle', 'sforce'"

From http://www.grc.com/sn/sn-078.htm,
by Steve Gibson.
"I can’t find any documentation on Microsoft’s site
anywhere, because we’re seeing a difference between
always-on and opt-out. That is, you would imagine that
always-on mode would be the same as opting out if you
weren’t having any opt-out programs. It turns out it’s
not the case. For example ... the IrfanView file viewer
... runs fine in opt-out mode, even if it has not been
opted out. But it won’t launch, Windows blocks it from
launching ... in always-on mode."

From http://www.grc.com/sn/sn-083.htm,
by Steve Gibson.
"... IrfanView ... won’t run with DEP turned on. It’s
because it uses an EXE packer, an executable compression
program called ASPack. And it makes sense that it
wouldn’t because naturally an executable compressor has
got to decompress the executable, so it allocates a
bunch of data memory into which it decompresses the
compressed executable, and then it runs it. Well, it’s
running a data allocation, which is exactly what DEP is
designed to stop. On the other hand, UPX, which is
actually the leading and most popular EXE compressor,
it’s DEP- compatible because those guys realized, hey,
when we allocate this memory, we should mark the pages
as executable."

Setting the System DEP Policy level to OptOut allows for any 32bit thread to call SetSystemDEPPolicy(0) which will disable DEP protection for the memory space allocated to that process. If the system's DEP Policy is set to AlwaysOn, invocations of SetSystemDEPPolicy will throw an error.

I knew about this document. It does not answer question 1 (except perhaps for "AlwaysOn ... System compatibility fixes for DEP do not take effect" - I am not sure how to interpret that). Regards, Peter.
–
Peter MortensenMay 29 '09 at 19:32

I have listened to all the "Security Now!" podcasts regarding DEP. It does not, I far as I know, answer question 1: why is there a difference between AlwaysOn and OptOut with no exceptions.
–
Peter MortensenMay 29 '09 at 19:37

That's unfortunate... I didn't recall it in that detail. Just one more chance: You also mean it has not been discussed in the listener q&a episodes coming after the "official" DEP episodes, right? (the transcripts help to browse quickly through them)
–
OlafMay 30 '09 at 7:17

OK, Steve Gibson actually observed the same difference as I did (episode 78 and episode 83). I am not sure if he found the reason or not. But I think Fabrice Roux did! - see beginning of this page. Regards, Peter.
–
Peter MortensenMay 30 '09 at 21:20

But then OptOut with no* exceptions (empty list of programs for which DEP does not apply) should be equivalent to AlwaysOn. This has been observed to not be the case on Windows XP 64 bit. UltraEdit 14 crashes on startup for AlwaysOn, but not for OptOut with no exceptions. I agree: the official documentation suggest the two cases to be equivalent. But they are not. The only change is to the boot.ini line and the problem is reproduciple. I would like to know the underlying reason for this behavior. Regards, Peter.
–
Peter MortensenMay 29 '09 at 20:09

That's interesting... I know that there is a DEP-aware option in Visual Studio, but what it does seems poorly documented. Perhaps if the app was compiled with this option switched on or off it may also be a contributing factor (in which case there's not too much you can really do about it).
–
Darth MelkorMay 29 '09 at 21:24

The DEP-aware option in Visual Studio merely shows that the developer claims that his code works with DEP turned on. It's explicit opt-in for that app.
–
Roger LipscombeJun 1 '09 at 15:03