Solution

How can I opt out of the Program Compatibility Assistant for my setup?

In order to prevent the Program Compatibility Assistant from appearing, you must include an embedded manifest that specifies a requested execution level for your setup executable. If you wrap the setup executable in a self-extracting package, you must also include an embedded manifest in the self-extracting package too. Once you do this, Windows Vista will treat your setup as Windows Vista-aware, and it will no longer show the PCA dialog when setup exits after a failure or cancellation.