(Note: This is a guide, not a tutorial; it assumes that you are familiar with the basics covered by the other tutorials on this site. In the future, when I get a chance, I may put up some downloadable examples. In the meantime, some of the existing addons on this forum can serve as examples.)

There are basically three main ways to do an app replacement: (to clarify, by "app replacement", I mean replacing something a way such that you can invoke the new app the same as before, without having to edit shortcuts, file associations, etc.)

Method 1: Directly replace the Windows executable (unclean)
If your app is standalone (e.g., Notepad2), you can just replace the appropriate file in I386 with your app.

If your app is not standalone, then you will need to create a launcher for it, install your app to %ProgramFiles% (or whereever), and replace the Windows executable with the launcher (which will then subsequently call your app)

Pro: This is the simple and obvious way.
Con: Replacing system files will usually result in digital signature warnings from Windows File Protection. This is thus an "unclean" method.

n.b.: If the new app has a digital signature, then you can do a direct replacement without getting signature errors. For example, it's possible to cleanly replace Calculator with Calculator Plus, as long as you do not modify the executable and thus invalidate the signature. In the specific case of Calculator Plus, it's probably best to solve the help file problem by patching accessor.inf (sample addon) instead of hacking the executable and thus invalidating the signature.

Method 2: Use an image "hijack" (clean)
Examples of apps that use this method include Microsoft Sysinternals Process Explorer and my Notepad2 replacement installer. This method uses the "Image File Execution Options" registry key to cause Windows to launch your app whenever someone tries to run the "hijacked" app.

For example, my Notepad2 2.x replacement installer uses the following key (a slightly different key is used for my NP2 3.x installer and newer, but the concept is the same):

The downside to this method is that it passes to your app the entire command line of the hijacked Windows app. For example, if I ran "notepad.exe foobar.txt", then Windows will launch "Notepad2.exe notepad.exe foobar.txt". Notice that the first parameter of the command line is no longer "foobar.txt", but is instead the name of the app that you hijacked, "notepad.exe". If your app doesn't care about what gets passed on the command line or if your app ignores that extra parameter, then this isn't a problem, but in the case of my Notepad2 replacement, this would result in Notepad2 trying to open "notepad.exe" instead of "foobar.txt". If this extra parameter poses a problem, then one way to work around it is by using a launcher that discards %1 (i.e., instead of passing %1 %2 %3 etc., the launcher will pass %2 %3 %4 etc.). Or if you are a programmer and have access to the source code of the app, you can tweak the app's command line parsing to have it discard that first parameter when appropriate (this is what I did with Notepad2).

Pro: Clean replacement that doesn't create digital signature warnings or upset WFP.
Con: Need for a launcher if that extra first parameter on the command line creates problems.

Method 3: Take advantage of path search precedence (clean)
When searching the PATH for an executable, Windows first searches the current directory and then the PATH variable, from left to right. This allows you to "replace" certain Windows apps by placing them in a location and adding that location to the start of the PATH environment variable, thus ensuring that Windows finds your app first, before it finds the old app.

For example, my method of replacing MAKECAB and EXPAND is by placing them in "%ProgramFiles%\Utilities" and then adding "%ProgramFiles%\Utilities" to the beginning of the PATH variable.

To add a path to the beginning of the PATH variable, you can either manually edit the registry (which carries the risk of clobbering any existing changes made to the PATH variable), use setx from Microsoft's resource toolkit (which carries the downside of converting the REG_EXPAND_SZ string into a REG_SZ string), or use my addpath tool (this is the tool that I use to set the environment variable for my own MAKECAB/EXPAND replacement addon)
Example: addpath pre "%ProgramFiles%\Utilities"

Pro: Clean replacement that doesn't create digital signature warnings or upset WFP.
Con: Does not work if the path to an app is explicitly specified (e.g., if someone explicitly runs "%SystemRoot%\system32\makecab.exe" instead of just "makecab.exe" or if the app is run from the directory of the old app (e.g., if you run makecab.exe from %SystemRoot%\system32). As a result of these limitations, this method makes sense mostly for command-line apps.

You never fail to impress. While we differ in philosophy - I prefer your method 1 "unclean" decription - the information is very well laid out and as usual, it contains a bunch of little things I never realized or had considered.

Personally, I never use method #1 (direct replacement). I always use method #3 (path precedence) for command-line apps (e.g., MAKECAB and EXPAND) or method #2 (image hijack) for all other apps (e.g., Notepad2). I avoid method #1 because I'm a perfectionist who likes to have zero WFP errors on my systems.

Why even bother leaving WFP on then? If you're truely a perfectionist, you wouldn't even have to use it..

And see, that would not necessarily be true. Code65536 himself might be a perfectionist, however a good majority of programs written for the Windows OS are not. WFP in addition to preventing normal human mistakes also helps to prevent programs that try to replace Windows programs (aka some virus, trojans, utilities that think they can do better). While it does not prevent the "well written" bad programs, it can stop the most casual program replacement._________________A mind is like a parachute, it only functions when it is open.
--Anonymous