I already have them all installed with both 32 and 64-bit versions, and I've narrowed it down to the Visual C++ 2010 x86 redistributable. I downloaded it and tried installing it over the top of the old one, but the installer gave me the same side-by-side configuration error as all other programs.

I could potentially uninstall the Visual C++ 2010 x64 redistributable (it gets further than x86 at least), but upon trying to uninstall the Visual C++ 2010 x86 version I get the error:

Error 1719.The Windows Installer Service could not be accessed. This can occur if you are running Windows in safe mode, or if the Windows Installer is not correctly installed. Contact your support personnel for assistance.

I've tried system restore, which didn't work, and I've been searching for a solution for over two hours now, and I'm pretty sure I just need to find some way of removing the faulty version of the C++ 2010 x86 redistributable, perhaps using a boot CD with an uninstaller.

After reading through some of the other answers on this site, I decided to download and install the Microsoft Visual C++ 2008 SP1 Redistributable Package (x86) from here. This did not however resolve the issue.

One of the answers though pointed to this Microsoft article which is largely for Vista but also has a paragraph for Other versions of Windows (the instructions are practically the same for Vista and other versions, the only difference being the expected location of the registry keys).

According to the article the location of the registry key for non-Vista versions is:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\SideBySide\Winners

Anyway, I had already checked my Windows\winsxs directory after reinstalling the C++ redistributable, and found that it had a folder with a name that closest matched the parameters in the Windows event text:
x86_microsoft.vc90.openmp_1fc8b3b9a1e18e3b_9.0.30729.1_none_118a7387f9d14a82

In this case it looked like the string 9.0.30729.1 in the folder name was the version of the file in that folder.

So I followed the instructions in the Microsoft article mentioned above, and changed my registry key to point to that version as shown in below screen shot.