The IT Manager

I have no idea what travesty I committed in a past life to deserve the pain I have recently experienced with Windows Updates1. However, I do know that my punishment is to know way more about the Windows Update module than any sane IT Manager has the right to know.

By way of atonement for my past sins, I offer the following in the hope that it helps some other poor bugger out there who might be in a similar situation.

What happened?

Our IT Estate has roughly 40 workstations. We use Shavlik to push out Windows patches. I noticed one month that Shavlik was pushing out patches but in many cases could not confirm that they had been installed properly. After a bit of digging around, we decided to have a go at patching those machines manually. That is when things went south.

The machines in question refused to push out updates using Windows Update (seriously… Microsoft… WTF!)

We Googled every single error code and tried every single internet “fix” that we could find. Nothing was working. It was time to admit defeat and do the unthinkable… the last resort… we had no choice… we had to get on the phone to Microsoft support.

What Microsoft found

On the first pass, even Microsoft gave up and recommended an OS re-install. Of course, we all laughed heartily at that suggestion (seriously… Microsoft… WTF!)

After the required amount of pushback, Microsoft relented and escalated internally.

For several more days, Microsoft worked remotely on one of our affected PCs and discovered that a number of patches had been “Staged”, but never installed. They were stuck in this “Staged” state and this was preventing all other patches from installing.

How did we get into this mess

This is our working theory. One word. McAfee.

It turns out that one of our EPO policies was actively preventing some Windows Updates from installing. If you have landed on this page because you have the same problem, you have to disable the following option:

How we got out of this mess

This is where it gets “interesting”. The solution on paper sounds simple:

Reset Windows Update:

Stop all Windows Update related services

Rename the SoftwareDistribution folder

Rename the catroot folder

Extract a list of all installed/staged packages to a text file using dsim

Restart Windows Update services

Parse the file containing the installed packages, extract any module name in the “Staged” state, and use dsim to remove those packages.

The problem is that the list of Staged packages was different on every single machine - not the sort of task you want to be doing by hand on that many machines.

The good news is that is is scriptable, but has to be done in phases.

Phase One

Run the following Powershell script on every affected machine (it can be run in a remote Powershell session without disturbing the target machine user):

Phase Two

I have a bit of background in Linux system admin, so I chose to use Cygwin bash scripts for this phase. The principle is pretty simple, so if you are not comfortable with bash, grep or sed, feel free to script in a language of your choice.

The following bash script: stagedupdateremoval.sh takes a generated package list file from phase one as a parameter, then uses grep to extract the list of “Staged” package names, and pipes that output to sed to create a Powershell script that can be run on the source machine to remove all of the Staged packages: