The IT Manager

I have no idea what travesty I committed in a past life to deserve the pain I have recently experienced with Windows Updates1. However, I do know that my punishment is to know way more about the Windows Update module than any sane IT Manager has the right to know.

By way of atonement for my past sins, I offer the following in the hope that it helps some other poor bugger out there who might be in a similar situation.

What happened?

Our IT Estate has roughly 40 workstations. We use Shavlik to push out Windows patches. I noticed one month that Shavlik was pushing out patches but in many cases could not confirm that they had been installed properly. After a bit of digging around, we decided to have a go at patching those machines manually. That is when things went south.

The machines in question refused to push out updates using Windows Update (seriously… Microsoft… WTF!)

We Googled every single error code and tried every single internet “fix” that we could find. Nothing was working. It was time to admit defeat and do the unthinkable… the last resort… we had no choice… we had to get on the phone to Microsoft support.

What Microsoft found

On the first pass, even Microsoft gave up and recommended an OS re-install. Of course, we all laughed heartily at that suggestion (seriously… Microsoft… WTF!)

After the required amount of pushback, Microsoft relented and escalated internally.

For several more days, Microsoft worked remotely on one of our affected PCs and discovered that a number of patches had been “Staged”, but never installed. They were stuck in this “Staged” state and this was preventing all other patches from installing.

How did we get into this mess

This is our working theory. One word. McAfee.

It turns out that one of our EPO policies was actively preventing some Windows Updates from installing. If you have landed on this page because you have the same problem, you have to disable the following option:

How we got out of this mess

This is where it gets “interesting”. The solution on paper sounds simple:

Reset Windows Update:

Stop all Windows Update related services

Rename the SoftwareDistribution folder

Rename the catroot folder

Extract a list of all installed/staged packages to a text file using dsim

Restart Windows Update services

Parse the file containing the installed packages, extract any module name in the “Staged” state, and use dsim to remove those packages.

The problem is that the list of Staged packages was different on every single machine - not the sort of task you want to be doing by hand on that many machines.

The good news is that is is scriptable, but has to be done in phases.

Phase One

Run the following Powershell script on every affected machine (it can be run in a remote Powershell session without disturbing the target machine user):

Phase Two

I have a bit of background in Linux system admin, so I chose to use Cygwin bash scripts for this phase. The principle is pretty simple, so if you are not comfortable with bash, grep or sed, feel free to script in a language of your choice.

The following bash script: stagedupdateremoval.sh takes a generated package list file from phase one as a parameter, then uses grep to extract the list of “Staged” package names, and pipes that output to sed to create a Powershell script that can be run on the source machine to remove all of the Staged packages:

It seems that every time we have a high profile terrorist attack, politicians, with alarming alacrity, seize on the opportunity to demand extra snooping powers. This used to be New Labour’s domain, but now it’s the turn of David Cameron. Our esteemed PM wants to allow our security services to view the content of encrypted messaging services:

Mr Cameron told ITV News: “I think we cannot allow modern forms of communication to be exempt from the ability, in extremis, with a warrant signed by the Home Secretary, to be exempt from being listened to.”

(That is very nearly a coherent sentence)

On the same day, The Independent spun similar comments made by Cameron into the following article:

David Cameron could block WhatsApp and Snapchat if he wins the next election, as part of his plans for new surveillance powers announced in the wake of the shootings in Paris.

The Prime Minister said today that he would stop the use of methods of communication that cannot be read by the security services even if they have a warrant. But that could include popular chat and social apps that encrypt their data, such as WhatsApp.

Apple’s iMessage and FaceTime also encrypt their data, and could fall under the ban along with other encrypted chat apps like Telegram.

“In extremis, it has been possible to read someone’s letter, to listen to someone’s call, to mobile communications,” Cameron said. “The question remains: are we going to allow a means of communications where it simply is not possible to do that? My answer to that question is: no, we must not.”

David, you are not the Prime Minister of North Korea. Policies like this will only lose you votes.

There are dozens of encrypted chat clients out there. Do you really want the security service spending all of their time play Whack-a-Mole with the online development community? The government and security services will be chasing their own tails for eternity trying to solve this problem. It is stupid idea, unenforcible, futile, and counter-productive.

Any half baked measure to introduce back door access or lower encryption standards will be an open invitation to hackers. It might give the government what they want, but we will all be worse off for it.

However, Dave can take heart that he’s not in this fight alone, the other countries where there are known domestic controls on the use of encryption are Russia, China, Mongolia, Vietnam, Pakistan, Iran, Kazakhstan, Belarus, Ukraine, Moldova, Israel, Tunisia and Morocco.