A world of hurt after McAfee mistakenly revokes key for signing Mac apps

Just allow untrusted certificates, one customer told.

A McAfee administrator accidentally revoked the digital key used to certify desktop applications that run on Apple's OS X platform, creating headaches for customers who want to install or upgrade Mac antivirus products.

A certificate revocation list [CRL] hosted by Apple Worldwide developer servers lists the reason for the cancellation as a "key compromise," but McAfee officials said they never lost control of the sensitive certificate which is used to prove applications are legitimate releases. The revocation date shows as February 6, meaning that for seven days now, customers have had no means to validate McAfee applications they want to install on Macs.

"We were told that as a workaround, we should just allow untrusted certificates until they figure it out," an IT administrator at a large organization, who asked that he not be identified, told Ars. "They're telling us to trust untrusted certs, and that definitely puts us at risk."

Bryan Barney, McAfee's executive vice president of product development, said the key was inadvertently revoked when an administrator was handling a development hardware upgrade. Instead of revoking his individual use key, the admin mistakenly revoked the code-signing keys Apple uses to help keep the Mac ecosystem free of malware. Company engineers are in the process of resigning their Mac apps with a new key, but until then, there are no good options for customers who want to install or upgrade their programs.

"It's not something we would want to tell people," Barney said when asked if it was true McAfee support personnel were telling customers to permit untrusted certificates "That is a workaround that would work, but it's not a workaround we'd be comfortable with."

Asked why applications haven't been signed a week after the key was revoked, Barney said the error was discovered only two days ago. In addition to generating a new key, engineers must also rebuild and resign applications and then perform quality-assurance testing to make sure the updated programs work properly. He didn't immediately have an estimate for when the problem would be resolved.

The episode is a graphic example of the complexities of administering the digital certificates at the heart of public key infrastructures used to validate software and websites and to encrypt email and other forms of Internet communication. Last week, a private key that security firm Bit9 uses to certify software was stolen by crooks and used to put a trusted seal of approval on malware that infected at least three Bit9 customers. A widely trusted key-signing certificate belonging to Adobe Systems was similarly compromised in September.

For now, customers of McAfee software for the Mac have no way to ensure the apps they're installing are genuine, and that's a problem.

"We might know that this is a one-off case, but we try to train the people that do our installs to be extra paranoid about this stuff," said the unnamed administrator. "They shouldn't have to get into the game where they have to pick and choose what's trustable and what's not. That defeats the purpose of the mechanism."

No, QA still needs to be involved. The company I work for has been burned a few times for making seemingly innocuous code changes that should have no impact, then distributing them without QA.

This, however, is not a code change. It's just a compile and sign with key B, rather than compile and sign with key A.

Other things have probably changed; source code isn't the only thing that goes into a binary. Apple released XCode 4.6 recently, possibly more recently than they last released their code, so their toolchain is probably different. Their internal build infrastructure may have changed. Lots of things in the environment could have changed, particularly at a company as big as McAfee, so the smart thing to do is check it with QA before releasing it, even if you haven't specifically made any code changes.

...Couldn't they just have Apple un-revoke the key? Problem solved. I know there's probably not a normal process for it but it's not like it couldn't be done, and they're a big enough customer that Apple would engage in discussions with them. If there's no compromise then there's no problem.

Has McAfee considered providing a list of SHA1 hashes for known-good versions of whatever install packages and update definitions are currently not available with a proper signature?

While just telling people to trust untrusted certificates is nuts, McAfee presumably knows what files are trusted(since they trusted them enough to sign them with their key back before they revoked it), and would then be in a position to allow customers to know whether an improperly-signed file is an unaltered version of what used to be a properly signed file(and thus as safe as it was before the revocation), or is an altered version signed with any untrusted key(in which case the customer should really be worrying)...

Also, good to hear that McAfee's key-handling process is robust enough that a single employee can break things. That doesn't sound like it could be worrisome at all.

No, QA still needs to be involved. The company I work for has been burned a few times for making seemingly innocuous code changes that should have no impact, then distributing them without QA.

This, however, is not a code change. It's just a compile and sign with key B, rather than compile and sign with key A.

Other things have probably changed; source code isn't the only thing that goes into a binary. Apple released XCode 4.6 recently, possibly more recently than they last released their code, so their toolchain is probably different. Their internal build infrastructure may have changed. Lots of things in the environment could have changed, particularly at a company as big as McAfee, so the smart thing to do is check it with QA before releasing it, even if you haven't specifically made any code changes.

Small nit about the article. It's not possible to "lose control" of a certificate, nor is a certificate sensitive. What you could lose control of, of course, is the cryptographically associated private key.

Barney Bryan, McAfee's executive vice president of product development, said the key was inadvertently revoked when an administrator was handling a development hardware upgrade. Instead of revoking his individual use key, the admin mistakenly revoked the code-signing keys Apple uses to help keep the Mac ecosystem free of malware.

I'm not familiar with Apple's code signing setup, are developer keys chained to the code-signing key? With the generation of a new code-signing key do they have to regenerate keys for all their dev infrastructure too?

...Couldn't they just have Apple un-revoke the key? Problem solved. I know there's probably not a normal process for it but it's not like it couldn't be done, and they're a big enough customer that Apple would engage in discussions with them. If there's no compromise then there's no problem.

Revoking a key needs to be a one-way operation, or else a hacker controlling a compromised machine could keep resurrecting the key he uses to sign malware. Besides, I'd be surprised if revocation lists have the kind of tombstone infrastructure needed to handle the syncing of deletes and undeletes.

No, QA still needs to be involved. The company I work for has been burned a few times for making seemingly innocuous code changes that should have no impact, then distributing them without QA.

This, however, is not a code change. It's just a compile and sign with key B, rather than compile and sign with key A.

Other things have probably changed; source code isn't the only thing that goes into a binary. Apple released XCode 4.6 recently, possibly more recently than they last released their code, so their toolchain is probably different. Their internal build infrastructure may have changed. Lots of things in the environment could have changed, particularly at a company as big as McAfee, so the smart thing to do is check it with QA before releasing it, even if you haven't specifically made any code changes.

This why you should have an automated build chain and keep archival copies of all the tools used to do the build. That way a release is not just for the source code, but also for all the tools used to build the exe. This how we do it when we build SOC's.

This why you should have an automated build chain and keep archival copies of all the tools used to do the build. That way a release is not just for the source code, but also for all the tools used to build the exe. This how we do it when we build SOC's.

They probably do have that, but the bottom line is that if the bits changed, it has to go through a full QA cycle. It's not just about catching the obvious regressions and bugs, but about catching the freak bugs that don't make any sense at all. Do enough software dev and you will see some of those, and McAfee big enough to understand that.

1)Why the heck are you using McAfee products to begin with?2)WHY THE HECK ARE YOU USING MCAFFEE PRODUCTS TO BEGIN WITH?

At least if you burn your cash, it provides warmth.

This time a million, I stopped using McAffee products after the THIRD TIME an update killed the networking in windows... the work around was to download the updated DAT,,, but to download it you need networking.. 3 times in a year... meaning testing there is done never.

The Mac OSX Gatekeeper was mainly implemented to force more Devs to sell through the mac App store with it's greedy 30% overhead. Mac users -- please just ignore Gatekeeper.

I found out McAfee competitor Norton had been blocking downloads of my simple PC games for several years. This whole service -- Gatekeeper, McAfee, Norton, etc. -- is just ScareWare or RansomWare. PC users -- please think for yourself and consider the motivation behind "Your PC may have virus!"

I found out McAfee competitor Norton had been blocking downloads of my simple PC games for several years. This whole service -- Gatekeeper, McAfee, Norton, etc. -- is just ScareWare or RansomWare. PC users -- please think for yourself and consider the motivation behind "Your PC may have virus!"

I'm not a fan of Norton or McAfee, but they are just sub-standard products, they are legit and not out to get you. There are some common things that uninformed devs like to do (like packing executables) that tend to set off the virus scanners, and it's worth it to learn how to avoid being mis-identified if you are distributing software.

They don't need to re-build anything. They just need to re-sign the binaries which have already been tested, re-package and deliver.

Provided that they store the release binaries somewhere, which is a good practice.

That was my first reaction. After some thought, I figured that they probably want to bump the version numbers to avoid confusion about which packages are correctly signed and which are not, and that may require a rebuild. Of course, some minimal QA testing is needed even for resigning and repackaging.

No, QA still needs to be involved. The company I work for has been burned a few times for making seemingly innocuous code changes that should have no impact, then distributing them without QA.

This, however, is not a code change. It's just a compile and sign with key B, rather than compile and sign with key A.

Other things have probably changed; source code isn't the only thing that goes into a binary. Apple released XCode 4.6 recently, possibly more recently than they last released their code, so their toolchain is probably different. Their internal build infrastructure may have changed. Lots of things in the environment could have changed, particularly at a company as big as McAfee, so the smart thing to do is check it with QA before releasing it, even if you haven't specifically made any code changes.

Well I doubt that McAffee or anyone working on a large code base is going to update to the newest compiler/libc version especially quickly and in any case you'd hope they'd have a documented process to get the toolchain set up, so should be able to reproduce the older environment quite easily (poor new developers if not). If they can't do that then yes just releasing a new product with a changed compiler/libc/whatever would be potentially rather bad.

Other question: Shouldn't you be able to strip the old certificate from the existing binary and resign it? Never tried, but I don't see why this wouldn't work.

No, QA still needs to be involved. The company I work for has been burned a few times for making seemingly innocuous code changes that should have no impact, then distributing them without QA.

This, however, is not a code change. It's just a compile and sign with key B, rather than compile and sign with key A.

Other things have probably changed; source code isn't the only thing that goes into a binary. Apple released XCode 4.6 recently, possibly more recently than they last released their code, so their toolchain is probably different. Their internal build infrastructure may have changed. Lots of things in the environment could have changed, particularly at a company as big as McAfee, so the smart thing to do is check it with QA before releasing it, even if you haven't specifically made any code changes.

Shouldn't it be possible to literally take the latest fully QA'd executable, extract the executable portion from the signed container, and resign it with the new key, and then send it down a more limited QA chain, since the actual executable code literally hasn't changed?

meaning that for seven days now, customers have had no means to validate McAfee applications they want to install on Macs.

And this is a bad thing?

I'm being 100% serious here. McAfee is the bane of Windows ITs everywhere, why in the gods' names would anyone want to install it on OS X?

Their mac malware-only product (no firewall, no app protection- just anti-virus) actually isn't bad- I administer it on about 500 macs at work. It's pretty easy to deploy and manage on OSX without touching the mcafee epo server- the agent installer is a shell script and the gui installer is a standard apple package, so you can deploy it in any number of ways- I use casper, but deploystudio or munki will have no issues with it, either.it never goes above about 3% cpu usage for background scanning, and it updates it's dat's reliably. <shrug> mcafee predates me at my current workplace, so i was basically forced to use it, but I have to say I've been pleasantly surprised.

Shouldn't it be possible to literally take the latest fully QA'd executable, extract the executable portion from the signed container, and resign it with the new key, and then send it down a more limited QA chain, since the actual executable code literally hasn't changed?

This is possible, with the "codesign" command line application (see man codesign)

No, QA still needs to be involved. The company I work for has been burned a few times for making seemingly innocuous code changes that should have no impact, then distributing them without QA.

This, however, is not a code change. It's just a compile and sign with key B, rather than compile and sign with key A.

Other things have probably changed; source code isn't the only thing that goes into a binary. Apple released XCode 4.6 recently, possibly more recently than they last released their code, so their toolchain is probably different. Their internal build infrastructure may have changed. Lots of things in the environment could have changed, particularly at a company as big as McAfee, so the smart thing to do is check it with QA before releasing it, even if you haven't specifically made any code changes.

At a company as big as McAfee, I'd expect them to be able to reproduce builds. The build toolchain itself should be under source control. You need to deal with this stuff anyway if you have multiple development branches. You don't want to force a new compiler version/SDK version that drops support for an older OS onto the stable release branch for some .z release.

...Couldn't they just have Apple un-revoke the key? Problem solved. I know there's probably not a normal process for it but it's not like it couldn't be done, and they're a big enough customer that Apple would engage in discussions with them. If there's no compromise then there's no problem.

Revoking a key needs to be a one-way operation, or else a hacker controlling a compromised machine could keep resurrecting the key he uses to sign malware. Besides, I'd be surprised if revocation lists have the kind of tombstone infrastructure needed to handle the syncing of deletes and undeletes.

Actually many certificate authorities support unrevoking, but only if 'Certificate Hold' is used as the reason for revocation. Placing a certificate on hold is rarely used though, except for testing the revocation process.

...Couldn't they just have Apple un-revoke the key? Problem solved. I know there's probably not a normal process for it but it's not like it couldn't be done, and they're a big enough customer that Apple would engage in discussions with them. If there's no compromise then there's no problem.

No.

Certificate revocation must be absolute and final otherwise there's no point to even having certificate revocation.

LOL. Waste of money - best anti-virus option out there -- pay attention and know what you are doing with your computer.

The anti-virus software makers have a business model and jobs because stupid Users are oblivious.

Aaron44126 wrote:

...Couldn't they just have Apple un-revoke the key? Problem solved. I know there's probably not a normal process for it but it's not like it couldn't be done, and they're a big enough customer that Apple would engage in discussions with them. If there's no compromise then there's no problem.

And No.

Apple has nothing to do with or control over the individual certificates. There are a few installed by default (3rd party companies) but these are usually added during software installs.

I'd tell you how to find these - but based on your comment - I'd guess you'd jsut want someone else to handle it anyway - so not going to bother.

So Apple messing with something after the fact is - 1 - pointless - and 2 - Apple has zero access to your computer. There is no secret backdoor on the desktop that Apple can just magiocally log into its 60 million Users computer and handle shit for you.

YOU Installed the software - it is your responsibility to contact McAfee and make them deal with fixing it.

I swear End computer Users should be licensed and have to pass a test.

From a paranoid point of view, what if the system administrator in question didn't "accidentally" revoke this key, but did it on purpose? This forces a hasty rebuild and resigning of much of McAfee's codebase. What if there is something new in it? Something that you wouldn't want signed? Even if that's not the case, the goal could even be as simple as getting the official word to disable signature checking for binaries.

The Mac OSX Gatekeeper was mainly implemented to force more Devs to sell through the mac App store with it's greedy 30% overhead. Mac users -- please just ignore Gatekeeper.

What I don't understand is how someone could make such ignorant comments and isn't an newbie to Ars.

The default setting for gatekeeper is "Mac App Store" and "Verified Developers" - the latter being apps that either don't belong on the Mac App Store or can't.

Like McAfee's apps - the Mac App Store enforces a sandbox that would render it useless, so Apple allows developers to buy a certificate from them to sign their apps with, and that lets them distribute the apps outside the Mac App Store.

Here are the things you CANNOT put on the Mac App Store:* Utilities (MAS apps are sandboxed, so you cannot access files randomly on the filesystem. Bit of a problem. In addition, I don't think you can request admin privileges if you need them)* Drivers (MAS apps are self-contained. You cannot put stuff outside your .app folder)* Apps costing more than $999.99. Yes, these exist. See AutoCAD - the LE version is on MAS, the full version is not (because it costs multiple thousands). It's actually something AutoDesk has complained about, too.* Demo applications (Must be put on your own website)

And the default will pretty much always have to be like that because guess who AREN'T in the Mac App Store? Microsoft. Adobe. I'm sure plenty of others, as well. Like McAfee (a virus scanner is pretty useless if it can only scan within its sandbox)

If you want to complain, go ahead and complain about something Gatekeeper forces you to do - namely spend $99/year for a developer account so you can get a signing certificate.

Am I the only one who was immediately reminded of the recent (and bizarre) hassles that the Linux Foundation experienced, when they were trying to get their UEFI Linux OS boot software for Windows 8 Certified hardware, signed with Microsoft's Secure Boot key?

Would it be going to far to suggest that this is an example of just how important it is to require that all UEFI/Secure Boot implementations, even those for "consumer" hardware, need to be designed in a robust, resilient, flexible manner (which doesn't appear to be the case at present)?