Gatekeeper: Apple’s next move to increase Mac security

While we’re still getting more information about Mac OS X 10.8 Mountain Lion, just announced this morning and available to registered developers as a preview build, one major feature of the new operating system deserves a closer look due to the implications it has for how users and software developers work with the platform.

Mac OS X has always been an open platform, and by that I mean there is nothing stopping you from installing or running any program you wish, all you need to do is write or obtain that program and OS X will happily run it. You may get a warning if you download a program from the internet, but it’s still up to the user to decide whether or not that program is trustworthy.

This makes developers and users lives easier in many ways, but it does leave novice users open to social engineering and accidental downloads of malicious programs. However, Apple has a good solution that should make most users happy while making the Mac more secure by default.

The problem with 3rd party applications

The problem for Apple and Mac users is, how do you secure a computer that will happily run whatever you tell it to run, while keeping the platform open? How do you prevent novice users from downloading and installing things from a malicious 3rd party that would essentially compromise the entire machine?

The actual answer is you can’t. If users are allowed to make judgement calls, there will always be users who end up making the wrong judgement call.

The only way for Apple to solve that issue would be to make the Mac work like iOS, where nothing runs at all unless it is signed, sandboxed and approved by apple.

However such a move would be the Mac equivalent of a nuclear bomb going off over developers & users heads, there is simply no way to force everything to move inside the sandbox & Mac App Store on short notice, and for some applications it may not be possible or even desirable to do so.

Mac App Store

The Mac App Store brought a new distribution mechanism to OS X, allowing developers to get their products in front of millions of Mac owners without having to run their own distribution channel and checkout/payment service.

There is an assumption that software downloaded from the Mac App Store is “safe”, and Apple has gone to great lengths to make that a reasonable assumption due to the application review process, the need to register with Apple to get products on to the store, and soon, mandatory kernel-enforced sandboxing to prevent applications from doing things they shouldn’t be able to do.

However, the Mac App Store has its own issues and limitations, as of right now a large number of existing Mac OS X applications can’t even be included in the Mac App Store due to restrictions on things like the use of kernel extensions/drivers, use of private APIs, and things that simply aren’t possible from inside the sandbox.

Enter Gatekeeper

With Mac OS X 10.8, Apple has shifted the default behavior for 3rd party software: if it isn’t signed by a developer certificate or downloaded from the Mac App Store (and therefore also signed), it won’t run unless you manually set the system to “opt-in”, not unlike Android’s checkbox to run applications from unverified sources.

This builds on existing Mac OS X features like the Quarantine bit, which flags a downloaded file as being potentially untrusted, and code signing, which ensures that applications haven’t been tampered with before running them.

Gatekeeper also has the ability to revoke a developer’s certificate, for cases where a developer distributes signed applications that are found to be malicious.

On balance Apple seems to be taking the right approach with Mountain Lion, if you want the ability to make your own decisions, there is a checkbox in System Preferences that will allow Mac OS X to run whatever you download.

However for novice users, the new default is a significant improvement. They can still make the wrong judgement call about an application, but they now have to explicitly enable that behavior rather than simply clicking through a popup warning.