OpenBSD takes on OpenSSL

Much of the Internet has been affected by the Heartbleed (Wikipedia) vulnerability in the widely used OpenSSL server-side software. The bug effectively allows anyone to collect random data from the memory of machines running the affected software, which was about 60% of encrypted sites worldwide. A massive cleanup effort ensued, but the vulnerability has been in the software for two years, so there’s no telling how much information was stolen in the interim.

The OpenSSL software is used not only to encrypt HTTPS connections to web servers but also to generate the certificates that undergird those connections as well as many PKIs. Since data could have been stolen over a period of two years, it should be assumed that certificates, usernames and passwords have been stolen as well. Pessimism is the only sure way.

In fact, any data that was loaded into memory on a server running a pre-Heartbleed version of the OpenSSL software is potentially compromised.

How to respond

So we should all generate new certificates, ensuring that the root certificate from which we generate has also been re-generated and is clean. We should also choose new passwords for all affected sites. I use LastPass to manage my passwords, which makes it much easier to use long, complicated and most importantly unique passwords. If you’re not already using a password manager, now would be a good time to start.

And this goes especially for those who tend to reuse their password on different sites. If one of those sites is cracked, then the hacker can use that same username/password combination on other popular sites and get into your stuff everywhere instead of just on the compromised site.

Forking OpenSSL

Though there are those who are blaming open-source software, we should instead blame ourselves for using software of unknown quality to run our most trusted connections. That the software was designed and built without the required quality controls is an entirely different issue.

An advantage of open-source software is that at least we can pinpoint exactly when a bug appeared. Another is that the entire codebase is available to all, so others can jump in and try to fix it. Sure, it would have been nice if the expert security programmers of the world had jumped in earlier, but better late than never.

The site OpenSSL Rampage follows the efforts of the OpenBSD team to refactor and modernize the OpenSSL codebase. They are documenting their progress live on Tumblr, which collects commit messages, tweets, blog posts and official security warnings that result from their investigations and fixes.

They are working on a fork and are making radical changes, so it’s unlikely that the changes will be taken up in the official OpenSSL fork but perhaps a new TLS/SSL tool will be available soon.

VMS and custom memory managers

The messages tell tales of support for extinct operating systems like VMS, whose continued support makes for much more complicated code to support current OSs. This complexity, in turn, hides further misuses of malloc as well as misuses of custom buffer-allocation schemes that the OpenSSL team came up with because “malloc is too slow”. Sometimes memory is freed twice for good measure.

Lots o’ cruft

This is all sounds horrible and one wonders how the software ran at all. Don’t worry: the code base contains a tremendous amount of cruft that is never used. It is compiled and still included, but it acts as a cozy nest of code that is wrapped around the actual code.

There are vast swaths of script files that haven’t been used for years that can build versions of the software under compilers and with options that haven’t been seen on this planet since before .. well, since before Tumblr. For example, there’s no need to retain a forest of macros at the top of many header files for the Metrowerks compiler for PowerPC on OS9. No reason at all.

There are also incompatibly licensed components in regular use as well as those associated with components that don’t seem to be used anymore.

Modes and options and platforms: oh my!

There are compiler options for increasing resiliency that seem to work. Turning these off, however, yields an application that crashes immediately. There are clearly no tests for any of these modes. OpenSSL sounds like a classically grown system that has little in the way of code conventions, patterns or architecture. There seems to be no one who regularly cleans out and decides which code to keep and which to make obsolete.

Security professionals wrote this?

This is to say nothing of how their encryption algorithm actually works. There are tales on that web site of the developers desperately having tried to keep entropy high by mixing in the current time every once in a while. Or even mixing in bits of the private key.

A lack of discipline (or skill)

The current OpenSSL codebase seems to be a minefield for security reviewers or for reviewers of any kind. A codebase like this is also terrible for new developers, the onboarding of which you want to encourage in such a widely used, distributed, open-source project.

Instead, the current state of the code says: don’t touch, you don’t know what to change or remove because clearly the main developers don’t either. The last person who knew may have died or left the project years ago.

It’s clear that the code has not been reviewed in the way that it should be. Code on this level and for this purpose needs good developers/reviewers who constantly consider most of the following points during each review:

Correctness (does it do what it should? Does it do it in an acceptable way?)

Patterns (does this code invent its own way of doing things?)

Architecture (is this feature in the right module?)

Security implications

Performance

Memory leaks/management (as long as they’re still using C)

Supported modes/options/platforms

Third-party library usage/licensing

Automated tests (are there tests for the new feature or fix? Do existing tests still run?)

Comments/documentation (is the new code clear in what it does? Any tips for those who come after?)

Living with OpenSSL (for now)

It sounds like it is high time that someone does what the BSD team is doing. A spring cleaning can be very healthy for software, especially once it’s reached a certain age. That goes double for software that was blindly used by 60% of the encrypted web sites in the world.

It’s wonderful that OpenSSL exists. Without it, we wouldn’t be as encrypted as we are. But the apparent state of this code bespeaks of failure to manage on all levels. The developers of software like this must be better than this. They must be the best of the best, not just anyone who read about encryption on Wikipedia.

OpenSSL will be with us for a while. It may be crap code and it may lack automated tests, but it has been tested and used a lot, so it has earned a certain badge of reliability and predictability. The state of the code means only that future changes are riskier but not that the current software is not usable.

Knowing that the code is badly written should make everyone suspicious of patches—which we now know are likely to break something in that vast pile of C code—but not suspicious of the officially supported versions from Debian and Ubuntu (for example). Even if the developer team of OpenSSL doesn’t test a lot (or not automatically for all options, at any rate—they may just be testing the “happy path”), the major Linux distros do. So there’s that comfort, at least.

““I’m looking at you, Fortune 1000 companies,” Marquess wrote. “The ones who include OpenSSL in your firewall/appliance/cloud/financial/security products that you sell for profit, and/or who use it to secure your internal infrastructure and communications. The ones who don’t have to fund an in-house team of programmers to wrangle crypto code, and who then nag us for free consulting services when you can’t figure out how to use it. The ones who have never lifted a finger to contribute to the open source community that gave you this gift. You know who you are.” […] As for Heartbleed, “the mystery is not that a few overworked volunteers missed this bug,” Marquess wrote. “The mystery is why it hasn’t happened more often.”(Emphasis added.)”