How Codenomicon Found The Heartbleed Bug Now Plaguing The Internet

You know that song lyric about the first cut being the deepest? It’s complete rubbish. Heartbleed taught us all that. Because the more we learn about this online data-security wound, the deeper that threat seems to go.

Discovered independently by Google engineer Neel Mehta and the Finnish security firm Codenomicon, Heartbleed has been called “one of the most serious security problems to ever affect the modern web.” I spoke with Codenomicon CEO David Chartier, who led the Finnish team that named and outed Heartbleed, to find out more about how his team discovered it, and how deep those vulnerabilities could go.

We All Bleed For Heartbleed

Codenomicon's David Chartier

Heartbleed actually started out really small. In fact, it was just a slight, accidental gaffe committed by one coder. Had it been caught immediately, it would have required just filling in a missing bit of code. But it wasn’t. And now, that error has propagated to compromise much of the Internet.

The main problem is it that affects OpenSSL, a widespread open-source security protocol used by as much as two-thirds of Web servers. The other issue is that it went largely undetected for two years—plenty of time to perpetuate across the Web and leave sites, services and accounts big and small open to infiltration. (As the National Security Agency has reportedly done, although the White House has denied that.)

The initial flood of news reports focused on the hackability of user logins, financial information, emails, photos, medical records, and much more. But Heartbleed’s reach could be bigger than anyone imagined. The OpenSSL flaw affects any server or client that uses it, and that means it could span a huge number of things—including routers and phones, as well as citywide or municipal infrastructure, such as emergency services, transit and utilities.

How Heartbleed Surfaced

Codenomicon first discovered Heartbleed—originally known by the infinitely less catchy name “CVE-2014-0160”—during a routine test of its software. In effect, the researchers pretended to be outside hackers and attacked the firm itself to test it.

“We developed a product called Safeguard, which automatically tests things like encryption and authentication,” Chartier said. “We started testing the product on our own infrastructure, which uses Open SSL. And that’s how we found the bug.”

The engineers found they could burrow in despite the cryptographic security layer, and were shocked at how much was up for grabs. They could access memory and encryption certificates, and pull user data and other records. “This is when we understood that this is a super significant bug,” Chartier said.

The revelation was startling, not only because of the access this hole could allow, but because of its insidious nature, Chartier said. “On top of that, we couldn’t find any forensic trail that we were taking this data.” The hack was completely untraceable.

But how did something this egregious and widespread go on undetected for two years? The error is buried in the code. The only reason Chartier’s team found the glitch is because Codenomicon uses a rigorous testing process using a very large number of test cases to find weaknesses, just like hardcore hackers do, Chartier explained.

“The vulnerabilities you find after many tests are often more interesting than the ones you find right away,” he said. “When you find one that’s difficult, it’s more interesting [to hackers] because they can write an exploit, and it will take more time to be found.”

The odds of finding the flaw were slight, yet Google’s Mehta discovered it practically simultaneously, during a routine security check in March. Chartier chalks it up to happenstance. “Google’s one of the leading companies in the world, and it’s constantly testing for vulnerabilities,” he said. The company has been known to take security testing very seriously, so much so that it even offers a bounty for exploits on projects like Chrome. This allows it to find flaws and fix them before hackers can take advantage of them.

But not every company takes security that seriously.

A Fail To Remember

Codenomicon, being a Finnish company, alerted the Finnish National Security Cyber Center of its findings. Commonly referred to as “CERT,” the group urged the OpenSSL Project to provide an update and release it to the public. This was just days after Mehta notified OpenSSL on April 1.

The news wasn’t broadcasted after the first discovery, as OpenSSL wanted “to give time for proper processes” to let vendors patch the hole before making it public. The plan was to make an announcement on April 9. But when two independent research teams coincidentally found the error, it suggested a greater risk, which prompted OpenSSL to accelerate the announcement to April 7.

The report blazed across both tech and mainstream media headlines. Chartier has been impressed with how online communities have disseminated the Heartbleed information. “We’re better off today than we were a week ago, because of getting the word out there,” he said. “It’s making the Internet safer and more secure.”

Unfortunately, the Web is not where this problem ends. Other networks also need to apply the software update in both server and client devices. This includes gadgets like phones, computers and other communication devices. It also include numerous other technologies in the broader world, particularly as it relates to the Internet of Things.

Because Heartbleed affects OpenSSL, which is widely adopted, it can affect an extensive range of categories, including connected homes, citywide transportation, emergency services, power grids and other utilities—pretty much any large scale, connected systems. But locking all of them down can be difficult.

Organizations must update to the patched version of OpenSSL, revoke encryption certificates that authenticate their sites and issue new ones. However, systems that haven’t gone through security and system testing may not be set up to handle update protocols efficiently. “There’s a lot of stuff out there that was built a long time ago,” said Chartier. “It wasn’t built for the type of stuff that’s coming out today.”

Security tests are essential for critical infrastructure, but unfortunately, there’s still a lot of room for improvement. “A lot of companies do a little performance testing, to see if [software] does what it’s supposed to,” he said. “But they don’t do enough security testing.”

Chartier thinks it could take up to a year or two before all or most of the old versions of OpenSSL out there get updated. In the meantime, things may get tricky.

At this point, many—though not all—of the largest vulnerable sites on the Web have patched OpenSSL against Heartbleed. With some of the smaller service providers and businesses, it may take a little more time. The most prudent users may want to assume that their data was compromised, and change those passwords on every site and service that has been secured. The Codenomicon chief recommends going through each provider, one by one, and “finding out if they used OpenSSL, and if they patched it.”

As for the companies and organizations, Chartier urges them to adopt more stringent security standards. “You need to put this type of testing into your build cycle,” he said. That’s the best chance at mitigating the risk—so threats don’t penetrate quite so deeply.