But the leaked data had been cached by major search engines, and the discovery triggered a frantic effort to remove the cached data before the flaw was publicized. Much of the exposed data would have normally been protected by SSL/TLS, but the nature of the vulnerability caused it to be exposed to the internet in unencrypted form.

It's unknown how much data may have been leaked, which may make it difficult for companies and users to decide what their most prudent reaction to this bug report should be.

Cloudflare specializes in improving the performance and redundancy of websites, as well as offering protection against attacks such as distributed denial-of-service. The discovery shows how a weak link in just a single widely used cloud service can have a vast impact on data security downstream.

Cloudflare has not released a list of affected domains. But Nick Sweeting, the co-founder and CTO of Blitzka Software, has created a list of 4.3 million websites that use Cloudlfare, and he aims to eventually narrow the list to only display sites left at risk by the coding error.

So far, Ormandy has found data on the web from Uber, 1Password, FitBit and OKCupid. 1Password, a widely used password manager, says the data that was exposed was encrypted in two other ways, thus making the Cloudflare bug of little consequence for its users.

"At the moment, we want to ensure and remind everyone that we designed 1Password with the expectation that SSL/TLS can fail," Jeffrey Goldberg, a security architect with AgileBits - which develops 1Password - writes in a blog post.

Shady Character

Cloudflare, in a detailed postmortem of what went wrong, says the error amounted to the inadvertent use of a single character - ">" instead of "=" - that caused a buffer overrun, which lead to the memory leak spotted by Ormandy.

Cloudflare accidentally introduced the bug into an HTML parser, John Graham-Cumming, the company's CTO, says in a blog post. The HTML parser modifies web pages that come through Cloudflare's edge servers for security reasons, such as obscuring email addresses from known scraping bots.

So far, it does not appear the bug was discovered by anyone else or exploited, Graham-Cumming says. Although SSL keys used by Cloudflare's customers were not leaked, the company itself has had a private key exposed as a result of the breach, as well as "a small number of secrets used internally at Cloudflare for authentication," Graham-Cumming says.

Cloudflare began using the private key to encrypt connections between its own machines, following former National Security Agency contractor Edward Snowden's leaks describing global government surveillance.

Other information that may have been disclosed includes cookies and authentication information, such as API keys and OAuth tokens.

Cloudflare customers were most impacted from Feb. 13 to Feb. 18, Graham-Cumming writes, noting that one in 3.3 million HTTP requests running through Cloudflare potentially resulted in a memory leak.

Funny Data, Then Whoa

Ormandy's first indication something was wrong came Feb. 17 as he was working on a project analyzing large public data sets. Some data he encountered was at first a mystery, he writes, but he notes that it isn't uncommon to come across malformed or garbage data.

"We fetched a few live samples, and we observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major Cloudflare-hosted sites from other users," he writes. "Once we understood what we were seeing and the implications, we immediately stopped and contacted Cloudflare security."

It was an unusual situation - and one that posed a great threat to personally identifiable information. "Seconds mattered here, emails to support on a Friday evening were not going to cut it," he writes.

Ormandy first reached Cloudflare via Twitter. As anyone in the security industry knows, if Ormandy wants to reach you urgently, something bad has happened. But fixing the damage involved much more than simply fixing the bug itself.

Cleanup Operation

By Feb. 21, with Cloudflare attempting to scrub spilled data that had been indexed by search engines such as Google, Yahoo and Bing, Ormandy says he was still discovering new sensitive data that had been leaked as a result of the coding error.

Now, however, Cloudflare says it's confident that there's no more cached data lurking. "We are disclosing this problem now as we are satisfied that search engine caches have now been cleared of sensitive information," Graham-Cumming writes.

All told, Cloudflare reports that it discovered 770 unique uniform resource identifiers, or URIs, that contained data from the memory leak, affecting 161 domains. All search engines have now removed the content, Graham-Cumming says.

Ormandy, however, has taken issue with the tone of Cloudflare's blog post. While he says it's an excellent postmortem, he believes that it "severely downplays the risk to customers."

About the Author

Kirk is a veteran journalist who has reported from more than a dozen countries. Based in Sydney, he is Managing Editor for Security and Technology for Information Security Media Group. Prior to ISMG, he worked from London and Sydney covering computer security and privacy for International Data Group. Further back, he covered military affairs from Seoul, South Korea, and general assignment news for his hometown paper in Illinois.

Operation Success!

Risk Management Framework: Learn from NIST

From heightened risks to increased regulations, senior leaders at all levels are pressured to
improve their organizations' risk management capabilities. But no one is showing them how -
until now.

Learn the fundamentals of developing a risk management program from the man who wrote the book
on the topic: Ron Ross, computer scientist for the National Institute of Standards and
Technology. In an exclusive presentation, Ross, lead author of NIST Special Publication 800-37
- the bible of risk assessment and management - will share his unique insights on how to:

Understand the current cyber threats to all public and private sector organizations;

Develop a multi-tiered risk management approach built upon governance, processes and
information systems;

Enter your email address to reset your password

Already have anISMG account?

Forgot Your Password Message:

Contact Us

Already have anISMG account?

Our website uses cookies. Cookies enable us to provide the best experience possible and help us understand how visitors use our website. By browsing bankinfosecurity.in, you agree to our use of cookies.