Seattle security expert helped uncover major design flaw on Internet

He and others worked in secret to repair problem

DANIEL LATHROP AND PAUL SHUKOVSK, Seattle Post-Intelligencer

By DANIEL LATHROP AND PAUL SHUKOVSKY, P-I REPORTERS

Published 10:00 pm, Sunday, August 3, 2008

Dan Kaminsky and a group of computer geniuses converged in a conference room at Microsoft to create, in absolute secret, a patch to fix a problem that left the Internet vulnerable to criminal attack.
Photo: Joshua Trujillo/Seattle Post-Intelligencer

Dan Kaminsky of Seattle doesn't wear a leotard and tights like other superheroes, but a few weeks ago he saved the Internet.

Kaminsky was typing away in bed in February tangled in a virtual snake pit of Ethernet cables when something sent a chill down his spine.

He had discovered a devastating design flaw that could turn the World Wide Web into a playground for criminals, leaving global commerce gridlocked.

"As soon as I saw it work the first time, I realized this was (the basis of) a universal attack," he said. "Anybody, any kid, had the ability to redirect your Internet."

He grabbed his cell phone and dialed his girlfriend, Web developer Crystal Williams.

"I broke the Internet," he told her.

"What do you mean you broke the Internet?" she replied. Because Kaminsky's job title is "director of penetration testing" for Seattle cybersecurity consultancy IOActive where he's the acknowledged expert on the "Black Ops" of Internet protocols, she assumed it wasn't a problem with his WiFi.

Kaminsky, 29, hadn't broken anything, exactly. It's more accurate to say he discovered a potentially devastating mistake in the design of the Internet itself.

That flaw, if it became public, would allow even an unsophisticated hacker secretly to swap one computer's Web address for another's in the Internet virtual address book, dubbed "Domain Name Services."

You might think you're logging into your bank, but you've actually been redirected to a bogus bank site posted by the Russian Mafia. Suddenly, you've got a zero balance.

Three weeks ago, Kaminsky's discovery caused a minor media stir. But before word leaked out, he had assembled a group of experts who worked in secret to craft a solution. By the time the public learned of the problem, most Internet users were already safe.

The reality of a major flaw being found and its solution orchestrated by a lone security researcher is a perfect example of what the country faces regarding threats to critical infrastructure, said Robert Jamison, the undersecretary for Homeland Security charged with overseeing civilian cyberdefenses.

"Basically if that vulnerability hadn't been solved as quickly as it was, you could have a lot of people going after it and exploiting that vulnerability ... There could have been a lot of damage," he said.

"We're not in a position to demand anything," said Chad Dougherty, the CERT researcher in the group assembled by Kaminsky. "Or to try to strong-arm vendors or researchers."

As a Web developer, Williams saw the problem right away.

"It's not that difficult to completely refashion the pages," she said. "If it looks like a duck, and it quacks like a duck but it's sending your credit card to info to China, that's probably a bad thing."

Because Domain Name Services information is stored on a local server by every Internet provider and corporate network, a process called caching, the attack is called "cache poisoning." Long known to be technically possible, it was believed to take weeks, making it infeasible for malicious hackers to exploit. Kaminsky found a way to do it in seconds.

His next call went to the godfather of DNS, Paul Vixie, a Bay Area programmer at the nonprofit Internet Systems Consortium. Vixie leads a large group of volunteer programmers who write and maintain BIND, the largest single Domain Name Services program.

He told Kaminsky that there was no way a design flaw could have been missed.

So Kaminsky explained, and Vixie, too, became very, very worried.

From Japan to Germany, experts in the arcane art of Internet names and numbers dropped everything to converge in a conference room on Microsoft Corp.'s Redmond campus March 31.

The group of about 16 experts created a spontaneous United Nations of computer geeks representing everything from government groups to nonprofit do-gooders to corporate giants such as Microsoft and Cisco Systems.

Secrecy was so tight that participants didn't know what Kaminsky had found until they sat down at the Redmond summit.

Kaminsky outlined the security hole, and they began to discuss their challenge: Find a fix and turn it into software "patches" while keeping the problem secret long enough to fix the software on thousands of computers around the planet.

Because of the emergency nature of their meeting, they had just one day to make their plan.

"They needed to ignore company boundaries and they needed to ignore private interests. They did it," Kaminsky said. "There was just no politics at this meeting. This was 'We have a problem.' "

Within hours they did the work of years, and, according to Kaminsky, found a fix "so sneaky" that they hoped the bad guys couldn't use it to discover the hole.

Now they had to make it work. And fast. July 8 would be their big day: the largest simultaneous software patch in history.

For Sandy Wilbourn, chief engineer at Bay Area software maker Nominum, the first inkling of trouble came in an e-mail on a business trip to Germany.

"I need to talk to you about something, but I can't send it over e-mail and I'm not comfortable talking about it over the phone," Wilbourn recalls the message saying.

When he called, the engineer remained laconic.

"Call Paul Vixie" was all he would say.

"People bring up security vulnerabilities all the time. At first your reaction is a little skeptical," Wilbourn said. But on the say-so of Vixie and Kaminsky, he went to Redmond.

Not only was their work to be done before July 8, but he couldn't tell anyone outside Nominum's small corps of engineers.

That stirred waves in the company. Wilbourn had a room full of people working on an emergency problem, but nobody could know what or why.

Then there were the customers. Major telecom firms don't change their computer systems lightly, but Wilbourn would need them to take a patch and couldn't tell them why until the patch was ready.

"That," he said, "always leads to an interesting dynamic."

By July 9, 15 percent of the servers tested were safe, and over the next two weeks the number crept up to half. Among them were many of Wilbourn's customers.

By late July, the secret was out. Hackers had developed "fully weaponized" attacks based on Kaminsky's discovery. But the biggest danger had passed. Most of the big providers – and their customers – were patched. The rest soon would be.

As of Friday, fully two-thirds of the servers tested were safe and the numbers were growing every day. But Kaminsky isn't resting easy.

"There exist threats to the continued correct running of the Internet," Kaminsky warned. "We have found one of them. There may be more."