Microsoft this week announced that each of the three finalists in the BlueHat Prize $250,000 security contest came up with ways to detect and stymie one of the most effective exploit methods now being used by hackers.

The three finalists -- two from the U.S., the other from Croatia -- took different tacks to block return-oriented programming, or ROP, a technique often used to sidestep DEP, or data execution prevention, one of Windows' primary anti-exploit technologies.

"It's an obvious reflection on the most pressing attack vector hitting systems right now," said Andrew Storms, director of security operations at nCircle Security, about the ROP subject of the finalists' entries.

"It seemed to us that to take an approach to block entire classes was the best way to engage with the research community and protect customers," Katie Moussouris, a senior security strategist lead at Microsoft, said in a news conference last year that announced the contest.

Winners Revealed at Black Hat

BlueHat Prize features a $200,000 first-place award, $50,000 for second, and a subscription to Microsoft's developer network, valued at $10,000, as the third-place prize. The three finalists will be flown to next month's Black Hat security conference in Las Vegas, where Microsoft will reveal the results July 26.

The finalists announced Thursday are: Jared DeMott, a security researcher employed by Florida-based Harris Corp., a major defense and aerospace contractor; Ivan Fratric, a researcher at the University of Zagreb in Croatia; and Vasilis Pappas, a PhD student at Columbia University.

All three worked alone -- contradicting earlier speculation that the contest was slanted toward teams -- and wrapped up their work one to two weeks before the deadline.

And each researcher tackled the same problem -- ROP -- and explained why in much the same way as Storms.

"I focused on ROP because it is the current state-of-the-art in exploit development and a burning issue in exploit prevention," said Fratric in an email reply to questions. "Furthermore, it is a very difficult problem to solve, so it was an interesting challenge."

DeMott echoed that. "I targeted ROP because it is currently the most-used technique to exploit fully-compiled software," he said, also in an e-mail.

But while DeMott, Fratric and Pappas all attacked ROP, they came up with different solutions.

DeMott, who calls his technology "/ROP" to match other Microsoft-made defenses, such as "/GS" and "/NXCompat," said his answer to ROP checks the target address of each return instruction, whether intended or not, and then compares it to a whitelist.

"/ROP is simple to understand and implement [and] it fits the current Microsoft paradigm," said DeMott. "It works with low overhead and finally, /ROP mitigates all known, practical ROP attacks."

Fratric's "ROPGuard" uses a somewhat similar technique to block ROP exploits, since his technology also checks each critical function call to determine if it's legitimate.

"Unless [the attacker] wants the attack to stay confined in the current process, [he or she] will need to call some 'special' functions to leverage the attack," said Fratric. "The attacker will need to call these functions from the ROP code, either directly or indirectly, and that makes these functions an ideal place to check if the attack is taking place or not."

Fratric said that ROPGuard could be applied at runtime for any process, even those already running.

"kBouncer," the name Pappas slapped on his defense, takes a different approach, and instead checks the control path leading to a system call.

"When ROP code is executing, control follows an unconventional path, which makes it easily detectable," said Pappas in an email Thursday. He called kBouncer a "lightweight form of control flow integrity."

Unlike /ROP and ROPGuard, kBouncer relies on a performance-monitoring feature found in newer Intel processors for its efficiency, said Pappas. "Although it does not require any specific hardware, it runs much better on these chips," he said. "Hopefully other CPU vendors will implement that functionality, too."

Low Impact Cure

All three claimed that their solutions would only minimally impact the performance of a Windows PC.

"The effect on memory and the CPU is minimal, about 3 percent to 4 percent on average," said DeMott.

Fratric said ROPGuard is even less processor-intensive. "It had an average CPU overhead of just 0.5 percent in my experiments," he said.

One of the BlueHat Prize stipulations was that a solution had to have a processor overhead of less than 5 percent.

Microsoft is expected to add one or more of the finalists' solutions -- or even some of those that didn't make the last cut -- to Windows. Contest participants will retain intellectual property rights to their work, but must license their technologies to Microsoft on a royalty-free basis.

"I suspect we will see changes in the next Windows 7 service pack," said Storms. "I'd put my money on January [2013]."

Microsoft issued its only service pack for Windows 7 in February 2011. Based on its previous practice, Microsoft will probably ship a second service pack for Windows 7 soon: It delivered SP2 for Windows XP just over three years after that edition's launch, and Vista SP2 two years and three months after the OS's debut.

Although it's possible that Microsoft could squeeze in the same protections into Windows 8 before its launch this fall, experts think that's unlikely. Microsoft could, of course, issue an update to the new operating system after it releases to add one or more of the defensive technologies.

Practical Application

The finalists were all optimistic that Microsoft could easily add their code to Windows.

"My guess is that it would be not that difficult," said Pappas. "kBouncer's main idea is straightforward and its transparency makes it easy to integrate. Even the prototype implementation, which I developed by myself, is already capable of protecting large and complex applications, like Adobe Reader and Internet Explorer."

ROP has been widely used by hackers, sometimes for spectacular results.

The Stuxnet worm, reportedly created by U.S. and Israeli coders to sabotage Iran's nuclear fuel enrichment facilities, used ROP extensively. In late 2010, attack code that exploited IE on Windows 7 went public; the attack exploited ROP vulnerabilities to sidestep Windows DEP and ASLR (address-space layout randomization), the two main anti-exploit defenses in the OS.

Earlier in 2010, a pair of researchers used ROP to hack Safari on Apple's iOS mobile operating system to win $15,000 at the Pwn2Own contest. It was the first time that ROP had successfully been used against an ARM processor-equipped device.

"Windows users would be safer if /ROP is adopted, because ROP attacks, as they are now, would fail," said DeMott. "Clever attackers will likely move to something more advanced, but that is the cat-and-mouse game we play in security."

Fratric chimed in with a similar take. "It will protect users from currently-used exploits. For how long, I can't answer, because as the protection technologies are developed, so are the exploitation techniques," Fratric said. "The protection I proposed is not perfect, but it raises the bar for the attacker and could raise it a bit more if some ideas I proposed are extended a bit."

Storms thought it noteworthy that Microsoft selected ROP defenses as the three finalists for BlueHat Prize, and said it showed that the company has ROP as a top priority.

"But couldn't they have chosen anything else for at least one of the finalists?" Storms asked.

DeMott pointed out the focus on ROP from a more personal angle: "I did not know that others would also address ROP attacks."