A new program aims to put the squeeze on malicious software that can disable, infiltrate or transform a computer system into an unwitting zombie — before the cyber attack ever materializes.

Instead of studying the signatures of software gone bad and released into the wild as computer viruses, worms, Trojan horses or other malware, a program called Korset has been designed to model the expected behavior of legitimate programs and immediately halt anything that veers into unexpected territory.

Avishai Wool, a professor of computer engineering at Tel Aviv University in Israel, likens the problem to a conveyor belt churning out colored cogs. If a villain throws a wrench into the machine, the sequence of cogs carried along the conveyor belt could change visibly — say from red-blue-blue to red-green-blue. Instead of a security guard trying to apprehend a potential saboteur, the Korset program is like a conveyor belt operator who sees the new sequence, says, “That can’t be right” and promptly turns off the machine before it can be damaged.

Wool and graduate student Ohad Ben-Cohen designed their two-phase program as a new line of defense for the Linux operating system. Linux hosts the majority of web and corporate e-mail servers around the world, as well as other vulnerable systems. Ben-Cohen presented the research at the Black Hat Internet security conference in Las Vegas last month.

In its initial registration phase, Korset analyzes the program requiring protection and produces a model of good behavior that describes every allowable action. In the subsequent monitoring phase, Korset looks over the program’s shoulder and ensures that it doesn’t deviate from that model. If so, the security detail stops the program.

“Any action taken by the program which is not in the model must be something malicious,” Wool said.

Ah, but could a particularly deceptive bit of malware sneak in under the guise of something legitimate? It’s possible, he said.

Despite the potential for overlooking danger, he said the approach is still a distinct improvement overtraditional statistical models. Under such schemes, an operator lets a computer program run under “normal” circumstances and records what the program is doing. Using statistical methods, a model can then describe how the target should behave under these normal circumstances. But if even a small number of legitimate functions are left unsampled, Wool said, eventually the statistically informed security force will kill a program it shouldn’t.

When semicolons and zombies attack
Modeling has long been a tricky business. No statistical model, for example, can describe with perfect accuracy all the e-mails you might legitimately receive, while labeling the rest as spam. You know this is annoyingly true if you’ve ever found a friend’s missing message amid sequestered e-mails touting Viagra, promoting affordable home mortgages or promising a million dollars to be wired from a bank account in Nigeria.

Similarly, British mathematician Alan Turing's research in the 1930s demonstrated the impossibility of accurately modeling every conceivable computer program. Fortunately, Wool said, computer programs are highly predictable and amenable to simplified models. His simplification is based on a program’s regular requests from the underlying operating system, known as system calls. Every program has a fixed repertoire of between a few hundred and a few thousand of these calls. “If it does what it is designed to do, it will emit a certain sequence of system calls and not another set,” he said. An aberrant system call sequence would be a dead giveaway that the program has a bug or is being attacked.

One common type of malware, called a buffer overflow attack, overwhelms programs that accept input, like the address line in a Web browser or a search window for a database. If the program has not been carefully written, a vandal with malicious intent could submit an entry that’s 30,000 characters long instead of the expected 30. Overflowing from its allotted space, the malware can override other program code and do considerable damage.

Another class of attack, named an SQL injection after the computer language commonly used for Web-based databases, uses unexpected characters to throw a program out of whack.

“Suppose you have a field that you can type a name into, and instead of a letter, you type in a quotation mark, parenthesis or semicolon,” Wool said. “If the program on the other side uses these special characters for its own purposes, say to indicate the end of a command, you basically throw it a curveball and it sees that something is ending when it shouldn’t and everything after that does something at a completely different level.”

For a third category, known as a denial of service attack, vandals command a large group of co-opted computers — the dreaded “army of zombies” — to barrage a victim’s Web site with a surge in traffic. Such an attack materialized during the recent conflict between Russia and the Republic of Georgia,
disabling many of Georgia’s governmental sites
, though exactly who launched the attack remains murky.

“At that point, Korset would not help,” Wool said, because the computers have already been taken over. But a Korset-ized computer could be protected from sinking to zombie status, making it harder for the attacker to muster the legions of digital undead to do the dirty work.

Staying ahead of computer villains
“It’s certainly impossible to figure out in advance what the bad guys are going to try,” Wool said. “And that is really the approach that is taken by intrusion detection systems.”

Like the human immune system, he said, such systems study traffic patterns. When they see a pattern known to emanate from a malicious source, they stop it.

“That’s fine, except that it is always reactive,” he said.

In order to stop the attack, the detection system must find the threat and then identify it as such, leaving an inevitable time delay from the malware’s debut until its deactivation.

“That’s a losing proposition in general,” he said. “The bad guys always have an advantage here.”

But with an approach that isn’t based on signatures, none of the traffic needs to be tracked in advance, and an attack can be stopped on what is known as Day 0.

Patrick Peterson, a computer security expert at IronPort, an independent business unit of San Jose, Calif.-based Cisco Systems, praised the Korset program and like-minded applications as being essential innovations for beefing up existing safeguards. “The bottom line is anti-virus software still has its place but criminals have become far too good at beating it to rely on this as a primary source of defense,” he said.

Based on reputation scores alone, around 80 percent of the data circulating through the Internet is considered harmful, Peterson said. One major advantage with technology like Korset in addressing all that, he said, is that it doesn’t rely on the airline terminal X-ray scanning approach to spotting such malware. “Criminals have spent 15 years getting past the X-ray antivirus scanner.”

The downside for an application like Korset may be the difficulty in picking up on well-cloaked actions or keeping up with huge institutions whose thousands of unique applications demand regular updating — and thus, re-registration with the security program. Every new method of protection system “often appears to be a silver bullet,” Peterson said. “But in reality it takes a lot of work.”

Nevertheless, he said he’s betting on the general approach as the promising future of security technology.

Wool and his collaborators have successfully included a Korset prototype into the Linux operating system and demonstrated in a proof-of-principle experiment that it can stop a simple attack. The team also has released its open-source code to promote further development of the software.

“It is our hope that this becomes mainstream and that this approach is adopted in standing distributions of operating systems,” Wool said. “If somebody like Microsoft looked at this work and decided to do something along the same type of line in the Windows operating system, that could have a tremendous effect on the safety of computers in the future.”