For at least a decade, the standard advice to every computer user has been to run antivirus software. But new, more commercial, more complex and stealthier types of malware have people in the industry asking: will antivirus software be effective for much longer? Among the threats they see are malware that uses the ability of the latest processors to run virtual machines that would be hidden from antivirus programs.

Antivirus vendors, though, don't seem to think they have a problem. "Probably every year people say antivirus software is dead," says Eric Chen, a research manager at Symantec. Graham Cluley, senior technology consultant at rival antivirus vendor Sophos, says: "We do, from time to time, see new ways of infecting computers ... But the way all those things actually would arrive at a computer would be through conventional methods." That is, no matter how a new piece of malicious code spreads or what it's going to try to do, it will still arrive at your machine as a piece of executable code that security software can detect before it does any damage.

Changing threats

Besides, says Cluley: "Some parts of the software community may underestimate what antivirus software is these days. It's the same name, but we're not doing it the way we did it 20 years ago." For one thing, he says - and Chen says the same - today's AV software relies far less on virus signatures - code unique to each bit of malware - to detect an intrusion than many people suppose, and is constantly evolving. A year ago AV software couldn't handle rootkits (programs that hide themselves and take control of the computer); now almost every product can.

Much good it does us. Pedro Bustamante, chief marketing officer for Spanish-based Panda Security, recently completed a survey of 1.5m consumer PCs and 17,809 computers in 1,206 company networks looking for all types of malware. Of the consumer PCs, only 37% had fully updated security protection - and of those, nearly a quarter were still infected. That's only 8.5% - but equates to 127,650 machines in the 1.5m sample. Of the corporate networks, 72% were infected.

Threats have also become more transient: viruses attack for a couple of weeks and then are never seen again.

Sophos and Symantec use heuristics to look for common behavioural patterns. And AV software is moving from blacklisting (barring bad stuff) to whitelisting (allowing only known good stuff).

Meanwhile, rather than email, the fastest-growing threat is so-called drive-by attacks - malware downloaded automatically when you visit an infected website. Unlike email threats, you can't avoid this by using common sense and avoiding dodgy sites; Cluley says Sophos is seeing 29,000 new infected web pages a day, and 80% of these are not "dangerous areas". Instead, they're legitimate sites on anything from pottery to the Miami Dolphins and the Bank of India.

The statistics on older types of malware alone are scary. Seven years ago, says Bustamante, there were maybe 100,000 to 300,000 viruses; now there are "millions and millions". And those use new techniques: 90% constantly modify their shells so their signature is never the same.

Even more important, malware writers have changed from amateurs who want bragging rights to a full-blown criminal - and commercial - market. The chains of events leading to attacks can involve as many as six or seven different players: one hacks the website, another writes the exploit toolkit, a third writes the botnet and rents it out at 4¢ per PC, and so on. If you know where to go, you can buy a one-day distributed denial of service (DDOS) attack for $100 (£50), send 10m spam emails for $600, send 1m spam IMs for $150, or pick up 50Mb of stolen bank and credit card accounts for as little as $30.

Today's malware is different, too: flamboyance has been replaced by stealth. The longer your virus can go undetected on a user's system the longer it can go on stealing money, bank accounts and credit card details. Today's malware writers want maximum effectiveness and a minimum of detection. Macs and Linux systems remain relatively untroubled because if you're after money, you go where the most users are. And that's Windows.

What lies beneath

Peter Gutmann, a researcher at the University of Auckland who presented the results of a study of the commercial market for malware at August's Defcon, estimates that a good virus programmer can make as much as $200,000 a year (here, a 660KB PDF). Alan Cox, an open-source security researcher, points out some additional possibilities. One is malware designed to sit under today's virtual machines. A proof-of-concept paper proposing such an attack, called Subvirt (PDF), appeared last year, written by three researchers from Microsoft and two from the University of Michigan. A presentation at last year's Black Hat security conference from Joanna Rutkowska, a researcher at Coseinc, a Singapore-based security company, covered a much leaner attack she called Blue Pill, which targets the virtualisation built into Windows Vista and into current processors from both AMD and Intel.

So far, these attacks are only theoretical. But, says Gerhard Eschelbeck, chief technology officer at the security company Webroot: "Probably by next year's Black Hat [in early August] we'll see malware leveraging virtual machine techniques in the real world."

Meanwhile the proliferation of short-lived viruses - Cluley says Sophos is seeing 9,000 new pieces of malware a month, or 300 per day - has forced Webroot to change how it works. No longer can it wait for customers to report a new virus. Instead, says Eschelbeck: "We have to hunt for malware on the internet as researchers."

New layer of security

Bustamante is certain that antivirus software is becoming less effective, along with the security suites they're packaged with. Panda's idea for the near future is adding a new layer of security that it calls collective intelligence. He calls it the "web 2.0 version of security": instead of keeping each user's computer separate, it's scanned from the "cloud". This approach, he says, allows much bigger signature files and can detect targeted attacks because all computers are visible in real time.

But even this approach won't last forever. Salvatore Stolfo, a professor of computer science at Columbia University, says the attackers "have the upper hand. They have all the time in the world, and they have great motivation to spend their time and energy to avoid detection."

Antivirus has a future, he says, but it may be in name only. "Basic implementation and strategy will change." Like the fraud detection in use by banks and credit card companies, "eventually, systems implanted in machines will learn your own personal behaviour and protect by detecting abnormalities". One has to hope so. Otherwise, the future looks bleak.

· If you'd like to comment on any aspect of Technology Guardian, send your emails to tech@guardian.co.uk

Sign up for the Guardian Today

Our editors' picks for the day's top news and commentary delivered to your inbox each morning.