For around 30 years we have lived under the spectre of HIV. In the early 1980s, the mysterious appearance of symptoms that would later be known as AIDS led to unprecedented efforts to unmask the cause. On 23 April 1984, Margaret Heckler, the US Secretary of Health and Human Services, told the world that scientists had identified the virus that was the probable cause of AIDS. She was correct. She also said that a vaccine would be "ready for testing in approximately two years." She was wrong.

Despite 28 years of research, there is still no vaccine that provides effective protection against HIV, and in that time around 25 million people have died of HIV-related causes. To understand why creating a vaccine is so hard, you need to understand HIV. This is no ordinary virus. Scientists who study it speak of it with a mix of weary frustration and awed reverence.

The virus is the most diverse we know of. It mutates so rapidly that people might carry millions of different versions of it, just months after becoming infected. HIV's constantly changing form makes it unlike any viral foe we have tried to thwart with a vaccine. "Almost every vaccine that's been developed protects against a small number of strains," says Gary Nabel, Director of the Vaccine Research Center at the US National Institute of Allergy and Infectious Diseases (NIAID).

Vaccines train the immune system to recognise part of a virus, creating a long-term armada of antibodies that seek and destroy the invader, should it ever show its face. For HIV, the most obvious target is gp120, the surface protein that it uses to attach itself to human cells. But gp120 also constantly changes shape, making it difficult to recognise. It also comes in clusters of three that are shielded by bulky sugar molecules, hiding it from the immune system.

On top of that, HIV targets immune cells, the very agents that are meant to kill it. And it can hide for years by shoving its DNA into that of its host, creating a long-term reservoir of potential infection.

So, creating an HIV vaccine is like trying to fire a gun at millions of shielded, moving targets. Oh, and they can eat your bullets.

Uphill struggle

So far, nature has provided little reassurance that a vaccine against HIV is even possible. For virtually every other microbe, there are people who naturally recover from their infections. "Nature itself provides the proof-of-concept experiment. It has told you that the body can inherently do this," says Anthony Fauci, an immunologist who heads NIAID. But when it comes to HIV, "we have the astounding reality that, with more than 30 million people living with the virus, there is not a single documented case of someone mounting an immune response to completely eliminate the virus from their body." Some people have the right genetic qualities to keep the virus in check, but no one clears it.

Given these challenges, it should be no surprise that vaccine research has been, to quote one researcher, a "Sisyphean onslaught of disappointments". Only three potential candidates have made it through clinical trials. Vaxgen's AIDSVAX vaccine, consisting of two fragments of the gp120, failed to provide any protection. Merck's v520 vaccine, consisting of a harmless cold virus carrying three HIV genes, fared even worse. It was meant to marshall immune cells called T-cells to kill off infected cells. It failed. Worse still, the trial had to be stopped early because vaccinated people seemed to be more susceptible to infections, for reasons we still do not fully understand.

The only sliver of success came in 2009. A Thai trial of more than 16,000 people – the largest one yet – had been testing a combination of two vaccines: a bird virus containing three HIV genes, which was meant to prime the immune system, and a tweaked version of the AIDSVAX vaccine to act as a booster. Since both vaccines had failed individually, critics argued that the trial was a waste of time and money. But to many people's surprise, the combination reduced the risk of infection by 31% – a statistically significant effect, though too low for a useable vaccine. (For comparison, the measles and polio vaccines are around 95% effective).

Some scientists were sceptical about the results, noting that the protection was short-lived and confined to people at low risk of infection. Others saw a ray of hope after years of frustration, a sign that a vaccine is in principle possible. Either way, the trial was confusing, especially since the vaccines did not reduce the levels of virus in infected people. Scientists are still trying to work out why the vaccine had any effect at all.

Renewed hope

But despite the muted results from existing trials, scientists in the field are unfazed. The reason, according to Wayne Koff from the International AIDS Vaccine Initiative, is that since the Thai results were announced, "the field that has begun to undergo a renaissance."

It turns out that many HIV patients carry secret weapons – "broadly neutralising antibodies" – that can attack a wide range of HIV viruses. For these patients, it is too late. Their infections are already in full swing and the virus can mutate around their defences. But the discovery proved that HIV's vaunted diversity is not the roadblock for vaccines that many scientists feared. If we taught the immune system to make these antibodies early, we could destroy the virus before it gained a foothold.

It is possible to isolate the right antibodies because we now know the shape of HIV's surface proteins, down to the atomic level. Nabel used this knowledge to identify parts that stay the same while the rest of the virus shifts and mutates. These non-mutating regions are likely to be vital areas that cannot change without causing problems. He searched patients' blood to find antibodies that target these conserved regions, and cells that make those antibodies. In 2010, he found three: two of which could neutralise 90% of HIV viruses. Other scientists, such as Dennis Burton of the Scripps Research Institute, have made similar discoveries using similar methods.

Nabel's vision is a cocktail of these super-antibodies that target different parts of the virus, cutting off its evolutionary escape routes. He hopes to start clinical trials of his first-generation antibodies by early 2013, and he says is close to producing a second-generation of even heavier hitters that he thinks are two to three years away from early trials.

Vaccine researchers are also working on ways of stimulating our T-cells to kill infected cells at an early stage. This was the strategy behind Merck's failed v520 vaccine, but scientists have since found better ways of smuggling viral genes into cells, and targeting them at the tissues most likely to be infected first. Both approaches would be complementary: "I think we would need a combination of broadly neutralising antibodies and a broad and robust T-cell response," says Koff.

Compelling need

There is no telling when, or indeed if, these strategies will yield results, but what is certain is that the need for a vaccine will not diminish. There are many ways of preventing HIV infection, including condoms, microbe-killing gels, and the use of treatments as soon as people get infected. "We'd be going in the right direction with the tools we already have," says Fauci. "But if we added a vaccine to the toolkit, even if it wasn't 90% effective, you could have a major additive effect. There really is a compelling need for one."

It may seem frustrating that decades of research have yielded nothing that satisfies this compelling need. But everything in the pipeline has depended on a steadily accumulating knowledge of the virus over those years. And as much as we know about the virus, and our immune response to it, there is still a great deal to learn. Also consider this. It took 47 years to create a vaccine for polio after the microbe behind it was identified. The measles vaccine took 42 years. The hepatitis B vaccine was a positive sprint at 16 years. "Twenty-eight years isn't an inordinate amount of time," says Fauci.

Dive into the awe-inspiring, beautiful, and quirky world of science news at Discover's Not Exactly
Rocket Science blog, written by Ed Yong. No previous experience required.