When the virus that causes Aids was identified in 1984, it was confidently predicted that a vaccine would be just around the corner. Yet 24 years on, science remains little closer to developing a means of protecting against HIV, a virus with which 33.2 million people live, and that kills 2.1 million every year.

Recent months have brought little good news. Two major vaccine trials have failed and a third was called off last week by the US National Institute of Allergy and Infectious Diseases (NIAID). Efforts to develop a microbicide, which women can apply vaginally to prevent infection, have also come to nothing.

The lack of progress has led some HIV experts, such as David Baltimore, a Nobel laureate, to question whether a vaccine will be created any time soon. Some pressure groups have even argued that vaccine research should be stopped entirely and funds reinvested elsewhere.

There is no question that these negative results are profoundly disappointing. The abandonment of the latest vaccine trial, however, may actually be cause for optimism. As researchers explained yesterday in the journal Science, it represents a tactical retreat, a change in priorities that could ultimately bring greater progress.

What has happened is that the failures, especially that of a vaccine trial known as Step, have convinced scientists that their basic approach to the problem is wrong. It has become clear that the way in which vaccines have been tested on monkeys in advance of human trials has not been up to the job. In particular, the genetically engineered version of HIV with which vaccinated monkeys are challenged does not predict well how humans will respond.

A new approach is needed and the NIAID has decided to stop throwing money at human vaccine trials until a better one is available. This is surely the right way forward. The need for a vaccine is so urgent that it is tempting to keep up the drive for clinical trials, but it is probably time to step back and take stock.

The money saved can be invested instead in basic HIV research, and in developing better monkey models for assessing vaccines - ones with a more realistic chance of success for testing on people. There is only a limited supply of money for HIV vaccine and microbicide research: the NIAID spends about $500 million (£250 million) a year, and the UK Medical Research Council about £12.5 million. Drug companies such as Merck are also involved, but all funds must be used wisely.

The progress of this research will have direct relevance for another issue in the news this week. When the Home Office released the latest animal experiment statistics on Monday, they showed a decline in the number of procedures performed on non-human primates.

Anti-vivisectionists argue that the failure of so many HIV vaccines tested on monkeys means that this trend should continue, and this type of research should stop. Animals, they say, are inadequate models for studying the virus as it affects humans.

That, however, would be the wrong conclusion to draw from the NIAID's criticism of the monkey experiments conducted to date. Non-human primates are not perfect models for assessing this pathogen, but as the only other group of animals that are susceptible to this viral family, they are the best tools that exist. What is required, as the NIAID paper makes clear, is not an end to the use of monkeys. It is to refine this animal model, so that it mirrors the infection process in humans more faithfully.

In the short term, that may mean fewer vaccine experiments on monkeys, though primate research in neuroscience will be unaffected. But, as HIV monkey models are improved, more procedures may ultimately be required. That will not make animal rights campaigners happy, but it is something to be welcomed if it improves the prospects for a much-needed vaccine.

Mark Henderson is Science Editor of The Times

Explore Health

• Expert Advice

• Health Features

• Mental Health

• Alternative Medicine

• Child Health

• Health Club