nontoxic.org.uk

Updated Sunday 27th September 2009 with links to further analysis. Updated again Saturday 10th October 2009 with Science Magazine’s take on the study.

You may have noticed in the news recently, some triumphant headlines concerning the “success” of an HIV vaccination trial which took place in Thailand over the last seven years. Vaccine heralds new dawn in the fight against Aids, proclaims the Independent, beginning the article with the words: “The scientific naysayers who claimed a vaccine against HIV would never be possible have received their comeuppance.” Aids vaccine found to cut risk of infection declared the Times. HIV breakthrough as scientists discover new vaccine to prevent infection exclaimed the Guardian. The Sun also weighed in with their typical measured and timid approach. First vaccine for AIDS developed, they unambiguously declared.

Out in Morocco (my current location), the news was relayed to me on my hotel room TV by Al-Jazeera, TV5, and DW-TV. So do we really have such a massive breakthrough? Do the claims stack up?

Well, certainly not to the extent that the world’s media appear to be claiming. We have, first of all, to eliminate the possibility that the results could have occurred as a result of bias, chance and confounding. Leaving aside confounding for a moment, we turn first of all to potential sources of bias, then look at the role of that funny beast, chance.

In looking for sources of bias, we look first of all at the trial design, and ask, as in secondary school physics lessons, “was it a fair test?” Colin Hockings over at blue-genes.net has this to say:

I found an interesting 2004 release from TAG, the Treatment Action Group, which campaigns ‘for larger and more efficient research efforts…towards finding a cure for AIDS’, saying that the Thailand trial (called RV144) is flawed for several reasons.

• A single experimental arm won’t allow the relative effects of the two vaccines to be tested. At the time, there had been no clinical trials of ALVAC vCP1521 efficacy and I don’t think that that has changed since then. If it’s still true then they have a very valid point: the experimenters cannot tell how much of the effect is from the ALVAC vaccine alone. TAG also had some concerns that the AIDSVAX gp120 boost may neutralise the other vaccine, based on experiments in macaque monkeys. It is therefore quite possible that ALVAC vCP1521 worked much better, but we won’t know until another large trial is completed.
• TAG also questioned the ethics of recruiting volunteers – who mostly claimed altruism as their motivation for joining – to a trial that would, at best, lead only to further trials (based on the arguments above).

The reason why such a large trial was performed that won’t actually answer any questions is fairly stupid. It was planned well before two clinical trials for AIDSVAX came back negative, and a trial for the ALVAC vaccine (called HVTN 501) was cancelled. Despite the drastically changed circumstances, RV144 went ahead, leaving us with tantalising evidence that we’re making progress, but not really changing the game much.

So not only were concerns raised about the way the trial was designed and how participants were recruited, there was little plausible prior reason why a combination of two vaccines should work when trials on the two separately came back negative. Media reports appeared to indicate that participants were recruited by getting one recruit and asking him or her to get his or her friends involved. It’s not clear how the participants were blinded as to whether they received the vaccine under scrutiny or placebo — a potential source of bias, though hopefully this will be clarified when the study is formally published in a respectable journal.

As for the thorny issue of chance, Martin Robbins over at layscience.net has done some analysis on the numbers being reported. We hear that 74 people out of 8,198 recipients of the placebo went on to contract HIV, compared with 51 people out of the 8,197 recipients of the vaccine. The headline figure being touted around (about 4 times in the space of 3 seconds in this Bloomberg report) is that this equates to a reduction of 31.2%. Other sources have quoted this as “30%”, or “one third”. Whichever way you report it, this figure is a red herring.

What matters is the p-value — the end result of any statistical test — which tells you the probability that the results you observed, or those that are more extreme, could have occurred by chance. Based on Martin’s analysis (and verified by another commentator), the resultant p-value comes out at 0.048, or 4.8%.

Now here’s where the problem lies. Convention has it that if a p-value is less than 0.05 (or 5%), then it has attained “statistical significance” and therefore we can call the test a success. But this is simply an arbitrary value. Why should a p-value of 0.048 be treated so hugely differently to a p-value of 0.052? The p-value represents a sliding scale of evidence strength, not a means to dichotomise results into those which are significant and those which are not, and its inventor never intended it to be used that way. At least, so argue epidemiologists such as Jonathan Sterne and George Davey Smith, and I very much agree with them.

So our p-value of 4.8% can really at best be considered moderate evidence that the results are not due to chance, and even then, concerns about the trial design will further weaken any conclusions that can be drawn. Furthermore, news reports suggest that around 30 other trials of HIV vaccines have been carried out, unsuccessfully. That this one has “attained significance” has been seen as an astounding breakthrough. Not really. If you roll a 20-sided die around 30 times, you wouldn’t be too surprised if you’d rolled a 1 at least once. If we’d obtained a lower p-value, I’d be more impressed.

So, in looking through the details of this trial (those that have emerged so far, at least), we can at best be cautiously optimistic, and thoroughly sceptical at worst. It feels like such a shame to have to be the one who rains all over this parade, but far worse than shattering optimism is providing false hope to vulnerable people, such as those highly at risk of contracting HIV. The media, in my opinion, have over the last few days done too much of the latter with hype and hyperbole without stopping to consider the facts.

Update (27th September 2009): “Dr. No” has shed more light on the numbers. Sadly, it really doesn’t make for impressive reading. Here are the main points of what he has to say:

• The “31.2% reduction” is the most flattering way of presenting the numbers, and even then, it’s not that impressive.
• A second way of looking at the figures is the absolute risk reduction. This figure describes the reduction in the number of infections in the group that received the vaccination compared to the group that didn’t as a estimated proportion of the total population. In other words, the proportion of the population that would be “saved from infection if the vaccination works”. This is only 0.3%. Very low indeed.
• A third (and perhaps more meaningful) way of looking at the numbers is the “number needed to treat”. This is an estimate of the number of people who would need to be vaccinated to save one person from becoming infected (again, if the vaccine works). This weighs in at a whopping 357 people. Let me make like an evangelical preacher for a second and say that again. 357 people. To save one person from being infected.

Again, it’s wonderful that people have the magnanimity to devote their lives to such life-saving research, and it feels like such a shame to be the party-pooper, but we really must see these figures in order not to get ahead of ourselves, not to let our vanity jump to conclusions that are not really justified, and not to give false hope to those at risk from contracting HIV.

Update (10th October 2009): Science Magazine have just reported on the study, now that independent analysts have had a chance to look at some of the data in detail. They are not best pleased.