nontoxic.org.uk

I can hazard cheezburger?

michaelgrayer — Thu, 17 May 2012 14:18:58 +0000

(alternate title: What’s a hazard ratio?)

Today I was faced with the not-entirely-straightforward task of explaining what a hazard ratio is. It’s a measure that pops up quite a lot in epidemiological statistics, when you’re conducting a study into survival times. It’s not immediately obvious, however, what it is, and when faced with the inevitable “so, what’s this hazard ratio all about then?” question, I struggled to find a good answer.

Wikipedia’s explanation is, frankly, rubbish (paraphrasing: “a hazard ratio is the ratio of the hazards”). Other attempts to explain it that I found online were similarly confusing. I found a very nice explanation online in the Pharmaceutical Statistics journal, which sadly falls down on two counts:

it’s still too long for “30-second elevator speech” purposes, and
it’s paywalled, so it’s useless to non-subscribers.

So, armed with this paper, I came up with the following even shorter explanation, which I now share with t’interwebz. I wrote it in the context of a “time-to-death” analysis, though survival analysis can be used to analyse times to almost any event you choose, not just mortality.

First of all, it would probably help to begin by explaining what a hazard is.

The hazard can be thought of as the instantaneous risk of dying at a given time point. This may vary over time, though how the hazard rises and falls over time is usually of secondary interest.

A survival analysis compares hazards between different groups of subjects in the study. One of the assumptions I made in this analysis is proportional hazards (you might encounter the phrase “assuming proportional hazards” in the epidemiological literature quite a bit). This means that we assume that although hazards may be different in different groups, and might for all we care rise/fall willy-nilly over time, when the hazard from one group changes over time, the hazard in all the other groups change by the same proportion.

In other words, the ratio between the hazards remains constant over the course of the entire study, even though the hazards themselves change over time. It’s this ratio that is the hazard ratio, and it summarizes mortality differences along the whole length of time being analysed.

I think that gives you the gist of it anyway. Feel free to rehearse it, tweak it, and drop it into your talks and lectures.

Crisis of confidence

michaelgrayer — Wed, 21 Dec 2011 23:54:00 +0000

So Crisis, a UK-based charity supporting the homeless, brought out a report highlighting differences in mortality between the homeless and the UK population in general. The headlines that accompanied it made shocking reading. “Homeless people in the UK revealed to have life expectancy of just 47″ from the Guardian is a typical example, highlighting the 30 year difference between that and the equivalent figure for the UK population. More on the media later.

Let me come clean and state my conclusions right from the off. There will be swearing.

Bullshit.

It’s not the life expectancy of the homelessness that’s 30 years lower than the UK average. It’s the average age at death. Now I know that that sounds like I’m splitting hairs over a trivial difference. Trust me, I’m not, as I will now explain.

Average age at death is just that. You look at all the deaths that happened in a population over a given period of time, and calculate the average age at which those deaths occurred. Relatively straightforward, but completely useless for comparing mortality between one population and another.

Why? Well, it could be telling you one of several things, and you have no way of knowing which it is. One explanation is, yes, that mortality is worse in the homeless population than in the population in general. But that (contrary to what the authors and Crisis seem to conclude) is not the only possible explanation. Another reason is that the alive homeless population could well be younger than the population in general.

If that explanation holds water, then another plausible narrative exists, other than the doom and gloom bandied about in the Crisis press release. This could well be an indicator of the success of charities such as Crisis and Shelter, and their less famous and more local equivalents, in tackling homelessness, so that those who are homeless are, to a considerable extent, getting rehoused and being helped back on their feet again. Infuriatingly, “average age at death” doesn’t tell you that, because it doesn’t take into account differences in the age-structure of the population. At all. Life expectancy does. And that’s not a trivial difference.

Consider the following question: what’s the average age of death among students? Now, I don’t know the exact figure, but I’d hazard a guess that it’s somewhere in the low 20s. Does that mean we should be getting hysterical about the 50 year gap between mortality amongst students and the UK population? Of course not. Thankfully, very few deaths occur among students. However, those that do, occur disproportionately to those in their twenties… because students are in their twenties. If you compare two populations with wildly different age-profiles, you have to account for those differences, otherwise you’re comparing apples with oranges and will draw silly conclusions.

A number of things really anger me about this study. One is that, well, in my opinion at least, every stage of this research and the chain of its dissemination failed somewhat. The researchers compared an indicator that was never ever fit for the purpose they were using it for, although they did refrain from calling it life expectancy, and did include a number of caveats explaining that perhaps “average age of death” isn’t quite as reliable an indicator as all that, and probably shouldn’t be used. But, infuriatingly, they went ahead and did it anyway, and in doing so they (probably unintentionally) set the trap. The Crisis press release took this one stage further, by continuing to avoid using the word life expectancy (good!) but removing all the caveats (arrgh!), thus priming the trap and making the bait look more attractive.

The media then snatched the bait, and went about reporting the “findings” with merry abandon. I’ve already mentioned the Guardian above, and the BBC and Telegraph‘s equivalent fervour about the shocking disparities in “life expectancy” are two more examples. One of the popular themes in these stories was comparing the figure to the life expectancy of the Democratic Republic of Congo, which the Telegraph and Guardian both did. Way to trivialise the situation in a country torn by civil war, with a per capita income of a paltry $280 a year, where malaria and pneumonia contribute to an under-5 mortality rate of a whisker under 20%. Classy, chaps, truly classy.

It gets worse. Even NHS Choices got it wrong. NHS Choices. Including the comparison to DR Congo. FFS. You should know better. But in some senses, you can’t fully blame the media. After all, a big, juicy, gossipy story was dangled in front of their noses.

But what really infuriates me is that despite all this, homelessness may still actually be a serious problem in the UK’s towns and cities, and actually, as I said before, a plausible story emerging from this data is that Crisis et al do a good job in ensuring that the homeless don’t remain so right through into old age. Some of the sub-plots that emerge in the report are on equally (if not more) dodgy ground. The narratives blaming drink and drugs as the main killers are based on data that are similarly unfit for purpose, for similar reasons. The data simply do not show that. I worry that those sub-plot conclusions are driven more by pre-conceived opinions about the habits of the homeless rather than actual data, and that they need to be challenged with proper evidence. That’s dabbling in speculation beyond my area of expertise though. I’d welcome anyone who actually knows more about that to enlighten me.

If you’re going to try and win hearts and minds, please do it with good data, fit for purpose, and construct narratives that actually describe what the data show. These kind of PR pieces may tug enough emotions to get people to part with some cash this holiday season, but do it at the expense of reinforcing prejudices and trivialising other problems. My Christmas donation this year will be to Cancer Research UK. They do a spectacular job on research into this horrific disease (or rather, very large group of diseases), often with terrific selflessness, passion and determination, and don’t resort to PR nonsense to big themselves up over Christmas (in fact they make a bloody good effort to combat it). I encourage you to do the same.

Merry Christmas.

How random variation fucks about with your presence of mind

michaelgrayer — Fri, 02 Dec 2011 13:32:48 +0000

This is a table I knocked up in Excel (other spreadsheet programs are available) to show how even a little bit of random variation can make spotting underlying trends really difficult, particularly if we look at data only selectively in little bits.

The data could represent a whole range of narratives; tumour size in a cancer patient (e.g. in testimonials of the kind produced by the chap who instigated this story), body weight during a dieting programme, global temperature (on a rather different timescale!), or just about anything to do with sports or the stock market (as XKCD would attest to).

Column 1 is time. I’ve called it “Days”. Frankly, depending on context, it could be anything: hours, months, years, decades.

Column 2 is the obvious trend with no random variation added. Our outcome (whatever it is) starts at 5 and goes up by 0.1 each day.

Column 3 is exactly the same but with some random noise added. Each data point is adjusted at random by anything up to 0.5 in either direction.

Column 4 is an edited version of Column 3 with some, er, rather optimistic comments added. They’re not unreasonable conclusions, based on those little data snippets.

Moral: if we really want to find a particular pattern in noisy data, we’ll find it. It doesn’t mean we’ve found the signal rather than the noise though.

Update: Tables are a bit visually crap, so here’s the same data in a graph:

Margins for error—let’s see more of them please!

michaelgrayer — Wed, 02 Mar 2011 16:57:01 +0000

Margins for error are an immensely important part of data analysis, yet are frequently ignored or misunderstood. When we make guesses, it’s very impossible to be completely certain about our guess, so they’re usually a “ballpark figure”. But similarly, making a guess isn’t usually an admission that we haven’t got the faintest idea and that we’re plucking a number at random. The question is not only “what’s the ballpark figure?” but also “how big is the ballpark?”.

Errors come in all forms. Possibly the simplest is rounding error. It’s tempting to think of seeing a figure of, say, £7.8million rounded to one decimal place and think “a-ha! this project cost exactly £7,800,000″ but that isn’t quite the case. To write that the project cost £7.8million actually means to say that it cost somewhere between £7,750,000 and £7,849,999.99. That’s a whole range of £100,000 in which the exact cost could lie. Often rounding errors don’t lead to misleading figures by themselves, but add up a lot of them and the margins for error can soon mount up.

I say often they don’t mislead on their own, but last week there was a headline that did exactly that. “The UK economy shrank by more than previously thought during the last three months of 2010″ reported the BBC, with similar stories in Reuters and other places. It turns out that this was due to a revision in GDP growth figures from -0.5% to -0.6% for the fourth quarter of 2010. But presented like that, we actually have very little idea what the actual extent of the revision was. It could have been from -0.54999% to -0.55001%, and would still have come up as a revision from -0.5% to -0.6%. In other words, there could have been hardly any movement at all. To be fair, by the same token it could also have been a revision of nearly two percentage points, but the point still rests: a difference of 0.1 in two figures that are rounded to one decimal place could still actually mean (to all intents and purposes) no difference at all.

Other sorts of error exist. Another one that applies to the GDP revision figures is standard error: given that we’re guessing the value of a figure based on a sample, what sort of margin for error do we expect from that? Was GDP shrinkage of 0.6% within the margin for error of the first guess? Without any information on this, it’s impossible to tell.

Another example of where knowing about a margin for error would be terribly useful but is frequently omitted as though it doesn’t matter is in the use of technology in sports officiating. A prime example arose in England’s World Cup cricket match with India earlier this week. England batsman Ian Bell was given not out despite the prediction software predicting that the ball which struck his pad would have gone on to hit the stumps. The reason for this was that the ball struck him more than 2.5 metres away from the stumps, and therefore was deemed “too far away” for the accuracy of the system to be trusted.

This rule, to a non-statistician, looks utterly ridiculous. In fact, to me, as both a statistician and an occasional sports referee, this also looks ridiculous but for different reasons. It’s all to do with margins for error.

The Hawkeye system (and similar technologies) work on a basic statistical principle. I couldn’t comment on the amazing technological wizardry they use to collect the data, but essentially, they collect lots of data on where the ball was at a lot of moments in time, judge when the ball hit an obstruction (in this case, Ian Bells leg-pad) and use that data to predict where the ball would have gone had it not hit that obstruction. There are an awful lot of moments where errors can creep into the analysis—and they will creep in. That part isn’t in question. The question is how big or small the accumulative effect of all those errors actually is. Are we talking micrometres or centimetres?

Firstly, errors can creep in during the data collection process. This process takes place between the point at which the ball lands on the floor and the point at which it hits the pad. This is the data that is subsequently used to predict the path of the ball. There will be some sort of margin for error for each time the tracking device detects where the ball is; the more times the tracker is able to do this, the more these errors will be smoothed out. In fact, there is a second rule governing when the results from the software may be called into question: the distance between where the ball hits the pitch and where it collides with the pad must be less than 40cm.

Secondly, the software has to know when to stop collecting data and start predicting. In other words, it has to accurately be able to predict when the ball collided with the batsman’s pad.

Finally, the software then has to use the data it has collected to churn out more data as to what the flight of the ball would have looked like had someone’s leg not got in the way. This means that the software would have had to apply some kind of function (I’m not a physicist, so I have no idea what that would be) to the data collected in the first stage, in order to get the predicted flight of the ball, and decide whether it would have gone on to hit the stumps. Errors may creep in here, as the function used will only be an approximation of what would have happened. Furthermore, any errors that crept in during the first two stages will be exacerbated: if the margin for error was 1mm based on the data alone then this margin will have crept up to several millimetres by the time the ball’s predicted flight gets to the stumps. This may not seem much, but given that the ball is only about 70mm wide, that’s a reasonable amount of doubt.

So it seems as though the rules-makers have brought in these “40cm” and “2.5m” limits in to try and account for margins-for-error. This is a case of right idea, wrong way of achieving it. In Ian Bell’s case, the predicted flight of the ball would have almost hit the dead centre of the wicket. Are we to assume that there is more doubt in this case than if the ball had struck him 2.4m away from the wicket and been predicted to simply graze the edge of the wicket?

The trouble is, without actually knowing the extent of the margin-for-error, there’s very little the rules-makers can sensibly do to account for it.

So anyway, back to journalism. Statistics, particularly when they’re based on guesses, need to have some kind of margin for error associated with them. It doesn’t even need to be that technical, just creating awareness that single figures might be complete guesses, subject to very rough rounding, or actually completely robust, and we as readers are left wondering which are which.

Academic bloat

michaelgrayer — Tue, 08 Feb 2011 10:43:34 +0000

As I’m coming up to the end of my PhD and the time where I finally submit that tome I’ve been writing (hopefully only three months to go!), I’m currently having to look around the job market for things to do afterwards. Yes, I’m looking for a job.

Generally this isn’t a good time to be looking for jobs, I’m told, but fortunately as a statistician looking for a research post in a university, things aren’t too bad. Except the motivation to work in a university gets utterly destroyed by today’s little gem I discovered on the academic jobs website, jobs.ac.uk.

In my search, two posts came up at the University of Birmingham’s School of Health and Population Sciences. The first one was a fairly bog-standard research fellow job—perhaps a grade above what I’d be looking for, but actually on the face of it not a bad match to my expertise. The advert is very clear about what the candidate is supposed to do, for example:

The post holder will set up, co-ordinate and manage a cohort study of new and existing chronic obstructive pulmonary disease (COPD) patients identified from 50 general practices.

The cohort is part of a programme of work which also includes a cluster randomised controlled trial to evaluate two methods of case finding for undiagnosed COPD in the community, and a series of observational and interventional studies to investigate the interrelationships between occupation and COPD. The cohort will include the recruitment of over 50 general practices in the West Midlands and the follow-up of ~2500 patients with COPD.

Great. Hard work, overseeing a whole study of over 2500 patients for a little over four-and-a-half years, but proper science. It’ll need a pretty awesome person specification to have the skills and expertise required to do that. And sure enough, the advert says so:

The successful candidate will have a good first degree in a health-related field (a PhD or equivalent experience would be an advantage), experience of managing a cohort study or similar large study in primary care/community settings, and good quantitative and qualitative research and analysis skills.

That’s pretty intense. Salary?

Starting salary in the range of £27,319 to £35,646 a year (potential progression on performance once in post to £37,839).

Hmm. Not bad, but with a PhD, and with the scale of the work that’s expected, one might expect something a little toward the upper end of that scale. All in all, on the face of it, not a bad proposition.

But wait, what’s this? There’s another advert in the same School (which seems to be the fashionably pointless en vogue name for a “Department” these days) for something called a “Research Facilitator“. Not heard of one of those before. I wonder what it is?

The Research Facilitator is a senior administrative post…

Ah. Administrative. Not really my cup of tea. Let’s continue.

…within the College of Medical & Dental Sciences Research & Knowledge Transfer (R&KT) Office…

Research and What Office?

…specifically occupying a core role within the Research Development Team. This post has particular affiliation to the School of Health and Population Sciences and associated research activity, occupying a senior administrative position within the School, with management responsibility for all research related admin activity.

So far, about half a dozen ways to say your job is to provide admin support for researchers in the medical and dental school.

The Research Facilitator within Health and Population Sciences is part of a small team of similar roles, each providing dedicated support to a specific Strategic Research Theme and having affiliation to a primary School within the College.

And you won’t be the only one, even within the School. Or College. No wait, definitely College. One Facilitator per School, several Schools per College. Makes perfect sense. Then it’s three Colleges to the Yard, Five-and-a-half yards to the Rod, Pole or Post-holder…

Following this, there is a whole load of bumph about how wonderful the School is. At the bottom of that paragraph is the only hint of what it is you’re actually supposed to do.

…the post holder will provide high level support to the Head of School and School Research Leads, supporting the development and delivery of the School’s R&KT activity/strategy.

What on Earth does that mean? Who are School Research Leads? Do we give them lots of forms to fill in? Is that what “supporting the development and delivery” of an “R&KT strategy” is? What qualifications are needed? Any ability to, y’know, actually do research?

And then, the killer blow…

Starting salary is normally in the range of £36,715 to £43,840, with potential progression once in post to £49,342 a year.

Let’s remind ourselves of the research fellow’s expected salary:

Starting salary in the range of £27,319 to £35,646 a year (potential progression on performance once in post to £37,839).

So the “facilitator” gets a higher salary than the one who actually does the research? So when I’m offered a brand new 10-page form to fill in by the facilitator (supposedly to make my research more, er, facilitated, I presume) it’s going to be a little hard not to feel at least a little bit resentful.

Here’s an idea: perhaps the facilitator vacancy could be scrapped, and a new research fellow post be created, so that, y’know, the salary bill for the department would be about £8-9k cheaper—and still more research would get done!

Arguing about religion is difficult.

michaelgrayer — Fri, 21 Jan 2011 19:59:06 +0000

Stupidly, I got myself involved in a Guardian Comment is Free thread, on an article about how the existence of beetles should give creationists pause for thought. Articles like this bring about in me a very mixed reaction. On the one hand, the author is quite right: the evolution of the beetle should give creationists pause for thought.

On the other hand though, whether it should or not is immaterial, given that it almost certainly won’t do so.

My other issue with articles like this is that they add to the idea that religion as a whole is purely about a literal interpretation of a holy book, and as such religion is automatically wrong. This almost entirely ignores the fact that (fundamentalism aside) religion is more about the shared experience of people and communities as human beings, and the interpretation of religious texts in, say, a sermon or address should sit firmly within the context that that shared experience brings about. I tried to explain this but found it increasingly difficult, annoying and time-consuming trying to patiently address the many (in my view fallacious) comments that were made.

The argument soon derailed into a whole stream of disparate sub-arguments. Trying to address them all was nigh-on impossible. Truth be told, this was probably because several people all had something to say at once, so even though an individual only had a few points to make, the whole thing ended up as a giant collective Gish Gallop.

What was also galling was that (as the article referred to the story of Noah) I couldn’t seem to move the argument beyond what other people perceived the literal interpretation of that story to be. When I suggested that the point of the story was as background to the “covenant” between God and Noah, that God would never again attempt to destroy sin by destroying humanity, this was simply thrown back. No, this story is clearly about God being a mass murderer, and arbitrarily saving one family whilst slaughtering the rest. No other interpretation was valid. Obviously.

The irony of this is that this is exactly the sort of rhetorical device a fundamentalist would use, only in attempted support of the exact opposite argument. Take the story, out of context, don’t read it in relation to contemporary human experience, insist that your interpretation is the right one because it is written so in the Bible. Conclusion: if you do/don’t (delete according to which side you adopt) follow [insert religion here, or even religion as a whole], you are a terrible person!

I don’t think this sort of argument helps anyone, really. A more useful debate that arises from this story could be a discussion about what such a covenant would imply: does it suggest that getting rid of sin completely is impossible and has undesirable consequences? What does that mean for our daily lives? Or something else? Let’s talk about it!

Another commenter raised the point about finding beauty and comfort in science, hence not needing religion. My opinion is that I see beauty and comfort in both. And not only beauty and comfort: challenge, discomfort, grief, angst, joy, frustration, excitement and many other things. Viewing religion purely as a “comfort” strikes me as somewhat patronising.

I hope to address some of the other points that were raised at a later date. I haven’t blogged too much about religion yet, certainly not as much as I’d like. Religion is something that I think receives a lot of criticism which is fair for a particularly heinous subset of it, but not the whole of it. My overall impression is that when it comes to taking a dim view of the societal impacts of religion, people’s (often excellent) standards of evidence slip drastically.

That the factual accuracy of events reported in the Bible doesn’t stand up to scientific evidence is not in question here. But does it then follow that religion is an entirely evidence-free discipline that merits the beating up it gets? That is more open to debate.

A battle of red herrings: let’s move on from over-population

michaelgrayer — Fri, 29 Oct 2010 15:58:40 +0000

Tomorrow, at the Upper Gulbenkian Gallery at the Royal College of Art, the 2010 “Battle of Ideas” debate will be taking place, on the subject of “overpopulation”. The theme of the debate will be “The great population debate: too many carbon footprints?” and the speakers will be Roger Martin of the Optimum Population Trust and Brendan O’Neill, editor of Spiked magazine. The debate promises to be one where one speaker presents a doom and gloom scenario (à la Thomas Malthus) about how dreadfully over-populated the planet is based on some back-of-an-envelope ecological footprint calculations, and the other speaker claiming that the first speaker is scaremongering and that we’ve heard these predictions of disaster before, but they’ve never come to light, so we shouldn’t be worried, and population control is coercive and unethical.

Sigh.

Neither side has it completely right. Reluctantly, if I had to pick one team to side with, it would have to be the OPT’s Malthusian view that curbing population growth is necessary, but probably not for their stated reasons. Endless population growth, of course, is unsustainable. The neo-liberal “people should just do as they please and we are here to propogate and pro-create” argument doesn’t hold water indefinitely. But I only agree with them in principle—the evidence that we have passed that limit of sustainability already is, frankly, poor. The ecological footprint calculations that the OPT produce are based on current levels of per capita consumption and carbon emissions: it is these that need to be reduced rather than population numbers per se. Monbiot, writing in the Guardian, gives an excellent précis of this argument.

Furthermore, OPT tends to over-egg the pudding when it comes to its proposed methods of curbing global population growth. The key driver of population growth is fertility, which can be brought under control through family planning policies, as OPT rightly points out. But oddly, OPT also targets teenage pregnancy in the UK as a major driver of population growth. In doing this, OPT muddles up the concepts of global population growth with country-specific population growth. The two are very different. Not only that, but teenage fertility isn’t even a major contributor to total birth rates in the UK, and total fertility is below replacement level in the UK anyway.

Taking that mix-up still further, OPT also targets migration in its policies to reduce population growth. It is true that population growth in the UK has been, in recent years, largely driven by net inward migration. But why this is of relevance to global population growth is anyone’s guess, since the people in question are alive already. Nevertheless, OPT devotes a lot of attention to arguing for “zero net migration”. It argues that this should be achieved by bringing immigration down to the level of emigration. It strikes me that the same could be achieved by raising emigration to the level of immigration.

I could go on, but in fact, all of this is a largely irrelevant sideshow, as there are some more fundamental issues that have been missed. Teenage fertility is of little consequence to global population growth. Migration is a major driver of population growth in some countries, but not globally, so “raise the drawbridge” policies of the type championed by OPT are of no consequence to global population growth at all. High fertility rates in poor countries are what currently drive population growth, but these are in countries where carbon footprints are tiny in comparison with richer nations, so are irrelevant to the question of carbon emissions. The population growth issue is therefore not an environmental concern, so should not be championed as such. Doing so deflects attention away from the real issue, which is changing behaviour so as to reduce carbon footprints of rich countries.

There are, however, very good reasons for promoting family planning and reduction of fertility in poor countries, but these are reasons of social justice, poverty eradication, women’s rights, and development, rather than environmental degradation. This is where I feel O’Reilly gets his argument wrong, as he overlooks both the non-environmental benefits of family planning and the fact that reduced family size is in fact a consequence of raised living standards, rather than a cause. Hans Rosling argues this point in many of his global development lectures. Here are two examples.

So this “overpopulation” debate looks set to be a squabble between “all these extra people are gonna suffocate our planet!” and “it’s my human right to procreate!”, neither of which really gets to the heart of the issues regarding population growth. The problem of treating population growth purely as an environmental issue is two-fold: it serves as a distraction from real environmental problems, and it fails to delve deeper into the underlying problems of which population growth is a consequence. Let’s move on from this battle over red herrings.

“More research is needed.” No it isn’t.

michaelgrayer — Fri, 08 Oct 2010 13:56:18 +0000

A couple of months ago, at around 11:30pm one evening, I had an idea for a journal article. The next morning, I started writing it, and by 11:30pm that evening, working non-stop, I’d finished. The article was in the form of a systematic review: a type of study that usually takes months of painstaking study of a multitude of medical databases. This one was somewhat less grand in scale: it took a mere 24 hours from inception to first draft. I then told the delightfully helpful Adam Jacobs from Dianthus Medical Ltd about it on Twitter, and he agreed to take a look at it. Less than a week later, following his suggestions, I had a manuscript ready to be submitted for publication in a scientific journal.

The reason for this particular review taking so little time to complete was that instead of painstakingly poring through large numbers of scientific articles in the analysis, my analysis took place on a grand total of zero articles. The reason for this was that in my analysis, I searched for literature on an utterly implausible intervention for treating a completely fictional disease, whose mechanism for working was based on an utterly ridiculous and made-up theory with absolutely no basis in reality.

My guess is that this is also the reason why my manuscript was rejected, which, I guess, is fair enough. But oddly enough, systematic reviews of this type do exist, and do get published in medical journals. Mostly, to be fair, the ailments involved in these studies are real. But the interventions and the underlying theories that describe how they are supposed to work are, well, real in the sense that some people have been known to try them, but based on little more than an odd combination of folklore and belligerence. I am, of course, talking about systematic reviews of complementary and alternative medical products and services.

Edzard Ernst, from the Peninsula School at Exeter University, is involved in the writing of many of these. In fact it was through a message on his Twitter account drawing attention to one such review that I got the idea to write this article. In conducting these reviews, he and his colleagues are doing a fantastic job in synthesising the scientific literature on a whole range of products and ailments. Occasionally, for very particular pairings of intervention and ailment (such as osteopathy and lower back pain, or St. John’s Wort and depression), they find that the evidence supports the claims. More often than not, however, the evidence does not back up the claims made—even if individual studies, when cherry-picked out of the body of literature, appear to be supportive.

Overall though, there is one criticism I have of systematic reviews of CAM treatments (and this is what I aimed to highlight with this article): it’s that they tend to say that “more research is needed” when faced with negative evidence, when in fact, the reverse is true. This problem is usually highlighted in the introductory paragraphs which describe the disease, the treatment, and how it works in theory. This last bit is the crucial part, since very little attention is paid to it in the writing of the review. This strikes me as odd: anyone could just make a theory up out of thin air, perform a systematic review, find no clinical trials investigating this made up theory, and conclude that more clinical trials are needed. This is what I’ve done in this article.

I argue that this is also exactly what has been done in some of these systematic reviews, and that this approach risks conferring a sense of legitimacy (via a “the jury is still out” message) onto products and services that really don’t deserve it.

Enough of the introductions, though. Here is my article, in self-published form. Thoughts and comments welcome.

Speed cameras don’t cause road casualties. Tell your friends.

michaelgrayer — Thu, 22 Jul 2010 18:26:25 +0000

Updated 23 July 2010.

Last week, the Taxpayers’ Alliance and the Drivers’ Alliance brought out a statistical report claiming that the introduction of speed cameras had failed in reducing the number of road accidents. A number of choice quotes about how speed cameras were nothing more than money-making conspiracies and how this analysis proved that speed was not the main cause of road accidents accompanied the press releases of this report.

The report was again featured on the BBC One O’clock News this lunchtime on a story about Swindon council’s decision to axe speed cameras in the town, in the context of budget cuts in local government. Quite how this made it into today’s news is anyone’s guess, given that Swindon council made that decision two years ago. [Update - the news item is now up on the BBC website.]

Anyway, what leapt out at me was a shot of a local resident [update: actually, on seeing the report again, it turns out to be the reporter] being shown a graph that claimed to show that the introduction of speed cameras had actually slowed the rate of decrease in the number of traffic accidents in the UK.

After a bit of digging, I found the report and the graph in question. Here it is:

At first glance this looks pretty convincing. There’s a really obvious red dotted line denoting when speed cameras were introduced, the slope of the green line (showing the data) looks different before 1990 to afterwards, and look! there’s a nice blue dotted line showing the pre-1990 line extrapolated beyond 1990 and it’s way lower than the actual data.

Does this prove that speed cameras have reduced the rate at which road casualty rates have been decreasing? Of course it doesn’t.

Helpfully, the report’s authors had cited the source of their data. One Google search later, I found the data upon which this graph was based. They come from the Department for Transport’s statistical yearbook “Transport Statistics Great Britain” (an imaginitive title if ever there was one; I’m sure it makes great bedtime reading). Having found the data (and to be fair, the latest edition of the yearbook has the road casualties data for 1956-1982 wiped out due to an editorial booboo; fortunately the 2008 edition doesn’t have that same cock-up) I repeated the same analysis myself (a linear regression, if you must know), and got the same line as in the TPA’s report. So far, so good.

But then I stopped and thought for a moment. What had the author (and now myself, separately) actually done, and was it a fair way of describing the data? They’d looked at the graph, noticed a kink around 1990, and compared the actual data with what would have happened had the earlier trend continued. So what would have happened? To cut a long story short, this trend line is rather optimistic, to say the least. It suggests that there would be no casualties from road accidents at all by the year 2012.

Something else bothered me. I had data from 1952 sitting in front of me in the yearbook. Why had they not included it in their analysis? Here’s the same data, extended back to 1952. [Just noticed an error - the Y-axis title should read "per billion passenger-km", not "per passenger-km". This error also appears in the TPA's graph. It doesn't really change the analysis though.]

This would suggest that 1978 would be a bit of a misleading point to start the graph from – the data point for 1978 (the first red line) is higher than the overall trend based on the 5-or-so years either side of that point. So, any decrease in the rate from that point will be faster than if you’d started your analysis elsewhere.

However, more to the point, the levelling-off from 1990 (the second red line) appears entirely consistent with the rest of the historical trend when taken back to around 1960: from 1960, road casualty rates started to plummet, then over the course of the next few decades, they’ve started to level off. This happened until around 2000, when they started to drop again. Far be it for me to suggest that this might perhaps have something to do with the introduction of speed cameras, which were introduced in the early 1990s, but weren’t widespread until the late 1990s…

Before writing this post, I emailed Jennifer Dunn, the contact person for methodological questions on this report at the Taxpayers’ Alliance, with a couple of my concerns. I was interested in why they’d based their regression on data from 1978 (when data from beforehand were available) to 1990 (which was before speed cameras were in common use), and why they’d extrapolated using a straight line, even though such a technique would “predict” road casualties to be zero by 2012. I got a reply which I would describe as very prompt, reasonably polite, quite firm and very deflective. The gist of it was that they’d plotted the graph, noticed a break in around 1990, performed a statistical test I’d not heard of before (the Chow Test, for what it’s worth), confirmed from the result of the test that it was a break, and added in the post hoc justification that it was all to do with speed cameras, because they’d been introduced at roughly the same time.

I wasn’t convinced, and I’m still not. There undoubtedly issues with speed cameras, I’m sure, and I would accept that they are probably not the only method for reducing road traffic casualties. But I am pretty sure that, even if they are overused, they are particularly useful in specific areas, and discarding them outright as Swindon council have been reported to have done, is probably, on balance, foolish. But to evaluate their effectiveness, we need good evidence, properly analysed, taking all variables into account. An unrealistic extrapolation and a big red “speed cameras introduced here” sign don’t help.

Follow-up

I got another polite but firm email from Jennifer Dunn this morning. She writes:

I think the graph confirms that if we had used data from as far back as the early 1960s we would have had similar results because the road casualty rate is declining rapidly year on year. We decided to go from a later period to try and control for dramatic changes in road technology. For example in 1952 there weren’t motorways. As your graph illustrates there were breaks in the pattern earlier in the series, but we have used a period sufficient to establish a trend and see how it changed in the early nineties. This was a report about a specific road safety policy, speed cameras and not a history of road safety. We therefore didn’t take the sample as far back as 1952.

This is partly valid. It’s totally fair not to go back as far as 1952 in the analysis, because, as Jennifer says, there is a break in the pattern in the early years. But that’s still no reason not to graph it. It also doesn’t explain why they didn’t take the sample back to around 1960, which is where the current trend started. It’s partially right to say that road casualty rates are declining rapidly from the early 1960s right through to the present day, but crucially, the rate at which they are declining is already slowing down by the 1970s and 1980s.

To demonstrate this, I performed the same statistical test as in the report, the Chow Test. Helpfully, and here’s where I really have to say “fair play” to the TPA, they do give a detailed description of how to perform one in the report. Following their instructions, I re-performed the test on the two time periods they mention in their study to check that I was doing it right. The exact numbers I got were very slightly different – and by “slightly different”, I mean disagreeing about the height of Everest by a couple of inches. In other words, not enough to alter the overall result of the test. It also transpired that the TPA had compared 1979-1990 (rather than 1978-2990, as they said in the text) with 1991-2007, but this discrepancy is forgiveable and makes absolutely no difference.

So, to demonstrate that the rates were already slowing down, I performed the same test, comparing the period 1962-1978 with 1979-1990. And whaddya know? The test concluded that there was a break in the time series, and two straight lines over the sub-periods were better at describing the data than a single one across both.

My attention was also drawn to another article from last month questioning the notion of whether speed cameras were “cash cows” for public finances. You can discuss the merits or otherwise of that article with its authors if you like, but I was particularly interested by the following quote from a spokesperson from the AA:

Spokesperson Andrew Howard explained to us where he felt the ‘cash cow’ claims had come from.

Until 2000, because authorities were unable to keep the revenue for fines, it cost police money to pursue anyone caught by the cameras. It was when this system changed that interpretation of the system changed.

He said: “From 2000 onwards the local authorities could effectively get the cost of running cameras back from fine revenue.

“That was where the cash cow claims started, because people started saying that councils make more money the more people they catch.

“To some extent they did, but that was because it cost more money to catch people, therefore they had to get funding to do that.”

So, although speed cameras were introduced in the early 1990s, local authorities did not keep the revenue from the fines until the year 2000. This would imply that there might be a “shock” in the time series in the year 2000. From the graph, it would certainly appear that from the year 2000, the rate of decline in road casulaties picks up again. So I performed another Chow Test, this time comparing 1991-1999 with 2000-2007. And, lo and behold, another “significant” result: two straight lines are better at describing the data than a single one across the whole of 1990-2007. You can have a look at my calculations for this test, the other one, and my re-doing of the TPA’s in the following screenshot.

I could go on, but it would be pointless. You could pretty much break the time series where you like, perform a Chow test, get a significant result, and come up with some post hoc justification for that result, just as both the TPA and I have just done. It’s not good evidence. Neither my analysis nor theirs has directly taken into account confounding variables, and has only considered the UK rates as a whole, not focussing on specific places where speed cameras have been introduced. Better evidence is available from a systematic review from the Cochrane Collaboration. Admittedly, the evidence is still not great. But certainly better than performing econometric tests on public health data until the Chows come home.

Post-script

Much as I’d love to end this post on a bad pun, I can’t end without at least giving some thanks to Jennifer Dunn at the Taxpayers’ Alliance for her willingness to engage with my concerns and for the manner of her email correspondence. I am also impressed (even if I still disagree with her about their appropriateness) about how transparent they have been about their statistical methods, so that we can all have a look, and decide whether or not their conclusions are justified. Thank you.

It also turns out that there’s another blogpost written last week, when the report came out, describing the foolishness of the extrapolation, here.

Spokesperson Andrew Howard explained to us where he felt the ‘cash cow’ claims had come from.Until 2000, because authorities were unable to keep the revenue for fines, it cost police money to pursue anyone caught by the cameras. It was when this system changed that interpretation of the system changed.

He said: “From 2000 onwards the local authorities could effectively get the cost of running cameras back from fine revenue.

“That was where the cash cow claims started, because people started saying that councils make more money the more people they catch.

“To some extent they did, but that was because it cost more money to catch people, therefore they had to get funding to do that.”

Amendments to Mr. Tredinnick’s bonkers EDMs

michaelgrayer — Sat, 26 Jun 2010 11:46:09 +0000

Some more follow-up to the Early Day Motions (EDMs) tabled in the House of Commons by David Tredinnick MP, that I blogged on last week.

Recap: David Tredinnick MP, staunch advocate of homeopathy and sundry “alternative” therapies, tabled four EDMs in parliament earlier this week. One was a motion condemning the British Medical Association’s stance that homeopathy should no longer be funded on the NHS. The other three were motions welcoming scientific “peer-reviewed” papers which at first glance appear to support homeopathy in the treatment of moderate to severe depression, insomnia, and (most worryingly) breast cancer. On closer examination, they do no such thing. Tredinnick was also on the Today Programme on BBC Radio 4 attempting to justify his position and was soundly torn apart by Simon Singh. Tredinnick made so many nonsensical statements that Singh did well to focus his argument on the few main points (given that the debate, such that it is, was being conducted in the public rather than scientific arena). The remainder of the points are dealt with here.

So what now? Firstly, the bad news. Or rather, the deeply depressing news, that Tredinnick (along with fellow ideologue Nadine Dorries) has been elected by the Conservatives to sit on the Select Committee for Health in the House of Parliament. The reasons why the appointment of Tredinnick and Dorries to this committee is regrettable are detailed here. However, the one saving grace, as Martin Robbins points out, is that Dorries has a pretty terrible track record when it comes to attending Select Committees (though of course that comes with the usual “past performance blah blah not indicative blah future results” disclaimer) and Tredinnick’s views are so far out of step with the more sensible members of that committee that they are unlikely to gather any momentum.

And now on to the second, and rather amusing development. Julian Huppert, MP for Cambridge, has tabled some rather extensive amendments to Tredinnick’s EDMs. The original amendments, set out in the formal amendment style (and hence a bit clonky to read), are available here, but since you have to scroll down the page to get to them, I’ve put them here, with the amended bits of the original crossed out.

If the amendments are adopted, EDM 284 would now look like this:

That this House expresses concern at motions 301, 301a, 301b, 301c, 301d, 301e and 301f at this year’s British Medical Association’s (BMA) Annual Representative Meeting, which calls for no further commissioning of, nor funding for, homeopathic remedies in the NHS; believes that the BMA has overstepped its remit by making such statements without proper consultation with its own membership that practice homeopathy and, more importantly, with the tens of thousands of patients who depend on homeopathy; thinks that an integrated NHS, which employs the best from the orthodox and complementary, and which empowers patients, could deliver better and more cost-effective outcomes at a time of financial prudence; and calls on the Government to maintain its policy of allowing decision-making on individual clinical interventions, including homeopathy, to remain in the hands of local NHS service providers and practitioners who are best placed to know their community’s needs. support for the right of the BMA to express its views about the efficacy and cost-effectiveness of any putative health treatment and the appropriateness of the NHS commissioning such treatments; notes that the motions of the 2010 BMA Annual Representative Meeting dealing with homeopathy endorse the findings of the recent report from the Fourth Report of the Science and Technology Select Committee of Session 2009-10 on Evidence Check 2: Homeopathy Fourth Report of Session, HC45, namely that the evidence base is clear that homeopathy is not effective beyond placebo and that scarce NHS funds should not be spent commissioning it at a time when, due to cost, the health service is not able to provide its patients even with treatments that have been shown to be effective in clinical trials; and further welcomes the view of the BMA that healthcare should be based on good evidence of effectiveness and cost-effectiveness.

…and EDM 285, like this:

That this House welcomes the study published in the International Journal of Oncology, 2010 Feb; 36(2): 395-403 which revealed that homeopathic remedies have a beneficial effect on breast cancer cells; notes that researchers at the University of Texas conducted an in vitro study to determine whether products prescribed by a clinic in India have any effect on breast cancer cell lines; further notes that the researchers studied four ultra-diluted remedies, carcinosin, phytolacca, conium and thuja against two human breast adenocarcinoma cell lines, MCF-7 and MDA-MB-231 and a cell line derived from immortalized normal human mammary epithelial cells, HMLE; observes that the remedies exerted preferential cytotoxic effects against the two breast cancer cell lines, causing cell cycle delay/arrest and apoptosis; believes that the findings demonstrate biological activity of these natural products when presented at ultra-diluted doses; and calls for further research in this important area. as an example of the failure of adequate scientific peer review because the paper provides no statistical analysis to support any conclusion, indicates that the experimental control, 87 per cent. alcohol solution, was itself toxic to the cell cultures, does not illustrate or explain the different chromatographic profiles of the solvent and the test substances, and does not provide sufficient data to allow proper evaluation of the study; notes that the lead author has retired and runs a homeopathy website which falsely claims that homeopathy is as effective as a conventional chemotherapy agent, Taxol, in treating breast cancer; further notes that one of the authors, Alison Pawlus, has publicly disowned the paper; regrets that isolated poor-quality studies are cited by proponents of homeopathy to endorse dangerous and exploitative cancer-curing claims in the face of overwhelming weight of scientific evidence against them; and agrees with the conclusions of the Science and Technology Select Committee’s Fourth Report of Session 2009-10, Evidence Check 2: Homeopathy HC45 that putting patients through pointless further clinical trials, and the spending of scarce public sector funds on research into homeopathy cannot be justified.

…and EDM 286 would now read like this:

That this House welcomes the double-blind study conducted at the outpatient clinic at Jundiai Medical School in São Paulo, Brazil, which consisted of patients with moderate to severe depression; notes that patients were randomly assigned to a double-blind treatment with individualised homeopathic Q-potencies or fluoxetine (Prozac); further notes that the non-inferiority analysis indicated that the homeopathic Q-potencies were not inferior as compared to fluoxetine in treatment of this sample; observes that the study is the first randomised controlled double-blind trial with a reasonable number of subjects to draw conclusions about the homeopathic treatment of depression; acknowledges that homeopathy is recognised as a medical specialty in Brazil; and calls on the Government to carry out further research into this area. notes the study published in August 2009 in the journal e-Cam by UC Adler et al, conducted in São Paulo on 91 patients with depression, which claimed that individualised homeopathic treatments were not inferior to fluoxetine, a selective serotonin reuptake inhibitor; further notes that only 55 patients completed the study and that the finding of non-inferiority, while statistically significant, was marginal; further notes that fluoxetine is considered by many to be no more or barely more effective than a placebo in such patients and that this study therefore merely provides further evidence that homeopathy treatment is no more effective than placebo treatment; and supports the findings of the Fourth Report from the Science and Technology Select Committee of Session 2009-10, Evidence Check 2: Homeopathy, HC45, namely that there are ethical problems in prescribing patients placebos without full candour, that the evidence base is clear that homeopathy is not effective beyond placebo and so scarce NHS funds should not be spent commissioning it at a time when, due to cost, the health service is not even able to provide its patients with treatments that have been clearly shown to be effective, and that putting patients through pointless further clinical trials, and the spending of scarce public sector funds on research into homeopathy cannot be justified.

…and the final EDM, number 287, would look like this:

That this House welcomes the randomised, double blind, placebo-controlled study at Durban University of Technology, South Africa, which evaluated the efficacy of homeopathic simillimum in the treatment of chronic primary insomnia; notes that the measurement tools used were a sleep diary (SD) and a sleep impairment index (SII); further notes that the SD data revealed that verum treatment resulted in a significant increase in duration of sleep throughout the study and a significant improvement in SII scores, compared to the placebo treatment which resulted in no significant increase in the duration of sleep; observes that an initial improvement occurred in the placebo group but this was not sustained and that a comparison between the results revealed a statistically significant difference; believes that homeopathic simillimum treatment of primary insomnia is effective compared with placebo in this study; and calls for further research in this area. notes the study conducted at Durban University of Technology, South Africa, by Naude et al, published in Homeopathy (2010) 99 63-68, which claimed that homeopathic treatment of primary insomnia was significantly more effective compared with placebo; further notes that only 33 patients were enrolled, which makes any conclusion statistically questionable, especially given the randomisation did not take account of the distribution of any clinically relevant variables between the resulting two groups, for example any age differences; further notes that there was not full blinding with respect to the dispenser of the treatment; recognises the dangers, inherent with publication bias, in selectively citing isolated small studies of dubious significance in the absence of any demonstration that infinitely diluted and vigorously shaken solutions have a rational and effective method of action; and supports the findings of the Fourth Report from the Science and Technology Select Committee of Session 2009-10, Evidence Check 2: Homeopathy, HC45, namely that there are ethical problems in prescribing patients placebos without full candour, that the evidence base is clearly that homeopathy is not effective beyond placebo and so scarce NHS funds should not be spent commissioning it at a time when, due to cost, the health service is not even able to provide its patients with treatments that have been clearly shown to be effective, and that putting patients through pointless further clinical trials and the spending of scarce public sector funds on research into homeopathy cannot be justified.

My only fear is that if these amendments are passed, they would start to make David Tredinnick, as the original author, look very sensible indeed…