1 Boring Old Man

There were four outcome variables in the CSR and in the published paper that were not specified in the protocol. These were the only outcome measures reported as significant. They were not included in any version of the protocol as amendments [despite other amendments], nor were they submitted to the institutional review board. The CSR … states they were part of an “analysis plan” developed some two months before the blinding was broken. No such plan appears in the CSR, and we have no contemporaneous documentation of that claim, despite having repeatedly requested it from GSK…

… although investigators can explore the data however they want, additional outcome variables outside those in the protocol cannot be legitimately declared once the study is underway, except as “exploratory variables”—appropriate for the discussion or as material for further study but not for the main analysis. The a priori protocol and blinding are the bedrock of a randomised controlled trial, guaranteeing that there is not even the possibility of the HARK phenomenon [“hypothesis after results known”]…

How researchers dupe the public with a sneaky practice called "outcome switching"

Vox: Science and Health

by Julia Belluz

December 29, 2015

For years, the drug company GlaxoSmithKline illegally marketed paroxetine, sold under the brand name Paxil, as an antidepressant for children and teenagers. It did so by citing what’s known as Study 329 — research that was funded by the drug company and published in 2001, claiming to show that Paxil is "well tolerated and effective" for kids. That marketing effort worked. In 2002 alone, doctors wrote 2 million Paxil prescriptions for children and adolescents. Years later, after researchers reanalyzed the raw data behind Study 329, it became clear that the study’s original conclusions were wildly wrong. Not only is Paxil ineffective, working no better than placebo, but it can actually have serious side effects, including self-injury and suicide.

So how did the researchers behind the trial manage to dupe doctors and the public for so long? In part, the study was a notorious example of what’s called "outcome switching" in medical research. Before researchers start clinical trials, they’re supposed to pre-specify which health outcomes they’re most interested in…

… "In Study 329," explains Ben Goldacre, a crusading British physician and author, "none of the pre-specified analyses yielded a positive result for GSK’s drug, but a few of the additional outcomes that were measured did, and those were reported in the academic paper on the trial, while the pre-specified outcomes were dropped."

These days, it’s easy to see whether researchers are engaged in outcome switching because we now have public clinical trials registries where they’re supposed to report their pre-specified outcomes before a trial begins. In theory, when journals are considering a study manuscript, they should check to see whether the authors were actually reporting on those pre-specified outcomes. But even still, says Goldacre, this isn’t always happening.

So with his new endeavor the Compare Project, Goldacre and a team of medical students are trying to address the problem. They compare each new published clinical trial in the top medical journals with the trial’s registry entry. When they detect outcome switching, they write a letter to the academic journal pointing out the discrepancy, and then they track how journals respond. I spoke to Goldacre to learn more.

"When we get the wrong answer, in medicine, that’s not a matter of academic sophistry — it causes avoidable suffering"

Julia Belluz: Why does outcome switching matter?

Ben Goldacre: This is an interesting example of a nerdy problem whose importance requires a few pages of background knowledge, and that’s probably why it’s been left unfixed for so long. But in short: Switching your outcomes breaks the assumptions in your statistical tests. It allows the "noise" or "random error" in your data to exaggerate your results [or even yield an outright false positive, showing a treatment to be superior when in reality it’s not].

We do trials specifically to detect very modest differences between one treatment and another. You don’t need to do a randomized trial on whether a parachute will save your life when you jump out of an airplane, because the difference in survival is so dramatic. But you do need a trial to spot the tiny difference between one medical intervention and another. When we get the wrong answer, in medicine, that’s not a matter of academic sophistry — it causes avoidable suffering, bereavement, and death. So it’s worth being as close to perfect as we can possibly be…