{"id":62868,"date":"2016-01-04T15:36:01","date_gmt":"2016-01-04T20:36:01","guid":{"rendered":"http:\/\/1boringoldman.com\/?p=62868"},"modified":"2016-01-04T15:37:28","modified_gmt":"2016-01-04T20:37:28","slug":"outcome-switching","status":"publish","type":"post","link":"https:\/\/1boringoldman.com\/index.php\/2016\/01\/04\/outcome-switching\/","title":{"rendered":"outcome switching…"},"content":{"rendered":"
In our reanalysis of Paxil Study 329<\/font> [Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence<\/a>], we challenged the fact that the published paper reached it’s conclusions about efficacy based on an analysis of outcome variables that were not included in the a priori<\/font><\/em> protocol:<\/div>\n
\n
There were four outcome variables in the CSR and in the published paper that were not specified in the protocol. These were the only outcome measures reported as significant. They were not included in any version of the protocol as amendments [despite other amendments], nor were they submitted to the institutional review board. The CSR … states they were part of an “analysis plan” developed some two months before the blinding was broken. No such plan appears in the CSR, and we have no contemporaneous documentation of that claim, despite having repeatedly requested it from GSK…<\/div>\n

<\/p>\n

… although investigators can explore the data however they want, additional outcome variables outside those in the protocol cannot be legitimately declared once the study is underway, except as “exploratory variables”—appropriate for the discussion or as material for further study but not for the main analysis. The a priori protocol and blinding are the bedrock of a randomised controlled trial, guaranteeing that there is not even the possibility of the HARK phenomenon [“hypothesis after results known”]<\/font><\/strong>…<\/div>\n<\/blockquote>\n
Their claim was that these variables were declared before the blind was broken, ergo they were a priori<\/font><\/em>. We had no evidence that was true, but even if it had been, we would’ve challenged using those variables anyway. When billions of dollars in commercial profit are on the line, taking a claim like that on faith is irrational. We couldn’t prove they "peeked" along the way, but the burden of proof is on their shoulders, not ours. The only guarantee that the final analysis is not jury-rigged is to insist on the analysis as declared in the a priori protocol. But what if they made a mistake in the protocol? Too bad. Do another trial. The whole point of an a priori<\/em> protocol is to eliminate even the possibility of outcome switching<\/em><\/font><\/strong> [as it has come  to be called]:<\/div>\n
\n
How researchers dupe the public with a sneaky practice called "outcome switching"<\/a><\/div>\n
Vox: Science and Health<\/div>\n
by Julia Belluz<\/div>\n
December 29, 2015<\/div>\n

For years, the drug company GlaxoSmithKline illegally marketed paroxetine, sold under the brand name Paxil, as an antidepressant for children and teenagers. It did so by citing what’s known as Study 329 — research that was funded by the drug company and published in 2001, claiming to show that Paxil is "well tolerated and effective" for kids. That marketing effort worked. In 2002 alone<\/a>, doctors wrote 2 million Paxil prescriptions for children and adolescents. Years later, after researchers reanalyzed<\/a> the raw data behind Study 329, it became clear that the study’s original conclusions were wildly wrong. Not only is Paxil ineffective, working no better than placebo, but it can actually have serious side effects, including self-injury and suicide.<\/p>\n

So how did the researchers behind the trial manage to dupe doctors and the public for so long? In part, the study was a notorious example of what’s called "outcome switching" in medical research. Before researchers start clinical trials, they’re supposed to pre-specify which health outcomes they’re most interested in… <\/p>\n

… "In Study 329," explains Ben Goldacre, a crusading British physician and author, "none of the pre-specified analyses yielded a positive result for GSK’s drug, but a few of the additional outcomes that were measured did, and those were reported in the academic paper on the trial, while the pre-specified outcomes were dropped." <\/p>\n

These days, it’s easy to see whether researchers are engaged in outcome switching because we now have public clinical trials registries<\/a> where they’re supposed to report their pre-specified outcomes before a trial begins. In theory, when journals are considering a study manuscript, they should check to see whether the authors were actually reporting on those pre-specified outcomes. But even still, says Goldacre, this isn’t always happening.<\/p>\n

So with his new endeavor the Compare Project<\/a>, Goldacre and a team of medical students are trying to address the problem. They compare each new published clinical trial in the top medical journals with the trial’s registry entry. When they detect outcome switching, they write a letter to the academic journal pointing out the discrepancy, and then they track how journals respond. <\/strong>I spoke to Goldacre to learn more. <\/p>\n

"When we get the wrong answer, in medicine, that’s not a matter of academic sophistry — it causes avoidable suffering"<\/p>\n

Julia Belluz: Why does outcome switching matter?<\/font><\/strong><\/p>\n

Ben Goldacre<\/font><\/strong>: This is an interesting example of a nerdy problem whose importance requires a few pages of background knowledge, and that’s probably why it’s been left unfixed for so long. But in short: Switching your outcomes breaks the assumptions in your statistical tests. It allows the "noise" or "random error" in your data to exaggerate your results [or even yield an outright false positive, showing a treatment to be superior when in reality it’s not].<\/p>\n

We do trials specifically to detect very modest differences between one treatment and another. You don’t need to do a randomized trial on whether a parachute will save your life when you jump out of an airplane, because the difference in survival is so dramatic. But you do need a trial to spot the tiny difference between one medical intervention and another. When we get the wrong answer, in medicine, that’s not a matter of academic sophistry — it causes avoidable suffering, bereavement, and death. So it’s worth being as close to perfect as we can possibly be… <\/p>\n<\/blockquote>\n

First off, three cheers for Ben Goldacre’s Compare Project<\/a>! I’m glad to see our Paxil Study 329 findings put into general use in this regard. Of course I didn’t believe the GSK explanation for outcome switching in that study the first time I read it, but the RIAT Initiative<\/a> wasn’t intended to be a critique or an indictment – rather an accurate republication. So we didn’t rest our argument on inuendo or inference – we stuck to the facts. I would recommend Ben’s army do the same. And Godspeed! I wrote them and suggested they include any protocol change in their project, including Income Switching<\/font><\/strong><\/em> [see the next post]…<\/div>\n","protected":false},"excerpt":{"rendered":"

In our reanalysis of Paxil Study 329 [Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence], we challenged the fact that the published paper reached it’s conclusions about efficacy based on an analysis of outcome variables that were not included in the a priori protocol: There were […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[5],"tags":[],"class_list":["post-62868","post","type-post","status-publish","format-standard","hentry","category-opinion"],"_links":{"self":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/62868","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/comments?post=62868"}],"version-history":[{"count":16,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/62868\/revisions"}],"predecessor-version":[{"id":62884,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/62868\/revisions\/62884"}],"wp:attachment":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/media?parent=62868"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/categories?post=62868"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/tags?post=62868"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}