1 Boring Old Man

We don’t need a lofty scientific explanation for why we should demand that Clinical Trials of medication follow [and report the results of] the a priori Protocol. Common sense and historical fact offer reasons enough. If I can regress to my monotonous diagram of the Clinical Trial process for a moment, the a priori Protocol [including the Statistical Analysis Plan] have to be both reviewed and approved before the study begins, and constitute the last verifiable insurance against things like HARking or p-hacking [see the hope diamond…] – study results potentially manipulated based on foreknowledge of the outcome:

For one thing, there’s no guarantee that a sponsor won’t just go around the blind [they’re paying the CRO doing the study] or, for another more likely scenario, select analytic techniques that produce the results they want. That’s the common sense part. The historical fact part is also self evident. This blog and many others are filled with examples, as are the court dockets. In fact, looking at the psychiatric literature of several decades, we don’t need statistics to know what happened. It’s hard to produce examples where this kind of distortion didn’t happen at some level. The recent article by Jureidini, Amsterdam, and McHenry [see the jewel in the crown… and why is that?…] just happens to be about a sample case where there were enough subpoenaed materials available to directly document the behind-the-scenes deceit [when you happen to have three researchers willing to go through thousands of pages to flag the ones that mattered].

But Dr. De Groot’s thoughtful analysis [see the hope diamond…] adds value beyond this obvious empirical evidence, even beyond the technical explanation. It goes to the heart of what statistic analyses really represent. Just because they involve numbers and formulas and generate numeric answers doesn’t mean that statistical analyses are like the familiar arithmetic, algebra, or calculus with computations producing distinct answers. In fact, if you took a statistics course, the teacher was likely a psychologist or a social scientist rather than someone from the math department. Statistical analyses are about conditional likelihoods [with the emphasis on conditional]. And De Groot is pointing out that, unlike the other mathematics, one absolute condition in confirmatory statistical analysis is blindness [with the emphasis on absolute].

• Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence
  In our paper, we were able to obtain the raw data as the result of over a decade of work coming from multiple foci. Analyzing the data using the manufacturer’s own a priori protocol we found that the reported efficacy of the drug disappeared and that the safety of the drug had been exaggerated.
• The citalopram CIT-MD-18 pediatric depression trial: Deconstruction of medical ghostwriting, data mischaracterisation and academic malfeasance
  In this case, even without the raw data, the internal corporate documents demonstrated how the a priori Protocol had been ignored, then perverted to report a positive result from this decidedly negative study.