1 Boring Old Man

The Goldacre et al COMPare study in a frenzied match of ping pong… seems straightforward. Find the Outcome Variables they said they were going to use and see if they used them. They frequently didn’t. But finding those preregistered [a priori] Primary and Secondary Outcome variables in the first place is no minor task. In fact, the COMPare team had to develop an algorithm for just that problem [see the COMPare full Protocol and the Adhering to the original plan section in the previous post]. It’s often ambiguous with either nothing to go on or several versions [Protocol, clinicaltrials.gov]. Some of the editors criticized COMPare‘s methodology. But the unclarity isn’t with COMPare. It’s with the inconsistent record keeping of the Sponsors. That shouldn’t be the problem it is. These Outcome Variables are a required part of the Protocol submitted to the Institutional Review Board that approved the study – before registration.

With the exception of the BMJ, the editors’ responses to the letters from COMPare pointing out the inconsistencies were either critical or dismissive. They talked about COMPare‘s methodology or intent, denied wrong-doing, brought up exceptions, etc. They did everything except flesh out the reason for the identified inconsistencies in their articles that COMPare was asking about. When we [Bernard Carroll, John Noble, Shannon Brownlee, the Lown Institute, and yours truly] were first beginning to work on turning Dr. Carroll’s Proposal into a Petition, I had written Peter Doshi with several questions about these matters. He responded but was similarly confused – said he was writing about this too [I presume this article]. He was finding the same mess we were trying to address.

Clinical Trials have been required by the FDA since 1962. ClinicalTrials.gov has been in existence since 1997, and required since 2007 for industry-funded trials. You would think that in this last 54 years with thousands of trials behind us, we’d have worked out something as simple as where to locate an accurate list of the a priori Primary and Secondary Outcome Variables for a clinical drug trial. But as this scenario clearly demonstrates, it just hasn’t happened [carefully note the use of the passive voice in this sentence]. "it just hasn’t happened" isn’t really accurate, in my opinion. I think it has been kept from happening using frenzied matches of ping pong similar to this one because it creates a heavily protected multi-billion dollar loophole in the system. So first, what is the loophole?

the multibillion dollar loophole

The FDA efficacy standards are simple. The Trial rises or falls based on results of the a priori Primary Outcome Variable[s]. They take Secondary Outcome Variables and other arguments into account in questionable cases, but the bedrock is a statistically significant result on the a priori Primary Outcome Variable[s]. And they require two such studies for approval. They have the Protocol and the results. They have the whole CSR and they either have or can have the whole IPD if they need it. None of the arguments the drug companies use to withhold information apply, since the FDA keeps their secrets. And the could potentially get in a heap of trouble if they lie, cheat, or steal. So although the standards for approval are set fairly low, one can generally accept an FDA approval as on the up and up. And that’s basically all they say, YES or NO. At some point down the line, they may [or may not] post their report on Drugs@FDA or you can get it with an FOIA request, but it takes a while. And I think those reports are generally only available for approvals drugs.

However, there’s no official oversight in published academic journal articles. Acceptance for publication is in the hands of the journal editor and editorial staff assisted by peer reviewers. They don’t have the CSR or the IPD. They don’t necessarily have the Protocol. What they have is the submitted article and a CONSORT statement. And of particular relevance, they don’t necessarily have a copy of the a priori Protocol guaranteed to be a priori – nor do they have any injunction to make any determination about the outcome of the trial. And so to the loophole. If a drug has been approved by the FDA, the company is only allowed to advertise it for the condition it was approved for [and advertising matters!]. Let’s say you do some trials for another indication and the FDA turns it down. You can’t advertise it for that indication, but you can submit it to a journal [with its different standards] and if it’s published, that’s a HUGE advertisement. The academic authors can present it at meetings. And the reprints from those publications can be disseminated far and wide.

Conclusions: Paroxetine is generally well tolerated and effective for major depression in adolescents.

And beyond that, in a published journal article, you can selectively present your data, make arguments that omit or discount contradictions, enlist the skills of trained statisticians and writers to lean the material in your direction. You can make it desirable to publish by ordering a gajillion reprints [the ones that go far and wide]. All you have to do is get it published [and having all those semi-famous authors on the byline helps with that too]. In the case of Paxil Study 329, there were over twenty such authors.

how to maintain a loophole

When someone challenges it, initiate a frenzied match of ping pong. Bring up all the exceptions, legitimate and otherwise. Discredit the critic. Ignore the critic. Make the critic’s eyes cross. In this case, do everything possible to keep all the ambiguity about the location of the bona fide a priori list of Primary and Secondary Outcome Variables alive. If they invent a ClinicalTrials.gov – ignore it. If they make it a requirement – insist on a long lag time before completely filling it out [and continue to ignore parts of it]. And if somebody like Ben Goldacre or Peter Doshi tries to nail things down, make them feel like this:

"…disputes are so detail oriented that my eyes crossed trying to follow what at times feels like a frenzied match of ping pong, each side’s latest rejoinder seeming to rebut their opponents’ last counterpoint."

The point is to either downplay the importance of these preregistered parameters or, in fact, change them to something that gives a more favorable result. And that is what has been happening in one version or another for literally decades. These arguments over whether they are a sine qua non or not have been going on that long. So…