The Randomized Clinical Trial of a psychoactive drug follows a monotonous and simple course:
-
a Protocol defines how the study is to be conducted and analyzed
-
the protocol is submitted to the Institutional Review Board [IRB]
-
on approval of the protocol, the trial is Registered on clinicaltrials.gov
-
the Trial is conducted, double blinded
-
the Results are Analyzed per the protocol, then submitted/published
Protocol » IRB » Registration » Trial » Results » Analysis. In the majority of cases, one could program the whole process into a PC at the beginning and simply get a printed report at the end. As I’ve often said, this is a product testing scenario, not scientific research. Yet every trial is written up differently. Why is that? The paper published yesterday deconstructing a 2004 Citalopram Clinical Trial Report opens a window into the answer [see the jewel in the crown…].
by Jon Jureidini, Jay Amsterdam, and Leemon McHenryInternational Journal of Risk & Safety in Medicine. 2016 28[1]:33-43.
is an analysis of…
by Karen Dineen Wagner, Adelaide S. Robb, Robert L. Findling, Jianqing Jin, Marcelo M. Gutierrez, and William E. HeydornAmerican Journal of Psychiatry. 2004 161:1079-1083.
The Jureidini et al paper is not available as a full text on-line yet, though it may become open access soon. However, if your interested, you can contact the corresponding author for a copy. In the meantime, I’ll briefly summarize some of their findings:
-
Who designed, analyzed, and wrote the article?Reviewing subpoenaed internal emails, it was clear that the protocol was written by under the direction of the sponsor’s Associate Medical Director, analyzed by the Forest Laboratory’s staff, and the first draft was written by a contracted ghost-writer, so it was first seen by the listed academic authors in its second draft. The later denial that the academic authors didn’t know it was ghost written was directly contradicted by the subpoenaed emails. Simply put, the academic authors were merely figure·heads for this industry produced article. This finding is well referenced in Jureidini et al.
-
How was the data analyzed and reported?Primary Outcome Parameter:Change from baseline to week 8 on the Children’s Depression Rating Scale-Revised [CDRS-R] total scoreThe Protocol specified that “Any patient for whom the blind has been broken will immediately be discontinued from the study and no further efficacy evaluations will be performed" There were nine such cases. Jureidini et al didn’t have the raw data, but they did have the Clinical Study Report. In fact, several versions. On 09/12/2001 – these cases were removed, acknowledged in a footnote. the p-value was 0.052. In a later version, these subjects were added back in, the footnote disappeared, and the p value was 0.038. It’s the second version that made it into the paper. Again, this finding is fully referenced in Jureidini et al. There were other irregularities, among them, a reported effect size of 2.9 [an outrageous number]. When confronted, it was revised to 0.32 without explanation of the error.Secondary Outcome Parameters:Clinical Global Impression [severity and improvement subscales], Kiddie Schedule for Affective Disorders and Schizophrenia [depression module], and Children’s Global Assessment Scale.
While they reported the CGI-S and CGI-I scales [not significant], they failed to report the other two. Instead, they reported the CDRS responders [as significant] – an outcome parameter that wasn’t even mentioned in the a priori Protocol.Adverse Events:By reporting the quantitative numbers, they were able to avoid mentioning agitation and a case of hypomania.
So we can return to my original question:
"… every trial is written up differently. Why is that?"
We all already know the answer. It’s the same reason the pharmaceutical companies have put up such a fight to keep the actual raw data from Clinical Trials a secret. It doesn’t have anything to do with patient confidentiality or commercially confidential information. They want to be able to continue to do exactly what they did here – fudge a little here, tweak a little there, leave out some of this and that, find a KOL on the take to sign on… As long as they have the option to work their spin machine, they can turn even the slightest signal of a medicinal effects into a significant effect into a blockbuster worth billions.
This negative study was used to support an FDA Approval for Citalopram [Celexa®] in adolescents, even though by that time, Dr. Karen Dineen Wagner, the first author, had been cited by the US Senate for unreported industry income. When the American Journal of Psychiatry discovered that this article was ghost written, they published a strong note [see the jewel in the crown…], but did not retract the study. The case against Forest Laboratories that brought all of these documents to light ultimately settled [see September 15, 2010: Drug Maker Forest Pleads Guilty: Will Pay More Than $313 Million to Resolve Criminal Charges and False Claims Act Allegations]. And unscathed, Dr. Karen Dineen Wagner is the President Elect of the American Academy of Child and Adolescent Psychiatry, the recipient of lifetime achievement awards for research in Mood Disorders in Children, and she was recently promoted to Chairman of the Department of Psychiatry at UTMB.
Dr. Wagner was involved in four clinical trials in the years leading up to the Black Box Warning being appended to all antidepressants for use in children and adolescents – an FDA decision that has weathered an impressive assault from many quarters. Two studies have recently been definitively shown to be improperly reported. Of the remaining two, the Sertaline [Zoloft®] trial has always been thought of as questionable [see tuning the quartet…]. It’s two negative studies that became positive when pooled together [?] [ergo false + false = true][?]:
-
-
2002: Fluoxetine for Acute Treatment of Depression in Children and Adolescents: A Placebo-Controlled, Randomized Clinical Trial. "Fluoxetine was superior to placebo in the acute phase treatment of major depressive disorder in child and adolescent outpatients with severe, persistent depression."by Emslie GJ, Heiligenstein JH, Wagner KD, Hoog SL, Ernest DE, Brown E, Nilsson M, and Jacobson JG
-
-
This paper by Jueidini, Amsterdam, and McHenry is a landmark study that brings the use of subpoenaed documents to an academic journal to show a clinical trial being actively distorted to fraudulently claim efficacy where none exists – a trial used to obtain FDA Approval for use of the drug in children. While many of us are wracking our brains trying to figure out how to get its message through to clinicians, we’re not the ones that need to be thinking about it. It is rightly in the domain of the government/FDA to press for data transparency to prevent any more repetitions of this kind of fraud. And it is rightly within the domain of the Department of Justice to pick up the ball at this point with this particular case. It’s what they’re for…
There is an obvious way to prevent that kind of data manipulation, cherry picking, moving of goalposts, HARKing, and glossing over of adverse events. All it would take is for the FDA to require that they analyze the data strictly according to the a priori protocol. That requirement would apply to any investigational new drug or to any approved drug being tested for a new indication. Corporations and investigators would be prohibited from reporting any analyses other than the FDA analyses. With an a priori protocol and plan of analysis there should be no room for self-serving “creativity” by the corporations.
As things stand, we have a Kabuki theater spectacle. The corporations don’t come clean about what they did and the FDA doesn’t call them out when agency analyses disagree with the corporate line. They may deny or delay approval, but the FDA doesn’t go to the literature like Jureidini, Amsterdam, and McHenry did here challenge the distorted corporate analyses reported in the literature.
This requirement would put an end to creative manipulation of the clinical trials literature. As Dr. Mickey often says, this is not high science but rather product testing. Thus, corporations cannot claim to be privileged for conducting the statistical analyses. We need a clinical trials equivalent of the Underwriters Laboratory. Before licensing, nobody takes the manufacturing corporation’s word for it concerning the safety and performance of X-ray machines or CT scanners or cardiac defibrillators. Why should we treat drugs any differently?
We might expect the pharmaceutical and device corporations to mount a First Amendment challenge to this proposal. It will fail, mainly because the public health is too important and because we know they have too often abused the public trust – thanks to dedicated critics like Jon Jureidini, Jay Amsterdam, and Leemon McHenry.
Sounds like “The Big Short”. Regulators not doing their job, or not having the law or resources to do their job.
“Sounds like ‘The Big Short’. Regulators not doing their job, or not having the law or resources to do their job.”
The biggest takeaway from the movie is that regulators are bought and paid for and often have a fraud agenda themselves, such as enforcing and promoting the CRA which was basically financially unsound. As scholars of human behavior we should not be naive enough to believe that the FDA after decades of fumbling and bumbling will suddenly get its act together. The only time it did was under enormous political pressure during the AIDS crisis when it scrapped its own onerous rules.
The real scandal is in peer review and academic publishing, which is peer promotion and logrolling. What Jurieidini, Amsterdam et. al. is considered “uncivil” in many publishing and academic bureaucracies. The FDA has nothing to do with the fact that 95% of scientific studies failing on attempts to replicate.
Colleagues .
Mickey’s horrified feelings are entirely justified Carroll’s suggestion that the FDA do the analysis is much more achievable than anything like my idea.
But it is the entire context 1) methods facilitating discovery,2)decision whether discovery is pursued 3) validation attempted by impeccable early investigations leading to definitive designs 4) outcome variables and analysis specified by mandatory public registration anteceding trial,5)impeccable clinical trials process with many safeguards, including treatment and design blind independent evaluation of subject status, 6) statistical analysis by several groups of independent experts with differing methods and public conclusions , 7)attempting to improve public health as the overriding criterion justifying any effort or decision– in contrast to increasing profitability by various tactics eg increasing market share ,increasing the period of commercial monopoly, price autonomy ,eliminating the negative and accentuating the positive,etc..
Attempting this immaculate process ,if managed (and therefore mismanaged) by private enterprise,is ludicrous.
The alternative of government management as under claims of socialism,have had some dreadful results== except apparently in Scandinavia where remarkable transparency , coupled with an individually identified , informed population seems promising,not definitive.
My hope lies in transparency– coupled with a trusted intellectual level , capable of explaining simply and accurately to a population capable of judging such open results .
That has never been a feature of either industry or government ( the more representative the better—usually). Also must deal with “who shall watch the watchers” which brings us back to educating/enabling a responsive population,
The trouble with this Utopian tract lies with the attempt to embrace it as a whole. Such revolutions are now impossible. Much more likely each part will be fought over separately and achieved,if at all, by public understanding and demand– in the face of the countervailing pressures of those who benefit from the status quo.
That’s why Barney’s suggestion to take all the analyses into the FDA’s hands needs a united front of the dissatisfied, as a practical demand on our current would be governors.
It also seems likely that a correct statistical analysis must embrace all the relevant data and generative procedures–step by step.
Utopian revolutions were always impossible. That’s why the American succeeded and the French ended up in rivers of blood. For years. That famous barricade in Les Miserables was not the original French Revolution but in the 1830s.
It’s funny to me that the same people who aspire to utopian societies openly mock the idea of a utopian marriage or family, which is still impossible but more theoretically possible since there are fewer moving parts and conflicting needs.
Perfection is always the enemy of the good.
The FDA is pretty much the opposite of perfection though and something like UL which has a much better track record should replace it.