1 Boring Old Man

NETWORK META-ANALYSIS: John Ioannidis

Let’s face it – for as much as Congress and the evening news pundits push us to look forward to bold new medical breakthroughs, for a lot of us, this is a time for looking back over where we’ve been and cleaning up a lot of misinformation and garbled, deceitful science. And while Stanford’s John Ioannidis recurrently warns us about sloppy/deceitful research [Why Most Published Research Findings Are False] and overdoing the Meta-Analyses and Systemic Reviews [The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses], he’s also an important resource for anyone interested in looking through the Clinical Trial Retrospectoscope. A given meta-analysis might address all the comparable studies of a single drug, or further, all the drugs aiming at the same target [using the same metrics]. A Network Meta-Analysis goes further. By using the Comparator Drugs in the various RTCs, it aims to create a heirarchy of relative efficacy. Obviously, the further one gets from the primary data, the more complex and abstract the mathematical operations become, and the more room there is for error, gibberish, or both. Ioannidis is right there at hand to at least outline the pitfalls and methodology of such an undertaking in an article understandable by mortals [Demystifying trial networks and network meta-analysis].

If there is a maestro of meta-analyses for the psychiatric drugs, it’s Oxford’s Andrea Cipriani, longtime contributor to the Cochrane Collaboration Systemic Reviews. This summer, his group published a Network Meta-Analysis of the SSRI/SNRI drugs in child and adolescent depression [Comparative efficacy and tolerability of antidepressants for major depressive disorder in children and adolescents: a network meta-analysis.][see also my antidepressants in kids? a new meta-analysis…] that concluded: "When considering the risk-benefit profile of antidepressants in the acute treatment of major depressive disorder, these drugs do not seem to offer a clear advantage for children and adolescents. Fluoxetine is probably the best option to consider when a pharmacological treatment is indicated."

If you read this blog from it’s psychiatry origins [2009] to the present, the over-riding focus has been on RCTs [Randomized Clinical Trials] of Psychiatric Drugs in the post-Prozac era. And of those, Paroxetine [Paxil] Study 329 would stand out as something of a near obsession [Efficacy of Paroxetine in the Treatment of Adolescent Major Depression: A Randomized, Controlled Trial]. I make no apology for that. As a clinician, I am a psychiatrist and psychoanalyst, and practiced psychotherapy throughout my career. So case study and n=1 are very much my cup of tea. But there are other reasons this study is so prominantly mentioned. It is virtually the only study where we have the raw data, and the internal documents that allow it to be investigated thoroughly. When we did our RIAT reanalysis [Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence], I was focused on  the efficacy analysis. That story has been often told here so I’ll move on to my point. They reported positive results in a negative trial by changing the Outcome Variables from those declared in the a priori Protocol. And spending several years immersed in that reanalysis taught me how essential pre-registration is in conducting a truly scientific study.

Lessons learned from Restoring Study 329: Transparent reporting, open databases and network meta-analyses as the way forward

by Sarah Barber and Andrea Cipriani

Australian & New Zealand Journal of Psychiatry. 2016 Nov 17.
[Epub ahead of print]

…According to the results from this study, psychiatrists who prescribe paroxetine to a 15-year-old with major depression are practising evidence-based medicine. Or at least they were, until a major re-analysis in 2015 found no difference between paroxetine and placebo when only outcome measures pre-specified in the original study protocol were considered [Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence]. The statistically significant findings in Study 329 original article, it emerged, relied on four new outcome measures, introduced post hoc by the sponsor. It is known that pharmaceutical industry-funded studies are more likely to show favourable efficacy results for investigational drugs than independent trials. Study 329 demonstrates how outcome-switching and selective reporting of results can be used to manipulate results…

This is an important issue, which is at the heart of the debate in the scientific literature about evidence based medicine and, therefore, evidence-based practice. Ben Goldacre and colleagues at the Centre for Evidence-Based Medicine at the University of Oxford have been systematically investigating the issue of outcome-switching in trials published in top medical journals and sharing their results on the website COMParetrials.org. They have found that poor trial reporting often masks inadequate design, and studies such as 329 are just examples of a widespread practice in all areas of medical research, not only psychiatry. This is despite the endorsement by prominent medical journals of the Consolidated Standards of Reporting Trials [CONSORT] Statement, a guideline for authors to prepare reports of trial findings, facilitating their complete and transparent reporting and aiding their critical appraisal and interpretation. Of course, good quality trials may be poorly reported. However, transparent reporting on inadequate trials will reveal deficiencies in the design if they exist, as in the case in Study 329. Among other things, CONSORT requires there to be "completely defined pre-specified primary and secondary outcome measures" and clear reporting of "any changes to trial outcomes after the trial commenced, with reasons". Both are absent in Study 329…

While we fully support the efforts of these groups, we wish to address the needs of practicing clinicians, who want to know which is the most reliable estimate of paroxetine in young people with major depressive disorder. Individual trials and also pairwise meta-analyses are not enough. Network meta-analysis has the great advantage over standard meta-analysis of comparing all treatments against each other [even if there is no trial data comparing the interventions directly] and ranking them according to their relative efficacy or tolerability with a precise degree of confidence. Another important advantage of network meta-analysis is that the effects of sponsorship bias are distributed [and therefore diluted] across the network. The case of Study 329 is a clear demonstration of this added value. A recent network meta-analysis has compared all antidepressants for major depression in children and adolescents [Comparative efficacy and tolerability of antidepressants for major depressive disorder in children and adolescents: a network meta-analysis]. The search strategy, though, was completed in May 2015, a few months before the publication of the re-analysis of study 329. As a result, Study 329 was included with the original 2001 data biased towards paroxetine; however, the overall results from the network meta-analysis showed that paroxetine was not statistically significantly different from placebo. This is because by combining direct with indirect evidence, the bias in studies that favour the investigational drug over placebo is mitigated by the findings of other indirect comparisons in the network. Interestingly, the final results of the network meta-analysis showed that, among all antidepressants, only fluoxetine was significantly more effective than placebo. This is probably a robust result, because it is consistent with the direct and indirect evidence comparing fluoxetine with all the other compounds in the network.

Even using the unrestored results from Study329, Paxil didn’t make the grade in the Network Meta-Analysis – a nice confirmation of our results. I also found the Prozac result interesting. It was approved for adolescents before the Black Box Warning and Lily fought [unsuccessfully] to keep it off the Prozac label. Prozac remains the main drug with FDA Approval for kids. A later Lexapro/Ccelexa FDA Approval has long been questioned, particularly recently since The citalopram CIT-MD-18 pediatric depression trial: Deconstruction of medical ghostwriting, data mischaracterisation and academic malfeasance came out. I’ve personally always suspected that the Prozac result was suspicious – thinking it simply made it under the wire before people caught on that so many studies were being jury-rigged. But this independent confirmation by Network Meta-Analysis suggests that it was indeed a valid result after all. Why it should be any different than the others doesn’t make much sense, but there it is…

The obvious drawback to Meta-Analyses and particularly the Network variety is that they need a lot of trials, so one is well along the path of a drug/class lifespan before such techniques can even be used. Further, one is at the mercy of the pharmaceutical industry’s choice of the Comparators being included in the studies. Bringing up an often unmentioned fact, in most cases, these short-term industry-funded studies conducted for approval and/or commercial purposes are all we really have – the only act in town. In spite of the fact that many, many patients are taking these drugs and staying on them for long periods of time, we still pore over those early brief trials as if they are definitive clinical roadmaps rather than, at best, simply starting places. Until we figures out a way to independently harvest the enormous untapped data pool generated from ongoing clinical usage, these often questionable approval trials retain their power to hold us in their grip [as they have for several decades].