The Cochrane Systematic Reviews were new to me, though they’ve been around for twenty years. The group is made up of a network of thousands of independent scientists who donate their time to do in-depth and comprehensive reviews of clinical trials using structured meta-analytic techniques – many of which have been developed by the Collaboration themselves. So beside bringing their scientific background to their work, reviewers are trained to use the many structured tools to make their reviews the gold standard in the world of clinical trial analysis.
Their logo shows their most characteristic graph – the forest plot with the mean and Confidence Intervals of a number of trials of the same thing. The size of the mean box in the center is proportional to the size of the trial and the horizontal line shows the 95% C.I. The bottom diamond is the weighted trial mean. The horizontal axis varies depending on what’s measured. The top graph is from the Tamiflu meta-analysis and shows the time to first alleviation of symptoms between Placebo and Zanamivir. One can see the means, the intervals, the significance [does it cross the vertical null line], and the sum of all studies at a single glance.
But that’s not all. One easy way to deal with studies you don’t like is simply not publish them, maybe even not even register them. In this Tamiflu story, there were many such missing studies, so the Cochrane authors spent almost four years along with many allies fighting to get their hands on the data from all the studies done. And Cochrane goes another step – they grade the quality of each clinical trial for signs of any number of sources of bias, and discard the uninterpretable trials from the dataset. In the case of Tamiflu, rather than used the data printouts, they used the CSRs [Clinical Study Reports], the actual forms where the data was being recorded because they felt they couldn’t trust the published results. Here’s their flow chart:.
I’ve only mentioned the broad strokes of the techniques they’ve developed to gather their data for analysis. It’s an obsessive and impressive process from start to finish. Is all of this necessary? If you’re a psychiatrist trying to make sense of the psychopharmacology trials of the last three decades, it’s a godsend. And as we’re seeing today, in the case of Tamiflu, a beyond-blockbuster success in the public health market, it was essential. Cochrane looks to be exposing Billions of dollars wasted, flowing into pharma coffers for no real public health gains. Now Roche and GSK having resisted this process as long as they could, is contesting this recent meta-analysis that counters their claims. So, I expect that we’ll be hearing about this one for some time.
I scrolled through my way-overly-simplified version of the Cochrane Collaborations’ efforts for a reason.
Governments Have Spent Billions of Dollars Stockpiling the Antiviral Drug for Use in Flu Outbreaks
Wall Street Journal
By Hester Plumridge
April 9, 2014
A new study into the effectiveness of one of the world’s leading flu-fighting drugs concludes it isn’t proven to reduce the spread of flu or flu complications, like pneumonia and hospital stays, raising questions about the medicine’s use against pandemics. Governments have spent billions of dollars stockpiling Roche Holding Ltd. ROG.VX -2.86% ‘s antiviral drug Tamiflu. The drug is used to treat flu, and to prevent pandemics and seasonal flu outbreaks. It is approved in more than 100 countries. The new study—conducted by the nonprofit Cochrane Collaboration, a global network of health-care academics, and published Thursday in the British Medical Journal—doesn’t question the drug’s ability to treat symptoms of flu. But in examining earlier trials and studies of the drug, the study found there is no indication it is effective in stopping the spread of flu or reducing complications—both key components of fighting a widespread flu outbreak…
The battle here was just getting the data to analyze, and it took them four years with major help from the British Medical Journal and others, including Ben Goldacre and the whole AllTrials community. It was a mega effort. The analysis was a walk in the park compared to the effort expended to get the data. They spent four years of their lives swimming upstream to produce these results. And now they have to listen to this:
Roche said it disagreed with Cochrane’s conclusions and criticized the decision to use data from just 20 of its trials and exclude real-world data from seasonal use and from the 2009 swine flu pandemic. "I believe our studies were run to a very high standard," said Barry Clinch, Roche’s global development leader for Tamiflu. Glaxo said it continued to believe the data from Relenza’s clinical trial program support its effectiveness against flu. The U.S. Federal Drug Administration said it considers the clinical trial data submitted for both drugs adequate to support their approval to treat and prevent influenza. The EMA said the review’s findings didn’t raise new concerns.
This is part science, but more comes under the heading of crime-fighting, the detectives at Cochrane using forensic techniques to get the data and analyze it. And what they found was plenty of evidence that a motivated crime had been committed, resulting in a few Billion Dollar haul. And now we’re going to hear the CYA responses of everyone concerned, and it’s unlikely that there will be much reprisal or policy change. To my way of thinking, this story is the payoff for the twenty years of effort expended to gather thousands of medical scientists together to do something good, something needed. And it must be discouraging for those involved to listen to those silly responses:
Cochrane reviewers fought for over four years to gain access to all the trial data from Roche and Glaxo, after their reviewers found in 2009 that the majority of the data on Tamiflu trials was unpublished. The reports’ authors call the current system of drug evaluation and regulation "deeply flawed," since even regulators may not have all the trial data. "The Cochrane review has made us aware of the very shaky ground on which clinical decisions have been made," said Fiona Godlee, editor of the BMJ. "We need the full data from clinical trials made available for all drugs in current use." The European Parliament earlier this month passed a law requiring drug companies to publish all clinical trials related to a drug, but it only covers drugs approved since January 2014. Roche and Glaxo both say they are now committed to publishing all clinical study reports on all their drugs.
But no matter what the concrete impact, this Tamiflu battle has already done a world of good as a paradigm that has driven the All Trials movement, the RIAT proposal, the recent legislative decisions in Europe, the rising momentum of the Data Transparency efforts. Fiona Godlee says, "The Cochrane review has made us aware of the very shaky ground on which clinical decisions have been made..." Ben Goldacre says, "This is a pivotal moment in the history of medicine. Trials transparency is finally on the agenda…" There’s little question that it will be a landmark in the medical history books, so there’s no need to be discouraged by the dancing of those being exposed. It’s the only dance they know…
hat tip to the Cochrane Collaborations…