The Cochrane Systematic Reviews were new to me, though they’ve been around for twenty years. The group is made up of a network of thousands of independent scientists who donate their time to do in-depth and comprehensive reviews of clinical trials using structured meta-analytic techniques – many of which have been developed by the Collaboration themselves. So beside bringing their scientific background to their work, reviewers are trained to use the many structured tools to make their reviews the gold standard in the world of clinical trial analysis.
Their logo shows their most characteristic graph – the forest plot with the mean and Confidence Intervals of a number of trials of the same thing. The size of the mean box in the center is proportional to the size of the trial and the horizontal line shows the 95% C.I. The bottom diamond is the weighted trial mean. The horizontal axis varies depending on what’s measured. The top graph is from the Tamiflu meta-analysis and shows the time to first alleviation of symptoms between Placebo and Zanamivir. One can see the means, the intervals, the significance [does it cross the vertical null line], and the sum of all studies at a single glance.
But that’s not all. One easy way to deal with studies you don’t like is simply not publish them, maybe even not even register them. In this Tamiflu story, there were many such missing studies, so the Cochrane authors spent almost four years along with many allies fighting to get their hands on the data from all the studies done. And Cochrane goes another step – they grade the quality of each clinical trial for signs of any number of sources of bias, and discard the uninterpretable trials from the dataset. In the case of Tamiflu, rather than used the data printouts, they used the CSRs [Clinical Study Reports], the actual forms where the data was being recorded because they felt they couldn’t trust the published results. Here’s their flow chart:.
I’ve only mentioned the broad strokes of the techniques they’ve developed to gather their data for analysis. It’s an obsessive and impressive process from start to finish. Is all of this necessary? If you’re a psychiatrist trying to make sense of the psychopharmacology trials of the last three decades, it’s a godsend. And as we’re seeing today, in the case of Tamiflu, a beyond-blockbuster success in the public health market, it was essential. Cochrane looks to be exposing Billions of dollars wasted, flowing into pharma coffers for no real public health gains. Now Roche and GSK having resisted this process as long as they could, is contesting this recent meta-analysis that counters their claims. So, I expect that we’ll be hearing about this one for some time.
Governments Have Spent Billions of Dollars Stockpiling the Antiviral Drug for Use in Flu OutbreaksWall Street JournalBy Hester PlumridgeApril 9, 2014
A new study into the effectiveness of one of the world’s leading flu-fighting drugs concludes it isn’t proven to reduce the spread of flu or flu complications, like pneumonia and hospital stays, raising questions about the medicine’s use against pandemics. Governments have spent billions of dollars stockpiling Roche Holding Ltd. ROG.VX -2.86% ‘s antiviral drug Tamiflu. The drug is used to treat flu, and to prevent pandemics and seasonal flu outbreaks. It is approved in more than 100 countries. The new study—conducted by the nonprofit Cochrane Collaboration, a global network of health-care academics, and published Thursday in the British Medical Journal—doesn’t question the drug’s ability to treat symptoms of flu. But in examining earlier trials and studies of the drug, the study found there is no indication it is effective in stopping the spread of flu or reducing complications—both key components of fighting a widespread flu outbreak…
Roche said it disagreed with Cochrane’s conclusions and criticized the decision to use data from just 20 of its trials and exclude real-world data from seasonal use and from the 2009 swine flu pandemic. "I believe our studies were run to a very high standard," said Barry Clinch, Roche’s global development leader for Tamiflu. Glaxo said it continued to believe the data from Relenza’s clinical trial program support its effectiveness against flu. The U.S. Federal Drug Administration said it considers the clinical trial data submitted for both drugs adequate to support their approval to treat and prevent influenza. The EMA said the review’s findings didn’t raise new concerns.
Cochrane reviewers fought for over four years to gain access to all the trial data from Roche and Glaxo, after their reviewers found in 2009 that the majority of the data on Tamiflu trials was unpublished. The reports’ authors call the current system of drug evaluation and regulation "deeply flawed," since even regulators may not have all the trial data. "The Cochrane review has made us aware of the very shaky ground on which clinical decisions have been made," said Fiona Godlee, editor of the BMJ. "We need the full data from clinical trials made available for all drugs in current use." The European Parliament earlier this month passed a law requiring drug companies to publish all clinical trials related to a drug, but it only covers drugs approved since January 2014. Roche and Glaxo both say they are now committed to publishing all clinical study reports on all their drugs.
You’re on fire, Dr. Nardo. It’s fun to watch someone being excited about statistical analyses, made possible by the fact that you and Goldacre are good story tellers.