the final lesson of Study 329: epilogue…

Posted on Saturday 8 September 2012

I really believe that the only way to insure that the debacle of the recent past with Clinical Trials like GSK’s Study 329 is to require public online posting of the raw data for any study published in any peer reviewed journal or submitted to the FDA. Mathematical and statistical analyses of the data is intended to enhance our understanding of the information, help us evaluate variability, differences, relationships, comparisons – not to be a way to obfuscate or beautify the complexity of real hard data. For starters, the graph on the left [individual Paroxetine HAM-D scores] doesn’t really convey the same thing as the graph on the right [the least squares mean with the 95% confidence limits of the mean]:
 
But that’s just an example that shows how the usual data display doesn’t really show the amount of variance among individuals in a dramatic way. The graph on the right is something of an illusion, but it’s an honest illusion. On the other hand, these two graphs below from the raw data available now on Study 329 show the kind of hanky-panky that is so easy to pull off in choosing how to display the information:
The investigators declared that one of their primary outcome variables was the percent responding using the criteria for Response equal HAM-D total score < 8 or < 50% Baseline at 8 weeks. Recall that in the paper, they used HAM-D total score < 8 for their primo outcome, number 1 in Table 2, a parameter unmentioned in their protocol. When you’ve got the data sitting in front of you, you can draw the graphs and see immediately why they made the switch and pulled HAM-D total score < 8 out of the hat. It just so happened that at 8 weeks, HAM-D total score < 8 was an unusually large value – an outlier. That’s hanky-panky! Had  Drs. Jureidini and Tonkin had the raw data available, they could have [and would have] demonstrated this little piece of black magic in a way that couldn’t have been brushed off with sarcasm and a nastygram [the lesson of Study 329: naked Emperors, fractious Queens…].

This is one of the reasons for requiring investigators to declare primary outcome variables before doing the study. Once you have the data in hand, you can play around with your data and the statistical package on your fancy computer and find something that looks the way you want it to look, even if the study is a bust – as was Study 329. This example is amateur night material if you’ve got the data, but can only be inferred but not nailed down with close reading of the paper without it. To repeat myself, "If you torture the statistics long enough, they’ll tell you anything." I’ve already shown another example in the adverse effects data. In the paper, hostility and emotional lability were buried in a table:

But they had a more data available than they let us see, and hostility and suicidality were in the serious range. This looks a whole lot more ominous when you can plot the numbers:

 

This is what the FDA does when they analyze the results, because the company is required to submit their raw data. And the Pharmaceutical companies know that, explaining this GSK position paper from shortly after they got the results of Study 329 [October 1998]:

I realize that by going over and over the results from Study 329, I’m pushing my luck, and becoming 1reallyboringoldman. But I’m determined to make this point crystal clear, even if I’m my only reader. That GSK position paper says everything needed to critique what has been wrong with the recent era of psychiatric Clinical Trials [and if you read the third from the last sentence, GSK had picked the abberent HAM-D total score < 8 before Sally Laden ever got to work].  What they can’t get through the FDA [even with its minimalistic standards], they publish in a peer reviewed journal and promote off-label use, handing out the reprints [see J&J Deal With The States ‘Is Huge’: Friede Explains]. It’s what’s wrong with ghost-writing. If they’re going to play with the numbers until they can figure out a way to make them reach their forgone conclusion, they need to do it themselves instead of hiring a "hit-man" like Sally Laden to do it for them. It’s their sleep we want disrupted [ghost-writers sleep just fine]. It’s the reason that the raw data needs to be published so that those of us who suspect misbehavior can do an audit, much as the I.R.S. audited Al Capone. I use these mobster examples because this kind of hanky-panky is scientific crime – in this case, crime against youth and crime against physicians…

So, believe it or not, I’ll shut up about Study 329 now and move on to something simple – what to do about it.
  1.  
    Bernard Carroll
    September 8, 2012 | 9:59 AM
     

    Your description of Glaxo’s thought process fits the expression HARKING – hypothesizing after results are known.

  2.  
    September 8, 2012 | 12:01 PM
     

    I hadn’t heard the acronym HARK [hypothesizing after results known], but I won’t forget it. It’s only really possible to get away with HARKing if no one else has access to the raw data. I guess LARK would do too [lying after the results known]. Thanks for the term…

  3.  
    jamzo
    September 8, 2012 | 1:03 PM
     

    Transparency for Clinical Trials — The TEST Act
    Jeffrey M. Drazen, M.D.
    N Engl J Med 2012; 367:863-864August 30, 2012

    “This legislation is important. The bill requires that any trial that could be used to support an application for FDA approval be registered in ClinicalTrials.gov and that the results be reported in a timely fashion. It requires that early-phase trials (those in which a drug is initially tested in humans) be registered. Thus, these trials will become public knowledge.”

    “We can make progress in medicine only if people are willing to put themselves at risk to test new diagnostic and therapeutic approaches. To recognize and reward these participants, and in keeping with the Declaration of Helsinki, clinical trials should be conducted in the open, with full public knowledge of the question asked, the intervention tested, and the results obtained. The TEST Act is another step toward this end, and we strongly support it.

  4.  
    September 8, 2012 | 6:21 PM
     

    Selling bulk reprints is a profit center for a journal, see http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3133891/ :

    “Additionally, reprint sales can be a financially attractive source of revenue for some publishers, which has the potential to generate about $700,000 from a single journal article.”

    (Apologies if you mentioned this elsewhere, Dr. Mickey.)

  5.  
    September 8, 2012 | 6:38 PM
     

    Martin Keller retired from Brown without explanation at the end of June http://www.pharmalot.com/2012/09/controversial-paxil-study-author-retires-from-brown/

Sorry, the comment form is closed at this time.