1 Boring Old Man

Is this trial misreported? Truth seeking in the burgeoning age of trial transparency

by Peter Doshi, associate editor

British Medical Journal. 2016 355:i5543

Peter Doshi has, as usual, done a great job of clearly laying out an issue and exploring all of its facets. It’s certainly one I haven’t been able to stop talking about since I knew it was there to be talked about. With great restraint, I’ll avoid referencing all the times [they might just fill this whole page]. He’s talking about the question of whether one needs to absolutely define the analytic parameters of a Clinical Trial before you begin the study, or can you make changes in the outcome variables or statistical methodology along the way. Doshi is particularly focused on what’s called "Outcome Switching"  [changing either the Primary or Secondary Outcome Variables after the study has begun]. And Doshi uses Ben Goldacre’s COMPare study as his example [comparin’…]. Goldacre’s group looked at recently published trials in top journals, and found that most of them had outcomes switched post hoc:

Goldacre et al wrote the journals about their findings and catalogued their responses:

This, from Peter’s article:

Lost in a maze of detail

The disputes are so detail oriented that my eyes crossed trying to follow what at times feels like a frenzied match of ping pong, each side’s latest rejoinder seeming to rebut their opponents’ last counterpoint. In one trial, COMPare’s datasheet shows that the group counted 83 secondary outcomes.34 Yet in its letter35 to NEJM, it mentions “29 pre-specified secondary outcomes” while the last entry before patient enrolment on ClinicalTrials.gov lists just two.36 By my count—based on the protocol37 dated before patient enrolment and posted on NEJM.org—there were over 200.

I shared the case with Curtis Meinert, Johns Hopkins professor and former editor of the journal Clinical Trials. Meinert guessed that fear of future accusations of bias was driving trialists to bad practices. “They don’t want to be accused of looking at something they didn’t specify, so they specify every freaking thing.”

What is the clinical significance?

I asked COMPare to elaborate on clinical impact. Beyond the technical violations of best practices in trial reporting, did the team discover any misreporting that concerned them from a clinical perspective? For COMPare, this question was out of scope. “On clinical impact, we’ve deliberately not examined that. We set out to assess whether outcomes are correctly reported in journals listed as endorsing CONSORT,” the team told me, referring to the well established guidelines for proper reporting of randomised trials. “We deliberately steered away from any value judgments on why someone reported something other than their prespecified outcomes, because we needed our letters to be factual, unambiguous, and all comparable between trialists and journals.”

But clinical and value judgments were central to journal editors’ defence. “COMPare objects to the format in which the data are communicated, but COMPare is silent about whether they dispute the key clinical message of the article,” NEJM editor Jeffrey Drazen said through a spokesperson. Annals echoed this sentiment: “COMPare’s methodology suggests a lack of recognition of the importance of clinical judgment when assessing pre-stated outcomes”…

Adhering to the original plan

The challenge of establishing outcome “switching” begins with determining trialists’ prespecified outcomes. But which document—protocol, statistical analysis plan, registry entry, or some combination of the above—details trialists’ true intentions? COMPare’s methods prioritise protocols over registry entries, a practice that troubles Elizabeth Loder, head of research at The BMJ, which requires reporting according to the registry entry. “I see a worrying trend away from trial registries to protocols … People seem to think that as long as you’ve registered the trial and given notice of its existence, the details don’t matter so much.”

Annals said that sometimes editors are faced with “a choice between an incomplete or non-updated registry record, with a reliable date stamp, and a more detailed and updated protocol document.” Such situations deeply trouble Deborah Zarin, a long time advocate of trial registration and director of ClinicalTrials.gov, who believes that trial registration is the foundation of the trial reporting system…

Even with trialists’ last testament established, complexity remains. Having herself conducted similar analyses that compare outcomes reported across different sources, Zarin noted that a key challenge is disagreement in the community over how detailed outcome prespecification must be. “From my perspective, prespecification sets the foundation for a statistical analysis; if there is no firmly prespecified outcome measure, then it’s unclear to me what any reported P value means,” she said, noting that ClinicalTrials.gov now requires investigators to delineate “the specific metric and the time point[s]” for all outcome measures.

Yet some trialists do not sufficiently define outcomes until after the data are collected [but not unmasked] and they finalise the statistical analysis plan, a situation the FDA apparently accepts. The FDA told The BMJ that it “does not require the statistical analysis plan to be in place prior to initiation of the trial.” Annals stated that “prespecification is certainly important, but holding blindly to this prespecification may not be good science. Sometimes it becomes clear to informed clinicians and scientists who understand the area that what was prespecified is not going to provide patients and clinicians with the information they need”…

That question has an answer. As long as the confusion remains, one can do whatever makes the outcome look most like you want it to look. And you can justify your choice with the kind of discussion that has Peter’s eyes crossing, continuing to play a frenzied match of ping pong until the end of days. So if you keep from being definite about the outcomes, you can pick the one that fits your fancy, and if challenged – bring on the fog. It’s confusing on purpose…