1 Boring Old Man

Is this trial misreported? Truth seeking in the burgeoning age of trial transparency

by Peter Doshi, associate editor

British Medical Journal. 2016 355:i5543

Outraged editors

The long awaited goal of universal registration of trials now seemed achievable.5 and medical journal editors issued an ultimatum: preregister your trial or forgo publication in our pages. “Honest reporting begins with revealing the existence of all clinical studies, even those that reflect unfavourably on a research sponsor’s product,” a group of influential editors declared. “Unfortunately, selective reporting of trials does occur, and it distorts the body of evidence available for clinical decision-making.” The declaration had enormous impact, and public trial registers remain a key mechanism to prevent investigators from hiding or spinning unfavourable results.

But more than a decade on, a small project from Oxford University’s Centre for Evidence Based Medicine seems to have journal editors eating their own words, with some of the world’s most powerful editors arguing that strict adherence to the registry entry’ or trial protocol may not always make sense…

Journal dissent

“Upon receipt of COMPARED initial communication, our editorial team (comprised of physicians and statisticians)thoroughly re-reviewed materials associated with the articles.” Annals’ editor in chief. Christine Laine, told The BMJ. “We concluded that the information reported in the articles audited by COMPare accurately represents the scientific and clinical intent detailed in the protocols.” In notices posted to the journal’s website, Annals editors acknowledged the good intentions of COMPare but warned people to be wary.

“Until the COMPare Project’s methodology’ is modified to provide a more accurate, complete and nuanced evaluation of published trial reports, we caution readers and the research community’ against considering COMPare’s assessments as an accurate reflection of the quality’ of the conduct or reporting of clinical trials”…

NEJM also flunked COMPare’s test. The journal’s editors refused to publicly engage with the group, rejecting all of  COMPare’s 20 letters, most with a tightly worded statement saying “we have not identified any clinically meaningful discrepancies between the protocol and the published paper that are of sufficient concern to require correction of the record… NEJM’s response has a certain irony. The journal’s editor, Jeffrey Drazen, has been a longtime supporter of trial registration, and NEJM was the first of its peers to publish trial protocols alongside trial publications, a practice now followed by JAMA, The BMJ, and most recendy-in part thanks to COMPare — Annals.

The BMJ did not escape criticism but ultimately got a greenlight. COMPare sent rapid responses for two of the three trials evaluated, one of which led to a correction. It was “an example of best practice,” the group said in a blog. What about JAMA and the Lancet? JAMA rejected all 11 letters the group sent, and the Lancet rejected some but published others…

Frankly, while "a more accurate, complete and nuanced evaluation of published trial reports" sounds really good, it’s well off the mark. The whole point of a clinical trial is objectification, hard core evidence-based medicine. Of course there are instances where common sense says that the preregistered outcome need to be reconsidered, but that’s not what’s on the table here. COMPare found that the majority of the articles had some outcome changed. And the changes weren’t flagged or explained.