1 Boring Old Man

Money back guarantees for non-reproducible results?

There are better solutions to the “reproducibility crisis” in research

by Eric J. Topol

British Medical Journal. 2016 353:i2770.

[full text on-line]

Money back guarantees are generally unheard of in biomedicine and healthcare. Recently, the US provider Geisenger Health System, in Pennsylvania, started a programme to give patients their money back if they were dissatisfied. That came as quite a surprise. Soon thereafter, the chief medical officer at Merck launched an even bigger one, proposing an “incentive-based approach” to non-reproducible results — what he termed a “reproducibility crisis” that “threatens the entire biomedical research enterprise.”

The problem of irreproducibility in biomedical research is real and has been emphasised in multiple reports.  In the same vein, the retraction of academic papers has been rising, attributable, in nearly equal parts, to irreproducible results or data that have been falsified. But this problem is not confined to basic science or animal model work from academic laboratories. Clinical trials, the final common pathway for the validation and approval of new drugs, have been plagued with serious drawbacks.

The bad science in clinical trials has been well documented and includes selective publication of positive results, data dredging, P hacking, HARKing, and changing the outcomes that were prespecified at the beginning of the study [below]. Indeed, the high prevalence of switching outcomes in drug industry funded trials led Ben Goldacre and colleagues at the University of Oxford to organise Compare, an initiative to track this considerable problem. Furthermore, the disparity between what appears in peer reviewed journals and what has been filed with regulatory agencies is long standing and unacceptable.

Bad science
• Data dredging — Mining a dataset with innumerable unplanned analyses
• P hacking — Repetitively analysing data in ways not prespecified to find a significant P value
• HARKing (hypothesising after the results are known) — Retrofitting the hypothesis after the results are known to portray an exploratory, retrospective analysis as if it was prospectively declared

Transparency is key

What is missing is the deep commitment—across academia and the life science industry—for open science and open data. Everyone asks for accountability of research findings, which can be vastly promoted by making them fully transparent. But compliance is poor. Even the mandatory requirement for publishing results on Clinicaltrials.gov within two years was evaded for 87% of 4347 clinical trials in academic centres.

When we start to see all the protocol, prespecified hypotheses, and raw data available for review, along with full disclosure of methods and analyses and what, if anything, changed along the course of experiments, be it at the bench or in clinical trials, we’ll have made substantive progress. A promising, low cost digital solution exists to capture all of the data and promote trust and reproducibility in biomedical research. Use of blockchain technology has recently been shown to provide an immutable ledger of every step in a clinical research protocol, and this could easily be adapted to…

Until we develop the right system, we don’t need or want money back guarantees on research reproducibility. But I’d be interested to pick up on that refund offer for my medications or any medical care that doesn’t work.