post-it notes…

Posted on Monday 17 August 2015

As a piece of their support of the movement for Data Transparency in Randomized Clinical Trials [RCTs], the BMJ featured a particular article [Restoring invisible and abandoned trials: a call for people to publish the findings] with this cover graphic [minus my added post-it notes] on their June 22, 2013 print edition:

Since then, I have learned a lot more about the details of Clinical Trial reporting and have talked about it here ad nauseum. Two of the authors, Peter Doshi and Tom Jefferson, went the extra mile in their quest to vet the studies on Roche‘s Tamiflu® for their Cochrane Collaboration review [Oseltamivir for influenza in adults and children: systematic review of clinical study reports and summary of regulatory comments] – spending literally years chasing down the raw data for the published and unpublished trials. Then they were joined by Kay Dickersin, David Healy, and Swaroop Vedula to collect even more data at large and launch the RIAT Initiative, a move supported by the BMJ and PLoS editors [see “a bold remedy”…]:

Restoring invisible and abandoned trials: a call for people to publish the findings
British Medical Journal
by Peter Doshi, Kay Dickersin, David Healy, S Swaroop Vedula, and Tom Jefferson
June 13, 2013
[full text on-line]
Restoring the integrity of the clinical trial evidence base
British Medical Journal
by Elizabeth Loder and Fiona Godlee [BMJ] and Virginia Barbour and Margaret Winker [PLoS Medicine]
June 13, 2013
[full text on-line]
In several dimensions, there’s still no consensus about what concretely constitutes Data Transparency. The overwhelming majority of RCTs are conducted, or at least managed, by the large Contract Research Organizations [CROs], whose job it is to follow the Study Protocol to the letter and insure the integrity of the blind. When the blind is broken, the raw data is passed on to the Sponsor who connects the subjects with their treatment, and usually does the analysis of the data in-house. What’s evolved is a system where the results are written up by a ghost-writer from a medical writing firm with variable input from the listed academic authors who serve as a ticket into the peer-reviewed medical literature. It’s hard for imagine how this system evolved to it’s current level of corruptability.

Data Transparency proposes to reform it by skipping the system altogether. Allow independent investigators to see the same thing the system sees as soon as the blind is broken and reanalyze it, checking the work of the Sponsor/Journal. So that’s the reason for my post-its upstairs. When the RIAT paper was written, I presume they looked for the clinical study report, the electronic patient level datasets, and the completed case report forms.

Having spent some time on a RIAT Team, here two+ years later, I would revise that cover slightly. The CSR is an exhaustive report on the study, but it’s not "raw." It’s something prepared by human beings, the same human beings responsible for the published article – thereby as prey to distortion and corruption as the paper itself. What they got as electronic patient level datasets are one type of IPD [individual patient data]. It’s the compiled database derived directly from the forms filled out as the study progressed – forms called the case report forms [CRFs]. Unless one suspects fraudulent transcription, the IPD is what matters – then you need the CRFs to check with. The CSR is only useful if it contains the IPD. Otherwise, it’s just the long form of the paper you already suspect has something awry.

So to my post-it notes. They add one other vital thing, the a priori study protocol [I’m 100% serious about the vital part]. Among many other things, it lays out what variables will be assessed and exactly how they will be analyzed. Most, if not all, RCT distortion involves not following the a priori study protocol [or having a biased protocol from the start]. So remember these post-it notes when evaluating a RCT or a RIAT reanalysis:

Sorry, the comment form is closed at this time.