As some of you know, a group of us are on a RIAT [restoring invisible and abandoned trials] team that has spent the last year working on the first shot at an independent team analyzing and reporting on someone else’s Clinical Trial. It has been a great learning experience for me, and has been the reason for so many posts about how trials work and my monotonous slides:
On the day there was a bruhaha in the comments, I had learned that there was much to do in a tight time-frame. Emails were buzzing across oceans and I was all caught up in those things, so I missed what was going on in my own back yard. When I say tight time-frame, I should say tight time-frame for me. My place in this hinges in part on my previous training in math and statistics [which ended the same year the first PC was built – 1974]. So every new task has a learning curve that’s long and tedious – catching up with newer statistical methodology and doing it using modern software [mainly the latter]. When I was doing it the first time, data was on punch-cards and software was something you wrote yourself [also on punchcards].
On 2 October the European Medicines Agency’s [EMA] published the final version of its policy on prospective release of clinical study reports [CSRs] of trials submitted by sponsors in support of Marketing Authorisation Applications [MAAs]PLOS: Speaking of MedicineBy Tom JeffersonNovember 3, 2014I have summarized its content and made preliminary comments here
The policy
In brief, from 1 January 2015 the proactive policy will join the current reactive policy and allow prospective access to incomplete CSRs at two levels: a “viewer” level of access and a “researcher” level. The former will allow on-screen viewing only with a simple registration procedure, while the latter will require proof of identity but allow download and OCR searches to be carried out on CSRs. Prospective seems to mean two things: after a regulatory decision has been taken, pharma will format and write CSRs for direct web posting, and once you have obtained access credentials, you will not have to ask every time you want a CSR and wait some months [and in some cases be turned down].Its meaning
The practical importance of the policy hinges on the dawning realization of the difference in accuracy, completeness and trustworthiness of published versions of trials compared to their CSR counterparts. Trials may not be published, some may not even be registered [especially the older ones] posing decades-old problems for readers, users and, from my point of view, evidence synthetisers. Reporting bias [cherry picking] is the broad term which encompasses the scores of different types of bias present in publications. The range from the well-known publication bias to the less known swamping bias [when important information on a trial is missed amidst a sea of other information]. CSRs in theory cut through all this by presenting exhaustive structured narratives and results of trials. This is essentially part of what regulators see.The policy has some as yet unclear legal aspects and sets out equally unclear rules for redaction of CSRs [based on the preservation of commercial interests and protection of participant privacy]. It also does not explain why some parts of the CSRs will not be released.
The proactive EMA policy will provide an easily accessible window to look at CSRs. When you read the statements “placebo controlled” or “matching placebo” or “double blind” in a journal article you can now go and check if that is really the case. And if, for example, the certificate of analysis1 is missing from the CSR that you are accessing, you can ask EMA why that is so, introducing an unheard of degree of accountability.
An ever growing body of scientific evidence shows that journals do not give the whole story [although some go to extraordinary lengths]. This point reflects simple arithmetic: the ratio of CSR pages to publication pages for the same trial can be as much as 8000 to 1. To maintain their credibility journals will have to either provide access to CSRs, or stop publishing trials altogether and offer commentaries on trials only. One further option, that of asking for the complete CSR to be made available for each trial on submission, is also at present unrealistic, as current peer reviewers are unskilled in making sense and reviewing a CSR, especially if time constrained. My experience also tells me that every trial should be looked at in the context of the whole trial programme, for example to avoid a piecemeal approach to the assessment of potential harms.
There may be other scenarios such as slowing the rate of publication of trials to allow thorough review of the linked CSR, the growth of a skilled cohort of well-paid peer reviewers or the publication of structured summaries linked to CSRs on the EMA webpage or a combination of these.
Evidence synthesis
Accessibility of CSRs, although not a panacea for reporting bias, should also improve the reliability of research synthesis. As more information on the planning, intended and real conduct of the trial will be available, a good review will be the one that searches regulatory and pharma websites, reconstructs and presents the whole trial programme of interest and includes and reviews CSRs. A good review will also contact the trials sponsors to obtain confirmation that there are no invisible trials and trawls through registers, black and grey literature to check that this is so.But all this will take a lot of time and require skills and tools to be developed. Businesses which produce reviews in months will require years to do the same, or fall behind.
It is likely that the whole infrastructure to produce synthesis studies will have to change and become much more professional, flexible and intensive, or fall behind. For the record, I don’t believe in automatic methods of data extraction to facilitate dealing with CSRs, but perhaps I am too old-school and will be proved wrong.
So, increased accountability should work for everyone and should keep CSR writers on their toes, open regulators to being quizzed, revolutionise publishing and produce better synthesis assessments.
The FDA must be reading this blog:
http://blogs.wsj.com/pharmalot/2014/11/07/fda-chastises-drug-maker-for-spotty-evidence-in-sleeping-pill-ads/?mod=WSJBlog
Steve Lucas
Mickey, thank you for all the work you’ve done, and the pleasure I’ve experienced in the years of reading your blog on psychiatry.
I hope you find a great deal of satisfaction in your new projects. I’ve always appreciated your brilliant information design calling out the key aspects of data. I’m sure you will help unearth important truths.
Thank you also for your representation of a responsible, healing-oriented psychiatry. You’re a rare physician in any specialty.
Welcome to Coursera? I hope that I eventually get as good as you flipping these tools around.
I was telling a student about this most honest, informative blog today, its diverse comments, most often earnestly touching, thinking out loud and sometimes expanding my view on dr Nardo’s output. I was delighted to tell her about Arby’s take on Peanuts Lucy’s “have fun, learn things!” and her quote from Tolkien’s The Hobbit: “It’s a dangerous business, Frodo, going out your door. You step onto the road, and if you don’t keep your feet, there’s no knowing were you might be swept off to.” This conversation led us to the article Reflections and Reconstruction by Kenneth J Gergen and John Kaye in the book Therapy as Social Construction edited by Sheila McNamee and Kenneth Gergen. The article opened with the poem Corsons Inlet, by A R Ammons, a poet I’d never even heard of till today, though easily available on internet. Psychiatry blogs and clinical research is not poetry, but all claim (differing) relevance to our lives. I thank dr Nardo with a few lines from Corsons Inlet.
I have reached no conclusions, have erected no boundaries
shutting out and shutting in,
separating inside from outside:
I have drawn no lines
as manifold events of sand change the dunes’ shape
that will not be the same shape tomorrow.
So I am willing to go along
to accept the becoming thought
to stake no beginnings and ends,
establish no walls.
Sadly, Berit, this looks like it is going to end.
I don’t really know how a dispute between two professionals with a backdrop of disputes between some regulars morphed into a discussion of the coarse nature of the internet that morphed into a license to behave badly because apparently “all the rest of the ‘net is doing it”.
I’d be nice if everyone took this opportunity to look into their own (increasingly black looking) hearts and ask themselves how they could be more civil in their comments. And, strangely, this is not largely directed at the physicians who can handle their own squabbles, and seem to be reacting, but at those that are baiting them, either stated up-front or hidden behind flowery speech.
And, before I face the wrath of you all about my own black heart, well, I already I know I have one; it’s one of the reasons I can recognize it, and I struggle with working on mine constantly.
I do not do this because there are internet rules to be followed, or even out of respect for the host, although that factors in, but for the simple reason of my own integrity and self-respect.
Which is probably why I won’t comment here again. This last comment already crossed a line I didn’t want to in myself – that of lecturing adults on behaving like adults.
I do not know when or where anything ends, Arby, but I know the Spanish poet/philosopher Jose Ortega Y Gasset once wrote that it always ends in shipwreck. Ibsen wrote that trolls are inescapeable, they are in us, our black hearts, as you poetically call it – to be reined in and tamed, I think, if we are to become humane while we can. Life is short. Nice meeting you, Arby!
Thank you, Berit. It has been nice meeting you too! Our exchanges have been a pleasure.
I do need to move on from posting comments but I look forwarding to reading yours here and elsewhere on the web.