by Asbjørn Hróbjartsson, The Nordic Cochrane Centre, Copenhagen, DenmarkBritish Medical Journal. 2013 347:f6927.Missing data in clinical trials is a major problem. One would expect the situation to be better for trials funded by large non-profit organisations that are free of commercial bias, and with policies for transparency and data sharing. However, in our experience with a non-profit trial funder we found this was not the case. Indeed, it was so bureaucratic that, despite the good intentions of the authors and others involved, accessing the data proved to be very difficult.
We wanted to study how treatment effects in randomised clinical trials were affected by a failure to blind the outcome assessors. We therefore systematically reviewed trials where the same outcome had been assessed by both a blind and a non-blind assessor within the same trial. One eligible study was a large, well performed trial of the effect of electrical deep brain stimulation for patients with advanced Parkinson disease, published in JAMA in January 2009. The trial was partially funded by the Cooperative Studies Program of the Department of Veterans Affairs’ Office of Research and Development. JAMA reported UPDRS motor scores [Unified Parkinson’s Disease Rating Scale III] evaluated by blind assessors, but did not report the scores evaluated by the non-blind assessors.
On 12 June 2009 we emailed the corresponding author to ask if the researchers would share the results of the unpublished non-blind UPDRS scores. We asked for summary statistics: means, standard deviations, and number of patients for the experimental group and for the control group—a total of six numbers. After a month we received a friendly reply stating that the data could not yet be released, and four days later a follow-up email asking us to fill in a data use agreement form that would be sent for approval to the Department of Veterans Affairs’ Office of Research and Development.
The emails we received during the prolonged exchange were all friendly, and the individuals involved were helpful and understood the need for data sharing, but they were hampered by inflexible, formalistic, and slow bureaucratic procedures. Since our first inquiry we communicated with four people, sent 25 emails, filled in four data use agreement forms, and waited one year and seven months. Governmental and other non-profit organisations that fund randomised trials need to address the problem of unpublished data, and to introduce policies that further advance medical science through appropriate responses to reasonable requests for unpublished data needed in meta-analyses.
We all have our stories about the maddening travels through bureaucratic hoops to get something very simple accomplished. I won’t bore you with the many examples that come to mind, except for my temporary insanity when I was in the military long ago. I had been through training rotations at a VA Hospital, and marveled at how many forms and procedures were required to do the simplest of things. But I wasn’t prepared for the military proper. After a time, I decided that the only reason we had wars was that there were a few people left over that weren’t involved in paper work, and they had to have something to do so they went to war. I proposed that with just a few more forms, we could end war. My resulting campaign, Form Formers for Peace, didn’t get me very far – about as far as a trip to the commander’s office for being a terminal smart aleck. My point is that such encounters can drive the best of us mad [both kinds].
With the Clinical Trial data, we tend to think that all the hoops are being put in place to hide something, and that is probably often true. But in this example, nobody was trying to hide anything. They were just hoops – those layers and layers of procedures that organizations create just by existing. It seems that Asbjørn Hróbjartsson was more composed than I, but his point was the same. Too many hoops. Way too many hoops. But in this case, at least it was just hoops. The main thread of this story isn’t about annoying hoops. It’s about impenetrable walls, walls unyielding to the most vigorous of jumps. They’ve only been breached by the tedious snail’s pace of the legal system.
When I stand back from all the negotiations and game·playing around access to the raw data from clinical trials, it all seems fairly simple. There’s only one confounding factor in full access to that information – protecting the identity of the participants. Other than that, if you’re going to publish an article based on the data [a proxy], it ought to be backed up by the data itself. Since the printouts of that data and the a priori Protocol already exist, and in this digital age, the data storage capacity and Internet accessibility is now essentially infinite, why not publish that information along with the article? Why put layers of bureaucracy in the way? All that’s really needed is a standardization of the procedures for anonymity and some data formats. Admittedly, there are forms of raw data that aren’t words and numbers – MRIs, EEG tracings, etc. But they’re somewhere. In those cases, the investigator might have to travel to where they are, or pay someone to transcribe them. That would be reasonable. But words and numbers – effortless.
If I stand back even further, I wonder why the subject anonymity is even the domain of the sponsor. Why should GSK have access to the subjects’ identities either? Since most trials are done by some version of a Clinical Research Organization, why not make registration, data archiving, anonymization, and data access their responsibility? I wouldn’t propose that for anything but the final phase III trials used for drug approval and journal publications – no trade secrets there, just treatment effect. If a journal article should be a true synopsis of the data, backing it up with the raw data seems the essence of evidence-based medicine. And why put the thick layer of bureaucracy in the way of access as was true in the uncontested case described above? Admitedly, they were requesting something extra, something unpublished, which might be expected to take longer. But 19 months?
"But the detail of the give and take of the problem solving which is reflected in the narrative of some of these clinical study reports is internal sensitive information which is nowhere reflected in the label. The process of getting these products approved with the regulatory agencies is a give and take of issues, challenges, reworking of data in response to regulator’s concerns, or concerns that we have identified or raised ourselves, which needs to be explained or articulated in documents that we submit to regulators to get products approved…"
The other thing affecting me is three days of watching back to back Nelson Mandela retrospectives. As he became a symbol for the anti·apartheid movement, he was offered a number of opportunities for a conditional release from prison – if you’ll do this, or if you’ll not do that, we’ll let you out. At the risk of being accused of a melodramatic comparison, he did the same thing as Gandhi and MLK. He said "no thanks" to the increasingly tempting offers along the way. When F.W. de Klerk finally released him in 1990, it was an unconditional release. When I say melodramatic – it may seem a false drama for me to analogize something as horrible as apartheid in South Africa with the chronic and widespread misuse of science in the presentation of the efficacy and safety of pharmaceuticals. I don’t think so. It’s a reliance on a precedent that shouldn’t have been set, a privilege that shouldn’t have been granted in the first place, one that benefits a few at the expense of many, and one that promises to do the same in the future. That’s not drama. That’s the truth. So why compromise?
I’ve wandered far from the bureaucratic annoyances encountered by Asbjørn Hróbjartsson into the more central problem of data secrecy [it seems like I can get there from almost any starting place these days]. So back to his point, I have had the opposite reaction in my own data quests. I thought that Freedom of Information requests to the FDA would be difficult, but they have been a snap. They do it for free if you’re an infrequent requester, and charge only a modest fee if you’re a heavy user. Fill out a form, make a few perfectly reasonable promises, and a disk arrives shortly with whatever they’ve got. So far, it has been a surprisingly easy task. Unlike the experience described above, they seem to see access as a service responsibility. I would think that the governmental agencies like the VA, the NIH, the NIMH, the CDC, etc. would all do well to follow the FDA’s lead. It’s all public data after all. With economic or population data, you can Google just about anything you want to find. Why is scientific information so different?
Why did it take 19 months to retrieve clinical trial data from a non-profit organisation?
There is no, and never has been, a non-profit organisation. It is a myth. 😉
But your post is too complex for me to comment now… I will have to read and digest a few times such is my lack of intellect. But my first intuitive response is that this is one of the most important pieces you have writtten. (and this is not devaluing your previous posts…)