by Rosa Ahn, Alexandra Woodbridge, Ann Abraham, Susan Saba, Deborah Korenstein, Erin Madden, W John Boscardin, and Salomeh Keyhani.BMJ 2017 356:i6770
Objective: To examine the association between the presence of individual principal investigators’ financial ties to the manufacturer of the study drug and the trial’s outcomes after accounting for source of research funding.Design: Cross sectional study of randomized controlled trials [RCTs].Setting: Studies published in “core clinical” journals, as identified by Medline, between 1 January 2013 and 31 December 2013.Participants: Random sample of RCTs focused on drug efficacy.Main outcome measure: Association between financial ties of principal investigators and study outcome.Results: A total of 190 papers describing 195 studies met inclusion criteria. Financial ties between principal investigators and the pharmaceutical industry were present in 132 [67.7%] studies. Of 397 principal investigators, 231 [58%] had financial ties and 166 [42%] did not. Of all principal investigators, 156 [39%] reported advisor/consultancy payments, 81 [20%] reported speakers’ fees, 81 [20%] reported unspecified financial ties, 52 [13%] reported honorariums, 52 [13%] reported employee relationships, 52 [13%] reported travel fees, 41 [10%] reported stock ownership, and 20 [5%] reported having a patent related to the study drug. The prevalence of financial ties of principal investigators was 76% [103/136] among positive studies and 49% [29/59] among negative studies. In unadjusted analyses, the presence of a financial tie was associated with a positive study outcome [odds ratio 3.23, 95% confidence interval 1.7 to 6.1]. In the primary multivariate analysis, a financial tie was significantly associated with positive RCT outcome after adjustment for the study funding source [odds ratio 3.57 [1.7 to 7.7]. The secondary analysis controlled for additional RCT characteristics such as study phase, sample size, country of first authors, specialty, trial registration, study design, type of analysis, comparator, and outcome measure. These characteristics did not appreciably affect the relation between financial ties and study outcomes [odds ratio 3.37, 1.4 to 7.9].Conclusions: Financial ties of principal investigators were independently associated with positive clinical trial results. These findings may be suggestive of bias in the evidence base.
Discussion
We found that more than half of principal investigators of RCTs of drugs had financial ties to the pharmaceutical industry and that financial ties were independently associated with positive clinical trial results even after we accounted for industry funding. These findings may raise concerns about potential bias in the evidence base.
Possible explanations for findings
The high prevalence of financial ties observed for trial investigators is not surprising and is consistent with what has been reported in the literature. One would expect industry to seek out researchers who develop expertise in their field; however, this does not explain why the presence of financial ties for principal investigators is associated with positive study outcomes. One explanation may be “publication bias.” Negative industry funded studies with financial ties may be less likely to be published. The National Institutes of Health [NIH]’s clinicaltrials.gov registry was intended to ensure the publication of all trial results, including both NIH and industry funded studies, within one year of completion. However, rates of publication of results remain low even for registered trials…Other possible explanations for our findings exist. Ties between investigators and industry may influence study results by multiple mechanisms, including study design and analytic approach. If our findings are related to such factors, the potential solutions are particularly challenging. Transparency alone is not enough to regulate the effect that financial ties have on the evidence base, and disclosure may compromise it further by affecting a principal investigator’s judgment through moral licensing, which is described as “the unconscious feeling that biased evidence is justifiable because the advisee has been warned.” Social experiments have shown that bias in evidence is increased when conflict of interest is disclosed. One bold option for the medical research community may be to adopt a stance taken in fields such as engineering, architecture, accounting, and law: to restrict people with potential conflicts from involving themselves in projects in which their impartiality could be potentially impaired. However, this solution may not be plausible given the extensive relationship between drug companies and academic investigators. Other, incremental steps are also worthy of consideration. In the past, bias related to analytic approach was tackled by a requirement for independent statistical analysis of major RCTs. Independent analysis has largely been abandoned in favor of the strategy of transparency, but perhaps the time has come to reconsider this tool to reduce bias in the analysis of RCTs. This approach might be especially effective for studies that are likely to have a major effect on clinical practice or financial implications for health systems. Another strategy to reduce bias at the analytic stage may be to require the publishing of datasets. ICMJE recently proposed that the publication of datasets should be implemented as a requirement for publication. This requirement is increasingly common in other fields of inquiry such as economics. Although independent analyses at the time of publication may not be feasible for journals from a resource perspective, the requirement to release the dataset to be reviewed later if necessary may discourage some forms of analytical bias. Finally, authors should be required to include and discuss any deviations from the original protocol. This may help to prevent changes in the specified outcome at the analytic stage…
-
The Randomized Controlled Trial [RCT] is a good way to determine clinical usefulness.
In 1962, the FDA was charged with requiring two Randomized Controlled Trials [RCTs] demonstrating statistical efficacy and all human usage data demonstrating safety in order to approve a drug for use. It’s a weak standard, designed to keep inert potions off the market. It was presumed that the medical profession would have a higher standard and determine clinical usefulness. That made [and makes] perfect sense. The FDA primarily insures safety and keeps swamp root and other patent medicines out of our pharmacopeia, but clinical usefulness should be determined by the medical profession and our patients. Not perfect, but I can’t think of a better system for approval. However, approval doesn’t necessarily correlate with clinical usefulness, or for that matter, long term safety. And then something unexpected happened. The Randomized Controlled Trials became the gold standard for everything – called Evidence Based Medicine. Randomized Clinical Trials are hardly the only form of valid evidence in medicine. That was a reform idea that kept people from shooting from the hip, but was also capable of throwing the baby out with the bathwater.
This structured procedure designed to dial out everything and isolate the drug effect [RCTs] became a proxy for the much more complex and varied thing called real life. RTCs have small cohorts of recruited [rather than help-seeking] subjects in short-term trials. Complicatred patients are eliminated by exclusionary criteria. The metrics used are usually clinician-rated rather than subject-rated. And the outcome is measured by statistical significance instead of by the strength of the response. Blinding and the need for uniformity eliminates any iterative processes in dosing or identifying target symptoms. It’s an abnormal situation on purpose, suitable for the yes-no questions in approval, but not the for-whom information of clinical experience.
-
It is ever going to be possible to create a system that insures that the industry sponsors will openly report on their RCT without exaggerrating efficacy and/or understating toxicity.
These RCTs were designed for submission to the FDA for drug approval. The FDA reviewers have access to the raw data and have regularly made the right calls. But then those same studies are written up by professional medical ghost writers, signed onto by KOL academic physicians with florid Conflicts of Interest and submitted to medical journals to be reviewed by peer reviewers who have no access to the raw data. The journals make money from selling reprints back to to the sponsors for their reps to hand out to practicing doctors. These articles are where physicians get their information, and discrepancies between the FDA version and the Journal versions are neither discussed, nor even easy to document.
So it’s not the FDA Approval that’s the main problem. It’s the glut of journal articles that have been crafted from those studies and been the substrate for advertising campaigns that have caused so much trouble. The basic Clinical Trials that were part of the Approval have been glamorized. And many trials that were unsuccessful attempts at indication creep have been spun into gold. It seems that every time there’s an attempt to block the fabulation of such trials, there have been countermoves that render the reform attempts impotent. So far, it’s been a chess game that never seems to get to check-mate.
Structured RCTs may well be the best method for our regulatory agencies use to evaluate new drugs. They cost a mint to do and about the only people who can fund them are the companies who can capitalize on success – the drug companies. But medicine doesn’t need to shouldn’t buy into the notion that they’re the only way to evaluate the effectiveness of medicinal products. As modern medicine has become increasingly organized and documented, there are huge caches of data available. And it’s not just patient data or clinic data. What about the pharmacy data that’s already being used by PHARMA to track physician’s prescribing patterns? And where are the departments of pharmacology and the schools of pharmacy in following medication efficacy and safety? or the HMOs? or the Health Plans? the VAH? What about the waiting room questionnaires? I’d much rather they ask about the medications the patient is on than being used to screen for depression. It’s really the ongoing data after a drug is in use that clinicians need anyway – more important than the RTC that gets things started.
Your post reminds me of a section that really stuck with me when I read an interview with Ben Goldacre about 3 years ago:
“Q: While reading Bad Pharma, I couldn’t help but think that we will look back at the way medicine is practiced today in the same way we look at something like bloodletting now.
A: That’s right. There’s this great line from Muir Gray (chief knowledge officer of the National Health Service in the UK) about John Snow, the great-grandfather of epidemiology who spotted the cholera epidemic in London. Gray says that in the 19th century, we made huge leaps in medicine with clean, clear water. In the 21st century, we’ll make the same leaps with clean clear information. That is where the action is. People will look back and say, ‘What on earth were you doing? Why were you relying on small crappy trials in unrepresentative patients? How could you expect to know what the true benefits of your treatments were? You must all be out of your mind.’
Q: So what will this revolution in medicine look like?
A: I think we have failed at getting a competent information architecture for evidence-based medicine. So many of these problems are only problems because medicine isn’t very good at finding out what works, bringing all the evidence together, and then making sure we get that information to the right person—the doctor, the patient—at the right time. If we were good at synthesizing and disseminating evidence to clinicians, marketing would be irrelevant. The only reason it matters that the majority of continuing medical education for doctors is funded by the pharmaceutical industry is because there’s no other better way of disseminating information to clinicians. Similarly, the problems of clinical trials being done in small numbers of unrepresentative people, comparing new treatments against nothing instead of the currently best-available treatment—all those problems should really be solved by embedding randomized controlled trials into everyday, routine clinical practice. You can run trials comparing one statin against another for almost no cost, using (National Health Service) electronic health records. If we did that, frankly, the fact that industry wants to fund trials against nothing—it would become irrelevant.”
http://www.macleans.ca/culture/books/talking-with-ben-goldacre-about-his-new-book-bad-pharma/
The issue of “proprietary data” is obviously a broad one, For example see the discussion here: http://ncjolt.org/wp-content/uploads/2014/09/Conley_Final2.pdf
I find this quote of particular interest:
“The only alternative to patents is not “open source”, -it is also trade secrets.
Virtually unmentioned by anyone is the fact that absent patents on isolated nucleic acids, and post Prometheus v. Mayo, fewer phenotype-biomarker associations will be published at all. Already there are companies based on proprietary tissue samples, from 23 and Me, to µBiome, to Dognition to Champions Oncology. Proprietary medical data and tissue collections provide a never expiring first mover advantage analogous to the customer and subscriber lists of Facebook, Google, Amazon and Target.”
http://www.ipwatchdog.com/2013/04/15/forward-looking-personalized-medicine-patent-law-and-science/id=39163/
This quote here was also interesting:
” If I ever do get my genome sequenced, it will be made publicly available or not at all (and I’ll be doing the sequencing, not some company).
FWIW, basically all of my geneticist friends have done it in spite of those concerns.”
http://www.ndnation.com/boards/showpost.php?b=backroom;pid=495743;d=all
Under be careful what you wish for:
http://vitalysthealth.org/09_12_2003/
And we get:
http://hcrenewal.blogspot.com/2015/09/politico-2015-ehr-sellers-using-gag.html
The health IT article cited by 1bom is Orwellian and I highly recommend everyone read it in its entirety. This stuff is right out of the movie Brazil. Judith Faulker is a menace to society.
1bom, NHS engaging with Google Deep Mind also comes to mind. But not sure how that fits in within what you have in mind. Who gets to see that data, and how it’s used, is still tightly controlled by the NHS. Though it appears that the work with Google does not involve excluding other vendors for future projects. Though the NHS Deep Mind work is more to detect early signs of trouble and alert clinicians using machine learning, one can see how such partnerships could also extend to looking at comparative drug effectiveness. I wonder though whether the creation of the Creative Commons licenses might be instructive? Especially in light of that geneticists comment I linked to above. Would people be willing to put their medical information in the public domain so that gatekeepering is less in the hands of large entities (be they private or public)? Part of the challenge might be that sharing such information publicly, especially genetic information, has potential impact on your kids even if you aren’t alive anymore. It’s not clear to me how much following the open data paradigm requires reconceptualizing medical privacy.
1bom,
“…and our seemingly unlimited desire to make obscene amounts of money through competitive advantage.” A large part of why I linked to that 2003 posting was that line.
Control of proprietary data and/or proprietary data analysis conveys competitive advantage. This is no less powerful a force outside of Pharma as within. The story of Linux seems to be one of harnessing collective individual action. In a way that made it difficult to hijack to monopolize and could stand up to legal challenges. And, importantly, something the collective could sustain.
Again, expanding far beyond RCTs is certainly feasible within our current approaches. But if you are looking for a system that can hold up robustly against getting warped for competitive advantage (financial or ideological) then I increasingly wonder about how much we would need to reconceptualze privacy. NOT because you couldn’t theoretically create openness and maintain equivalent privacy but because you need large entity buy in to achieve that and it’ll never happen in practice.
My concern about expanding data collection beyond RCTs to other sources is that placebo comparisons are not possible. Thus, the effectiveness of a pill above and beyond a placebo is unknown. Irving Kirsch has presented excellent evidence that the pills we call “antidepressants” owe their efficacy simply to the placebo effect, and the penetration of the double blind due to the presence of side-effects. This conclusion would not be possible by analyzing pharmacy data, no matter how much data is available.
I’m wondering if financial bias is the worst kind of bias. You do see some dissenting work by some with apparent pharma COI. The kind of bias that scares me the most from my own experience is that related to the experimenter’s own personal demons. I saw this first hand at the academic centers involved the junk science leading up to the McMartin trial, where rational inquiry was treated as heresy.
@ Eric — Excellent point about data collection from ‘other sources.’ On a similar note, one of the issues with EHRs– at least the one that I worked on– is that they are so opaque that removing errors and other forms of bad data is almost impossible. There was a ton of junk we put into ours when we set it up– hundreds of records of test data. Without the test data, we would never have been able to write the manual for the system, or even figure out how it worked, since the developer never answered the phone. Some of it was almost indistinguishable from real patient data.
Personally, when it comes to technology, I generally prefer my phone to be a phone and my camera to be a camera. There’s nothing wrong with using your phone as a camera, or a guitar tuner, or any number of other devices, but usually the result will not be as good. It’s a backup.
Likewise, it seems to me that the EHR can never be set up to be a research tool. The method of data collection has to be part of the study design, yes? Therefore, it’s probably not a good idea to consider a medical record as a potential swiss-army-knife data collector.
So it would be fine to collect data from an EHR and make some broad, general inferences for administrative purposes, just not for serious research. And even then, when we tried to use the EHR to figure out something simple like the rate of missed appointments (important for grant purposes), we never could get reliable numbers. One clinician would get dinged for having a 40% no-show rate, but then one of the OTHER databases would show that this number was only 15%, etc.
@1BYM — Great link. And I am well aware of the “gag” clause in EHR contracts, and this is one of several reasons why I only post my full name here once a year or so. As an independent contractor, I am pretty sure that my contract did not include that language. But it’s remotely possible that somewhere in my contract I agreed to abide by terms of my clinic’s contract. And I have posted entire pages of the ludicrous document I created which describes the data entry procedures for our EHR, described the errors that resulted, and named the developer.
@ Dr. OBrien: “Experimenter’s personal demons” = Oh, yeah. One of my professors was targeted by the McMartin trial.
I don’t know if it’s the most pervasive kind of bias, but it’s probably the most insidious, perhaps because it’s covert. And there’s a horrible veneer of self righteousness and arrogance that goes along with that.
You frequently mention “Brazil” and Dr. H mentioned “Back to the Future” in the previous thread, but this idea sounds more like “Forbidden Planet.” The idea of ‘Monsters from the Id’ may seem superficially quaint and dated but… well, maybe not so much. There’s a ghastly veneer of arrogance and self righteousness that works with that. No one is immune to it.
“Such portions then of the Krell science as I may, from time to time, deem suitable and safe, I shall dispense to Earth. Other portions, I shall withhold. And in this, I shall be answerable exclusively to my own conscience and judgment.” — Dr. Morbius.