The Trolley Problem With Science
Discover MagazineBy NeuroskepticJune 20, 2013Imagine a scientist who does an experiment, and doesn’t like the results. Perhaps the scientist had hoped to see a certain pattern of findings and is disappointed that it’s not there. Suppose that this scientist therefore decided to manipulate the data. She goes into the spreadsheet and adds new, made-up data points, until she obtains a statistically significant result she likes, and publishes it. That’s bad.
Now, start this scenario over. Suppose that rather than making up data, the scientist throws it out. She runs the experiment again and again [without changing it], throwing out the results every time they’re wrong, until eventually, by chance, she obtains a statistically significant result she likes, and publishes it. Is that bad?
Yes, but isn’t it less obviously bad than data fabrication? I’m talking about an intuitive level. We feel that fabrication is clearly outrageous, fraudulent. Cherry-picking is bad, no-one denies it, but it doesn’t generate feelings to the same extent. Cherry-picking goes on in science, and I don’t know a scientist who doubts that it’s more common than fraud. Yet we treat fraud much more harshly, regardless of the extent of the manipulation or the amount of money and prestige at stake. One fabricated point of data is misconduct; a thousand unpublished points of data is merely a ‘questionable practice‘. Why?
I think it’s a trolley problem.
Imagine an out of control trolley was going to smash into five people and kill them. You happen to be standing by a lever that would divert the trolley onto another track, where it would hit a person, and kill him. Do you pull it? Many people say that they would not pull the lever, even though this means that four extra deaths occur. They wouldn’t pull it because they don’t like the idea of committing the act of killing someone. Even though the decision not to act causes more harm, it doesn’t feel as bad, because it’s [in some intuitive sense] an act of omission. Even people who do pull the lever feel conflicted about it.
I think the mentality is the same in science. Data fabrication is like pulling the lever – it’s committing deception. Cherry-picking is a sin of omission – you wait for the right data to come to you, and report that, omitting to report the rest of the data that points to a different conclusion. The end result is the same – misleading results. But they feel different.
The way he frames the Trolley Problem addresses something very real in the world of Clinical trials, the strategic place of "missing studies." It’s the "All" the AllTrials petition and the "I" in the RIAT proposal ["I" for Invisible] – studies that have gone missing. It’s a pretty obvious problem. If you do five clinical trials and have one that’s significant at the p<0.05 level and the others show no significant difference, it’s a bust – unless you only publish the significant study. And that’s what has been done over and over. Here’s a recycled piece from an earlier blog post [gone missing…] to make that point real:
by Erick H. Turner, Annette M. Matthews, Eftihia Linardatos, Robert A. Tell, and Robert RosenthalNew England Journal of Medicine. 2008 358:252-260.
[full text on-line]
Background: Evidence-based medicine is valuable to the extent that the evidence base is complete and unbiased. Selective publication of clinical trials — and the outcomes within those trials — can lead to unrealistic estimates of drug effectiveness and alter the apparent risk–benefit ratio.Methods: We obtained reviews from the Food and Drug Administration [FDA] for studies of 12 antidepressant agents involving 12,564 patients. We conducted a systematic literature search to identify matching publications. For trials that were reported in the literature, we compared the published outcomes with the FDA outcomes. We also compared the effect size derived from the published reports with the effect size derived from the entire FDA data set.Results:Among 74 FDA-registered studies, 31%, accounting for 3449 study participants, were not published. Whether and how the studies were published were associated with the study outcome. A total of 37 studies viewed by the FDA as having positive results were published; 1 study viewed as positive was not published. Studies viewed by the FDA as having negative or questionable results were, with 3 exceptions, either not published [22 studies] or published in a way that, in our opinion, conveyed a positive outcome [11 studies]. According to the published literature, it appeared that 94% of the trials conducted were positive. By contrast, the FDA analysis showed that 51% were positive. Separate meta-analyses of the FDA and journal data sets showed that the increase in effect size ranged from 11 to 69% for individual drugs and was 32% overall.Conclusions:We cannot determine whether the bias observed resulted from a failure to submit manuscripts on the part of authors and sponsors, from decisions by journal editors and reviewers not to publish, or both. Selective reporting of clinical trial results may have adverse consequences for researchers, study participants, health care professionals, and patients.
Is that the problem of the PHARMA Executive? Is he in a double-bind? an impossible situation? driven by the shape of his self system? Hell no. He’s not in a double bind, he’s just making a choice about which way to cheat – fraud or cherry picking. Neuroskeptic is 100% correct that from the outside we might see the latter as a sin of omission – a lesser sin. But more to the point, the Executive is way much less likely to get caught by not publishing. Until I saw Ben Goldacre’s first Ted Talk nine months ago, I had never even thought about missing trials as a strategy. And having spent some time trolling the FDA files and clinicaltrials.gov, I have nothing but respect for the authors of that study up there for their stick-to-it-ness. The Invisible in Restoring Invisible and Abandoned Trials was well chosen term – they really are.
The kind of double bind the PHARMA Marketing Executive lives with isn’t like that of Styron’s Sophie or Neuroskeptic’s Trolley Switchman, it’s the one we call Conflict of Interest. It’s a conflict between getting caught jury-rigging a study or getting caught burying the negative ones. Like the other kinds of double binds, there are two wrong choices, and going missing is simply the safer pick. But this kind of double-bind has a right choice – give up on the ineffective drug and look for another. That’s why we call it a Conflict of Interest, because to make the right choice is not in the PHARMA Marketing Executive’s or his company’s best interest. Neither Sophie nor the Switchman have criminal intent. Our Executive does, and his bind is more in the range of the risk assessment of a thief measuring the odds of getting caught.
The Real Double Binds:
Who is in the real double bind? The practicing doctor and the patient. As a doctor I’m reading the respected peer reviewed journal that I’ve counted on since medical school, and I read "Paroxetine is generally well tolerated and effective for major depression in adolescents." I’m counting on the editor, the reviewers, the academic physicians at the honored universities who wrote the article and tested the drug to give me the straight story. But I now know that some of these clinical trials have been jiggered with. I know that sometimes they hide the negative studies. Is this one of the bad ones? Are there even any good ones? Which way to turn?
As a patient, I’m used to trusting the doctor. But I know some doctors are "KOLs" or are influenced by Drug Reps. I know that some of the information given to doctors has been tampered with. But I’m really depressed and desperate for relief. My friend said try exercise and meditation, but it hasn’t helped. I’m afraid of medication and afraid not to take it. Which way to turn?
The American Diabetes Association is on board- with a similar RIAT idea. I found this link from one of the comments at Davidhealy.org (Johanna)
http://www.diabetes.org/for-media/2013/american-diabetes-association-incretin-therapy.html
So an association is calling for patient level data.
Half the data gets tossed, and the drugs barely beat placebo – again and again, after half the data is tossed!
The trolley double-bind.
I’m truly lost with this analogy…
If I understand, it becomes a moral dilemma somehow…
As if people die if you present all the data, but fewer die if you do not?
What?
Duane
I suppose if the researcher had the false impression that the drug might save lives…
This is certainly not the case with antidepressants…
1 in 10 adults on them, with increase in suicide rates.
“Save lives” by improving quality of life?
Only is becoming fat and asexual is an improvement.
No, the double-bind analogy of the researcher being at the switch of the trolley car just doesn’t work for me… The dilemma of having to pay the mortgage would be better.
Duane
OK, the author simply saying that omission is easier than an action – even if omission ends with the same result. Apologies for the confusion, back-to-back comments.
Duane
a-non, you state: The American Diabetes Association is on board- with a similar RIAT idea. I found this link from one of the comments at Davidhealy.org (Johanna)
Amazingly, after more than 2 decades, the ADA has never advocated for investigation of rDNA synthetic insulin–used by all Type 1 diabetics. The original mantra was “it’s just like the human body makes” . . . but no proof was required. The only requirement seems to be that it (rDNA insulin) lowered blood glucose and didn’t kill the patient immediately. Natural products–created in nature–are no longer available to American diabetics; but no investigation or validation requirements are on the horizon. Interestingly, “dead-in-bed syndrome” and “hypoglycemia unawareness” have entered the vernacular (10 years ago medicine was denying that either of these existed) . . . but conveniently, the onus can still be placed squarely on the patient or the disease–never the insulin.
Dr. Nardo,
You have openly stated on this site that one’s ethics are critical in medicine/psychiatry when focusing on those like Nemerof. You have also said that the fraud has to stop in psychiatry.
You are using an example of the dangers of diabetes treatment due to shoddy science as well as psychiatric drugs.
If you are going to be somewhat of a conscience of psychiatry, do you think it is right to remain silent or fudge the evidence or “cherry pick” the types of psychiatric treatments you will study and report on objectively based on the true, factual evidence rather than old boy network cronyism, which appears to be the case with ECT and other probable new barbaric practices on the horizon? There are many studies/expert opinions out there like Sackeim, Bentall and Read and many others as opposed to fraudulent infomercials by the current ECT pushers like Max Fink, Michael Taylor, Edward Shorter and David Healy who should know better.
Here is one woman’s experience of psychiatric drugging and ECT that led to diabetes, the typical brain damage, permanent loss of memory and disability of such horrific treatments that has caused her to be a firm advocate against ECT like all too many other victims.
http://www.madinamerica.com/2013/04/did-electroshock-save-my-life/
I realize I risk raising the ire of you and certain ECT advocates here, but doesn’t your ethical stance on either cherry picking or omitting the negative data altogether apply to barbaric practices like ECT too rather than the cherry picked, twisted anecdotal “evidence” of those like David Healy and Max Fink I posted on another recent article here?
Since many psychiatric drugs are going off patent and have already been exposed for being useless and dangerous, I am concerned that the more such drugs get demonized, the more people will be forced into even more dangerous ECT and other worse brain damaging/disabling “treatments” all too typical of psychiatry. So, I question if demonizing the drugs like you, David Healy and others do will just cause more women, children and the elderly to be subjected to forced ECT as is now being done at an appalling, increasing rate that also increases stroke and death rates. One article/review of David Healy’s book I posted here discussed all the dangerous treatments of psychiatry that initially win great support and enthusiasm only to be found to be life destroying and barbaric in the long run as with all types of lobotomy including ECT.
I reviewed Michael Taylor’s book, HIPPOCRATES CRIED, more and found it to be mostly another infomercial for ECT with Taylor part of the select ECT pushing crowd that includes Max Fink, David Healy, Edward Shorter and other cheerleaders here. I am also appalled to find that catatonia was pushed into the DSM 5 by Max Fink as another ploy to push ECT as does David Healy. The same is true for the push to include melancholia, which appears to me to occur in people who have suffered stress breakdowns possibly from toxic relationships and other severe stressors. In my opinion, the last thing anyone needs is brain damage, amnesia and the trauma of forced psychiatric treatment abuse when already in crisis since this just causes massive retraumatization according to many victims of it.
So, I challenge you with the question if you can be consistent with your own ethics to carefully examine ALL the evidence of ALL psychiatric treatments objectively and truthfully and honestly report the results wherever they lead. Perhaps you have your own blind spots, double binds, difficult choices and certain loyalties that make it difficult to see or admit the truth believing you are choosing the lesser of evils when it isn’t really your choice to make?
“All is takes for evil to win is for good people to say and do nothing.”
This applies to ECT and Max Fink just like it applies to SSRI’s and Nemerof in my opinion.
Correction:
It appears others here (not Dr. Nardo) have used the diabetes analogy for harmful medical/psychiatry practice as did David Healy in his latest post on harmful medical practices.
FYI
the institute of psychiatry at maudsely has been a focus of attention recently due to the lecture by nemeroff
the link below takes you to a report of their DSM 5 conference on June 10
http://brc.slam.nhs.uk/our-blog/brcu-blog/2013/dsm-5-at-the-iop
One of my clients had been a diabetic since childhood and she was not happy with Avandia. It simply did not work as well for her as insulin and she felt powerless to do anything about it.
In what ways are the hypotheses about the effectiveness of psychiatric drugs for various categories of “mental illness” as described in the DSM falsifiable?
Fun fact: the trolley problem was designed by woman philosopher and ethicist Philipa Foot in 1967.
FYI
i think you would find this paper is a very good read
DxSummit.org
The Global Summit on Diagnostic Alternatives: An Online Platform for Rethinking Mental Health
DSM-5?s Somatic Symptom Disorder: From Medical Enigma to Psychiatric Sphinx
Sarah Kamens • June 2, 2013
The author would like to thank Frederick Wertz, Suzy Chapman, Donald Kamens, Ben Kamens, and Charles Olbert for their input on this manuscript.
This paper was presented at the Interdisciplinary Graduate Student Symposium in Philosophy & Psychology, Fordham University, March 19, 2013.
http://dxsummit.org/archives/712
If you have the time and inclination would be curious to see you dissect this:
http://blogs.plos.org/mindthebrain/2013/06/20/do-rising-rates-of-antidepressant-prescription-translate-into-lower-rates-of-suicide-evidence-from-29-countries/#comment-97767
Particularly since he had waxed so positive on Gibbons:
http://blogs.plos.org/mindthebrain/2012/12/26/the-antidepressant-wars-a-sequel-how-the-media-distort-findings-and-do-harm-to-patients/
In this example the patient and doctor have been funneled into an unhealty interaction because of lack of boundries. So the establisment of definative fences would make a better interaction. So I guess the real question is how to establish the proper informational fence where neither party gets funneled into one trap or another. How could this interaction be made healthy?
http://davidhealy.org/we-have-a-dream-getting-engaged-to-a-doctor/
Great article, Jamzo. Was just studying a bit on ontology and contemplating the hubris of bio-psychiatry writ large with its attempt to define what a person essentially is and how “mentally ill” they are by reifying psychiatry’s fickle history of etiologies; and using them as an excuse to dig into the brain (writ large) without sufficiently examining their own reductionist efforts to define what is valid and without considering a person’s account of themselves as being meaningful, or even relevant beyond the psychiatrist’s prerogative to judge it legitimate or not based on his perception of the height of his perch.
Bigotry. Self-idolatry.
“Amazingly, after more than 2 decades, the ADA has never advocated for investigation of rDNA synthetic insulin–used by all Type 1 diabetics. The original mantra was “it’s just like the human body makes” . . . but no proof was required. The only requirement seems to be that it (rDNA insulin) lowered blood glucose and didn’t kill the patient immediately. Natural products–created in nature–are no longer available to American diabetics; but no investigation or validation requirements are on the horizon. Interestingly, “dead-in-bed syndrome” and “hypoglycemia unawareness” have entered the vernacular (10 years ago medicine was denying that either of these existed) . . . but conveniently, the onus can still be placed squarely on the patient or the disease–never the insulin.”-Melody
I wonder just what the purpose of those associations are for to begin with. When the black box is open are we going to find far less than we expected? I hope Dr. John Ioannidis is wrong.
“Lies, Damned Lies, and Medical Science” The Atlantic Nov. 2010 David H. Freedmen
http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/308269/