impossible situations…

Posted on Thursday 20 June 2013


By Neuroskeptic
June 20, 2013

Imagine a scientist who does an experiment, and doesn’t like the results. Perhaps the scientist had hoped to see a certain pattern of findings and is disappointed that it’s not there. Suppose that this scientist therefore decided to manipulate the data. She goes into the spreadsheet and adds new, made-up data points, until she obtains a statistically significant result she likes, and publishes it. That’s bad.

Now, start this scenario over. Suppose that rather than making up data, the scientist throws it out. She runs the experiment again and again [without changing it], throwing out the results every time they’re wrong, until eventually, by chance, she obtains a statistically significant result she likes, and publishes it. Is that bad?

Yes, but isn’t it less obviously bad than data fabrication? I’m talking about an intuitive level. We feel that fabrication is clearly outrageous, fraudulent. Cherry-picking is bad, no-one denies it, but it doesn’t generate feelings to the same extent. Cherry-picking goes on in science, and I don’t know a scientist who doubts that it’s more common than fraud. Yet we treat fraud much more harshly, regardless of the extent of the manipulation or the amount of money and prestige at stake. One fabricated point of data is misconduct; a thousand unpublished points of data is merely a ‘questionable practice‘. Why?

I think it’s a trolley problem.

Imagine an out of control trolley was going to smash into five people and kill them. You happen to be standing by a lever that would divert the trolley onto another track, where it would hit a person, and kill him. Do you pull it? Many people say that they would not pull the lever, even though this means that four extra deaths occur. They wouldn’t pull it because they don’t like the idea of committing the act of killing someone. Even though the decision not to act causes more harm, it doesn’t feel as bad, because it’s [in some intuitive sense] an act of omission. Even people who do pull the lever feel conflicted about it.

I think the mentality is the same in science. Data fabrication is like pulling the lever – it’s committing deception. Cherry-picking is a sin of omission – you wait for the right data to come to you, and report that, omitting to report the rest of the data that points to a different conclusion. The end result is the same – misleading results. But they feel different.
Neuroskeptic presents us with an interesting variant on the theme of the double-bind [impossible situation] because it adds another variable – volition and responsibility. So no matter what you do, you’re wrong – a variation on the classic paradigm of Sophie’s Choice by William Styron. In the usual double bind, the right thing to do in an impossible situation is nothing. But in Sophie’s Choice, that option was taken from her. Choose which child will be killed, but if you don’t choose, both will be killed. In Neurospeptic’s Trolley Problem, there’s a right choice [pull the lever], but you become a murderer in the process. If you make the wrong choice [do nothing], you are guilty of murder five times over, but you have plausible deniability – "I was paralyzed", something like that.

The way he frames the Trolley Problem addresses something very real in the world of Clinical trials, the strategic place of "missing studies." It’s the "All" the AllTrials petition and the "I" in the RIAT proposal ["I" for Invisible] – studies that have gone missing. It’s a pretty obvious problem. If you do five clinical trials and have one that’s significant at the p<0.05 level and the others show no significant difference, it’s a bust – unless you only publish the significant study. And that’s what has been done over and over. Here’s a recycled piece from an earlier blog post [gone missing…] to make that point real:

by Erick H. Turner, Annette M. Matthews, Eftihia Linardatos, Robert A. Tell, and Robert Rosenthal
New England Journal of Medicine. 2008 358:252-260.
[full text on-line]

Background: Evidence-based medicine is valuable to the extent that the evidence base is complete and unbiased. Selective publication of clinical trials — and the outcomes within those trials — can lead to unrealistic estimates of drug effectiveness and alter the apparent risk–benefit ratio.
Methods: We obtained reviews from the Food and Drug Administration [FDA] for studies of 12 antidepressant agents involving 12,564 patients. We conducted a systematic literature search to identify matching publications. For trials that were reported in the literature, we compared the published outcomes with the FDA outcomes. We also compared the effect size derived from the published reports with the effect size derived from the entire FDA data set.
Results:Among 74 FDA-registered studies, 31%, accounting for 3449 study participants, were not published. Whether and how the studies were published were associated with the study outcome. A total of 37 studies viewed by the FDA as having positive results were published; 1 study viewed as positive was not published. Studies viewed by the FDA as having negative or questionable results were, with 3 exceptions, either not published [22 studies] or published in a way that, in our opinion, conveyed a positive outcome [11 studies]. According to the published literature, it appeared that 94% of the trials conducted were positive. By contrast, the FDA analysis showed that 51% were positive. Separate meta-analyses of the FDA and journal data sets showed that the increase in effect size ranged from 11 to 69% for individual drugs and was 32% overall.
Conclusions:We cannot determine whether the bias observed resulted from a failure to submit manuscripts on the part of authors and sponsors, from decisions by journal editors and reviewers not to publish, or both. Selective reporting of clinical trial results may have adverse consequences for researchers, study participants, health care professionals, and patients.
    First off, this article only looks at the studies submitted to the FDA. The FDA rule is that the sponsor is required to submit at least two studies showing significant efficacy, but is required to submit all studies done to look at safety…  So here’s what they found when they looked at whether the studies submitted to the FDA were published in the peer-reviewed literature:
    The Published, conflicts with FDA decision category refers to studies that the FDA said were negative or questionable that were published as positive [dodgy studies]. The Not published categories refers to the studies gone missing. They go on to show us which pharmaceutical companies were the worst offenders:
    So that’s a 45% fudge factor [11 dodgy + 22 gone missing ÷ 74 total trials]. That’s totally horrible! Grand Jury horrible!
So let’s go back and pretend analyze the hapless victim at the lever in the Trolley Problem. If he’s an internally secure, self assured guy, he pulls the lever without a thought. If anyone later calls him a murderer, he looks at them like they’ve got rocks for brains and says, "I saved four lives!" But if he’s a person whose self image is determined by the opinions of others, real or imagined, he really is paralyzed by what he thinks others will think [or what his overly harsh conscience will think]. How he is viewed [externally or internally] over-rides his logic [and his basic arithmetic].

Is that the problem of the PHARMA Executive? Is he in a double-bind? an impossible situation? driven by the shape of his self system? Hell no. He’s not in a double bind, he’s just making a choice about which way to cheat – fraud or cherry picking. Neuroskeptic is 100% correct that from the outside we might see the latter as a sin of omission – a lesser sin. But more to the point, the Executive is way much less likely to get caught by not publishing. Until I saw Ben Goldacre’s first Ted Talk nine months ago, I had never even thought about missing trials as a strategy. And having spent some time trolling the FDA files and clinicaltrials.gov, I have nothing but respect for the authors of that study up there for their stick-to-it-ness. The Invisible in Restoring Invisible and Abandoned Trials was well chosen term – they really are.

The kind of double bind the PHARMA Marketing Executive lives with isn’t like that of Styron’s Sophie or Neuroskeptic’s Trolley Switchman, it’s the one we call Conflict of Interest. It’s a conflict between getting caught jury-rigging a study or getting caught burying the negative ones. Like the other kinds of double binds, there are two wrong choices, and going missing is simply the safer pick. But this kind of double-bind has a right choice – give up on the ineffective drug and look for another. That’s why we call it a Conflict of Interest, because to make the right choice is not in the PHARMA Marketing Executive’s or his company’s best  interest. Neither Sophie nor the Switchman have criminal intent. Our Executive does, and his bind is more in the range of the risk assessment of a thief measuring the odds of getting caught.


Parenthetically, the two bad choices in this Conflict of Interest aren’t mutually exclusive. GSK jury-rigged Paxil Study 329 and didn’t publish the two negative trials of Paxil in adolescents [377 & 701] until five years later when they were under fire [Evaluation of suicidal thoughts and behaviors in children and adolescents taking paroxetine]…

The Real Double Binds:

Who is in the real double bind? The practicing doctor and the patient. As a doctor I’m reading the respected peer reviewed journal that I’ve counted on since medical school, and I read "Paroxetine is generally well tolerated and effective for major depression in adolescents." I’m counting on the editor, the reviewers, the academic physicians at the honored universities who wrote the article and tested the drug to give me the straight story. But I now know that some of these clinical trials have been jiggered with. I know that sometimes they hide the negative studies. Is this one of the bad ones? Are there even any good ones? Which way to turn?

As a patient, I’m used to trusting the doctor. But I know some doctors are "KOLs" or are influenced by Drug Reps. I know that some of the information given to doctors has been tampered with. But I’m really depressed and desperate for relief. My friend said try exercise and meditation, but it hasn’t helped. I’m afraid of medication and afraid not to take it. Which way to turn?

The real solution for Sophie? for the Switchman? for the Doctor? for the Patient? is for none of them to be in their respective situations in the first place. We can’t help Sophie or the Switchman, but perhaps we should talk more about how doctors and patients can proceed in these current days of confusion instead of just decrying them. For the long haul, we can’t accept the situation continuing. So support the AllTrials petition and the RIAT proposal. They are thus far two solid paths towards reclaiming the knowledge base that has become corrupted for all of us…
  1.  
    a-non
    June 21, 2013 | 4:06 AM
     

    The American Diabetes Association is on board- with a similar RIAT idea. I found this link from one of the comments at Davidhealy.org (Johanna)
    http://www.diabetes.org/for-media/2013/american-diabetes-association-incretin-therapy.html
    So an association is calling for patient level data.

  2.  
    June 21, 2013 | 9:03 AM
     

    Half the data gets tossed, and the drugs barely beat placebo – again and again, after half the data is tossed!

    The trolley double-bind.
    I’m truly lost with this analogy…

    If I understand, it becomes a moral dilemma somehow…
    As if people die if you present all the data, but fewer die if you do not?
    What?

    Duane

  3.  
    June 21, 2013 | 9:10 AM
     

    I suppose if the researcher had the false impression that the drug might save lives…

    This is certainly not the case with antidepressants…
    1 in 10 adults on them, with increase in suicide rates.

    “Save lives” by improving quality of life?
    Only is becoming fat and asexual is an improvement.

    No, the double-bind analogy of the researcher being at the switch of the trolley car just doesn’t work for me… The dilemma of having to pay the mortgage would be better.

    Duane

  4.  
    June 21, 2013 | 9:18 AM
     

    OK, the author simply saying that omission is easier than an action – even if omission ends with the same result. Apologies for the confusion, back-to-back comments.

    Duane

  5.  
    Melody
    June 21, 2013 | 9:28 AM
     

    a-non, you state: The American Diabetes Association is on board- with a similar RIAT idea. I found this link from one of the comments at Davidhealy.org (Johanna)

    Amazingly, after more than 2 decades, the ADA has never advocated for investigation of rDNA synthetic insulin–used by all Type 1 diabetics. The original mantra was “it’s just like the human body makes” . . . but no proof was required. The only requirement seems to be that it (rDNA insulin) lowered blood glucose and didn’t kill the patient immediately. Natural products–created in nature–are no longer available to American diabetics; but no investigation or validation requirements are on the horizon. Interestingly, “dead-in-bed syndrome” and “hypoglycemia unawareness” have entered the vernacular (10 years ago medicine was denying that either of these existed) . . . but conveniently, the onus can still be placed squarely on the patient or the disease–never the insulin.

  6.  
    Donna
    June 21, 2013 | 1:38 PM
     

    Dr. Nardo,

    You have openly stated on this site that one’s ethics are critical in medicine/psychiatry when focusing on those like Nemerof. You have also said that the fraud has to stop in psychiatry.

    You are using an example of the dangers of diabetes treatment due to shoddy science as well as psychiatric drugs.

    If you are going to be somewhat of a conscience of psychiatry, do you think it is right to remain silent or fudge the evidence or “cherry pick” the types of psychiatric treatments you will study and report on objectively based on the true, factual evidence rather than old boy network cronyism, which appears to be the case with ECT and other probable new barbaric practices on the horizon? There are many studies/expert opinions out there like Sackeim, Bentall and Read and many others as opposed to fraudulent infomercials by the current ECT pushers like Max Fink, Michael Taylor, Edward Shorter and David Healy who should know better.

    Here is one woman’s experience of psychiatric drugging and ECT that led to diabetes, the typical brain damage, permanent loss of memory and disability of such horrific treatments that has caused her to be a firm advocate against ECT like all too many other victims.

    http://www.madinamerica.com/2013/04/did-electroshock-save-my-life/

    I realize I risk raising the ire of you and certain ECT advocates here, but doesn’t your ethical stance on either cherry picking or omitting the negative data altogether apply to barbaric practices like ECT too rather than the cherry picked, twisted anecdotal “evidence” of those like David Healy and Max Fink I posted on another recent article here?

    Since many psychiatric drugs are going off patent and have already been exposed for being useless and dangerous, I am concerned that the more such drugs get demonized, the more people will be forced into even more dangerous ECT and other worse brain damaging/disabling “treatments” all too typical of psychiatry. So, I question if demonizing the drugs like you, David Healy and others do will just cause more women, children and the elderly to be subjected to forced ECT as is now being done at an appalling, increasing rate that also increases stroke and death rates. One article/review of David Healy’s book I posted here discussed all the dangerous treatments of psychiatry that initially win great support and enthusiasm only to be found to be life destroying and barbaric in the long run as with all types of lobotomy including ECT.

    I reviewed Michael Taylor’s book, HIPPOCRATES CRIED, more and found it to be mostly another infomercial for ECT with Taylor part of the select ECT pushing crowd that includes Max Fink, David Healy, Edward Shorter and other cheerleaders here. I am also appalled to find that catatonia was pushed into the DSM 5 by Max Fink as another ploy to push ECT as does David Healy. The same is true for the push to include melancholia, which appears to me to occur in people who have suffered stress breakdowns possibly from toxic relationships and other severe stressors. In my opinion, the last thing anyone needs is brain damage, amnesia and the trauma of forced psychiatric treatment abuse when already in crisis since this just causes massive retraumatization according to many victims of it.

    So, I challenge you with the question if you can be consistent with your own ethics to carefully examine ALL the evidence of ALL psychiatric treatments objectively and truthfully and honestly report the results wherever they lead. Perhaps you have your own blind spots, double binds, difficult choices and certain loyalties that make it difficult to see or admit the truth believing you are choosing the lesser of evils when it isn’t really your choice to make?

    “All is takes for evil to win is for good people to say and do nothing.”

    This applies to ECT and Max Fink just like it applies to SSRI’s and Nemerof in my opinion.

  7.  
    Donna
    June 21, 2013 | 1:41 PM
     

    Correction:

    It appears others here (not Dr. Nardo) have used the diabetes analogy for harmful medical/psychiatry practice as did David Healy in his latest post on harmful medical practices.

  8.  
    jamzo
    June 21, 2013 | 2:09 PM
     

    FYI

    the institute of psychiatry at maudsely has been a focus of attention recently due to the lecture by nemeroff

    the link below takes you to a report of their DSM 5 conference on June 10

    http://brc.slam.nhs.uk/our-blog/brcu-blog/2013/dsm-5-at-the-iop

  9.  
    wiley
    June 21, 2013 | 3:09 PM
     

    One of my clients had been a diabetic since childhood and she was not happy with Avandia. It simply did not work as well for her as insulin and she felt powerless to do anything about it.

    In what ways are the hypotheses about the effectiveness of psychiatric drugs for various categories of “mental illness” as described in the DSM falsifiable?

  10.  
    wiley
    June 21, 2013 | 3:12 PM
     

    Fun fact: the trolley problem was designed by woman philosopher and ethicist Philipa Foot in 1967.

  11.  
    jamzo
    June 21, 2013 | 4:43 PM
     

    FYI

    i think you would find this paper is a very good read

    DxSummit.org

    The Global Summit on Diagnostic Alternatives: An Online Platform for Rethinking Mental Health

    DSM-5?s Somatic Symptom Disorder: From Medical Enigma to Psychiatric Sphinx

    Sarah Kamens • June 2, 2013

    The author would like to thank Frederick Wertz, Suzy Chapman, Donald Kamens, Ben Kamens, and Charles Olbert for their input on this manuscript.

    This paper was presented at the Interdisciplinary Graduate Student Symposium in Philosophy & Psychology, Fordham University, March 19, 2013.

    http://dxsummit.org/archives/712

  12.  
    Annonymous
    June 21, 2013 | 4:50 PM
     
  13.  
    a-non
    June 21, 2013 | 6:27 PM
     

    In this example the patient and doctor have been funneled into an unhealty interaction because of lack of boundries. So the establisment of definative fences would make a better interaction. So I guess the real question is how to establish the proper informational fence where neither party gets funneled into one trap or another. How could this interaction be made healthy?
    http://davidhealy.org/we-have-a-dream-getting-engaged-to-a-doctor/

  14.  
    wiley
    June 21, 2013 | 6:57 PM
     

    Great article, Jamzo. Was just studying a bit on ontology and contemplating the hubris of bio-psychiatry writ large with its attempt to define what a person essentially is and how “mentally ill” they are by reifying psychiatry’s fickle history of etiologies; and using them as an excuse to dig into the brain (writ large) without sufficiently examining their own reductionist efforts to define what is valid and without considering a person’s account of themselves as being meaningful, or even relevant beyond the psychiatrist’s prerogative to judge it legitimate or not based on his perception of the height of his perch.

    Bigotry. Self-idolatry.

  15.  
    a-non
    June 22, 2013 | 7:14 AM
     

    “Amazingly, after more than 2 decades, the ADA has never advocated for investigation of rDNA synthetic insulin–used by all Type 1 diabetics. The original mantra was “it’s just like the human body makes” . . . but no proof was required. The only requirement seems to be that it (rDNA insulin) lowered blood glucose and didn’t kill the patient immediately. Natural products–created in nature–are no longer available to American diabetics; but no investigation or validation requirements are on the horizon. Interestingly, “dead-in-bed syndrome” and “hypoglycemia unawareness” have entered the vernacular (10 years ago medicine was denying that either of these existed) . . . but conveniently, the onus can still be placed squarely on the patient or the disease–never the insulin.”-Melody
    I wonder just what the purpose of those associations are for to begin with. When the black box is open are we going to find far less than we expected? I hope Dr. John Ioannidis is wrong.
    “Lies, Damned Lies, and Medical Science” The Atlantic Nov. 2010 David H. Freedmen
    http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/308269/

Sorry, the comment form is closed at this time.