{"id":37734,"date":"2013-06-20T19:46:26","date_gmt":"2013-06-20T23:46:26","guid":{"rendered":"http:\/\/1boringoldman.com\/?p=37734"},"modified":"2013-06-21T01:22:12","modified_gmt":"2013-06-21T05:22:12","slug":"37734","status":"publish","type":"post","link":"https:\/\/1boringoldman.com\/index.php\/2013\/06\/20\/37734\/","title":{"rendered":"impossible situations…"},"content":{"rendered":"
\n
\n
The Trolley Problem With Science<\/a>
Discover Magazine<\/font><\/strong><\/div>\n
By Neuroskeptic<\/div>\n
June 20, 2013<\/div>\n

Imagine a scientist who does an experiment, and doesn’t like the results. Perhaps the scientist had hoped to see a certain pattern of findings and is disappointed that it’s not there. Suppose that this scientist therefore decided to manipulate<\/strong> the data. She goes into the spreadsheet and adds new, made-up data points, until she obtains a statistically significant result she likes, and publishes it. That’s bad.<\/p>\n

Now, start this scenario over. Suppose that rather than making up data, the scientist throws it out<\/strong>. She runs the experiment again and again [without changing it], throwing out the results every time they’re wrong, until eventually, by chance, she obtains a statistically significant result she likes, and publishes it. Is that bad?<\/p>\n

Yes, but isn’t it less obviously<\/em> bad than data fabrication? I’m talking about an intuitive level. We feel that fabrication is clearly outrageous, fraudulent. Cherry-picking is bad, no-one denies it, but it doesn’t generate feelings to the same extent. Cherry-picking goes on in science, and I don’t know a scientist who doubts that it’s more common than fraud<\/a>. Yet we treat fraud much more harshly, regardless of the extent of the manipulation or the amount of money and prestige at stake. One fabricated point of data is misconduct; a thousand unpublished points of data is merely a ‘questionable practice<\/a>‘. Why?<\/p>\n

I think it’s a trolley problem<\/a><\/strong>.<\/p>\n

Imagine an out of control trolley was going to smash into five people and kill them. You happen to be standing by a lever that would divert the trolley onto another track, where it would hit a person, and kill him. Do you pull it? Many people say that they would not pull the lever, even though this means that four extra deaths occur. They wouldn’t pull it because they don’t like the idea of committing <\/em>the act of killing someone. Even though the decision not<\/em> to act causes more harm, it doesn’t feel as bad, because it’s [in some intuitive sense] an act of omission<\/em>. Even people who do pull the lever feel conflicted about it.<\/p>\n

I think the mentality is the same in science. Data fabrication is like pulling the lever – it’s committing<\/em> deception. Cherry-picking is a sin of omission<\/em> – you wait for the right data to come to you, and report that, omitting to report the rest of the data that points to a different conclusion. The end result is the same – misleading results. But they feel different.<\/div>\n<\/blockquote>\n
Neuroskeptic presents us with an interesting variant on the theme of the double-bind [impossible situation] because it adds another variable – volition and responsibility. So no matter what you do, you’re wrong – a variation on the classic paradigm of Sophie’s Choice<\/font><\/strong> by William Styron. In the usual double bind, the right thing to do in an impossible situation is nothing. But in Sophie’s Choice, that option was taken from her. Choose which child will be killed, but if you don’t choose, both will be killed. In Neurospeptic’s Trolley Problem, there’s a right choice [pull the lever], but you become a murderer in the process. If you make the wrong choice [do nothing], you are guilty of murder five times over, but you have plausible deniability<\/em> – "I was paralyzed", something like that.<\/div>\n

The way he frames the Trolley Problem addresses something very real in the world of Clinical trials, the strategic place of "missing studies<\/font><\/strong>." It’s the "All" the AllTrials<\/font><\/strong><\/a> petition and the "I" in the RIAT<\/font><\/strong> proposal ["I" for Invisible<\/em>] – studies that have gone missing. It’s a pretty obvious problem. If you do five clinical trials and have one that’s significant at the p<0.05 level and the others show no significant difference, it’s a bust – unless you only<\/u> publish the significant study. And that’s what has been done over and over. Here’s a recycled piece from an earlier blog post [gone missing…<\/a>] to make that point real: <\/p>\n

\n
Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy<\/a><\/div>\n
by Erick H. Turner, Annette M. Matthews, Eftihia Linardatos, Robert A. Tell, and Robert Rosenthal<\/div>\n
New England Journal of Medicine.<\/font><\/strong> 2008 358:252-260.
[full text on-line<\/a><\/strong>]<\/div>\n

<\/p>\n

Background<\/font><\/u><\/strong>: Evidence-based medicine is valuable to the extent that the evidence base is complete and unbiased. Selective publication of clinical trials — and the outcomes within those trials — can lead to unrealistic estimates of drug effectiveness and alter the apparent risk–benefit ratio.<\/div>\n
Methods<\/font><\/u><\/strong>: We obtained reviews from the Food and Drug Administration [FDA] for studies of 12 antidepressant agents involving 12,564 patients. We conducted a systematic literature search to identify matching publications. For trials that were reported in the literature, we compared the published outcomes with the FDA outcomes. We also compared the effect size derived from the published reports with the effect size derived from the entire FDA data set.<\/div>\n
Results<\/font><\/u><\/strong>:Among 74 FDA-registered studies, 31%, accounting for 3449 study participants, were not published. Whether and how the studies were published were associated with the study outcome. A total of 37 studies viewed by the FDA as having positive results were published; 1 study viewed as positive was not published. Studies viewed by the FDA as having negative or questionable results were, with 3 exceptions, either not published [22 studies] or published in a way that, in our opinion, conveyed a positive outcome [11 studies]. According to the published literature, it appeared that 94% of the trials conducted were positive. By contrast, the FDA analysis showed that 51% were positive. Separate meta-analyses of the FDA and journal data sets showed that the increase in effect size ranged from 11 to 69% for individual drugs and was 32% overall.<\/div>\n
Conclusions<\/font><\/u><\/strong>:We cannot determine whether the bias observed resulted from a failure to submit manuscripts on the part of authors and sponsors, from decisions by journal editors and reviewers not to publish, or both. Selective reporting of clinical trial results may have adverse consequences for researchers, study participants, health care professionals, and patients.<\/div>\n<\/blockquote>\n