{"id":67003,"date":"2016-07-23T00:36:04","date_gmt":"2016-07-23T04:36:04","guid":{"rendered":"http:\/\/1boringoldman.com\/?p=67003"},"modified":"2016-10-16T20:23:17","modified_gmt":"2016-10-17T00:23:17","slug":"wonky-science","status":"publish","type":"post","link":"https:\/\/1boringoldman.com\/index.php\/2016\/07\/23\/wonky-science\/","title":{"rendered":"comparin’…"},"content":{"rendered":"\n
Note: I am pleased to announce that Google has moved me from<\/font> This site may have been hacked<\/font><\/strong> to<\/font> This site is not mobile friendly<\/font><\/strong> status. I see this as a real step up in the world and will look into the mobile friendly issue the next time I’ve got nothing else to do. I do appreciate that Google is now monitoring such things. It’s going to mean a safer more useful Internet – a good thing…<\/font><\/em><\/sup><\/div>\n
<\/p>\n\n\n\n
<\/td>\n<\/tr>\n
Figure A1. A continuum of experimental exploration and the corresponding continuum of statistical wonkiness. On the far left of the continuum, researchers find their hypothesis in the data by post hoc theorizing, and the corresponding statistics are “wonky”, dramatically overestimating the evidence for the hypothesis. On the far right of the continuum, researchers preregister their studies such that data collection and data analyses leave no room whatsoever for exploration; the corresponding statistics are “sound” in the sense that they are used for their intended purpose. Much empirical research operates somewhere in between these two extremes, although for any specific study the exact location may be impossible to determine. In the grey area of exploration, data are tortured to some extent, and the corresponding statistics are somewhat wonky.<\/sup><\/font><\/td>\n<\/tr>\n<\/table>\n

Sometimes, a good cartoon can say things better than volumes of written word. This one comes from the exaplanatory text accompanying a republication and translation of Adrian de Groot<\/font>‘s classic paper on randomized trials explaining why they must be preregistered [see The Meaning of “Significance” for Different Types of Research<\/a>, the hope diamond…<\/a>, Why we need pre-registration<\/a>, For Preregistration in Fundamental Research<\/a>]. It’s the central point in the proposal suggested by Dr. Bernard Carroll’s Healthcare Renewal blog [CORRUPTION OF CLINICAL TRIALS REPORTS: A PROPOSAL<\/a>].<\/p>\n

Our journals are filled with articles where the data has been tortured<\/font><\/em> [center above] or had the outcome moved to fit the data <\/font><\/em>[left above]. But RCTs [Randomized Clinical Trials] are intended to test an already defined hypothesis<\/font><\/em>, not make one up. They’re like Galileo’s famous experiment<\/a> [right above]. Define the conditions in advance, then do the experiment to see if those conditions are met. And the only way to insure that the trial follows and is analyzed by those preregistered conditions is to publicly declare them before<\/font><\/em> the experiment is done and afterwards<\/font><\/em> publicly post the analyses done by the preregistered methods. Anything else ends up in wonky·land<\/font><\/em>. Comes now this…<\/div>\n
\n
The COMPare Trials Project.<\/font><\/strong><\/em><\/div>\n
Ben Goldacre, Henry Drysdale, Anna Powell-Smith, Aaron Dale, Ioan Milosevic, Eirion Slade, Philip Hartley, Cicely Marston, Kamal Mahtani, Carl Heneghan.<\/div>\n
www.COMPare-trials.org<\/a>, 2016.<\/div>\n<\/blockquote>\n
We know Ben Goldacre<\/font> from his books<\/a>, his TED talk<\/a>, and his AllTrials<\/a> campaign, but I think his finest achievement is his current enterprise – The COMPare<\/a> Project. The idea is simple. Compare the a priori <\/em>Protocol defined outcome variables with those in a published journal article. I personally discovered the  importance of that working on our Paxil Study 329 article, and ever since have gone looking for protocols on the Clinical Trials that have come along. Sometimes they’re listed on clinicaltrials.gov [and sometimes not]. But even if they’re there, there’s rarely enough to do the proper protocol defined analysis. I’ve never<\/u> found a full a priori<\/em> portocol except in cases where it has been subpoenaed in litigation. So I wondered how Goldacre’s group was getting them. Here’s what he says<\/a>:<\/div>\n
\n
"Our gold standard<\/a> for finding pre-specified outcomes is a trial protocol that pre-dates trial commencement, as this is where CONSORT states outcomes should be pre-specified. However this is often not available, in which case, as a second best, we get the pre-specified outcomes from the trial registry entry that pre-dates the trial commencement. Where the registry entry has been modified since the trial began, we access the archived versions, and take the pre-specified outcomes from the last registry entry before the trial began."<\/em> He explains this further in the FAQ<\/a> on their website…<\/div>\n<\/blockquote>\n

He has some hard working medical students and volunteer faculty working on his team and they checked all the trials in five top journals over a four month period last winter, comparing protocol defined outcomes against published outcomes. Here’s what they found<\/a>:<\/p>\n\n\n\n
TRIALS
CHECKED<\/sup><\/font><\/strong><\/td>\n		TRIALS WERE
PERFECT<\/sup><\/font><\/strong><\/td>\n		OUTCOMES NOT
REPORTED<\/sup><\/font><\/strong><\/td>\n		NEW OUTCOMES
SILENTLY ADDED<\/sup><\/font><\/strong><\/td>\n<\/tr>\n
67<\/font><\/strong><\/td>\n9<\/font><\/strong><\/td>\n354<\/font><\/strong><\/td>\n357<\/font><\/strong><\/td>\n<\/tr>\n<\/table>\n

And when they wrote the editors about the discrepancies, only 30% were published. And when authors do respond, they are sometimes combative or defensive [sometimes the COMPare guys get it wrong and apologize, but that’s not often]. I won’t go on and on. The web site is simple and has all the info clearly presented. Epidemic Outcome Switching! Check and Mate!<\/font><\/strong><\/p>\n

They haven’t published yet, but we look forward to what’s coming. I personally think the COMPare Project has landed on the center of the problem. We’ve complained about not being able to see the data itself, but to have this much distortion of the specified outcome variables is even more basic. There is no justification<\/font><\/em> for this level of scientific misconduct…<\/div>\n","protected":false},"excerpt":{"rendered":"

Note: I am pleased to announce that Google has moved me from This site may have been hacked to This site is not mobile friendly status. I see this as a real step up in the world and will look into the mobile friendly issue the next time I’ve got nothing else to do. I […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[5],"tags":[],"class_list":["post-67003","post","type-post","status-publish","format-standard","hentry","category-opinion"],"_links":{"self":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/67003","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/comments?post=67003"}],"version-history":[{"count":34,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/67003\/revisions"}],"predecessor-version":[{"id":68699,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/67003\/revisions\/68699"}],"wp:attachment":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/media?parent=67003"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/categories?post=67003"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/tags?post=67003"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}