{"id":55972,"date":"2015-04-15T10:00:01","date_gmt":"2015-04-15T14:00:01","guid":{"rendered":"http:\/\/1boringoldman.com\/?p=55972"},"modified":"2015-04-15T11:46:28","modified_gmt":"2015-04-15T15:46:28","slug":"55972","status":"publish","type":"post","link":"https:\/\/1boringoldman.com\/index.php\/2015\/04\/15\/55972\/","title":{"rendered":"one of the many ways…"},"content":{"rendered":"
\n
Is science broken?<\/a><\/div>\n
the psychologist<\/font><\/strong><\/div>\n
British Psychological Society<\/font><\/strong> <\/div>\n
17th March 2015<\/div>\n
[see also Is Science Broken? Let’s Ask Karl Popper<\/a>] <\/div>\n

A lively debate was held at London’s Senate House<\/a> yesterday with panellists from neuroscience and psychology discussing the question: is science broken? If so, how can we fix it? The discussion covered the replication crisis<\/a> along with areas of concern regarding statistics and larger, more general problems…<\/p>\n

<\/div>\n

Neuroskeptic, a Neuroscience, Psychology and Psychiatry researcher and blogger<\/a>, gave a personal perspective on problems with science, speaking of the events which led him to lose faith in the research in the field. He said that as undergraduate students people are taught to do statistics in a very particular way, but once a person begins PhD research things change vastly. After gathering some results for his PhD research, Neuroskeptic found he had one significant result out of seven tasks performed by his participants. He said: ‘I thought back to my undergraduate days and thought “what if you do a Bonferroni correction across all the tasks?”. I got the idea that I’d suggest this to my supervisor but don’t think I ever did, I realised that just wasn’t how it was done. I was very surprised by this. I learned as an undergraduate you do a Bonferroni correction if you have multiple tasks. I started to wonder if we aren’t doing this who else isn’t doing it? I began to lose faith in research in the field.’<\/p>\n

Neuroskeptic said he wondered whether there was a good reason that multiple comparisons correction was not used. He added: ‘I still don’t think there’s a good reason we can’t do that. We have come to the tacit decision to accept methods which we would never teach undergraduates were a statistically good idea, but we decide that we’re happy to do them ourselves. That’s how I got on the road to blogging about these issues.’<\/p>\n

<\/div>\n<\/blockquote>\n
Neuroskeptic<\/a> is something of a blogger’s blogger, maintaining his anonymity on his personal blog for years, and now as a blogger for Discovery Magazine<\/font><\/strong>. He writes about a variety of topics, and they’re usually interesting whether they’re in your field or not. His nom de plume<\/font><\/em>, Neuroskeptic<\/em>, was a good choice. He not a "neuro-cynic," but rather a person who doesn’t believe in absolute truth just like his namesake, Pyrrho of Ellis<\/a>, the founder of Skepticism<\/font><\/em> in ancient Greece [as opposed to Dogmatism<\/font><\/em>] [see <\/font><\/em>my old Greek<\/em>…]<\/a>.  Neuroskeptic<\/em> brings his skeptial attitude to everything he writes.  I linked to his blog about this topic, Is Science Broken?<\/font><\/em>, in case you’re interested, but I wanted to talk about the specific example he’s using here, the Bonferroni Correction<\/font><\/em>, as it relates to Clinical Trials<\/font><\/em>. <\/div>\n

My own biostatistics and research experience was in another medical field over forty years ago, so when I began to look at the math of clinical trials, it was familiar but only just. Besides coursework, my only hands-on experience was using ANOVA to partition the variance of interactions of effects, so there was  much to learn. But I do have a Bonferroni Correction<\/font><\/em> story to tell from those days. During an Immunology fellowship, my clinical work was with a Rheumatology Section. Rheumatology is like Psychiatry in that there are many conditions where the etiology [cause] was and is unknown. In the 1960s, Rheumatologists were collecting large databases on every patient they saw to develop criteria for diagnoses [sound familiar?]. Databases were new, as were the mainframe computers that held the data entered with punch cards and stored on tapes. Statistics were run with home-grown Fortran programs that ran over-night [if you were lucky]. Bill Gates hadn’t yet made it to high school. Excel was something you did in sports. And correcting for multiple variables was something kind of new.<\/p>\n

One afternoon, the statistician and clinical staff blocked out a two hour conference to show us the results from the clinical database they were collecting [with great pride]. It was one of those after-lunch conferences where the eyelids are hard to hold open. Towards the end, the statistician showed us a thick stack of computer print outs with all the significant findings – disorders across the top, parameters down the side, cells filled with probabilities. Then he said something like, "Of course we had to correct the statistics for multiple measurements.<\/em>" I don’t remember the term Bonferroni Correction<\/font><\/em>, but I do remember what he did. He divided all those p-values by the number of things measured, and then he showed a slide of what significance remained from that thick stack of printouts. It evaporated, and left a table that fit on one readable slide. I was pretty impressed, but he seemed deflated watching his fine p-values go up in smoke.<\/p>\n

The logic behind correcting for multiple variables is pretty sensible, and simple. If you do an experiment and measure one outcome variable, p<0.05 means there’s less than 1 in 20 odds that the result happened by chance. However, if you measure 20 outcome variables, one will come out p<0.05 by chance alone. The Bonferroni Correction<\/font><\/em> is to divide each p-value by 20 [the number of outcome variables] – so you’d need a p< 0.0025 [0.05 ÷ 20] to claim the same level of significance. With 10 outcome variables, you would need p<0.005 [0.05 ÷ 10]. Piece of cake? Well Neuroskeptic <\/em>is absolutely right. Many [if not most] Clinical Trials just ignore this correction altogether. Others try to explain not using it, like this from Morrison et al<\/font><\/em> [Cognitive therapy for people with a schizophrenia spectrum diagnosis not taking antipsychotic medication: an exploratory trial<\/a> reported in slim pickings…<\/a> recently]:<\/div>\n
\n
\n
"Dependent t tests were used to analyse changes in outcome measures for the normally distributed variables; non-parametric analyses using Wilcoxon’s signed ranks test were used for skewed data. Tests of significance were two-tailed, but no correction was made for multiple comparisons given that this was a feasibility study in which we were less concerned about type 1 error<\/font><\/strong>."<\/em><\/div>\n
[Note:<\/font><\/strong> A type I error<\/font><\/em> is a false positive] <\/div>\n<\/blockquote><\/div>\n
Well, we have a really impressive false positive problem, that’s for sure. The Bonferroni Correction<\/font><\/em> is very tough on results – a harsh test. There have been other methods developed that are gentler, but they’re not used very much either. Another point: the method of correction, like any piece of the analysis, should be declared in the a priori<\/em> protocol, and that’s rarely done. The reason is obvious. Post hoc<\/em>, knowing the results, you can pick your correction method [if you even pick one] to fit how you want things to come out. So  Neuroskeptic<\/em> is absolutely correct, this is an almost institutionalized problem in Clinical Trials – just one of the many ways people get control of what their data says – like correction for attrition, or study design, or choice of statistical tests, etc. It’s why Data Transparency is so vital – so you can see under the places where deceitful analysis can change things but remain hidden…<\/div>\n
and break science…<\/div>\n","protected":false},"excerpt":{"rendered":"

Is science broken? the psychologist British Psychological Society 17th March 2015 [see also Is Science Broken? Let’s Ask Karl Popper] A lively debate was held at London’s Senate House yesterday with panellists from neuroscience and psychology discussing the question: is science broken? If so, how can we fix it? The discussion covered the replication crisis […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[5],"tags":[],"class_list":["post-55972","post","type-post","status-publish","format-standard","hentry","category-opinion"],"_links":{"self":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/55972","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/comments?post=55972"}],"version-history":[{"count":31,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/55972\/revisions"}],"predecessor-version":[{"id":55995,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/55972\/revisions\/55995"}],"wp:attachment":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/media?parent=55972"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/categories?post=55972"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/tags?post=55972"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}