1 Boring Old Man

Is science broken?

the psychologist

British Psychological Society

17th March 2015

[see also Is Science Broken? Let’s Ask Karl Popper]

A lively debate was held at London’s Senate House yesterday with panellists from neuroscience and psychology discussing the question: is science broken? If so, how can we fix it? The discussion covered the replication crisis along with areas of concern regarding statistics and larger, more general problems…

Neuroskeptic, a Neuroscience, Psychology and Psychiatry researcher and blogger, gave a personal perspective on problems with science, speaking of the events which led him to lose faith in the research in the field. He said that as undergraduate students people are taught to do statistics in a very particular way, but once a person begins PhD research things change vastly. After gathering some results for his PhD research, Neuroskeptic found he had one significant result out of seven tasks performed by his participants. He said: ‘I thought back to my undergraduate days and thought “what if you do a Bonferroni correction across all the tasks?”. I got the idea that I’d suggest this to my supervisor but don’t think I ever did, I realised that just wasn’t how it was done. I was very surprised by this. I learned as an undergraduate you do a Bonferroni correction if you have multiple tasks. I started to wonder if we aren’t doing this who else isn’t doing it? I began to lose faith in research in the field.’

Neuroskeptic said he wondered whether there was a good reason that multiple comparisons correction was not used. He added: ‘I still don’t think there’s a good reason we can’t do that. We have come to the tacit decision to accept methods which we would never teach undergraduates were a statistically good idea, but we decide that we’re happy to do them ourselves. That’s how I got on the road to blogging about these issues.’

My own biostatistics and research experience was in another medical field over forty years ago, so when I began to look at the math of clinical trials, it was familiar but only just. Besides coursework, my only hands-on experience was using ANOVA to partition the variance of interactions of effects, so there was  much to learn. But I do have a Bonferroni Correction story to tell from those days. During an Immunology fellowship, my clinical work was with a Rheumatology Section. Rheumatology is like Psychiatry in that there are many conditions where the etiology [cause] was and is unknown. In the 1960s, Rheumatologists were collecting large databases on every patient they saw to develop criteria for diagnoses [sound familiar?]. Databases were new, as were the mainframe computers that held the data entered with punch cards and stored on tapes. Statistics were run with home-grown Fortran programs that ran over-night [if you were lucky]. Bill Gates hadn’t yet made it to high school. Excel was something you did in sports. And correcting for multiple variables was something kind of new.

One afternoon, the statistician and clinical staff blocked out a two hour conference to show us the results from the clinical database they were collecting [with great pride]. It was one of those after-lunch conferences where the eyelids are hard to hold open. Towards the end, the statistician showed us a thick stack of computer print outs with all the significant findings – disorders across the top, parameters down the side, cells filled with probabilities. Then he said something like, "Of course we had to correct the statistics for multiple measurements." I don’t remember the term Bonferroni Correction, but I do remember what he did. He divided all those p-values by the number of things measured, and then he showed a slide of what significance remained from that thick stack of printouts. It evaporated, and left a table that fit on one readable slide. I was pretty impressed, but he seemed deflated watching his fine p-values go up in smoke.