{"id":59622,"date":"2015-09-01T14:59:08","date_gmt":"2015-09-01T18:59:08","guid":{"rendered":"http:\/\/1boringoldman.com\/?p=59622"},"modified":"2015-09-01T16:20:49","modified_gmt":"2015-09-01T20:20:49","slug":"originator-bias","status":"publish","type":"post","link":"https:\/\/1boringoldman.com\/index.php\/2015\/09\/01\/originator-bias\/","title":{"rendered":"originator bias?…"},"content":{"rendered":"\n

In our recent project, I had to bone up on my statistics. It was actually pretty interesting in that the statistical tests themselves haven’t changed all that much since my hard science days. But it wasn’t like riding a bicycle exactly, more like going to a class reunion where there’s an awkward start, but with a little catching up, the old familiarity returns.  While the statistics came back quickly, the implementation was all new. SPSS was unrecognizable. The newer SAS required SAS programming training. But then there’s R [just "R"], a free Open Source, command line statistical package put together by the academic community that’s a thing of great beauty. But learning the various procedures, each carrying the idiosyncrasies of its individual creator, meant going through a number of tutorials along the way.<\/p>\n

That’s a very long introduction to this – a lot of the tutorials had examples from studies done by social psychologists. After all, who teaches the Statistics courses? Often statistics professors come from that very discipline. And over and over, working through the examples, I thought about the softness<\/font><\/em> of the experiments compared to medicine [even psychiatry]. I don’t mean that disparagingly. It’s the nature of their subject matter. The study examples were kind of interesting in their own right, and I think it prepared me for this report about an article in Science<\/font><\/strong> [Estimating the reproducibility of psychological science<\/a>] that was a major undertaking – having 100 studies from their main journals repeated by other unrelated groups and comparing the outcomes. I wasn’t as surprised as the press seemed to think I ought to be at the low reproducibility figures:<\/span><\/div>\n
\n
Many Psychology Findings Not as Strong as Claimed, Study Says<\/a><\/div>\n
New York Times<\/font><\/strong><\/div>\n
by Benedict Carey<\/div>\n
AUG. 27, 2015 <\/div>\n

The past several years have been bruising ones for the credibility of the social sciences. A star social psychologist was caught fabricating data, leading to more than 50 retracted papers. A top journal published a study supporting the existence of ESP that was widely criticized. The journal Science pulled a political science paper on the effect of gay canvassers on voters’ behavior because of concerns about faked data.<\/p>\n

Now, a painstaking years long effort to reproduce 100 studies published in three leading psychology journals has found that more than half of the findings did not hold up when retested. The analysis was done by research psychologists, many of whom volunteered their time to double-check what they considered important work. Their conclusions, reported Thursday in the journal Science, have confirmed the worst fears of scientists who have long worried that the field needed a strong correction…<\/p>\n

The vetted studies were considered part of the core knowledge by which scientists understand the dynamics of personality, relationships, learning and memory. Therapists and educators rely on such findings to help guide decisions, and the fact that so many of the studies were called into question could sow doubt in the scientific underpinnings of their work.<\/p>\n

“I think we knew or suspected that the literature had problems, but to see it so clearly, on such a large scale — it’s unprecedented,” said Jelte Wicherts, an associate professor in the department of methodology and statistics at Tilburg University in the Netherlands…<\/div>\n<\/blockquote>\n
\n
Psychologists Welcome Analysis Casting Doubt on Their Work<\/a><\/div>\n
New York Times <\/font><\/strong><\/div>\n
by Benedict Carey<\/div>\n
AUG. 28, 2015 <\/div>\n

The field of psychology sustained a damaging blow Thursday: A new analysis found that only 36 percent of findings from almost 100 studies in the top three psychology journals held up when the original experiments were rigorously redone.<\/p>\n

After the report was published by the journal Science, commenters on Facebook wisecracked about how “social” and “science” did not belong in the same sentence.<\/p>\n

Yet within the field, the reception was much different. Along with pockets of disgruntlement and outrage — no one likes the tired jokes, not to mention having doubt cast on their work — there was a sense of relief. One reason, many psychologists said, is that the authors of the new report were fellow researchers, not critics. It was an inside job…<\/div>\n<\/blockquote>\n
\n
Estimating the reproducibility of psychological science<\/a><\/div>\n
by the Open Science Collaboration<\/div>\n
Science<\/font><\/strong> 349,aac4716 [2015].<\/div>\n

Abstract:<\/font><\/strong><\/u>
Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.<\/p>\n

<\/p>\n

Conclusion:<\/font><\/strong><\/u>
After this intensive effort to reproduce a sample of published psychological findings, how many of the effects have we established are true? Zero. And how many of the effects have we established are false? Zero. Is this a limitation of the project design? No. It is the reality of doing science, even if it is not appreciated in daily practice. Humans desire certainty, and science infrequently provides it. As much as we might wish it to be otherwise, a single study almost never provides definitive resolution for or against an effect and its explanation. The original studies examined here offered tentative evidence; the replications we conducted offered additional, confirmatory evidence. In some cases, the replications increase confidence in the reliability of the original results; in other cases, the replications suggest that more investigation is needed to establish the validity of the original findings. Scientific progress is a cumulative process of uncertainty reduction that can only succeed if science itself remains the greatest skeptic of its explanatory claims.<\/p>\n

The present results suggest that there is room to improve reproducibility in psychology. Any temptation to interpret these results as a defeat for psychology, or science more generally, must contend with the fact that this project demonstrates science behaving as it should. Hypotheses abound that the present culture in science may be negatively affecting the reproducibility of findings. An ideological response would discount the arguments, discredit the sources, and proceed merrily along. The scientific process is not ideological. Science does not always provide comfort for what we wish to be; it confronts us with what is. Moreover, as illustrated by the Transparency and Openness Promotion [TOP] Guidelines [http:\/\/cos.io\/top<\/a>], the research community is taking action already to improve the quality and credibility of the scientific literature. <\/p>\n

We conducted this project because we care deeply about the health of our discipline and believe in its promise for accumulating knowledge about human behavior that can advance the quality of the human condition. Reproducibility is central to that aim. Accumulating evidence is the scientific community’s method of self-correction and is the best available option for achieving that ultimate goal: truth. <\/div>\n<\/blockquote>\n
I rearranged the frequency plots from the figure to clarify the central point. The effect sizes fell by a half and the number that were statistically significant by two thirds. I guess they expected some fall in reproducability, but nothing quite so dramatic. It’s a wake up call for their field, actually for all of us – replication being the gold standard in scientific experimentation and analysis:<\/div>\n

<\/p>\n

Reading through this paper, I don’t think there was so much of the kind of problem we so often run into in the Clinical Trials of medication I follow in this blog. There was open sharing of protocols, materials, and methodology between the original investigators and the groups repeating the studies – no ghost writers or jury-rigged and obfuscated analyses. And yet the replication rate was still a lot lower than anticipated.<\/p>\n

Even in this situation absent the on-purpose<\/font><\/em> biases we deal with in many of the pharmaceutical trials, it seems like there’s an intrinsic bias present when someone conducts a study of their own design. I’ll bet the bias<\/em>-ologists have a name for it. Looking only at the Effect Sizes, in a repeat study by a non-originator,<\/font><\/em> the net strength of the effect generally falls, often precipitously – and it’s not just for the weaker studies, but to a lesser extent, all across the range:<\/p>\n

<\/p>\n

I realize that I’m shamelessly co-opting this data for my own purposes, but I just thought it was striking that in this study-of-studies which is likely not so suffused with the on-purpose<\/font><\/em> biases we’re looking for in RCTs of medications, the results of testing a pet<\/em> hypothesis [or drug?] tend towards inflation [even without obvious "cheating"]. This may be a well known phenomenon that some commenter can tell us all about, but it’s not so well known by me. It’s all the more reason to be pristine in conducting a trial or experiment and in looking for independent confirmation – replication. Meta-analysis won’t correct for this kind of originator<\/em> bias in that it’s usually a meta-analysis of a group of pet<\/em> hypotheses…<\/p>\n

I obviously spent some time thinking about this report. The authors seemed worried that they would discredit their discipline with this low reproducibility finding. I felt the opposite, impressed that they were examining the precision of their metrics. Because of the subjectivity of the social sciences, it felt like familiar territory to my own corner of things, psychotherapy, where confirmation is so ethereal and replication is king. But I also thought that it was a humbling reminder that our scientific evidence-based<\/em> tools [our graphs, and tables, and statistics, etc.] are just crude attempts to simplify and objectify the world around us – mere proxies for the infinite variability of the nature we’re trying to understand…<\/div>\n","protected":false},"excerpt":{"rendered":"

In our recent project, I had to bone up on my statistics. It was actually pretty interesting in that the statistical tests themselves haven’t changed all that much since my hard science days. But it wasn’t like riding a bicycle exactly, more like going to a class reunion where there’s an awkward start, but with […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[5],"tags":[],"class_list":["post-59622","post","type-post","status-publish","format-standard","hentry","category-opinion"],"_links":{"self":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/59622","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/comments?post=59622"}],"version-history":[{"count":45,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/59622\/revisions"}],"predecessor-version":[{"id":59669,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/59622\/revisions\/59669"}],"wp:attachment":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/media?parent=59622"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/categories?post=59622"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/tags?post=59622"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}