New York TimesBy MICHAEL SUK-YOUNG CHWEJanuary 31, 2014SCIENCE is in crisis, just when we need it most. Two years ago, C. Glenn Begley and Lee M. Ellis reported in Nature that they were able to replicate only six out of 53 “landmark” cancer studies. Scientists now worry that many published scientific results are simply not true. The natural sciences often offer themselves as a model to other disciplines. But this time science might look for help to the humanities, and to literary criticism in particular. A major root of the crisis is selective use of data. Scientists, eager to make striking new claims, focus only on evidence that supports their preconceptions. Psychologists call this “confirmation bias”: We seek out information that confirms what we already believe. “We each begin probably with a little bias,” as Jane Austen writes in Persuasion, “and upon that bias build every circumstance in favor of it.”
Despite the popular belief that anything goes in literary criticism, the field has real standards of scholarly validity. In his 1967 book “Validity in Interpretation,” E. D. Hirsch writes that “an interpretive hypothesis,” about a poem “is ultimately a probability judgment that is supported by evidence.” This is akin to the statistical approach used in the sciences; Mr. Hirsch was strongly influenced by John Maynard Keynes’s “A Treatise on Probability.” However, Mr. Hirsch also finds that “every interpreter labors under the handicap of an inevitable circularity: All his internal evidence tends to support his hypothesis because much of it was constituted by his hypothesis.” This is essentially the problem faced by science today…
It’s a danger the humanities have long been aware of. In his 1960 book “Truth and Method,” the influential German philosopher Hans-Georg Gadamer argues that an interpreter of a text must first question “the validity — of the fore-meanings dwelling within him.” However, “this kind of sensitivity involves neither ‘neutrality’ with respect to content nor the extinction of one’s self.” Rather, “the important thing is to be aware of one’s own bias.” To deal with the problem of selective use of data, the scientific community must become self-aware and realize that it has a problem. In literary criticism, the question of how one’s arguments are influenced by one’s prejudgments has been a central methodological issue for decades…
Austen might say that researchers should emulate Mr. Darcy in Pride and Prejudice, who submits, “I will venture to say that my investigations and decisions are not usually influenced by my hopes and fears.” At least Mr. Darcy acknowledges the possibility that his personal feelings might influence his investigations. But it would be wrong to say that the ideal scholar is somehow unbiased or dispassionate. …the textbook “scientific method” of dispassionately testing a hypothesis is not how science really works. We often have a clear idea of what we want the results to be before we run an experiment. …science as a lived, human process is different from our preconception of it. He was trying to give us a glimpse of self-understanding, a moment of self-doubt.
When I began to read the novels of Jane Austen, I became convinced that Austen, by placing sophisticated characters in challenging, complex situations, was trying to explicitly analyze how people acted strategically. There was no fancy name for this kind of analysis in Austen’s time, but today we call it game theory. I believe that Austen anticipated the main ideas of game theory by more than a century. As a game theorist myself, how do I know I am not imposing my own way of thinking on Austen? I present lots of evidence to back up my claim, but I cannot deny my own preconceptions and training. As Mr. Gadamer writes, a researcher “cannot separate in advance the productive prejudices that enable understanding from the prejudices that hinder it.” We all bring different preconceptions to our inquiries, whether about Austen or the electron, and these preconceptions can spur as well as blind us…
What a thoughtful piece about the illusion of scientific detachment. The ancient Greeks thought if they could only master the rules of logic, enumerate all the logical fallacies, they would be able to reach absolute truths. They were the Dogmatists [back in a time when it meant the search for truth rather than what dogmatic means now]. With logic – every single day lawyers joust in courtrooms with carefully crafted legal arguments whose logic impeccably proves opposite points in the same case [and leaves the choice up to the everymen in the jury]. In science, we have our own versions a bit beyond simple logic with our data and its surrogates neatly arrayed in tables and graphs leading to conclusions wrapped in a cloak of statistical and mathematical proof.
Game theory was a big part of my graduate business program. One very big difference is that I was educated BC (before computers) and we balanced the numbers with other information. Low probabilities combined with a flawed biased study were discarded without any consideration. The concept that a single positive number, from a single occurrence, was by itself an indication of a trend or used in the decision process, was discounted out of hand.
Today we see low probability studies with poor design being embraced as gospel due to some insignificant number that will raise revenue. I bristle when talking with doctors that talk about raising revenue and economics, as if this is their only medical consideration.
Troubling is how doctors will fly like moths to a flame over some new treatment if it will increase their income. Looking at the data is not considered if the study fits their bias and raises revenue.
Recently we have seen a blow back due to changes in certain guidelines and doctors have been told to change their prescribing habits only to find all of their patients are “special” and need to continue on their existing schedules.
My understanding is that medicine is designed to treat the tails of the bell curve that makes up our population. Small movements towards the center of this curve produce outsized changes in patient population and thus higher revenue for the doctor and drug company. Only when we see patient populations approaching impossible numbers does someone ask if this is in fact a reliable marker, and then and only then, are studies done to prove the need for treatment.
We need to go back to those BC days and ask if the results are logical and can be reproduced before embracing treatment. Some quick and dirty math would debunk many of the medical claims we see today, and lead to a healthier population.
Steve Lucas
The evidence will continue to accumulate that Ioannides is correct – most published research is false. Some of it is more false than others. There are some science editors who can look at a paper and realize the the data and graphics are too perfect to be true. For the past two weeks I have been researching rhinovirus exacerbations of asthma, and I am shocked about the lack of advancement in knowledge over the past 30 years. I should qualify that by saying the basic science and measurements have advanced but at the clinical level – the only difference is that we don’t load people up on theophylline any more.
The best test of a medicine is what happens in the real world sometimes long after it has gone through the politics at the FDA. Prednisone for example – is off label for treating exacerbations of asthma due to rhinovirus but it also happens to be the standard of care.
The numbers are usually only adequate during post marketing surveillance.
Wonderful post. More and more I think that the pleasure principle is governing our scientific enterprise, and the reality principle has been forcibly (and commercially) banished to the repressed.