a false economy…

Posted on Thursday 15 May 2014

by Susanna Every-Palmer and Jeremy Howick
Journal of Evaluation in Clinical Practice. 2014 May 12. [Epub ahead of print]

Evidence-based medicine [EBM] was announced in the early 1990s as a ‘new paradigm’ for improving patient care. Yet there is currently little evidence that EBM has achieved its aim. Since its introduction, health care costs have increased while there remains a lack of high-quality evidence suggesting EBM has resulted in substantial population-level health gains. In this paper we suggest that EBM’s potential for improving patients’ health care has been thwarted by bias in the choice of hypotheses tested, manipulation of study design and selective publication. Evidence for these flaws is clearest in industry-funded studies. We argue EBM’s indiscriminate acceptance of industry-generated ‘evidence’ is akin to letting politicians count their own votes. Given that most intervention studies are industry funded, this is a serious problem for the overall evidence base. Clinical decisions based on such evidence are likely to be misinformed, with patients given less effective, harmful or more expensive treatments. More investment in independent research is urgently required. Independent bodies, informed democratically, need to set research priorities. We also propose that evidence rating schemes are formally modified so research with conflict of interest bias is explicitly downgraded in value.
hat tip to pharmagossip…   
I don’t respond positively to the term Evidence Based Medicine [EBM]. I’ve most often heard it used to elevate the Randomized Clinical Trial [RCT] or somebody’s Guideline to the level of binding truth. This article points us to Sackett et al as a classic description:
It’s about integrating individual clinical expertise and the best external evidence
by David L Sackett , William M C Rosenberg , JA Muir Gray , R Brian Haynes , and W Scott Richardson
British Medical Journal. 1996 312:71-72.

Evidence based medicine is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research. By individual clinical expertise we mean the proficiency and judgment that individual clinicians acquire through clinical experience and clinical practice… By best available external clinical evidence we mean clinically relevant research, often from the basic sciences of medicine, but especially from patient centred clinical research into the accuracy and precision of diagnostic tests [including the clinical examination], the power of prognostic markers, and the efficacy and safety of therapeutic, rehabilitative, and preventive regimens…

Good doctors use both individual clinical expertise and the best available external evidence, and neither alone is enough. Without clinical expertise, practice risks becoming tyrannised by evidence, for even excellent external evidence may be inapplicable to or inappropriate for an individual patient…

Evidence based medicine is not “cookbook” medicine. Because it requires a bottom up approach that integrates the best external evidence with individual clinical expertise and patients’ choice, it cannot result in slavish, cookbook approaches to individual patient care. External clinical evidence can inform, but can never replace, individual clinical expertise, and it is this expertise that decides whether the external evidence applies to the individual patient at all and, if so, how it should be integrated into a clinical decision. Similarly, any external guideline must be integrated with individual clinical expertise in deciding whether and how it matches the patient’s clinical state, predicament, and preferences, and thus whether it should be applied. Clinicians who fear top down cookbooks will find the advocates of evidence based medicine joining them at the barricades…

Some fear that evidence based medicine will be hijacked by purchasers and managers to cut the costs of health care. This would not only be a misuse of evidence based medicine but suggests a fundamental misunderstanding of its financial consequences…

Evidence based medicine is not restricted to randomised trials and meta-analyses. It involves tracking down the best external evidence with which to answer our clinical questions… It is when asking questions about therapy that we should try to avoid the non-experimental approaches, since these routinely lead to false positive conclusions about efficacy. Because the randomised trial, and especially the systematic review of several randomised trials, is so much more likely to inform us and so much less likely to mislead us, it has become the “gold standard” for judging whether a treatment does more good than harm…
Reading that article, I found myself nodding with approval until I got to the part highlighted at the end. Of course, that should be right [an RTC "is so much more likely to inform us and so much less likely to mislead us"] and it’s an easy sermon to preach. But it hasn’t been at all right, certainly not in psychiatry. RCTs became the mechanism for something that can only be characterized a massive, orchestrated scam. The Every-Palmer/Howick article addresses how this should-be-true statement became so perverted. Their article begins with a simple point:
If EBM were the revolutionary movement it was hailed as, we would expect more than benefits demonstrated in specific cases. We would expect population-level health gains, such as those that occurred after the introduction of antibiotics, improved sanitation and smoking cessation. Unfortunately, there is little evidence that EBM has had such effects.
But they get to their main point talking about antipsychotics and antidepressants:
The story so far suggests improved patient outcomes and EBM’s ability to identify superior treatments to replace less effective alternatives. However, the reality is different. Ten years after atypicals had saturated the market, large independent trials known by the acronyms CATIE [Clinical Antipsychotic Trials of Inter- vention Effectiveness], CUtLASS [Cost Utility of the Latest Antipsychotic Drugs in Schizophrenia Study], and EUFEST [European First Episode Study] have demonstrated that the atypical agents are in fact no more effective, no better tolerated and are less cost effective than their typical predecessors.

In relation to depression, independent meta-analyses pooling unpublished as well as published data now show that SSRIs are no more effective than placebo in treating mild-to-moderate depression, the condition for which they have been most commonly prescribed

So how is it that for over a decade we were convinced by the evidence into thinking these treatments were superior? How could there have been ‘an evidence myth constructed from a thousand randomized trials’ and how did we fall for it?
We all know where they go next – a compendium of the methods used by industry to make their clinical trials come out in their favor – things like selective publication and manipulation of study design. I’ll add the selective use of data analytic techniques to suggest efficacy where none exists [statistical significance that is clinically trivial] or guidelines by expert opinion rather than scientific evidence. They offer some suggestions:
It is beyond the scope of this paper to discuss practical solutions in great detail, however, we make the following suggestions:
  1. The sensible campaign to formalize and enforce measures ensuring the registration and reporting of all clinical trials [see http:// www.alltrials.net/] should be supported – otherwise trials that do not give the answer industry wants will remain unpublished.
  2. More investment in independent research is required. As we have described, it is a false economy to indirectly finance industry-funded research through the high costs of patented pharmaceuticals.
  3. Independent bodies, informed democratically, need to set research priorities.
  4. Individuals and institutions conducting independent studies should be rewarded by the methodological quality of their studies and not by whether they manage to get a positive result [a ‘negative’ study is as valuable as a ‘positive’ one from a scientific point of view].
  5. Risk of bias assessment instruments such as the Cochrane risk of bias tool should be amended to include funding source as an independent item.
  6. Evidence-ranking schemes need to be modified to take the evidence about industry bias into account. There are already mechanisms within EBM evidence-ranking schemes to up- or downgrade evidence based on risk of bias. For example, the Grading of Recommendation Assessment, Development and Evaluation [GRADE] system allows for upgrading observational evidence.
I’m not sure that the highlighted portions of 5. and 6. above are enough. Those articles that we’re talking about really shouldn’t be published at all by a journal that is peer reviewed. They’re not evidence-based medicine, they’re misleading advertisements at best. Unfortunately, the solution is much more complex because industry has become such a major funding source for research, and that funding depends on wealth generated by the drug sales fueled by these studies. It’s a vicious cycle without a readily apparent solution at this point except number 2 above…
    May 15, 2014 | 11:18 AM

    On the subject of finding out what is working, every now and then something interesting comes in out of the blue. Warfarin for Long-term Psychosis Remission?. Sorry, login required after jump.

    May 15, 2014 | 12:19 PM

    it is time for criminal charges to be filed for the frank malfeasance, negligence, and frank profit mongering over reponsible standards of care being blatantly abandoned just for the sake of selling drugs.

    Check out Ronald Pies recent post at http://www.psychcentral.com/blog from yesterday and tell me that is not a covert sales pitch to use antipsychotics for acute grief issues. Hallucinations indeed, what a load of crap being sold to get patients on Abilify and Seroquel, and wait for it, as solo drugs!

    May 15, 2014 | 4:38 PM

    Someone on another blog linked to this page


    and I was struck by all the NIMH studies that concluded that the medicines being used in psychiatry right now are harmful, make people more sick, make people more likely to die, disable people unnecessarily, and don’t really do all that much for the symptoms they’re supposed to “treat”.

    EBM’s indiscriminate acceptance of industry-generated ‘evidence’ is akin to letting politicians count their own votes. Given that most intervention studies are industry funded, this is a serious problem for the overall evidence base. Clinical decisions based on such evidence are likely to be misinformed, with patients given less effective, harmful or more expensive treatments. More investment in independent research is urgently required. Independent bodies, informed democratically, need to set research priorities. We also propose that evidence rating schemes are formally modified so research with conflict of interest bias is explicitly downgraded in value.

    No shit. NIMH? Hello?!

    May 15, 2014 | 6:21 PM

    Don’t get me started on Ronald Pies.

    Arby, throw a bomb into metabolism and some fallout is going to be perceived as beneficial. This is the history of psychiatric drugs.

    May 15, 2014 | 7:52 PM


    I’m not a physician, a scientist or a patient (or in pharmacy any longer), yet I find this sort of thing fascinating. It is when drug companies create a FrankenMed just so they can get a patent, along with 57 varieties of me-too drugs, that troubles me.

    Bernard Carroll
    May 15, 2014 | 10:02 PM

    Here is a delightfully trenchant commentary on EBM by the irrepressible Bruce Charlton. Bruce used to be editor of the journal Medical Hypotheses until he left that position in a disagreement with the publisher over a matter of principle. There are lessons here for all who would promote top-down science – Dr. Insel at NIMH, are you listening?

    James O'Brien, M.D.
    May 16, 2014 | 1:34 PM

    More evidence that organized psychiatry keep focusing on new territory beyond its biggest Venn Diagram and ignoring the most seriously ill.

    I am 1 million times more interested in treating seriously ill schizophrenics than treating situational problems at the margins of the science.

    Here’s my MOC question for the NIMH and APA:

    1. We should be most focused on treating the seriously mentally ill especially those who are dangerous.

    2. We should expand mental illness and treatment to include nearly everyone.

    Choose one answer only.

    May 16, 2014 | 2:59 PM

    NIMH and APA’s quick and terse reply:


    May 16, 2014 | 3:49 PM

    Arby, that paper on warfarin and psychosis (tiny number of subjects, uncertain diagnoses, anecdotal evidence) is larded with baloney, including this nugget:

    The researchers note that patients with schizophrenia commonly have a reduction of hippocampal volume, which is often explained as being caused by a trigger (such as use of illicit drugs or a previous traumatic event) and/or a predisposing condition that impairs neuronal plasticity.

    There is fairly substantial evidence that the so-called hippocampal deficit in schizophrenia is caused by antipsychotic drugs.

    Gad Mayer
    May 16, 2014 | 4:53 PM

    Regarding Arby’s link:
    My assumption is that this is a case of confounding::
    A. People with severe mental illness often don’t adhere to their medications, including warfarin.
    B. Many people are prescribed warfarin but only those who adhere to it stay in the specialty clinic.
    C. Those diagnosed with schizophrenia who stay on warfarin long term are less severely mentally ill than those who are lost to followup in the clinic.
    What you have in the end, is a select group of high functioning people who were diagnosed with schizophrenia but are less ill than most. From this the authors, prematurely, in my opinion, hypothesize elaborate biological explanations.

    May 16, 2014 | 7:54 PM

    Although the researchers are a bit self-promoting in the warfarin/psychosis article, I didn’t read it as presenting anything other than an observation that warranted more investigation. What I found fascinating about the article was the observational part. How science used to work. Take something you see and try to understand it. And, even though it seems off-point to Dr. Nardo’s post, the tangential connection in my mind is that it is in sharp contrast to EBM which doesn’t sound much like real science after reading his post and the articles he links to.

    This isn’t to say that somewhere along the line the observation won’t be made into an assumption by the researcher or some other group. Or, that after a few more studies and few changes to the molecules in warfarin to create another patented drug it won’t be extrapolated and promoted as the wonder cure for psychosis, migraines, bed-wetting, and old age. However, that is just bad science, politics and greed. I don’t think where it started is bad. My opinion of no consequence is, that the only population with psychosis that warfarin works to ameliorate their psychosis in, is the population where it actually does so.

    Btw, I don’t mind articles I mention on blogs being critiqued, even if ripped to shreds. It just helps me learn.

    Gad Mayer
    May 17, 2014 | 3:37 AM

    Arby, I can only agree with you on the value of good clinical observation (and creative theory). But clinical observation can be misleading, and this is where the next stages of clinical investigation should be followed, that is, EBM. This is the way medicine should work. Just because components of EBM were misused and EBM itself used as a misleading slogan, doesn’t mean that the principles of EBM are wrong and should be discarded (I’m not implying that you said otherwise, just stressing the point).

    James O'Brien, M.D.
    May 17, 2014 | 9:23 AM

    If the APA meeting were held on the Titanic, their priority would be mopping up coffee stains in the carpets of the first class cabins.

    Not a lot of “big picture” strategizing going on…

    May 18, 2014 | 3:03 PM

    I don’t think that this is limited to PHRMA.

    CBT is evidence-based and psychodynamic therapies are not, but frequently the people who get included in CBT trials have much cleaner diagnoses than those seen in clinical practice.

    I’m not knocking CBT in particular. I think it can be very helpful depending on the population being served, but evidence-based doesn’t necessarily make it superior to dynamic therapies.

    That doesn’t seem to be about where the funding comes from. Not sure what exactly–unless managed care insurers are funding it.

    May 18, 2014 | 4:48 PM

    Gad Mayer, I have no issues with the science of medicine that I can proclaim, since I lack knowledge of it. I have to rely on the good faith of others to shut down bad science or correct me if I am wrong when I stray too far into this subject.

    My only gripe is with the business of medicine and it is the bain of my existence, since I see it playing out exactly like what I’ve seen in the business world. It is a nasty thing. To Dr. Nardo’s point, most of the people I’ve worked with are normal, decent individuals. If you sat in a room with them, you can talk about their families, their hobbies and their hopes and dreams. You can even get them to say that they see what you see. However, this isn’t just to mollify you; they will actually offer evidence that only they have knowledge of, in support of it. Yet, get these individuals acting corporately and they will act like monsters. And this, for a very simple reason, because the behavior is rewarded. And, because the behavior is rewarded it becomes ok to them.

    I can avoid poor practitioners, yet I am finding it increasingly hard to avoid the EBM, gov’t and managed care edicts. So, I can look forward to a good majority of the caring doctors who are going to be punished and rewarded on the basis of them, behaving in the way they must, whether or not it is in a patient’s best interest. This is why everything needs to be a sugar-coated lie. I used to wonder why so much time was spent on lies and spin for the people beneath them who they can just order around. But, it just now occurred to me that it is more for those that would order us around than to keep the masses quiet. For those whose consciences would bother them, they can just choose to believe whatever the current lie is, pretending that these parties always know what is best for everyone and they can go on believing that they aren’t acting like monsters.

    Sorry, my epiphany is probably not new to anyone here, but it is new thinking in the business world where most managers are condemned as jerks or worse. Yet, I never agreed with this condemnation because I’ve liked all of my managers as people.

Sorry, the comment form is closed at this time.