quite a week…

Posted on Thursday 10 May 2012

    He who studies medicine without books sails an uncharted sea,
         but he who studies medicine without patients does not go to sea at all…
    Sir William Osler
In March, I looked into the story of how the authors of the DSM-III, DSM-IIIR, and DSM-IV [Robert Spitzer and Allen Frances] came to be at odds with the current DSM-5 Directors [David Kupfer and Darrel Regier][see dangerous men…]. It started in April 2007 when Dr. Spitzer asked to look at the minutes of the DSM-5 Task, and after a nine month delay was turned down, citing reasons of confidentiality. Finally, in June 2008, after an article in which the outgoing APA president praised the openness of the DSM-5 group, Spitzer unloaded in an article in the Psychiatric Times. In a subsequent series of articles, Spitzer continued to attack the secrecy of the DSM-5 Task Force. He asked Allen Frances to join him but Frances declined, though he agreed with the complaint. But then in May, 2009, after hearing about the Psychosis Risk Syndrome at a party at the APA meeting in San Francisco, Allen Frances weighed in with an article of his own in the Psychiatric Times which contained these prophetic paragraphs:
A Warning Sign on the Road to DSM-V:
Beware of Its Unintended Consequences
Psychiatric Times
By Allen Frances
June 26, 2009

The DSM-V goal to effect a “paradigm shift” in psychiatric diagnosis is absurdly premature. Simply stated, descriptive psychiatric diagnosis does not now need and cannot support a paradigm shift. There can be no dramatic improvements in psychiatric diagnosis until we make a fundamental leap in our understanding of what causes mental disorders. The incredible recent advances in neuroscience, molecular biology, and brain imaging that have taught us so much about normal brain functioning are still not relevant to the clinical practicalities of everyday psychiatric diagnosis. The clearest evidence supporting this disappointing fact is that not even 1 biological test is ready for inclusion in the criteria sets for DSM-V. Fortunately, the NIMH is now embarked on a fascinating effort to effect the real paradigm shift of basing diagnosis on biological findings. Unfortunately, this is years [if not decades] from fruition… So long as psychiatric diagnosis is stuck at its current descriptive level, there is little to be gained and much to be lost in frequently and arbitrarily changing the system. Descriptive diagnosis should remain fairly stable until, disorder by disorder, we gradually attain a more fundamental and explanatory understanding of causality…

Indeed, there has been only 1 paradigm shift in psychiatric diagnosis in the past 100 years—the DSM-III introduction in 1980 of operational criteria sets and the multiaxial system. With these methodological advances, DSM-III rescued psychiatric diagnosis from unreliability and the oblivion of irrelevancy. In the subsequent evolution of descriptive diagnosis, DSM-III-R and DSM-IV were really no more than footnotes to DSM-III and, at best, DSM-V could only hope to join them in making a modest contribution. Descriptive diagnosis is simply not equipped to carry us much further than it already has. The real paradigm shift will require an increase in our knowledge—not just a “rearrangement of the furniture” of the various descriptive possibilities…

I think of the APA response to Dr. Frances’ article as a nasty-gram written by Dr. Alan Schatzberg, then President of the APA [under investigation at the time by the U.S. Senate for financial impropriety]. It did say that the DSM-III and DSM-IV were outdated and hadn’t kept up with current thinking and the advances of science, but then they accused Drs. Spitzer and Frances of having financial motives behind their complaints:
Setting the Record Straight:
A Response to Frances Commentary on DSM-V
Psychiatric Times
By Alan F. Schatzberg, MD, James H. Scully Jr, MD, David J. Kupfer, MD, Darrel A. Regier, MD, MPH
July 1, 2009

The DSM-III categorical diagnoses with operational criteria were a major advance for our field, but they are now holding us back because the system has not kept up with current thinking. Clinicians complain that the current DSM-IV system poorly reflects the clinical realities of their patients. Researchers are skeptical that the existing DSM categories represent a valid basis for scientific investigations, and accumulating evidence supports this skepticism. Science has advanced, treatments have advanced, and clinical practice has advanced since Dr. Frances’ work on DSM-IV. The DSM will become irrelevant if it does not change to reflect these advances…
Dr. Spitzer responded, continuing his theme of the dangers of the DSM-5 Task Force’s policy of secrecy:
APA and DSM-V:
Empty Promises
Psychiatric Times
By Robert L. Spitzer, MD
July 2, 2009

The debate over DSM-V has unfortunately taken an ugly turn with the APA leadership suggesting that Dr. Frances’s and my motivation for critiquing DSM-V is financial. People familiar with this controversy might recall that it all began when I asked Darrel Regier if I could look at the minutes of DSM-V Task Force meetings so that I could keep up with the ongoing process. He explained that he could not do this because of confidentiality agreements that all DSM-V participants have been required to sign. Because of my strong belief that DSM has been and should always be a completely open process, I started my effort to get APA to change its ways. Read Dr Frances’ commentary on DSM-V and the APA’s response For brevity’s sake, I will limit my comments regarding APA’s response to Dr Frances’ commentary to the core issue of transparency. APA continues to maintain the empty rhetoric that the DSM-V process is the “most open and inclusive ever”…

And the rest is history – a three year long debate ensued, often contentious, about the whole process. Those three articles were all published in one week in the month following the APA Annual Meeting in San Francisco. That was quite a week! Here are a few good references about the ongoing story [A Moment of Crisis in the History of American Psychiatry, Inside the Battle to Define Mental Illness, DSM-V: Getting Closer to Pathologizing Everyone?].


In the three years since that week in 2009, a lot has happened. It’s no longer a rhetorical conflict that involves a handful of psychiatrists – it involves the entire specialty of psychiatry, the other mental health professions, the psychopharmaceutical industry, the clinical research industries, the medical reimbursement industry, and help seeking patients far and wide. Of course, Drs. Spitzer, Frances, and Kupfer didn’t cause the conflict any more than the Archduke Francis Ferdinand and his Assassin caused World War I. Their differences were just a focal point for something much bigger than all of them. But as is the case in such situations, the something-much-bigger tends to get submerged in the bluster that follows.

Dr. Spitzer got mad first, and the thing that made him mad was the secrecy [and the process] of the DSM-5 Task Force. Dr. Frances agreed with him about the secrecy, and didn’t care much for the process either, but he stayed out of the fray until he heard the kind of thing the DSM-5 Task Force was thinking about adding eg the Psychosis Risk Syndrome. That played into his own concerns about medicating kids. Then he got mad too and spoke out. Why was that the last straw for him? Speaking of last straws, why did I get so noisy around that same time myself? I wasn’t in these guys league, having been an early casualty in this same DSM-III Revolution – not really on the other side but close enough for government work. I was five or six years retired, thinking little about psychiatry. But in the summer of 2009, two things happened. I started seeing patients as a volunteer and did a review of psychopharmacology as part of that. And I continued to read about Senator Grassley’s investigation into psychiatrists in high places who were crooks – one of whom was the Chairman of a Department I’m still a part of. I’d lived with the dramatic changes in psychiatry after leaving academia and adapted. I’d had a fine career, though it felt a bit like being in exile.

I know what made me so angry. I found out what Dr. Frances and Spitzer couldn’t possibly not have also known – that all was not as it appeared. Corruption was prevalent in our ranks, our literature, and our treatment recommendations to patients. I’m guessing that’s a part of why the secrecy bothered Dr. Spitzer and the Psychosis Risk Syndrome bothered Dr. Frances, among the other things they knew about because they’d been DSMers. They knew that the cuurrent directions in psychiatry had opened to door for rampant corruption and they were both aware of a coming crisis [the one we're in right now]. Did Drs. Kupfer and Regier know too? Were they part of the problem? They would’ve had to put cotton in their ears and wear dark glasses not to know. For one thing, corruption and secrecy are virtually synonyms.


At last, I reach the point of this post. Dr. Frances says above in his opening salvo, "descriptive psychiatric diagnosis does not now need and cannot support a paradigm shift." That’s in the center of this in my mind. While the DSM-III Revolution was, on the surface, a move to make psychiatric diagnosis more scientific and more reliable, it was also driven to exorcise unproven ideology from the diagnostic system and psychiatry at large – at that time specifically psychoanalysis. And that’s what happened. And then…

Psychiatry Should Stay Comfortable In Its Own Skin
No Good Comes From Overselling Our Science Base
DSM-5 in Distress : Psychology Today
by Allen J. Frances, M.D.
June 2, 2011

But there is one source of great and continuing frustration in our field. We are in the midst of a neuroscience revolution that has provided a miraculous and tantalizing window into normal brain functioning. But the vast accumulation of basic science knowledge revealing the mechanisms of normal brain functioning has shed relatively little light on the far greater complexity of what causes psychopathology. As a result, the neuroscience revolution has so far had almost no impact on how we diagnose and treat our patients. The inherent difficulty in translating from basic to clinical science guarantees that we will make only slow progress in unraveling the multitudinous heterogeneity of brain malfunctions that cause mental illness. DSM 5 initially got into trouble because it was ambitious to jump-start a "paradigm shift" in psychiatry – well before there was sufficient scientific knowledge to make this possible. We would not have been burdened by all the dangerous DSM 5 suggestions for unproven diagnoses if its workgroups had not been given the green light to be recklessly creative in promoting their pet innovations… Psychiatry does itself no good when we oversell ourselves…

Psychiatry should live comfortably within its own skin, not make excessive claims. We are largely successful at doing what we do best in our current clinical work. We are eager to advance and incorporate the ever advancing scientific understanding of mental disorders and how best to treat them. But [except for Alzheimer's], psychiatry is likely decades away from anything resembling a paradigm shift. It’s always best to modestly under-promise and then strive to over deliver. The sad tale of DSM 5 is a succession of overblown promises and then disappointing and potentially dangerous under performance. Psychiatry should work hard at what we do well – without reaching beyond our current grasp or raising expectations we can’t possibly fulfill

It’s unquestioned that down some road at some future time, biological causes or factors are going to be part of the mental illness nosology. It’s equally unquestioned that one identical twin can be the picture of mental health and the be other be as sick as a goat – that some mental illnesses of significance can comes from biography. The conundrum is that neither of those things should  matter in the diagnostic system of psychiatry as it was conceived by Dr. Spitzer et al in 1980. In his system, causes or mechanisms only counted if they were known ["except for Alzheimer's"]. After sixteen years of Neurosis, Freud’s mental mechanisms had to go the way of Reich’s Orgone Box [a disillusionment from Spitzer's youth]. The DSM-5 Task Force had missed a very large point, as had many others. They mistook a failing of Robert Spitzer’s DSM-III Revolution for its essence – a failing so common in revolutions that it ought to be part of the definition. What was good about Spitzer’s direction was to aim for descriptive categories that were reliably grounded in observable phenomena – kappa was king. What was unfortunate was that, like most revolutions, there was another agenda. The old ways had to be ferreted out and exiled – expectable, but it lead to trouble.

The soft spot of the DSM-III was nowhere more evident than in the creation of the category of Major Depressive Disorder. It was in the area of depression that both the psychologically minded and the biologically oriented had made the most progress. In certain depressions, there was a statistically valid marker [not digital as everyone wished - but evidence nonetheless] and somewhat robust treatments [also not digital as everyone wished]. In the biopsychosocial realm, the relationship of some depressions was well understood in relationship to attachment and loss, and the mental mechanisms of some depressions as well as pathological grief had achieved a level of fairly clear clinical usefulness. All of those things hinged on the careful clinical discrimination of the depressions – aka diagnosis. In his zeal to make sure that his DSM-III was free of the problems of the past, Spitzer’s Major Depressive Disorder blunted the very real possibilities of the kind of advances psychiatry actually longs for. The Czar had to be killed and the Red Guard had to re-educate the "Roaders." Any fractionation of depressive diagnosis might have opened the door to Neurosis. So the successes of the past, at the time in their infancy, went the way of the bath water – Depressive Neuroses and Melancholia alike – both descriptively definable. If he didn’t like the names, he could’ve changed them. And not-psychological became biological in the minds of many, who then flourished. That should have been expected and happened relatively quickly.

The stated goal of this DSM-5 Task Force was to insert yet another unproven ideology into the diagnostic system, and thereby reframe psychiatry [A Research Agenda for DSM-V]. I call that ideology clinical neuroscience, borrowing the term from Dr. Tom Insel, Chief of the NIMH, but you could call it neurobiology, or biological psychiatry, or brain science. Whatever you call it, it’s the belief that problems mental are brain/biology problems and that proof is just around the corner. In their prequel, the DSM-5 leaders predicted that the DSM-5 itself would be solidly grounded in biology by the time it was released, though they had to back off from that prediction recently [Neuroscience, Clinical Evidence, and the Future of Psychiatric Classification in DSM-5].

Like the psychoanalysts in the 1950′s and 1960′s, the neuroscientists of the DSM-5 Task Force were so sure that they saw the future clearly that they lost sight of their Task [and relied on Force instead]. Their job was to carefully improve the terrain map of the desert, correcting the errors of earlier cartographers, adding new features only where justified and well documented. Instead, they gave in to their dreams. They failed to notice along the way that their critics weren’t their old enemies, but were rather the people that put them on the map in the first place. And when the second of the former Task Force chiefs, Allen Frances, joined the first, Robert Spitzer, in trying to point out their folly, they instead allied themselves with an APA President, Alan Schatzberg, who was himself a big part of the problem. They failed to see that some of the criticism was directed at his kind of thinking which had opened the door to run-away corruption, that instead of closing old loopholes, they were opening new ones. They ignored the advice about process, transparency, and detail, and ended up with a set of Field Trial outcomes that have us obsessing about their flawed methodology rather than gaining any clarification at all from their work product. Main line diagnoses like Schizophrenia, Major Depressive Disorder, and Generalized Anxiety Disorder with reliability well less than half the way between chance and full agreement among clinicians? Groan…

They were so busy dreaming together of their paradigm shift in cloistered workgroups, they failed to attend to the organizational necessities of such a project; they failed to listen to the wisdom of their elders; and they allied themselves with the wrongest of crowds – obvious to anyone who read the newspaper of the time. In a single week three years ago, they declined two life-lines and decided to go down with the ship.They may keep spinning their story and publish their book. Some people might even use it. But instead of their grand plan of making it more scientific, they fueled the opposite impression – and they should’ve known. If there’s any lessons at all for the future, one is that nosology should ride on the trailing edge of innovation and hypothesis, looking for things that are in need of clarification and correcting previous errors rather than involving itself with the whimsey and passion of the leading edge. The other lesson is that matters diagnostic are for clinicians, not dreamers and researchers…
         but he who studies medicine without patients does not go to sea at all…
Mickey @ 4:04 pm
Filed under: politics
updated table…

Posted on Wednesday 9 May 2012

DSM-5 Field Trials Generate Mixed Results
Medscape
by Deborah Brauser
May 8, 2012

Preliminary results are mixed for the recently completed field trials for the upcoming Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), according to the first public presentation of the findings here at the American Psychiatric Association’s (APA’s) 2012 Annual Meeting. Diagnostic criteria for autism spectrum disorder, posttraumatic stress disorder (PTSD), and attention-deficit/hyperactivity disorder (ADHD) in children were found to be very reliable. Conditions that did not do well included major depressive disorder (MDD), in adults and in children, and general anxiety disorder (GAD). According to Darrel Regier, MD, vice-chair of the DSM-5 task force, the poor scores for MDD may be attributable to "co-travelers," such as PTSD, major cognitive disorder, or even a substance use disorder, which often occur concurrently with depression. "Patients often don’t come in a single, simple diagnosis in clinical practice," Dr. Regier told Medscape Medical News. "If you have a patient with PTSD and major depression, the one that will probably get the most attention is PTSD. And that’s what we found in our study." He added that analysis of the field trials is ongoing and that the results presented at the meeting are preliminary.

Members of the task force said they hope to publish the full results "within a month." However, the third and final public comment period for the manual opened last week and ends on June 15. Although the entire period is 6 weeks long, the public may only have 2 weeks to comment after the publication of the field trials’ findings. From all accounts, the manual is still on track for publication right before next year’s APA Annual Meeting in San Francisco…

It’s a long article with lots of quotes, mostly explaining [away] the results. It had some more kappas, so I updated my table. The table has all the data from all the articles I’ve seen so far:


Disorder   DSM-5 (95% CI)     DSM-IV     ICD-10     DSM-III  

Major neurocognitive disorder .78 (.68 – .87) .66 .91
Autism spectrum disorder .69 (.58 – .80) .85 .77 -.01
Post traumatic stress disorder .67 (.59 – .74) .59 .76 .55
Child attention deficit disorder .61 (.51 – .72) .59 .85 .50
Complex somatic disorder .60 (.41 – .78) .45* .42*
Bipolar disorder .54 (.43 – .65) .69
Oppositional defiant disorder .41 (.21 – .61) .55 .66
Major Depressive Disorder (in adults) .32 (.24 – .40) .59 .53 .80
Generalized anxiety disorder .20 (.02 – .36) .65 .30 .72
Disruptive mood dysregulation disorder .50 (.32 – .66)
Schizophrenia .46 .76 .79 .81
Mild neurocognitive disorder .50 ( .40 – .60)
Schizoaffective Disorder .50 .54 .51 .54
Mild traumatic brain injury .46 (.28 – .63)
Alcohol use disorder .40 (.27 – .54) .71 .80
Hoarding .59 (.17 – .83)
Binge Eating .56 (.32 – .78)
Major Depressive Disorder (in kids) .29 (.15 – .42)
Borderline personality disorder .58 (.46 – .71)
Mixed anxiety/depressive disorder .06
Conduct Disorder .48 .57 .78 .61
Antisocial Personality Disorder .22
Obsessive Compulsive Disorder .31
Attenuated Psychosis Syndrome .46 (0-?)
[* see comment]
Mickey @ 12:08 am
Filed under: politics
box scores and kappa…

Posted on Tuesday 8 May 2012

Lou Gehrigs Box ScoresMy Dad had been a ball player, but by the time I came along, he was a Coach – my uncle too. People on the street called them both ‘Coach’ and it sounded like ‘Doctor’ or ‘Mr. President’ and I felt proud. They had grown up poor. Sports were a way to escape working in the mines, so it had to be a consuming interest for it to succeed – and it was. I grew up with sports and the Stats as the most important things going. The first part of the newspaper my Dad read were the Box-Scores from last night’s games in whatever sport was in season, in preparation for the evening game on the radio. I remember when television came and you could actually see the game, it was different. I was used to seeing the game in fantasy with pictures from the baseball cards, sports page, and stats from the box scores all dancing in my head. Being able to actually see the game played took some getting used to.

Babe Ruth, Shoeless Joe JacksonThere were live games too at the local park [Coach's kids got in free]. In the stands, there were lots of people who scored the game on an elaborate form in the program. They kept up with every pitch, every hit, and drew pictures with special symbols of the play. They were the ones who knew their Stats and would predict things [like in the recent movie Moneyball]. It was like the players weren’t real people. They were Stats too – ERAs, RBIs, Steals. "He leads the league in whatever"

When I got to Statistics in school, it was mostly about probability – how probable is it that something is significant, or is it just chance. I had to get used to that. In Baseball, everything had been significant. This was a different kind of Stats. Even in a Master’s program in Statistics back when, it was mostly about probability – a kind of yes-no science. With Baseball, there was so much more. That Babe Ruth or Shoeless Joe were probably going to get hits wasn’t enough. What kind of hit was the question. A double? a triple? a homer? an RBI? Could you count on them in a clinch? It was more than probability, it was effect size [how strong was Babe's mojo?] and reliability [how often did Shoeless Joe get on base? steal?]. I’m just learning about those kinds of Stats in medicine – things like NNT [number needed to treat], OR [odds ratio], Effect Size, and Kappa. And Kappa is on the front burner right now since the DSM-5 Field Trials have been completed. Kappa is about reliability.

Robert Spitzer was not well suited for the psychoanalytic training he chose. Psychoanalysis and psychotherapy are about intuition, and by report, Spitzer was remarkably short on intuition, so he was a perfect person to focus on the concrete descriptive criteria for psychiatric diagnosis. Along the way, he teamed up with Statisticians Joseph Fleiss of Columbia and Jacob Cohen at NYU who had studied inter-rater reliability, and began to use their versions of Kappa, a statistical method for quantification:
Robert Spitzer [and the ghost of Emil Kraepelin] 
A Re-analysis of the Reliability of Psychiatric Diagnosis
By ROBERT L. SPITZER and JOSEPH L. FLEISS
British Journal of Psychiatry. 1974  125:341-347.

Classification systems such as diagnosis have two primary properties, reliability and validity. Reliability refers to the consistency with which subjects are classified; validity, to the utility of the system for its various purposes. In the case of psychiatric diagnosis, the purposes of the classification system are communication about clinical features, aetiology, course of illness, and treatment.  A necessary constraint on the validity of a system is its reliability. There is no guarantee that a reliable system is valid, but assuredly an unreliable system must be invalid. Studies of the reliability of psychiatric diagnosis provide information on the upper limits of its validity. This paper discusses some of the difficulties in appraising diagnostic reliability, offers a re-analysis of available data from the literature, and suggests a possible course of action to improve psychiatric diagnosis…
That "course of action" was to use Kappa to test agreement of raters using given same criteria, and thus the nidus of the DSM-III was in place. At its core, Kappa quantifies the interval between what pure chance would bring two raters and total agreement between them:
 
So if "chance" agreement was 50:50 and they actually agreed on 3 out of 4, kappa would equal (0.75 – 0.50)÷(1.00 – 0.50) = 0.50. Things get more complex when you throw in multiple raters and multiple categories with varying prevalence, but the concept remains the same – quantifying the interval between chance [kappa = 0] and perfect agreement [kappa = 1]. In his 1974 article, Spitzer said [self-evident…]:
There are no diagnostic categories for which reliability is uniformly high. Reliability appears to be only satisfactory for three categories: mental deficiency, organic brain syndrome [but not its subtypes], and alcoholism. The level of reliability is no better than fair for psychosis and schizophrenia and is poor for the remaining categories.
So here’s his original table color coded by yours truly based on what he said in the article:
[I assume that Chronic Brain Syndrome was a slip-up] Spitzer’s criteria were pretty clear. And this is Robert Spitzer’s show – he invented it in collaboration with the very statisticians that created kappa. In the article, he describes the conditions for each of the studies he reports above. All were rated by psychiatrists. Some were tightly controlled, but others were rated by admitting psychiatry  residents. Spitzer concludes:
The reliability of psychiatric diagnosis as it has been practised since at least the late 1950′s is not good. It is likely that the reasons for diagnostic unreliability are the same now as when Beck et al studied them. They found that a significant amount of the variability among diagnosticians was due to differences in how they elicited and evaluated the necessary information, and that an even larger amount was due to inherent weakness and ambiguities in the nomenclature. Since that time there have been two major innovations which may provide solutions to these problems.

Several investigators have developed structured interview schedules which an interviewer uses in his examination of the patient. These techniques provide for a standardized sequence of topics, and ensure that variability among clinicians in how they conduct their interviews and in what topics they cover is kept to a minimum. For rating the pathology observed, these schedules contain precoded items which explicitly define the behaviours to be rated rather than relying on technical terms which have different meanings to different clinicians.

With respect to improving the nomenclature, the St.Louis group (Feighner et al) has offered a system limited to 16 diagnoses for which they believe strong validity evidence exists, and for which specified requirements are provided. Whereas in the standard system the clinician determines to which of the various diagnostic stereotypes his patient is closest, in the St. Louis system the clinician determines whether his patient satisfies explicit criteria. For example, for a diagnosis of the depressive form of primary affective disorder the three requirements are dysphoric mood, a psychiatric illness lasting at least one month with no other pre-existing psychiatric condition,and at least five of the following eight symptoms: poor appetite or weight loss; sleep difficulty; loss of energy; agitation or retardation; loss of interest in usual activities or decrease in sexual drive; feelings of self-reproach or guilt; complaints of or actually diminished ability to think or concentrate; and thoughts of death or suicide.

A consequence of the St. Louis approach is the necessity for an ‘undiagnosed psychiatric disorder’ category for those patients who do not meet any of the criteria for the specified diagnoses. In actual use, this category is applied to 20-30 per cent of newly-admitted in-patients. These two approaches, structuring the interview and specifying all diagnostic criteria, are being merged in a series of collaborative studies on the psychobiology of the depressive disorders sponsored by the N.I.M.H. Clinical Research Branch. We are confident that this merging will result not only in improved reliability but in improved validity which is, after all our ultimate goal.

And so the DSM-III and all that followed was created: kappa, the Feighner criteria, structured interviews.

In Baseball, the uniforms change, the stadiums grow, the ticket and hot dog prices escalate with the players’ salaries, and the records very slowly fall. But the box scores stay the same and the distance from home to first base never changes. When you have a losing season, you don’t argue with the Stats, you get a new Manager, add a relief pitcher, but you sure don’t fight with the way the game is played.

The DSM-5 tried to change the game with a ‘paradigm shift’ to clinical neuroscience as the sole organizer of psychiatry, and it just didn’t fly. Now they’re telling us that there’s some reason to redefine the meaning or interpretation of kappa. Foul Ball! So I colored in Dr. Frances’ table according to Spitzer, circa 1974. Here are the box scores:
Mickey @ 12:49 am
Filed under: politics
Village Consumed by Deadly Storm…

Posted on Monday 7 May 2012

In Annie Proulx‘s Shipping News, the protagonist, Quoyle is learning to write headlines for the local paper:
    Billy: It’s finding the center of your story, the beating heart of it, that’s what makes a reporter. You have to start by making up some headlines. You know: short, punchy, dramatic headlines. Now, have a look, what do you see? [Points at dark clouds at the horizon]
    Billy: Tell me the headline.
    Quoyle: Horizon Fills With Dark Clouds?
    Billy: Imminent Storm Threatens Village.
    Quoyle: But what if no storm comes?
    Billy: Village Spared From Deadly Storm.
One can do almost anything with a headline. I’m periodically scanning the Internet for more information  about the DSM-5 Field Trials, and the Google just doesn’t have much to say. The Scientific American Blog, Dr. Frances’ comments, and this on a University of Pennsylvania CME Site, MedPage Today, with a memorable headline:
Most DSM-5 Revisions Pass Field Trials
MedPage Today
By John Gever, Senior Editor
May 07, 2012

Field testing of proposed new diagnostic criteria for many mental disorders showed that, for the most part, they were at least adequately reliable, according to initial reviews presented here. But some new disorders that had appeared set for inclusion in DSM-5, the forthcoming fifth revision of the American Psychiatric Association’s [APA] influential diagnostic manual, will now probably be relegated to a research-use-only classification because interrater reliability was so poor. Among them is the controversial "attenuated psychosis syndrome," intended to capture patients with low-level hallucinations and thinking disturbances.

In some cases, it appeared that the field trials themselves had failed – that too few patients were found with a given diagnosis to provide meaningful data, or that other problems cropped up that made the results uninterpretable. Results of the field trials – a critical step in the development of DSM-5 – were presented here at the APA’s annual meeting. The trials are essentially the last step before the final round of meetings and decisions that will set DSM-5 in its published form. Although DSM-5 will be a "living document" that will be revised on an ongoing basis, the paper version must go to the printer in December and therefore the contents must be finalized in the next few months, explained David Kupfer, MD, of the University of Pittsburgh, the task force chairman for the effort…

Darrel Regier, MD, the APA’s research director, explained that the trials were intended primarily to establish reliability – that different clinicians using the diagnostic criteria set forth in the proposed revisions would reach the same diagnosis for a given patient. The key reliability measure used in the academic center trials was the so-called intraclass kappa statistic, based on concordance of the "test-retest" results for each patient. It’s calculated from a complicated formula, but the essence is that a kappa value of 0.6 to 0.8 is considered excellent, 0.4 to 0.6 is good, and 0.2 to o.4 "may be acceptable." Scores below 0.2 are flatly unacceptable.

Kappa values for the dozens of new and revised diagnoses tested ranged from near zero to 0.78. For most common disorders, kappa values from tests conducted in the academic centers were in the "good" range:

  • Bipolar disorder type I: 0.54
  • Schizophrenia: 0.46
  • Schizoaffective disorder: 0.50
  • Mild traumatic brain injury: 0.46
  • Borderline personality disorder: 0.58
In the "excellent" range were autism spectrum disorder [0.69], PTSD [0.67], ADHD [0.61], and the top prizewinner, major neurocognitive disorder [better known as dementia], at 0.78. But some fared less well. Criteria for generalized anxiety disorder, for example, came in with a kappa of 0.20. Major depressive disorder in children had a kappa value of 0.29. A major surprise was the 0.32 kappa value for major depressive disorder. The criteria were virtually unchanged from the version in DSM-IV, the current version, which also underwent field trials before they were published in 1994. The kappa value in those trials was 0.59.

But a comparison is not valid, Regier told MedPage Today. The patient samples were very different – in the DSM-IV trials, patients with psychiatric comorbidities were excluded, whereas they were allowed in the DSM-5 tests. Also, the clinicians in the DSM-IV trials received much more elaborate training before seeing patients than did those in the DSM-5 trials. Helena Kraemer, PhD, a statistician at Stanford University who helped set the general DSM-5 trial structure, said an important goal was to test the criteria "with real clinicians and real patients." She said the kappa values in DSM-IV were "inflated" because of the patient exclusions and clinician training.

Jan Fawcett, MD, of the University of New Mexico, who headed the DSM-5 workgroup that decided on the depression criteria, said that the disorder may simply be hard to diagnose reliably because of the many comorbidities that come with it. "I was not happy with the kappas that came out," he said. "But it may be that that is the reliability of that diagnosis in the real world."

Diagnoses that did very poorly in the trials included either because of low kappa values or confidence intervals so wide as to be meaningless:

  • Mixed anxiety-depressive disorder in adults and children
  • Attenuated psychosis syndrome
  • Obsessive-compulsive personality disorder [separate from obsessive-compulsive disorder]
  • Antisocial personality disorder
  • Nonsuicidal self-injury [e.g., cutting]
Although the kappa value for attenuated psychosis was actually in the "good" range at 0.46, the 95% confidence interval extended to zero, probably reflecting a failed trial. Most of the others had low point values for kappa – approaching zero for mixed anxiety-depression in adults – as well as wide confidence intervals. These conditions had been considered seriously for inclusion in the main body of DSM-5, but now will probably either be ditched entirely or placed in Section III [what had been the appendix in DSM-IV], for diagnoses that could be clinically useful but need more research-based refinement…
I could’ve thought of a hundred things to say about the DSM-5 Trials, but Most DSM-5 Revisions Pass Field Trials wouldn’t have been on the list. And considering the import of the trials, I would’ve given the kind of access these people had to the "big dogs" to a more mainstream media source. About all I got from this article other than a few more dismal numbers was the sense that the powers that be on the DSM-5 Task Force are going to deal with things with spin:
  • But a comparison is not valid, Regier told MedPage Today. The patient samples were very different – in the DSM-IV trials, patients with psychiatric comorbidities were excluded, whereas they were allowed in the DSM-5 tests.
  • Also, the clinicians in the DSM-IV trials received much more elaborate training before seeing patients than did those in the DSM-5 trials.
  • Helena Kraemer, PhD, a statistician at Stanford University who helped set the general DSM-5 trial structure, said an important goal was to test the criteria "with real clinicians and real patients."
  • She said the kappa values in DSM-IV were "inflated" because of the patient exclusions and clinician training.
At least, one person gave a semi-straight explanation with less curve on the ball:
  • Jan Fawcett, MD, of the University of New Mexico, who headed the DSM-5 workgroup that decided on the depression criteria, said that the disorder may simply be hard to diagnose reliably because of the many comorbidities that come with it. "I was not happy with the kappas that came out," he said. "But it may be that that is the reliability of that diagnosis in the real world."
The DSM-5 Task Force has skipped  the gold standard of having two sets of Field Trials to dial in on validity and reliability. They’ve attempted to redefine the meaning of the kappa values [see self-evident…]. They’ve maligned their predecessors, saying the DSM-IV Field Trials were too controlled, or that the raters were too trained, evoking the "naturalistic" argument used for other flops like STAR*D. They’ve been preoccupied with off-beat categories that are falling out like flies. The only thing I can find to agree with in this article is Dr. Fawcett’s comment about Major Depressive Disorder [which came in at 0.32 for adults and 0.29 for kids]. It’s not a reliable diagnosis because it doesn’t exist. Never has. So as headlines go, so far, I’d propose that Dr. Allen Frances’ previous posts be collected in a book called, Imminent Storm Threatens Village. And I’d name the Chapter about the Field Trials, Village Consumed by Deadly Storm. Most DSM-5 Revisions Pass Field Trials just isn’t in the mix. It’s going to take a really strong person to do the right thing here, which is postpone this whole show and bring in someone to get the enterprise on the right path. If they want the DSM-5 to be something that’s actually used, something psychiatry can be proud of, that’s the only outcome that makes a bit of sense…
Mickey @ 8:34 pm
Filed under: politics
a fork in the road…

Posted on Monday 7 May 2012

Newsflash From APA Meeting: DSM 5 Has Flunked its Reliability Tests
Needs To Be Kept Back For Another Year
DSM5 in Distress: Psychology Today
by Allen J. Frances, M.D.
May 6, 2012

The whole purpose of having a manual of psychiatric diagnosis is to promote diagnostic agreement. DSM III was an important milestone because it saved the credibility and relevance of psychiatry at a time when it was ridiculed for low reliability. Everyone knew that the reliability achieved in DSM field testing far exceeds what is possible in clinical practice, but DSM III took the major step of proving that reliability could be achieved at all. Until now, the DSM’s have facilitated communication across the clinical/research interface, promoted research, and provide credibility in the court room.

But bad news has just been reported from the annual meeting of the American Psychiatric Association in Philadelphia. The hard won credibility of psychiatric diagnosis is compromised by the abysmal results reported by the DSM 5 Field Trials. This failure was clearly predictable from the start:
[1] The writing of the DSM 5 criteria sets was far too raw and imprecise to be ready for the rigors of field testing. The ambiguity cried out for expert editing; without which reasonable reliability is impossible;
[2] The design of the field trial was byzantine in complexity and could never be done on schedule:
[3] Constant delays in starting and completing Stage 1 of the study forced DSM 5 to cancel the planned Stage 2 that was meant to clean up the poorly performing criteria sets identified in the first stage.
[4] With stage 2 cancelled without explanation, it looks like even the worst diagnoses are being given a social pass; and, most absurd,
[5] The design was totally off point, failing to ask the only question that really counted [the impact of DSM 5 on rates].
The results of the DSM 5 field trials are a disgrace to the field. For context, in previous DSM’s, a diagnosis had to have a kappa reliability of about 0.6 or above to be considered acceptable. A reliability of 0.2-0.4 has always been considered completely unacceptable, not much above chance agreement:

Kappa Values from the DSM-5 Field Trials


[reformatted from the original]

No predetermined publication date justifies business as usual in the face of these terrible Field Trial results [which are even more striking since they were obtained in academic settings with trained and skilled interviewers, highly selected patients, and no time pressure. The results in real world settings would be much lower]. Reliability this low for so many diagnoses gravely undermines the credibility of DSM 5 as a basis for administrative coding, treatment selection, and clinical research.

What can be done to salvage this deplorable mess:
[1] DSM 5 has never had anyone on board who could write a clean, consistent, unambiguous criteria set. DSM 5 received either no editing at all or amateur editing. Getting the words right is certain not enough- but If you can’t get them right, nothing else can ever be safe.
[2] For DSM 5 to retrieve credibility, it complete the second planned stage of its field testing. If doing the job right must delay publication so be it. Public trust must trump private publishing profits and it is self defeating for APA to publish a book no one can trust.

I have been consistently pessimistic and critical about DSM 5 since my first piece on it 3 years ago. The sad thing is I can still be so surprised. Each step of the way I predict it will fail in one or another way. But then I discover that DSM 5 has managed to fail in ways that go beyond my poor imagination. This assault on reliability was predicted, but its scope exceeds even my jaundiced fears and creates a DSM 5 emergency.

They really are pretty terrible. I might have predicted Mixed Anxiety Depression Disorder, but Schizophrenia 0.46? Major Depressive Disorder 0.32? Generalized Anxiety Disorder 0.20? Antisocial Personality Disorder 0.22? Alcohol Use Disorder 0.40? Obsessive Compulsive Disorder 0.31? Most of those criteria weren’t even changed very much, if at all. I did some frequency distributions of the values from the table:
Sure looks like things are running downhill to me. I have to admit that I was speechless when I saw all of this. I’ve been focusing on the content of the diagnostic changes, and haven’t paid as much attention to the process, but I now see why Dr. Frances has been so worried. I hate to actually say this, but my diagnosis here is sloppy. That’s the only way I can explain the outcome in formerly stable categories. They explained it as due to more rigor. I’d see it more as Rigor Mortis. I’m going to let it sit for a while, maybe review my Kappa a bit, before holding forth. This feels like a fork in the road to me…
Mickey @ 11:50 am
Filed under: politics
major depressive disorder κ=0.30?…

Posted on Sunday 6 May 2012

It sure is quiet from up there in Philadelphia. I’m in the Georgia Mountains, but if a pin dropped in Philly at the APA, I think I’d hear it all the way down here. Waiting for news about the DSM-5 Field Trials is like listening to paint dry. The only word out that I’ve seen is from this Scientific American blogger:

… Darrel Regier, vice chair of the APA’s DSM-5 Task Force, presented kappas for various DSM-5 diagnoses — the first publicly released results from the field trials. Fortunately, the kappas for many of the DSM-5 diagnoses look strong. Field trials of the new autism spectrum disorder [ASD], for example — which collapses DSM-IV diagnoses for autistic disorder, Asperger’s and other developmental conditions into one category — yielded a kappa of 0.69. However, two pitiful kappas shocked me. The kappa for generalized anxiety disorder was about 0.2 and the kappa for major depressive disorder was about 0.3.

These numbers are way too low according to the APA’s own scales — and they are much lower than kappas for the disorders in previous versions of the DSM. Regier and other members of the APA emphasized that field trial methodology for the latest edition is far more rigorous than in the past and that kappas for many diagnoses in earlier editions of the DSM were likely inflated. But that doesn’t change the fact that the APA has a problem on its hands: its own data suggests that some of the updated definitions are so flawed that only a minority of psychiatrists reach the same conclusions when using them on the same patient. And the APA has limited time to do something about it..

Although the APA has been working on the DSM-5 for more than 11 years now, field trials only started within the last year. While reporting my feature, I asked members of the APA why they waited so long to conduct the field trials. After all, only one year remains until scheduled publication of the DSM-5 and we still do not know whether the revised diagnoses are reliable and whether they are a genuine improvement over their predecessors. I never received a satisfactory answer..

To make an analogy, consider a baker who spends months developing a recipe for the ultimate chocolate cake in his head and — a day before he has to deliver the cake — finally tries out the recipe only to discover that the cake tastes awful. He has one day to come up with something else. The APA has placed itself in a similarly desperate position. The final drafts of the new manual are due December of this year, which means the APA has less than 8 months to implement what it has learned from the field trials if it wants to publish on schedule. New field trials would take years to arrange and at least one additional year to conduct. Either the association delays publication of the DSM-5 for several more years, revises the diagnoses yet again and conducts new field trials — or it goes forward with the current schedule and publishes a significantly flawed DSM-5.

If the APA has a plan of action — beyond vague statements like “continuing to analyze our data” — the association did not make it clear at the symposium. The presenters hardly seemed troubled by the alarming results. Even worse, they sometimes came off as oblivious …

Major Depressive Disorder κ=0.30? Generalized Anxiety Disorder κ=0.20? Little wonder that there’s so little noise coming our way. Those numbers are abysmal for two mainstream diagnoses in the DSM pantheon. What makes this even more confusing is besides their being low, the criteria for them are little changed from the DSM-IV versions. The provided explanation ["[the] field trial methodology for the latest edition is far more rigorous than in the past and that kappas for many diagnoses in earlier editions of the DSM were likely inflated"] isn’t particularly helpful.  Rereading their prequel [DSM-5: How Reliable Is Reliable Enough?] didn’t offer much of an explanation of more rigor. This blogger certainly wasn’t impressed. Later in the presentation, he confronted the speaker on a mis-statement, packed up, and walked out. He seemed to think they were blase`, "oblivious." Certainly there’s nothing about the lousy numbers in Kupfer’s article from the day before [DSM Field Trials Providing Ample Critical Data]. Surely they’re not going to pretend that this is an acceptable reliability outcome and steam ahead? I guess we just wait for more information for the moment [and given their track record, it's not likely to come from the Task Force itself]…
hat tip to Soulful Sepulcher
Mickey @ 8:23 pm
Filed under: politics
self-evident…

Posted on Sunday 6 May 2012

On the eve of the  DSM-5 Update of the Field Trial results at the APA Annual Meeting in Philadelphia, the Task Force director published  this article:
DSM Field Trials Providing Ample Critical Data
Psychiatric News
by David J. Kupfer, M.D.
May 4, 2012

As of this month, the 12-month countdown to the release of the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders [DSM-5] officially begins. While the developers of DSM-5 will continue to face several deadlines over the coming year, the progress that has been made since APA’s 2011 annual meeting has been nothing short of remarkable. One of the most notable and talked-about recent activities of the DSM revision concerns the implementation and conclusion of the DSM-5 Field Trials, which were designed to study proposed changes to the manual. This included both the examination of specific diagnostic criteria as well as overarching changes applicable across disorders, like the integration of dimensional measures and diagnostic severity scales. The DSM-5 Field Trials used two study designs — a large-scale design implemented across 11 medical-academic centers and a second design that was created specifically for solo and small-group clinicians in routine clinical practice [RCP] settings.

Both types of field trials provided landmark opportunities to study DSM in a more rigorous manner than field assessments in previous iterations of DSM. Specifically, they provided the rare opportunity to better understand the reliability, clinical utility, and feasibility of proposed changes in both high-volume research-practice settings and in everyday clinical care settings. This means that findings from the DSM-5 Field Trials will be generalizable to the actual patient groups diagnosed with the manual—unlike the tightly controlled field trials in DSM-IV that used preselected patients with predetermined diagnoses. Recruiting actual clinicians and patients, rather than relying on DSM Task Force members as participants, also reduces our chances of having biased findings.

Additionally, we examined the proposed changes to DSM among a wide variety of populations, varying in age, diagnosis, and ethnic background. And diagnostic interviews were conducted not just by psychiatrists but psychologists, counselors, psychiatric nurses, and family and marriage therapists; this is particularly important given the range of users who will use DSM-5 in their daily care of patients. These diagnostic interviews were not conducted using structured interviews. Rather, we wanted clinicians to assess patients as they normally would, again increasing the generalizability of results and helping us better understand how DSM-5 might perform out in the “real world.” We anticipate that findings from the DSM-5 Field Trials will help address questions from the 13 work groups about their proposed revisions, such as whether the criteria are reliable and understandable…

But there’s no actual accounting of the results in the article. We were warned about this in January starting with this article from the Task Force:
DSM-5: How Reliable Is Reliable Enough?
by Helena Chmura Kraemer, David J. Kupfer, Diana E. Clarke, William E. Narrow, and Darrel A. Regier
American Journal of Psychiatry 2012 169:13-15.
[full text on-line]

Reliability will be assessed using the intraclass kappa coefficient κI. For a categorical diagnosis with prevalence P, among subjects with an initial positive diagnosis, the probability of a second positive diagnosis is κI+P[1–κI], and among the remaining, it is P[1–κI]. The difference between these probabilities is κI. Thus κI=0 means that the first diagnosis has no predictive value for a second diagnosis, and κI=1 means that the first diagnosis is perfectly predictive of a second diagnosis…

From these results, to see a κIfor a DSM-5 diagnosis above 0.8 would be almost miraculous; to see κIbetween 0.6 and 0.8 would be cause for celebration. A realistic goal is κIbetween 0.4 and 0.6, while κIbetween 0.2 and 0.4 would be acceptable. We expect that the reliability [intraclass correlation coefficient] of DSM-5 dimensional measures will be larger, and we will aim for between 0.6 and 0.8 and accept between 0.4 and 0.6. The validity criteria in each case mirror those for reliability…

… followed by this response from Dr. Frances:

Two Fallacies Invalidate The DSM 5 Field Trials
Psychiatric Times
by Allen Frances
January 9, 2012

The designer of the DSM 5 Field Trials has just written a telling commentary in the American Journal of Psychiatry. She makes two very basic errors that reveal the fundamental worthlessness of these Field Trials and their inability to provide any information that will be useful for DSM 5 decision making.

[1] The commentary states: “A realistic goal is a kappa between 0.4 and 0.6, while a kappa between 0.2 and 0.4 would be acceptable.” This is simply incorrect and flies in the face of all traditional standards of what is considered ‘acceptable’ diagnostic agreement among clinicians. Clearly, the commentary is attempting to greatly lower our expectations about the levels of reliability that were achieved in the field trials- to soften us up to the likely bad news that the DSM 5 proposals are unreliable. Unable to clear the historic bar of reasonable reliability, it appears that DSM 5 is choosing to drastically lower that bar – what was previously seen as clearly unacceptable is now being accepted. Kappa is a statistic that measures agreement among raters, corrected for chance agreement. Historically, kappas above 0.8 are considered good, above 0.6 fair, and under 0.6 poor. Before this AJP commentary, no one has ever felt comfortable endorsing kappas so low as 0.2-0.4. As a comparison, the personality section in DSM III was widely derided when its kappas were around 0.5. A kappa between 0.2-0.4 comes dangerously close to no agreement. ‘Accepting’ such low levels is a blatant fudge factor – lowering standards in this drastic way cheapens the currency of diagnosis and defeats the whole purpose of providing diagnostic criteria. Why does this matter? Good reliability does not guarantee validity or utility – human beings often agree very well on things that are dead wrong. But poor reliability is a certain sign of very deep trouble. If mental health clinicians cannot agree on a diagnosis, it is essentially worthless. The low reliability of DSM 5 presaged in the AJP commentary confirms fears that its criteria sets are so ambiguously written and difficult to interpret that they will be a serious obstacle to clinical practice and research. We will be returning to the wild west of idiosyncratic diagnostic practice that was the bane of psychiatry before DSM III..

[2] The commentary also states: “one contentious issue is whether it is important that the prevalence for diagnoses based on proposed criteria for DSM-5 match the prevalence for the corresponding DSM-IV diagnoses” …. “to require that the prevalence remain unchanged is to require that any existing difference between true and DSM-IV prevalence be reproduced in DSM-5. Any effort to improve the sensitivity of DSM-IV criteria will result in higher prevalence rates, and any effort to improve the specificity of DSM-IV criteria will result in lower prevalence rates. Thus, there are no specific expectations about the prevalence of disorders in DSM-5.” This is also a fudge. For completely unexplained and puzzling reasons, the DSM 5 field trials failed to measure the impact of its proposals on rates of disorder. These quotes in the commentary are an attempt to justify this fatal flaw in design. The contention is that we have no way of knowing what true rates of a given diagnosis should be – so why bother to measure what will be the likely impact on rates of the DSM 5 proposals. If rates double under DSM 5, the assumption will be that it is picking up previous false negatives with no need to worry about the risks of creating an army of new false positives…

There are plenty of heady articles around about the vicissitudes of Kappa that might leave one math-phobic for life. Perhaps, the right place to start was with its introduction to psychiatric diagnosis by Robert Spitzer in 1974. He had actually looked into reliability earlier [1967] and so in 1974 he published a meta-analysis of six studies retrospectively calculating Kappa:
A Re-analysis of the Reliability of Psychiatric Diagnosis
By ROBERT L. SPITZER and JOSEPH L. FLEISS
British Journal of Psychiatry. 1974  125:341-347.
Here’s Spitzer’s 1974 table of his calculated Kappa values using the DSM-II [before the Revolution] derived from the data of six previously published studies of inter-rater reliability:


[reformatted][0.46 is the Mean of all values]]

Here’s a frequency distribution of all of the Kappa values in Spitzer’s table from the six studies…

And here are the two versions of the interpretation of Kappa from Dr. Kraemer’s and Dr. Frances’ articles in January of this year [above]…

Kappa
    <0.20   >0.20 & <0.40   >0.40 & <0.60   >0.60 & <0.80   >0.80
Kraemer et al   negative   acceptable   realistic   celebration   miraculous
Frances   negative   ~ no agreement   poor   fair   good

Some choices are self-evident. They’ve already been made. Had Dr. Spitzer used Dr. Kraemer’s criteria, he wouldn’t have needed to revise much of anything in the first place…
Mickey @ 10:18 am
Filed under: politics
not a good time…

Posted on Saturday 5 May 2012

"I didn’t know whether to laugh or cry. Laugh — because there is no way task force and work group members can be made to refrain from discussing the developing DSM-V with their colleagues. Cry — because this unprecedented attempt to revise DSM in secrecy indicates a failure to understand that revising a diagnostic manual — as a scientific process — benefits from the very exchange of information that is prohibited by the confidentiality agreement. Such exchange of information with colleagues was encouraged with the development of DSM-III, DSM-III-R, and DSM-IV."
Robert Spitzer July 18, 2008

He was talking about this confidentiality agreement signed by members of the DSM-5 Task Force:

I will not, during the term of this appointment or after, divulge, furnish, or make accessible to anyone or use in any way… any Confidential Information. I understand that ‘Confidential Information’ includes all Work Product, unpublished manuscripts and drafts and other pre-publication materials, group discussions, internal correspondence, information about the development process and any other written or unwritten information, in any form, that emanates from or relates to my work with the APA task force or work group.

That’s how the protest of the DSM-5 began. Dr. Robert Spitzer, the father of the DSM-III and DSM-IIIR had requested some information about the DSM-5 process and they essentially said that it was none of his business, evoking the confidentiality agreement. He went ballistic and began to write about their closed shop. Was it injured pride? justified outrage? Probably both. He asked Dr. Allen Frances, who had been in charge of the DSM-IV to join him in protesting but Frances declined. Later, hearing about the Psychosis Risk Syndrome, he changed his mind and became  an outspoken and global critic, not just about the process, but about their work product. Failing to engage the powers within organized psychiatry, he took his campaign to the streets, or at least to other mental health disciplines impacted by this revision, with great success [DSM5 in Distress]. Today, in Philadelphia at the APA Annual Meeting, We’ll hear the results the DSM-5 Field Trials – tests of inter-rater variability in their proposed changes. We’ll likely hear that the Attenuated Psychosis Syndrome and the Mixed Anxiety Depression Disorder flunked [DSM-5 Draft Criteria Open for Public Comment]. I would expect there will be others with very low values since they’ve already written that we should lower our expectations [don’t expect too much…]. Later in the week, we’ll hear the specifics about the Disorders.

Meanwhile, protesters will be marching on the convention with greater diversity and numbers than usual – adding people protesting the current DSM-5 Task Force to the psychiatry survivor and other groups that usually gather. This tradition of protests at the APA has roughly the same timeline as the DSM-III Revolution and the two are inter-twined. It arose in the 1970s from the early Gay Pride Movement and protests against Involuntary Hospitalization and Biological Treatments:

As a veteran of the Civil Rights Movement in my youth, I used up my own protest energies early on. I believed in that cause unambiguously. Since then, I’ve almost always been cursed with ambivalence – seeing something in both sides of most polarities – feeling uncomfortable with the zealots on either side of the fence. It’s not that I don’t have opinions, I just don’t trust them. In this week’s scene in Philadelphia, in spite of opinions and preferences, I can get on either side of a few of the elements in the mix. But sometimes, I feel that old unambiguous feeling from long ago, so I snuck some pictures into my opening collage that really don’t belong there. That’s DSM-5 Chairmen Darrel Regier and David Kupfer on the left at a hearing earlier in the year. The other picture is from last October’s Chair Summit – a CME neuroscience conference lead by Department Chairs in Psychiatry and Neurology.

There is Charlie Nemeroff [until recently chair at Emory, now at Miami] – too toxic for public display but still in game. Next is Martin Keller [until recently chair at Brown], notorious PHARMA front-man of Paxil Study 329. And next to him, Alan Schatzberg [until recently chair at Stanford] – a widely discredited PHARMA guy, but still an active player. Jeffery Lieberman, APA president elect, and Schatzberg recently moderated an American Psychiatric Foundation summit with industry about how to make it easier for the Pharmaceutical Companies  to introduce new CNS drugs onto the market [APF Convenes Unique Pipeline Summit]. While Regier and Kupfer are less tarnished, their ideological taint is pernicious [and they don't seem to even know it's there]. I threw in Tom Insel, Director of the NIMH, since he’s also in Philadelphia talking to reporters creating his own illusions:

    Thomas R. Insel, a psychiatrist who directs the National Institute of Mental Health, sees psychiatry heading toward what he calls "clinical neuroscience." Psychiatrists should study the brain the way cardiologists study the heart, he said. "There is no biochemical imbalance that we have ever been able to demonstrate. What we think about are changes in circuitry and how the brain is processing information."
So returning to my comments from the Future of an Illusion, I still see way too many psychiatrists in power captured by an illusion [and way too many who have capitalized on that illusion] to be hopeful about what’s going on inside the Convention Hall in Philadelphia this week. I find myself feeling unambivalently negative about the current leadership in psychiatry that’s still running on the inertia of a three decade long unholy alliance with the Pharmaceutical Industry in spite of repeated wake-up calls. It’s not a good time…

Note: An earlier version had an error, confusing J. Lieberman and M. Keller in the photo.
Mickey @ 6:00 pm
Filed under: politics
unusual things…

Posted on Friday 4 May 2012

Well Dr. Gibbons’s meta-analysis of suicidality and efficacy in Major Depressive Disorder on SSRIs [Prozac, Effexor], the subject of 24 previous posts on this blog, is the absolute paradigm for the gift that keeps on giving. In this issue of The Scientist, he’s interviewed in an offering about access to full clinical trial data.

… But even clinical study reports include some level of synthesis. At the finest level of resolution are the raw, unabridged, patient-level data. Getting access to either set of results, outside of being trial sponsors or drug regulators, is a rarity. Robert Gibbons, the director of the Center for Health Statistics at the University of Chicago, had never seen a reanalysis of raw data by an independent team until a few years ago, when he himself was staring at the full results from Eli Lilly’s clinical trials of the blockbuster antidepressant Prozac.

FDA, time, Gibbons had questioned the belief that antidepressants are linked to an increased risk of suicide. Previous meta-analyses by independent reviewers on suicidal thoughts and behaviors among people taking the drugs had for the most part relied on summary data, Gibbons says. At a meeting at the Institute of Medicine a few years ago, Gibbons spoke with a senior investigator at Eli Lilly and brought up the idea of doing a full workup of the original data. Much to his surprise, shortly after the meeting Gibbons was in possession of the numbers. “We haven’t seen anybody get these kinds of data,” he says. He decided to push his luck. Gibbons had served as an expert witness for Wyeth, and he approached attorneys for the pharmaceutical company to ask if they would also share data from trials of the company’s antidepressant Effexor. “They got back to me, and they were agreeable to provide all their adult data,” he recalls.

Gibbons and his colleagues went to work reanalyzing the data. “Everything was exquisitely well documented,” he says. The raw data allowed them to take into account each person’s depression severity and to determine individual outcomes rather than averages. Their results, published earlier this year, ended up bucking much of the published literature on antidepressants. For one, they found no link between Prozac and suicide risk among children and young adults. And secondly, they found that Prozac appeared to be more effective in youth, and antidepressants far less efficacious in the elderly, than previously thought. “I think these kinds of analyses and the discrepancies in the findings are good reason to be concerned about our reliance on traditional meta-analyses,” Gibbons says. Although some of his results reflect negatively on the drugs, others are clearly very positive. There’s been an understanding for some time that publication bias is a real occurrence, and that it often favors the drug…

For the life of me, I still can’t figure out what he means by "The raw data allowed them to take into account each person’s depression severity and to determine individual outcomes rather than averages." Each study reports the number of people meeting Response and Remission criteria, so the source of his claim that his meta-analysis had some other outcome tidbits that made them special is beyond me yet. Dr. David Healy had a fine time with this article:
May Fools’ Day
by David Healy
May 3, 2012

Following the long-standing tradition, dating back at least to Chaucer, of playing practical jokes on May 1, The Scientist clearly thought it would be a good idea to show the outside world that science doesn’t always have to be stuffy and picked the appropriate day to demonstrate the point [http://the-scientist.com/2012/05/01/data-diving/]. Sadly, the joke has gone unnoticed, which defeats the point. Practical jokes of this type need to be celebrated. Hopefully this belated recognition will bring some comfort to the author and editors, and encourage them to continue bringing this important tradition into an arena that desperately needs it. To celebrate May 1, the journal ran an article called Data Diving, ostensibly on a subject that is at the heart of healthcare debates today – access to clinical trial data. In the presence of full and unfettered access, scientific data will supposedly sing out clear and true [Chant clear – they might have said in Chaucer’s day]. Without access, companies can claim their trials show whatever the company wants them to show. Without access, companies get to charge whatever they want for their products…

The centerpiece of Data Diving, the joke, features the work of Robert Gibbons, who has supposedly had unfettered access to patient level data from the trials of Lilly’s Prozac and Wyeth’s Effexor [see Coincidence a fine thing]. Just as Doshi and Jefferson’s access overturned a myth – that Tamiflu has a significant clinical effect, so also Gibbons’ access to the “data” seemingly has overturned myths. In this case, those pesky myths that antidepressants don’t really work and especially that they cause suicide. It was, it turns out, lack of access to the data in the first place that led us to these mistaken beliefs. Companies don’t engage in conspiracies, we are being told, they are masters of the cock-up, and if given a choice of feet to shoot themselves in will opt for both feet. It needs independent academics like Robert Gibbons to  wade in and put a stop to their self-injurious behavior.

The give away that this is a May Fool’s Day joke is that very few articles in recent years in the psychiatric or any other literature  have received such withering critique as the Gibbons’ articles to which The Scientist refers – see for example the sequence from April 14 to 16 by boringoldman. The journal’s fact checkers would never have missed this, and by picking such an egregious piece are clearly letting us in on the joke:
    an anatomy of a deceit 1… introduction
    an anatomy of a deceit 2… the fog comes on little cat feet
    an anatomy of a deceit 3… readin’ and writin’ and ‘rithmetic
    an anatomy of a deceit 4… the letter
    an anatomy of a deceit 5… the purloined letter
    an anatomy of a deceit 6… anticipated and forestalled
There is high parody here in that of course Gibbons didn’t have access to anything the original authors of the Prozac papers didn’t already have access to, in that the original authors were all company people with presumably even fuller access than Gibbons was later given. Gibbons has also managed to avoid incorporating or otherwise handling data in the public domain that could be readily accessed anyway that show incontrovertibly that Prozac and Effexor can not only trigger suicide but that on balance the harms outweigh any benefits. By ignoring relevant accessible data in favor of data no-one else has access to, there is here an almost complete inversion of the standard access to data argument. In the new spirit of openness and perhaps to continue the joke, The Scientist might consider asking Dr Gibbons to make his data fully available on the journal’s website…

I appreciate the linking, but even more I appreciate that others are looking at this bizarre  and highly publicized meta-analysis. Dr. Gibbons seems dead-set on undermining the black box warning against using these drugs casually, particularly in children and adolescents – and the press appears delighted to help him after hearing his heroic narrative of how his special access to the data allowed him to achieve results that exceed all of the original studies! Special indeed.

I was a late arrival to awareness of the systematic deceit in our psychiatric scientific literature, so most of the corrupted studies I’ve looked at have been in retrospect. I came into this one almost on the day Gibbons’ first article was published and have read the media coverage that accompanied his findings "hot off the press." It feels different when it’s happening in real-time – darker. Unless my own analysis of these articles is in left field [along with everyone else's], the authors have to know themselves that they are promoting a jury-rigged piece of science – and Dr. Gibbons is promoting it a lot. I can’t figure any way to fold the  paper and conclude that their results are simply in error. It appears to me for all the world like conscious artifice – thus my title, an anatomy of a deceit.

And I could understand not publishing criticism from some blogger like me, but the Archives of General Psychiatry has sent the more credentialed to time out too. I’m in good company:
There may be an unexpected sequel to this joke. A scandal brewing over the Gibbons’ articles – some unreconstructed academics it seems just don’t “get it”. Not only are Gibbons’ data inaccessible they complain but it seems criticism of the Gibbons’ paper is out of bounds. The Archives of General Psychiatry has steadfastly refused to publish letters from a number of groups pointing out the lethal flaws in Gibbons papers.
In the article from The Scientist as elsewhere, Gibbons is claiming that the reason his analysis is right and everyone else is wrong is that he has raw data that the rest of us don’t have. What makes that claim ludicrous, as Dr. Healy points out, is that he doesn’t show us that data – neither the raw data nor, in the case of youth, even some graph or table. The point of The Scientist article above is that with full access to clinical trial data, you find unusual things. Then Gibbons reports his unusual things but doesn’t give us the raw data. Remarkable. Thus Dr. Healy’s challenge, "The Scientist might consider asking Dr Gibbons to make his data fully available on the journal’s website," which I see as a real and reasonable request. Dr. Healy is now decades into dealing with recurrent attempts to discount the suicidality and low efficacy of the SSRI’s, particularly in youth. I’m a newbie. I’m not used to it yet, and I hope I never have to get used to it.

The great irony in this is that the statistical approach to this problem really doesn’t even matter. Assume for a moment that the unproven and unlikely argument that the SSRIs really are effective enough to actually prevent suicides in depressed kids as suggested by Drs. Gibbons now or Dr. Nemeroff in his 1991 testimony at the FDA:
September 20, 1991

I would suggest to you that I have as little confidence in these anecdotal reports as I do in the anecdotal report of Teicher, and that, in fact, there is no substitute for controlled prospective double-blind clinical trials…

In conclusion, there is simply no scientific evidence whatsoever, no placebo-controlled double-blind study that has established a cause-and-effect relationship between antidepressant pharmacotherapy of any class and suicidal acts or ideation.

As Drs. Potter and Fawcett have suggested, limiting the availability of antidepressants could have a very profound adverse effect in terms of increasing the morbidity and, in fact, mortality associated with untreated depression.

Dr. Charles Nemeroff,
Professor and Chair, Department of Psychiatry,
Emory University, Atlanta Georgia

That doesn’t invalidate the known cases where the well documented paradoxical reaction has lead to unnecessary suicides. Those kids are just as dead, and they and their families should’ve been warned that it could happen. Even if the results of the controlled prospective double-blind clinical trials guaranteed an efficacy that would decrease depression related suicidality in the majority [which they don't], the warning about the minority possibility would still be just as important, improving outcomes even further by short circuiting senseless tragedies. Neither Nemeroff before him nor Gibbons in the present make this obvious argument. So, I can only conclude that their motives are to promote more widespread prescribing by denying and invalidating a very real danger. That is a misuse and abuse of science and our literature…
Mickey @ 9:20 am
Filed under: politics
the future of an illusion V…

Posted on Thursday 3 May 2012

If it becomes discredited — and indeed the threat to it is great enough — then your world collapses. There is nothing left for you but to despair of everything, of civilization and the future of mankind.
Sigmund Freud: The Future of an Illusion 1927

In his book, The Future of an Illusion, Freud was having a fantasied argument with someone who was defending an irrational religious illusion, and he was pointing out that the anticipated consequences of giving up such an illusion is apocalypse and catastrophe. We’ve certainly seen that phenomenon at work in the world religions in recent years. But it seems bigger than just religion, we hear it nightly in political debates – grave predictions that the end is just around the corner if we don’t… <something or another>. If I look at the world events of my lifetime, it appears to me that the formula, the loss of an illusion will lead to an apocalypse, might be totally backwards – that it’s the persistence of illusions that leads to apocalypse. Indeed, shared illusion seems to be an essential ingredient in most catastrophic consolidations of power, either defending one’s own [illusions] or attacking someone else’s [illusions].

The history of the DSM-III and subsequent revisions is a pretty good example of the future of illusions based on fears of apocalypse. In the 1970s, psychiatry was under attack from a number of directions. Dr. Szasz, a psychiatrist, published The Myth of Mental Illness – which was a fairly global criticism, but his primary focus was on the abuse of power throughout the history of psychiatry in the area of Involuntary Hospitalization and forced treatment. He felt that psychiatrists had become agents of the State. He saw Mental Illness as a fictitious construct, not an object of medical concern and his argument rested on the absence of any biomarkers linking Mental Illness to medicine proper. Like many in that era, he saw Freud’s psychoanalysis as something of a cult, based on illusion. But there were other directions of attack. The psychologists and others were battling over third party payments. Internally, there was also a battle about reimbursements – analysts and other psychotherapists were billing insurance for long, expensive treatments. The same complaint was leveled at long [or any] psychiatric hospitalizations. The future of coverage for psychiatric care was in question  and catastrophe was imminent.

The DSM-III was thus more than a revision in the diagnostic scheme, it was the antidote to a coming cataclysm – a compromise for all seasons. What had been seen as the the psychoanalytic illusion was replaced with the neo-Kraepelinian illusion – anchoring psychiatry to medicine [and its finances]. All things mind were replaced with all things medical. In the minds of many, medical meant brain. For others, it was conveniently ambiguous. The change was widely welcomed as it resolved so many problems. Involuntary treatment became more an issue for the legal system and made moot by the disappearance of mental hospitals [the largest collections of institutionalized mental patients are now in prisons]. Psychoanalytic training continues, training people from all mental health disciplines primarily to do psychodynamic psychotherapy rather than the full analysis of Freudian days. The other mental health professions, the insurers, and the drug companies all embraced the new DSM system each for their own reasons. But its appeal wasn’t its medicalization, it was that it provided a framework for coding and its ambiguity allowed all comers to find a way to use it for their own purposes and not really change courses.

Rather than being a definitive diagnostic compendium, it had more the quality of a negotiated peace [I likened it to the treaty of versailles in an early post because it contained the kernels of trouble up ahead]. So over the last thirty years, main stream psychiatry has become almost exclusively medical/brain oriented while maintaining the illusion of etiologic neutrality in the diagnostic system. Psychiatric research flowered, mostly funded by the pharmaceutical industry in the form of endless trials of medications targeted at DSM diagnostic groups, run primarily by the new clinical research industry. Treatment with psychotropics escalated geometrically as more clones were introduced. The 1990s were declared the Decade of the Brain. Then the first decade of the twenty-first century was either the dawn of the Age of Clinical Neuroscience or the single most corrupt merger of science and industry in medical history – depending on your vantage point.

Why the powers that be in psychiatry chose to co-opt the not-terribly-useful-but-comfortably-ambiguous  DSM fully into the neurobiology camp at the very same time that the dark side of the Age of Pharmacology was all over the evening news and their pharmaceutical allies raced to full retreat is beyond my understanding. Instead of a course correction to put balance in a diagnostic system that badly needs it as a show of good faith or even contrition, the DSM-5 Task Force came out of the gate with the message that their fully validated brain science was at last a near reality. Then they began working on adding even more reasons to the DSM to over-medicate patients. Maybe they just didn’t see what was coming. My only other shot at an explanation is that they began to believe their own illusions and are just too insular to hear that no one was clapping – in spite of the heads-up by Drs. Spitzer and Frances, in spite of an anti-psychiatry movement that may put the 1970s to shame when it’s said and done, in spite of the widespread exposure of impropriety, in spite of an empty pipeline, in spite of no particularly exciting tangible results in the research arena.

So here we sit in the midst of the future of an illusion, which is always the same – a time of disillusionment. And there’s apocalypse all around. What should we do? Like anyone reading this, I have some thoughts about that but I don’t trust any of them, nor think it’s the time for some grand new plan. The best principle of Crisis Intervention is to insert a pause if possible, to take a deep breath, and to let the emotional drivers die down so you can think rationally. To borrow a term from the recent DSM-5 debates – it’s a time for bereavement – a time to look at what we’ve lost in the last thirty years that really mattered. I think the answers are easy. We’ve lost integrity. That part needs little clarification as it’s so obvious. But there’s something else, we’ve lost a lot of our clinician class. If you’re in Mental Health, you know who I’m talking about. Some were analysts, but most weren’t. They are those people who students and residents emulate, want to become – the ones that we all wanted to be like when we showed up forty years ago. They’re the people we referred to if they were seeing new patients, independent of their stated interests. They were no longer valued because they didn’t often do research or bring in any money, so their ranks gradually declined. British neuropsychopharmacologist Dr. David Nutt gave a lecture last year entitled, "no psychiatry without psychopharmacology." A more useful topic might have been "no psychiatry without psychiatrists" – meaning clinicians who are skilled at getting at the essence of things, no matter what their orientation [some of the best clinicians I've ever known actually spent their afternoons in the lab with rats and monkeys]. The DSM-III and later hasn’t encouraged those clinical skills, nor has the current standard of practice. And it shows when you’re around young psychiatrists who talk too much about new drugs and technology rather than sick people.

So the task at hand for psychiatry is to get out of the illusion business, to clean up our corruption and conflicts  of interest, to restore integrity to our journals and research, to keep our DSM-5 from perpetuating idiosyncratic fantasies, to become true experts on our drugs [like side effects, true efficacy, withdrawal syndromes, etc], to fight for the means to repopulate our clinician class, to remember with respect that we are just one mental health discipline among the many, and to restore our primary alliance with the people we call patients rather than the various industries that profit from their [and our] vulnerabilities. And while we’re at it, there are a number of rotten apples near the top of our tree that need to plucked yesterday. Okay, I admit it. I do have a few suggestions after all. Apocalypse be damned. It’s of our own making…
Mickey @ 5:01 pm
Filed under: politics