Village Consumed by Deadly Storm…

Posted on Monday 7 May 2012

In Annie Proulx‘s Shipping News, the protagonist, Quoyle is learning to write headlines for the local paper:
    Billy: It’s finding the center of your story, the beating heart of it, that’s what makes a reporter. You have to start by making up some headlines. You know: short, punchy, dramatic headlines. Now, have a look, what do you see? [Points at dark clouds at the horizon]
    Billy: Tell me the headline.
    Quoyle: Horizon Fills With Dark Clouds?
    Billy: Imminent Storm Threatens Village.
    Quoyle: But what if no storm comes?
    Billy: Village Spared From Deadly Storm.
One can do almost anything with a headline. I’m periodically scanning the Internet for more information  about the DSM-5 Field Trials, and the Google just doesn’t have much to say. The Scientific American Blog, Dr. Frances’ comments, and this on a University of Pennsylvania CME Site, MedPage Today, with a memorable headline:
Most DSM-5 Revisions Pass Field Trials
MedPage Today
By John Gever, Senior Editor
May 07, 2012

Field testing of proposed new diagnostic criteria for many mental disorders showed that, for the most part, they were at least adequately reliable, according to initial reviews presented here. But some new disorders that had appeared set for inclusion in DSM-5, the forthcoming fifth revision of the American Psychiatric Association’s [APA] influential diagnostic manual, will now probably be relegated to a research-use-only classification because interrater reliability was so poor. Among them is the controversial "attenuated psychosis syndrome," intended to capture patients with low-level hallucinations and thinking disturbances.

In some cases, it appeared that the field trials themselves had failed – that too few patients were found with a given diagnosis to provide meaningful data, or that other problems cropped up that made the results uninterpretable. Results of the field trials – a critical step in the development of DSM-5 – were presented here at the APA’s annual meeting. The trials are essentially the last step before the final round of meetings and decisions that will set DSM-5 in its published form. Although DSM-5 will be a "living document" that will be revised on an ongoing basis, the paper version must go to the printer in December and therefore the contents must be finalized in the next few months, explained David Kupfer, MD, of the University of Pittsburgh, the task force chairman for the effort…

Darrel Regier, MD, the APA’s research director, explained that the trials were intended primarily to establish reliability – that different clinicians using the diagnostic criteria set forth in the proposed revisions would reach the same diagnosis for a given patient. The key reliability measure used in the academic center trials was the so-called intraclass kappa statistic, based on concordance of the "test-retest" results for each patient. It’s calculated from a complicated formula, but the essence is that a kappa value of 0.6 to 0.8 is considered excellent, 0.4 to 0.6 is good, and 0.2 to o.4 "may be acceptable." Scores below 0.2 are flatly unacceptable.

Kappa values for the dozens of new and revised diagnoses tested ranged from near zero to 0.78. For most common disorders, kappa values from tests conducted in the academic centers were in the "good" range:

  • Bipolar disorder type I: 0.54
  • Schizophrenia: 0.46
  • Schizoaffective disorder: 0.50
  • Mild traumatic brain injury: 0.46
  • Borderline personality disorder: 0.58
In the "excellent" range were autism spectrum disorder [0.69], PTSD [0.67], ADHD [0.61], and the top prizewinner, major neurocognitive disorder [better known as dementia], at 0.78. But some fared less well. Criteria for generalized anxiety disorder, for example, came in with a kappa of 0.20. Major depressive disorder in children had a kappa value of 0.29. A major surprise was the 0.32 kappa value for major depressive disorder. The criteria were virtually unchanged from the version in DSM-IV, the current version, which also underwent field trials before they were published in 1994. The kappa value in those trials was 0.59.

But a comparison is not valid, Regier told MedPage Today. The patient samples were very different – in the DSM-IV trials, patients with psychiatric comorbidities were excluded, whereas they were allowed in the DSM-5 tests. Also, the clinicians in the DSM-IV trials received much more elaborate training before seeing patients than did those in the DSM-5 trials. Helena Kraemer, PhD, a statistician at Stanford University who helped set the general DSM-5 trial structure, said an important goal was to test the criteria "with real clinicians and real patients." She said the kappa values in DSM-IV were "inflated" because of the patient exclusions and clinician training.

Jan Fawcett, MD, of the University of New Mexico, who headed the DSM-5 workgroup that decided on the depression criteria, said that the disorder may simply be hard to diagnose reliably because of the many comorbidities that come with it. "I was not happy with the kappas that came out," he said. "But it may be that that is the reliability of that diagnosis in the real world."

Diagnoses that did very poorly in the trials included either because of low kappa values or confidence intervals so wide as to be meaningless:

  • Mixed anxiety-depressive disorder in adults and children
  • Attenuated psychosis syndrome
  • Obsessive-compulsive personality disorder [separate from obsessive-compulsive disorder]
  • Antisocial personality disorder
  • Nonsuicidal self-injury [e.g., cutting]
Although the kappa value for attenuated psychosis was actually in the "good" range at 0.46, the 95% confidence interval extended to zero, probably reflecting a failed trial. Most of the others had low point values for kappa – approaching zero for mixed anxiety-depression in adults – as well as wide confidence intervals. These conditions had been considered seriously for inclusion in the main body of DSM-5, but now will probably either be ditched entirely or placed in Section III [what had been the appendix in DSM-IV], for diagnoses that could be clinically useful but need more research-based refinement…
I could’ve thought of a hundred things to say about the DSM-5 Trials, but Most DSM-5 Revisions Pass Field Trials wouldn’t have been on the list. And considering the import of the trials, I would’ve given the kind of access these people had to the "big dogs" to a more mainstream media source. About all I got from this article other than a few more dismal numbers was the sense that the powers that be on the DSM-5 Task Force are going to deal with things with spin:
  • But a comparison is not valid, Regier told MedPage Today. The patient samples were very different – in the DSM-IV trials, patients with psychiatric comorbidities were excluded, whereas they were allowed in the DSM-5 tests.
  • Also, the clinicians in the DSM-IV trials received much more elaborate training before seeing patients than did those in the DSM-5 trials.
  • Helena Kraemer, PhD, a statistician at Stanford University who helped set the general DSM-5 trial structure, said an important goal was to test the criteria "with real clinicians and real patients."
  • She said the kappa values in DSM-IV were "inflated" because of the patient exclusions and clinician training.
At least, one person gave a semi-straight explanation with less curve on the ball:
  • Jan Fawcett, MD, of the University of New Mexico, who headed the DSM-5 workgroup that decided on the depression criteria, said that the disorder may simply be hard to diagnose reliably because of the many comorbidities that come with it. "I was not happy with the kappas that came out," he said. "But it may be that that is the reliability of that diagnosis in the real world."
The DSM-5 Task Force has skipped  the gold standard of having two sets of Field Trials to dial in on validity and reliability. They’ve attempted to redefine the meaning of the kappa values [see self-evident…]. They’ve maligned their predecessors, saying the DSM-IV Field Trials were too controlled, or that the raters were too trained, evoking the "naturalistic" argument used for other flops like STAR*D. They’ve been preoccupied with off-beat categories that are falling out like flies. The only thing I can find to agree with in this article is Dr. Fawcett’s comment about Major Depressive Disorder [which came in at 0.32 for adults and 0.29 for kids]. It’s not a reliable diagnosis because it doesn’t exist. Never has. So as headlines go, so far, I’d propose that Dr. Allen Frances’ previous posts be collected in a book called, Imminent Storm Threatens Village. And I’d name the Chapter about the Field Trials, Village Consumed by Deadly Storm. Most DSM-5 Revisions Pass Field Trials just isn’t in the mix. It’s going to take a really strong person to do the right thing here, which is postpone this whole show and bring in someone to get the enterprise on the right path. If they want the DSM-5 to be something that’s actually used, something psychiatry can be proud of, that’s the only outcome that makes a bit of sense…
  1.  
    Steve
    May 7, 2012 | 8:55 PM
     

    Wait, did I read that right? You’re saying that there’s no such thing as MDD?

  2.  
    May 7, 2012 | 9:05 PM
     

    MDD is on a par with Congestive Heart Failure in my book – a collage…

  3.  
    Steve
    May 8, 2012 | 10:23 AM
     

    But isn’t that like saying that there is no such thing as cancer because cancer is an amalgam of diseases? I know that many folks do say that about cancer, but it always strikes me as somewhat dishonest. Cancer may be a grouping of many things, but that doesn’t mean that there is no such thing as cancer. We can always use reduction to say that something is a grouping of other things, but it doesn’t follow that it isn’t also X.

  4.  
    May 8, 2012 | 12:58 PM
     

    It’s now possible to be successfully treated for some cancers so that the cancer doesn’t return. The patient need not be labeled with Major Cancer Disorder and told that he/she will have to continue treatment for life.

Sorry, the comment form is closed at this time.