1 Boring Old Man

Billy: It’s finding the center of your story, the beating heart of it, that’s what makes a reporter. You have to start by making up some headlines. You know: short, punchy, dramatic headlines. Now, have a look, what do you see? [Points at dark clouds at the horizon]
Billy: Tell me the headline.
Quoyle: Horizon Fills With Dark Clouds?
Billy: Imminent Storm Threatens Village.
Quoyle: But what if no storm comes?
Billy: Village Spared From Deadly Storm.

Most DSM-5 Revisions Pass Field Trials
MedPage Today
By John Gever, Senior Editor
May 07, 2012

Field testing of proposed new diagnostic criteria for many mental disorders showed that, for the most part, they were at least adequately reliable, according to initial reviews presented here. But some new disorders that had appeared set for inclusion in DSM-5, the forthcoming fifth revision of the American Psychiatric Association’s [APA] influential diagnostic manual, will now probably be relegated to a research-use-only classification because interrater reliability was so poor. Among them is the controversial "attenuated psychosis syndrome," intended to capture patients with low-level hallucinations and thinking disturbances.

In some cases, it appeared that the field trials themselves had failed – that too few patients were found with a given diagnosis to provide meaningful data, or that other problems cropped up that made the results uninterpretable. Results of the field trials – a critical step in the development of DSM-5 – were presented here at the APA’s annual meeting. The trials are essentially the last step before the final round of meetings and decisions that will set DSM-5 in its published form. Although DSM-5 will be a "living document" that will be revised on an ongoing basis, the paper version must go to the printer in December and therefore the contents must be finalized in the next few months, explained David Kupfer, MD, of the University of Pittsburgh, the task force chairman for the effort…

Darrel Regier, MD, the APA’s research director, explained that the trials were intended primarily to establish reliability – that different clinicians using the diagnostic criteria set forth in the proposed revisions would reach the same diagnosis for a given patient. The key reliability measure used in the academic center trials was the so-called intraclass kappa statistic, based on concordance of the "test-retest" results for each patient. It’s calculated from a complicated formula, but the essence is that a kappa value of 0.6 to 0.8 is considered excellent, 0.4 to 0.6 is good, and 0.2 to o.4 "may be acceptable." Scores below 0.2 are flatly unacceptable.

Kappa values for the dozens of new and revised diagnoses tested ranged from near zero to 0.78. For most common disorders, kappa values from tests conducted in the academic centers were in the "good" range:

• Bipolar disorder type I: 0.54
• Schizophrenia: 0.46
• Schizoaffective disorder: 0.50
• Mild traumatic brain injury: 0.46
• Borderline personality disorder: 0.58

In the "excellent" range were autism spectrum disorder [0.69], PTSD [0.67], ADHD [0.61], and the top prizewinner, major neurocognitive disorder [better known as dementia], at 0.78. But some fared less well. Criteria for generalized anxiety disorder, for example, came in with a kappa of 0.20. Major depressive disorder in children had a kappa value of 0.29. A major surprise was the 0.32 kappa value for major depressive disorder. The criteria were virtually unchanged from the version in DSM-IV, the current version, which also underwent field trials before they were published in 1994. The kappa value in those trials was 0.59.

But a comparison is not valid, Regier told MedPage Today. The patient samples were very different – in the DSM-IV trials, patients with psychiatric comorbidities were excluded, whereas they were allowed in the DSM-5 tests. Also, the clinicians in the DSM-IV trials received much more elaborate training before seeing patients than did those in the DSM-5 trials. Helena Kraemer, PhD, a statistician at Stanford University who helped set the general DSM-5 trial structure, said an important goal was to test the criteria "with real clinicians and real patients." She said the kappa values in DSM-IV were "inflated" because of the patient exclusions and clinician training.

Jan Fawcett, MD, of the University of New Mexico, who headed the DSM-5 workgroup that decided on the depression criteria, said that the disorder may simply be hard to diagnose reliably because of the many comorbidities that come with it. "I was not happy with the kappas that came out," he said. "But it may be that that is the reliability of that diagnosis in the real world."

Diagnoses that did very poorly in the trials included either because of low kappa values or confidence intervals so wide as to be meaningless:

• Mixed anxiety-depressive disorder in adults and children
• Attenuated psychosis syndrome
• Obsessive-compulsive personality disorder [separate from obsessive-compulsive disorder]
• Antisocial personality disorder
• Nonsuicidal self-injury [e.g., cutting]

^{Although the kappa value for attenuated psychosis was actually in the "good" range at 0.46, the 95% confidence interval extended to zero, probably reflecting a failed trial. Most of the others had low point values for kappa – approaching zero for mixed anxiety-depression in adults – as well as wide confidence intervals. These conditions had been considered seriously for inclusion in the main body of DSM-5, but now will probably either be ditched entirely or placed in Section III [what had been the appendix in DSM-IV], for diagnoses that could be clinically useful but need more research-based refinement…}

• But a comparison is not valid, Regier told MedPage Today. The patient samples were very different – in the DSM-IV trials, patients with psychiatric comorbidities were excluded, whereas they were allowed in the DSM-5 tests.
• Also, the clinicians in the DSM-IV trials received much more elaborate training before seeing patients than did those in the DSM-5 trials.
• Helena Kraemer, PhD, a statistician at Stanford University who helped set the general DSM-5 trial structure, said an important goal was to test the criteria "with real clinicians and real patients."
• She said the kappa values in DSM-IV were "inflated" because of the patient exclusions and clinician training.

• Jan Fawcett, MD, of the University of New Mexico, who headed the DSM-5 workgroup that decided on the depression criteria, said that the disorder may simply be hard to diagnose reliably because of the many comorbidities that come with it. "I was not happy with the kappas that came out," he said. "But it may be that that is the reliability of that diagnosis in the real world."