1 Boring Old Man

Newsflash From APA Meeting: DSM 5 Has Flunked its Reliability Tests
^{Needs To Be Kept Back For Another Year}
DSM5 in Distress: Psychology Today
by Allen J. Frances, M.D.
May 6, 2012

The whole purpose of having a manual of psychiatric diagnosis is to promote diagnostic agreement. DSM III was an important milestone because it saved the credibility and relevance of psychiatry at a time when it was ridiculed for low reliability. Everyone knew that the reliability achieved in DSM field testing far exceeds what is possible in clinical practice, but DSM III took the major step of proving that reliability could be achieved at all. Until now, the DSM’s have facilitated communication across the clinical/research interface, promoted research, and provide credibility in the court room.

But bad news has just been reported from the annual meeting of the American Psychiatric Association in Philadelphia. The hard won credibility of psychiatric diagnosis is compromised by the abysmal results reported by the DSM 5 Field Trials. This failure was clearly predictable from the start:
[1] The writing of the DSM 5 criteria sets was far too raw and imprecise to be ready for the rigors of field testing. The ambiguity cried out for expert editing; without which reasonable reliability is impossible;
[2] The design of the field trial was byzantine in complexity and could never be done on schedule:
[3] Constant delays in starting and completing Stage 1 of the study forced DSM 5 to cancel the planned Stage 2 that was meant to clean up the poorly performing criteria sets identified in the first stage.
[4] With stage 2 cancelled without explanation, it looks like even the worst diagnoses are being given a social pass; and, most absurd,
[5] The design was totally off point, failing to ask the only question that really counted [the impact of DSM 5 on rates].
The results of the DSM 5 field trials are a disgrace to the field. For context, in previous DSM’s, a diagnosis had to have a kappa reliability of about 0.6 or above to be considered acceptable. A reliability of 0.2-0.4 has always been considered completely unacceptable, not much above chance agreement:

Kappa Values from the DSM-5 Field Trials

^{[reformatted from the original]}

No predetermined publication date justifies business as usual in the face of these terrible Field Trial results [which are even more striking since they were obtained in academic settings with trained and skilled interviewers, highly selected patients, and no time pressure. The results in real world settings would be much lower]. Reliability this low for so many diagnoses gravely undermines the credibility of DSM 5 as a basis for administrative coding, treatment selection, and clinical research.

What can be done to salvage this deplorable mess:
[1] DSM 5 has never had anyone on board who could write a clean, consistent, unambiguous criteria set. DSM 5 received either no editing at all or amateur editing. Getting the words right is certain not enough- but If you can’t get them right, nothing else can ever be safe.
[2] For DSM 5 to retrieve credibility, it complete the second planned stage of its field testing. If doing the job right must delay publication so be it. Public trust must trump private publishing profits and it is self defeating for APA to publish a book no one can trust.

I have been consistently pessimistic and critical about DSM 5 since my first piece on it 3 years ago. The sad thing is I can still be so surprised. Each step of the way I predict it will fail in one or another way. But then I discover that DSM 5 has managed to fail in ways that go beyond my poor imagination. This assault on reliability was predicted, but its scope exceeds even my jaundiced fears and creates a DSM 5 emergency.