1 Boring Old Man

Well, the APA has come and gone with its DSM-5 Track. Frankly, after the report of the results of the Field Trials, I didn’t pay very much attention to their report on the various changes in the Disorders. The Field Trials themselves were so devastating that everything else seemed immaterial. In the absence of the bedrock of Medical Diagnosis based either on known etiology or a pathognomonic biomarker [some laboratory test that defines the disorder], psychiatry has traditionally followed other paths – settling on Descriptive Criteria in 1980. The two parameters that are most in question in such a system are Validity and Reliability. We have to struggle with both parameters with every diagnostic entity because we don’t have the lynchpins available in physical medicine. There’s no shame in that, but the onus is on our specialty to prove its diagnoses nonetheless. And when push comes to shove, our lynchpin has been Reliability for the last thirty years. Can two psychiatrists evaluate a given patient and come up with the same diagnosis? Without Reliability, our diagnostic system is faulty by definition. It’s all we’ve really got. And we measure it with Kappa – zero is what chance would bring and one is perfect agreement.

Looking at the DSM-5 Web Site, I can’t find that the Field Trial results have been posted, so in the absence of their posting, here’s my collected version for your after-the-dust-has-settled review:

And this is a simple set of distribution graphs for this and previous Field Trials:

On the right is a distribution of the Kappa values from our old system [DSM-II] from Dr. Spitzer’s 1974 meta-analysis of five reliability studies – values considered at the time unacceptable and a reason to radically revise the whole diagnostic system. We set our own standard back in those days, our own benchmark. Dr. Spitzer and his statistician colleagues essentially created and tested Kappa to be the Reliability coefficient for psychiatric diagnosis. If we are going to have any integrity, we can’t change the standards after the results are in.

The protest lead by Dr. Spitzer and Dr. Frances has had two prongs. The one we’ve all followed has been changes like the Attenuated Psychosis Syndrome, dropping the Bereavement Exclusion, or changing the bar in diagnosing Autism – content issues. But both former revision czars have also worried about the process – secrecy, schedules, Field Trials, etc. These Field Trial results focus us on the second set of problems. The Field Trials are not only disappointingly weak, unacceptably low, they are uninterpretable. Mainstream diagnoses like Schizophrenia, Major Depressive Disorder in kids and adults, Generalized Anxiety Disorder, Obsessive Compulsive Disorder – they have Kappas dragging the bottom. Those criteria haven’t changed since the DSM-IV. That can only be an indictment of either the whole DSM system or the DSM-5 Field Trials themselves. There’s no other interpretation, and certainly no interpretation that remotely says that everything is fine, right on schedule, business as usual.

And to make matters worse, there were supposed to be two sets of Field Trials. One to test their new criteria and a second to test the tweaking after the first one. Sounds like science to me. But they cancelled the second trial because of scheduling problems and chose to put their eggs all in one basket – the Field Trials reported in the Philadelphia APA meeting recently. Those results are both weak and uninterpretable.

Psychiatry has been plagued with scientific distortion for a long time. It’s a matter of public record that any number of studies and articles have overstepped the rational bounds of scientific enterprise – overblowing the results of clinical trials, withholding information about adverse events, renaming things like suicidality – spinning the data rather than just reporting it. It’s time for that to stop. We know it, but more importantly, the rest of the medical community and our patients know it.