another possibility…

Posted on Sunday 13 November 2016

We  live in an age of Meta-Analyses and Systematic Reviews [studies of studies]. Some, like watchdog John  Ioannidis, lament their frequency as redundant and/or unnecessary [The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses]. But at least in the domain of neuroscience, I see these meta-analyses as an attempt at oversight in an area where advanced technologies have been applied to clinical problems without yielding much in the way of clarity. An example:
Meta-analyses of Neuroimaging Studies
by Veronika I. Muller. PhD; Edna C. Cieslik. PhD; llinca Serbanescu. MSc.; Angela R. Laird. PhD; Peter T. Fox. MD; and Simon B. Eickhoff. MD
JAMA Psychiatry. Published online November 9, 2016.

Importance During the past 20 years, numerous neuroimaging experiments have investigated aberrant brain activation during cognitive and emotional processing in patients with unipolar depression [UD]. The results of those investigations, however, vary considerably; moreover, previous meta-analyses also yielded inconsistent findings.
Objective To readdress aberrant brain activation in UD as evidenced by neuroimaging experiments on cognitive and/or emotional processing.
Data Sources Neuroimaging experiments published from January 1, 1997, to October 1, 2015, were identified by a literature search of PubMed, Web of Science, and Google Scholar using different combinations of the terms fMRI [functional magnetic resonance imaging], PET [positron emission tomography], neural, major depression, depression, major depressive disorder, unipolar depression, dysthymia, emotion, emotional, affective, cognitive, task, memory, working memory, inhibition, control, n-back, and Stroop.
Study Selection Neuroimaging experiments [using fMRI or PET] reporting whole-brain results of group comparisons between adults with UD and healthy control individuals as coordinates in a standard anatomic reference space and using an emotional or/and cognitive challenging task were selected.
Data Extraction and Synthesis Coordinates reported to show significant activation differences between UD and healthy controls during emotional or cognitive processing were extracted. By using the revised activation likelihood estimation algorithm, different meta-analyses were calculated.
Main Outcomes and Measures Meta-analyses tested for brain regions consistently found to show aberrant brain activation in UD compared with controls. Analyses were calculated across all emotional processing experiments, all cognitive processing experiments, positive emotion processing, negative emotion processing, experiments using emotional face stimuli, experiments with a sex discrimination task, and memory processing. All meta-analyses were calculated across experiments independent of reporting an increase or decrease of activity in major depressive disorder. For meta-analyses with a minimum of 17 experiments available, separate analyses were performed for increases and decreases.
Results In total, 57 studies with 99 individual neuroimaging experiments comprising in total 1058 patients were included; 34 of them tested cognitive and 65 emotional processing. Overall analyses across cognitive processing experiments [P > .29] and across emotional processing experiments [P > .47] revealed no significant results. Similarly, no convergence was found in analyses investigating positive [all P > .15], negative [all P > .76], or memory [all P > .48] processes. Analyses that restricted inclusion of confounds [eg, medication, comorbidity, age] did not change the results.
Conclusions and Relevance Inconsistencies exist across individual experiments investigating aberrant brain activity in UD and replication problems across previous neuroimaging meta-analyses. For individual experiments, these inconsistencies may relate to use of uncorrected inference procedures, differences in experimental design and contrasts, or heterogeneous clinical populations; meta-analytically, differences may be attributable to varying inclusion and exclusion criteria or rather liberal statistical inference approaches.
This meta-analysis focuses on neuroimaging studies of patients where DSM/ICD criteria were used to define the target cohort of depressed subjects. While this classification has survived for now 36 years, I know of no evidence that these criteria define anything like a discrete clinical entity.
Criteria Related to the Investigation Participants
Included experiments statistically contrasted neural activation between an adult [>18 years] UD group [based on DSM-IV-TR and DSM-5 or International Statistical Classification of Diseases and Related Health Problems, Tenth Revision] and a group of healthy controls…
The studies examined were ones that compared brain activity measured by PET scans or fMRI studies between their depressed cohort and normals. That presumes a hypothesis that the experimental subjects would have something detectable in their brains either causing or as a result of their depression. Thus there were two bedrock assumptions: [1] a unitary clinical entity [2] that has something to do with the brain.
A total of 57 studies with 99 individual neuroimaging experiments, comprising 1058 patients were included in this analysis. There were 34 cognitive processing experiments and 75 emotional processing experiments; 50 experiments reported increased brain activity in UD, and 49 experiments reported decreased brain activity in UD.

Meta-analyses Across Emotional Experiments
None of the 9 emotional meta-analyses revealed any significant results {all emotional: 65 experiments [P > .69]; in- creases: 33 experiments [P > .47]; decreases: 32 experiments [P > .581; negative valence: 33 experiments [P > .76]; negative valence increases: 19 experiments [P > .12]; positive valence: 19 experiments [P > .15]; emotional faces: 32 experiments [P > .80]; negative emotional faces: 18 experiments [P > .75]; sex discrimination: 17 experiments [P > .41]}. Figure 4A displays the distribution of foci of the emotional analyses.

Meta-analyses Across Cognitive Experiments
None of the 4 cognitive meta-analyses revealed any significant results {all cognitive: 34 experiments [P > .63]; increases: 17 experiments [P > .29]; decreases: 17 experiments [P > .97]; memory: 19 experiments [P > .48]}. Figure 4B displays the distribution of foci of the cognitive analyses.

Meta-analyses Controlling for Confounds
Analyses restricted to [1] patients not receiving medication, [2] patients without comorbidity, and [3] patients without late-life or geriatric depression revealed similar results … When restricting the analyses to experiments using corrected statistics [COR], the analyses across experiments of negative emotional processing revealed significant convergence in the left thalamus extending into hippocampus [x = -18, y = -36, z = -4; 5 experiments contributing]. All other analyses did not reveal significant convergence {COR all emotional: 38 experiments [P > .82]; COR increases emotional: 20 experiments [P > .27]; COR decreases emotional: 18 experiments [P > .23]; COR all cognitive: 23 experiments [P > .61]}…
Said the accompanying editorial:
Müller et al provide a technically sophisticated and informative set of meta-analyses examining altered brain activity in adults with symptomatic unipolar depression. The striking overall finding of their analyses is the lack of consistent group differences across studies. The absence of replicable effects across studies remained even when they addressed a number of potentially key confounds, such as examining only patients not receiving medication, patients without comorbidities, and patients without late-life or geriatric depression…
Don’t think that reading this blog post is the definitive take on this comprehensive and complex meta-analysis. Read  the article and its supplements. What you’ll find is a window into the heavy duty math and numerous assumptions that go into these neuroimaging studies. What the article says is that we don’t know anything about the altered brain activity in unipolar depression. Reading this, I’m not even sure that we know if there is any altered brain activity in unipolar depression. The random scattering of hot and cold spots from the studies along with the dramatically insignificant p-values isn’t much of a confidence builder. The authors give a laundry list of possible reasons for the non-replicability and non-convergence, suspecting that methodological problems top the list. I would start my own list at the input end of things and quote historian Edward Shorter from Before Prozac:
"Bottom Line: Major Depression doesn’t exist in Nature. A political process in psychiatry created it…"
Major Depressive Disorder, Unipolar Depression, hasn’t played out as an entity, it’s a symptom complex [see which nail?…]. Is it a brain disease? So far, there’s some indirect evidence that there are perhaps several biological entities in the mix in small doses [Melancholia, Depressive Episodes in Manic Depressive Illness], and who knows what else? The point being that there may be methodological problems with these neuroimaging studies, sure enough. But that aside, the questionable diagnostic and etiological speculations before they even turn on the MRI machine are probably more to the point. I would venture that a strong possibility to be considered is that they’re chasing a biomedical etiology in a heterogeneous population of people with a symptom complex primarily generated in the biographical and psychosocial domain of living…
    Bernard Carroll
    November 13, 2016 | 12:10 PM

    I think you nailed it, Dr. Mickey. The heterogeneity of what DSM called major depression created noise sufficient to mask any signal from the really good studies, such as the one by Wayne Drevets and colleagues in 1992. Back in the day, we talked about selecting patients for such studies very carefully. We thought it best to enroll only clear-cut cases for a first look with new technologies like PET or fMRI scans. We also did our best to apply plausible clinical distinctions such as melancholic versus nonmelancholic versus atypical cases. We did that even while recognizing that the reliability of such distinctions was not perfect for all cases. But by contrasting patients at the ends of the spectrum we might learn something, while leaving it for later to resolve issues in the less-clear middle group.

    The story told in this review article underlines the futility of studies that ignore clinical heterogeneity among depressed patients. You could add waste to futility, too – waste of time, waste of funding, waste of clinical research resources and infrastructure. Robert Spitzer and the DSM-III crowd have a lot to answer for.

    November 13, 2016 | 4:42 PM

    It is a waste. Reading through this article and its supplements, one has the sense of some solid scientists who put some real sweat equity into doing it right. While some of their statistical comparisons are beyond any place I’ve been or am likely to go, they did a good job of making it apparent how much this kind of research needs standardization – and the thrust of their recommendations had to do with improving this methodology. But as to their topic of the brain in Unipolar Depression – how many different ways do we have to be shown that the category is not an entity, not a disease, not really even a disorder? It’s a large group of people with similar symptoms who have been inappropriately “lumped” together.

    November 13, 2016 | 5:30 PM

    “By using the revised activation likelihood estimation algorithm, different meta-analyses were calculated.”

    I would not put any more faith in meta-analyses that the spreadsheet analyses and selection of variables that preceded them.

    Does any clinical psychiatrist need a meta-analysis of research imaging results to tell them that depression is a heterogenous group of disorders? The only factor that I have ever seen produce better results was depression severity (HAM-D scores >30). The trade-off for most researchers is that it would take years to accumulate a large enough group of those folks who were also not severely personality disordered.

    I would also not agree that major depression has not played out as an entity. It plays out every time I see somebody who is severely depressed, depressed and psychotic, or depressed and suicidal. I am counting on that literature and inductive reasoning to come up with a strategy to treat that person. Much more often than not I am correct. The severely depressed are not great placebo responders and they generally have failed multiple treatments and providers before I see them. The only logical conclusion is that there is something there in that literature despite the noise of polygenes and epigenetic effects.

    Bernard Carroll
    November 13, 2016 | 7:02 PM

    George, I recognize what you say but I would caution that severity and typology (subgroup) are not the same constructs. But you are right to say that when you encounter a patient with severe symptoms that include mood-congruent psychotic features and who is incapacitated then you pretty much know what ballpark you are in.

    November 15, 2016 | 1:12 AM

    Barney, I recognize the focus on constructs that are used to differentiate the idea of depression. I don’t know if you read Kendler’s paper earlier this year in the Am J Psychiatry on diagnoses as an indexing effort rather than something more specific. Being interested in human consciousness and the computational ability of the human brain – is there a factor in the conscious state of the psychiatrist that is not explicit in making these diagnoses and treating the problem? We do have the qualifier that reading the criteria are not enough. But it is clear that psychiatrists vary in their diagnoses and treatment practices.

    I think that Kendler is correct. Criteria in a manual get us into the ballpark. After that – experience treating thousands of people in that ballpark lead to refined diagnostic and treatment procedures. Severity just makes things clearer.

    Nobody really adequately discusses that process. I posed the following question to an expert in epidemiology and statistics: “In your review critical of antidepressants I have treated many times the number of patients in these studies with much better results. At what point can we say that meta-analyses do not really reflect the treatment condition or psychiatric expertise?”

    He never responded to me. As far as I know – nobody ever addresses this obvious point. Psychiatrists and Big Pharma are lumped together as ineffective based on highly biased studies and meta-analyses that have nothing to do with clinical practice

    A big part of that variance is that a diagnosis by a real psychiatrist in not an indexing procedure and treatment in not a clinical trial. And more importantly – the difference between my much larger and better results and some clinical trial or meta-analysis is more than the placebo effect.

    Bernard Carroll
    November 15, 2016 | 11:13 AM

    George, I agree with your emphasis on idiographic versus nomothetic approaches to diagnosis. What you say about skilled clinicians reflects their capacity for pattern recognition, which is essential if we are to avoid diagnosis by checklist. And, of course, clinicians vary greatly in their ability to discern clinical patterns. I have long taught that the proper use of formal diagnostic criteria is to verify that the idiographic diagnoses made by clinicians conform to a minimum nomothetic data set. But the clinical diagnosis comes first, and it can be influenced by factors that are outside the diagnostic criteria – like family psychiatric history or the course of past episodes. Such “external” factors can change the Bayesian prior probabilities of competing diagnoses. Good clinicians are attuned to such “external” factors.

    It’s important as well to keep clearly in mind the distinction between defining a disorder and diagnosing a case. Those are not coterminous operations, and that’s what Kendler was getting at with his comments about indexing. When there is disagreement between the clinician’s idiographic diagnosis and the checklist criteria then a clinical override may be contemplated. That should not happen very often, and it always raises the specter of a goofy clinician (yes, they do live among us). In such cases, deferral of the diagnosis may be preferable or a second opinion could be requested. Needless to say, none of these nuances are captured by the brute force of EHRs.

Sorry, the comment form is closed at this time.