1 Boring Old Man

Generally, folk heros are from the past like John Henry and represent some laudable principle – like man versus machine. Back in the upside-down years of the 60s, Arlo Guthrie had a shot with his endless comical antiwar song, Alice’s Restaurant. The premise, however, was dead serious. When enough people get behind a needed change, that’s when things finally happen. We’ve seen some of that with Data Transparency and distorted Clinical Trial reports alread [though not near enough].

I happen to agree with Arlo that the hard work of righting a wrong is in keeping it on the front burner [as in the Civil Rights Movement, Women’s Rights, Gay Rights], and the distortion of the scientific record by corruptly reported Clinical Trials is a wrong of sufficient magnitude to dog until it’s past history. While Psychiatry lead the pack there for a while, it’s a problem throughout medicine that needs our full attention – and I personally have come to believe that without Data Transparency, it will never happen. One of the major frustrations with these journal articles is that you can’t get at the data itself to evaluate things yourself, at least that’s been my major frustration. Even if you have an academic connection that allows you access to the articles themselves rather than just the abstracts, they often omit key parameters that would allow you to get under the carefully crafted rhetoric even with the crude tools described in the first two john henry’s hammer posts to see what’s going on.

from Adjunctive brexpiprazole 1 and 3 mg for patients with major depressive disorder following inadequate response to antidepressants: a phase 3, randomized, double-blind study.

"Mean change in MADRS total score for the efficacy population also showed improvement for brexpiprazole 3 mg versus placebo (−1.52; 95% CI, −2.92 to −0.13; P = .0327) but did not reach the level of statistical significance required for multiple comparisons according to the prespecified statistical analysis. The mean improvement for brexpiprazole 1 mg versus placebo was less than that for 3 mg (−1.19; 95% CI, −2.58 to 0.20; P = .0925)."

from Efficacy and safety of adjunctive brexpiprazole 2 mg in major depressive disorder: a phase 3, randomized, placebo-controlled study in patients with inadequate response to antidepressants.

"Similar results were seen for brexpiprazole versus placebo in the efficacy population (LS mean = −8.27 vs −5.15; LS mean difference =−3.12 [95% CI, −4.70 to −1.54], P = .0001)."

They give us all the summary data we could ever want on all of their secondary outcome variables, but not the primary outcome variable in either article [a μ or so with the "good article," but no SEM or σ]. So I went to the FDA Medical Report, hoping that since they approved the drug for this indication, they would give us more. No such luck. Maybe I missed something [and I’d appreciate someone else taking a look], but I ended up with a table that was barely populated and a mighty lonely spreadsheet:

Instead, they give us the difference between the means…

(μ₁–μ₂)

and the 95% confidence limits – the formula for that is:

CI_[95%]=(μ₁–μ₂)±1.96×√((σ₁²÷n₁)+(σ₂²÷n₂))

There’s just no way I can figure to deconstruct those values, work it backwards, and end up with the summary data [even though I know they had it and used it to generate something I didn’t particularly want]. In the two articles, they give us four tables of summary data for secondary outcome variables – two with their jury-rigged numbers in the articles and two with the real numbers in the supplements, but for their declared primary outcome variables, slim pickings.

Admittedly, some claim that reporting the difference in the MEANs is an Effect Size and don’t convert it to a Standardized MEAN Difference [Cohen’s d], but I’m not in that particular some [and they aren’t either, since the same people didn’t do that in their Schizophrenia articles from the same company with the same ghost-writing firm – even reporting the Cohen’s d in their best showing article]. So what’s a fellow who’s trying to write a guide for other elm-tree vetters going to do in this situation? Is John Henry’s Hammer dead in the water? I say where there’s a will, there’s a way. Stay tuned.

When I read Spielmans et al’s meta-analysis of the augmentation with Atypical Antipsychotics in treatment resistant depression [see the extra mile…], I ran across these lines:

"The quality of data reporting varied across studies. For continuous outcomes, effect sizes were computed from means and standard deviations when possible. When these were not provided, effect sizes were computed based on means and p-values, or p-values only."

7.7.7.2 Obtaining standard errors from confidence intervals and P values: absolute (difference) measures

… The first step is to obtain the Z value corresponding to the reported P value from a table of the standard normal distribution. A standard error may then be calculated as

SE = intervention effect estimate/Z.

As an example, suppose a conference abstract presents an estimate of a risk difference of 0.03 (P = 0.008). The Z value that corresponds to a P value of 0.008 is Z = 2.652. This can be obtained from a table of the standard normal distribution or a computer (for example, by entering =abs(normsinv(0.008/2) into any cell in a Microsoft Excel spreadsheet). The standard error of the risk difference is obtained by dividing the risk difference (0.03) by the Z value (2.652), which gives 0.011.

Still a little heady for John Henry’s hammer, but promising. The Z value is the area under the normal distribution curve used to compute p-values. But while looking around about the Z value, I ran across the Mother Lode – Psychometrica, a German web site tailor made for my purposes. It has an Internet Calculator that takes the Z value and the sample size [n₁+n₂] and out comes Cohen’s d. And on top of that, I found another Internet Calculator that produces a Z Score from a p-value [remember, you have to divide the p-value by 2 first!] requiring neither tables nor spreadsheet visits. So, armed with my two new calculators, I decided to try it out first on something where I had arrived at Cohen’s d and the CI_[95%] by other means. So, back to the Brexpiprazole in Schizophrenia Trials [truncated with the new stuff in green]. You’ll note that I’m using my p-values rather than the ones in the articles because one of theirs appears to be in error [Kane 1mg], and that I changed the  sign on Cohen’s d and the CI_[95%] :

I agree with John Henry and Arlo! Those numbers are close enough for elm tree work in my book! I had no clue if this would pan out, so my hat’s off to the Calculator makers. Why do you divide by two? For that matter, how does it work? I don’t know those answers yet. Magic?

So moving right along – What about that other dataset where I didn’t know the the Effect Sizes? [The CI_[95%] are from the formula in john henry’s hammer: continuous variables I…. This one’s also truncated with the new stuff in green]:

I doubt if anyone is much surprised that the 3mg dose in that first study had a trivial Cohen’s d – right near inert. Why else would they make it so hard to generate it? But when you take away the Protocol sleight of hand and look at the Effect Size of 0.204, that p-value of 0.0327 doesn’t look very impressive at all. And that’s the point. Of course p and d are still just numbers. But adding the Effect Size [d] to the mix definitely deepens our understanding of the reported values well beyond the p-value. If the authors don’t give an Effect Size, and the editors don’t insist on an Effect Size, we need to be able to easily generate it ourselves.