1 Boring Old Man

^{WARNING AGAIN: Sometimes the devil really is in the details. Here too, we need to get pretty far into the details…}

Prior to opening the blind, the sponsor and investigators developed a plan to analyze the efficacy data. The plan described a definition of responders and called for additional measures of effectiveness. These included the depression items from the HAM-D and K-SADS-L instruments, and the plan provided for a status of remission…

Primary Efficacy Variables

The protocol defined the primary efficacy parameters for comparing the efficacy of each active treatment with that of placebo to be:

• The change from baseline in total HAM-D score at endpoint of the acute phase.
• The percentage of responders at the endpoint of the acute phase.

Initially the protocol defined a "responder" as a patient whose HAM-D at endpoint was at least 50% lower than the baseline score. This definition was to be used as an "operational" criteria for entry into the continuation phase.

Prior to opening the blind, the sponsor and the investigators developed an analytical plan. Among other issues, this agreed plan included a definition of a "responder" and a "remission" status. The intent was to provide a robust definition of "response" and to describe a status of "remission" in order to provide a rigorous anchor point in analyzing relapses in the continuation phase.

The agreed analytical plan described a "responder" as a patient whose HAM-D score was 8 or less or was reduced from baseline by at least 50%. The remission status was defined as a HAM-D score of 8 or less. The agreed analytical plan also called for the following measure of effectiveness to be included in the analysis: the 9-item depression subscale of the K-SADS-L, the depression item from both the HAM-D and the K-SADS-L, and two methodologies for analyzing the clinical global improvement score: 1) the mean scores and 2) the proportion of patients with rating of "1" or "2" ("very much" or "much improved" respectively). The initial protocol described the K-SADS-L and CGI instruments as secondary measures.

The protocol defined as secondary measures the behavior and functional instruments. These included the Autonomic Function Checklist (AFC), the Self- Perception Profile (SPP), and the Sickness Impact Scale (SIP). The agreed analytical plan included a time to sustained response and various subsidiary covariate analyses of response as secondary analyses.

If you look back at the last post, the name changing, depression-related variables and declared a priori  make more sense. They simply cover-up the fact that they’ve changed the outcome variables by changing the meaning of a priori [without really letting you in on the change]. The terms depression-related variables and plan of analysis appear to have come from an email exchange among the ghost writer and the two SKB authors during editing. And speaking of changes, remember that this is part of Appendix B of that self-same Protocol:

You might need to take a look at the table again. I’m going to resist the temptation to go on and on about how deceitful that was, since Study 329 will likely become the enduring symbol for why we strictly adhere to the before starting the study meaning of a priori in Clinical Trials going forward. But since we were looking to reanalyze this study, we felt obligated to look for documentation that this plan of analysis actually occurred as described in the CSR. The volumes of archived documents available from the various lawsuits yielded only the Draft Minutes of an April 22, 1997 Teleconference call a few months before the blind was broken. After repeated requests to GSK, we received another – a copy of an email documenting that an SAS programmer was coding for some of the suggested new analysis, again from the period right before the blind was broken. From the perspective of our reanalysis of the data from Study 329, our path was clear. The point was to aim towards adding a correct analysis of the study to the medical literature, so we planned to use the outcome variables outlined by the original a priori protocol [from before starting the study] and to only mention the late addition non-protocol variables in passing without analysis.

Parenthetically, in the three years since I first happened onto the data from Study 329, there has been an intense debate about Data Transparency – the AllTrials Initiative, the EMA, the FDA/NIH, Journals [BMJ], even some PHARMAs, etc. But there’s anything but a consensus about what Data Transparency actually means. The CSR [Clinical Study Report] is often mentioned as the thing to release into the Public Domain for Data Transparency. Every time I read that, I cringe [see post-it notes…]. These CSRs seem to vary a lot from trial to trial, but many, like this one, don’t have the raw data included. In this case, that data was listed as Appendices, but they weren’t released along with the narrative in 2004. And nobody much noticed for 8 years!  The CSR, like the published paper, is only an authored proxy for the data itself. In fact, in the case of Study 329, the CSR is as flawed as the paper itself, if not even more so. Data Opacity reigned.

In this case, we are asked to believe that the authors, uninfluenced by any foreknowledge of the results, decided to completely revamp the outcome analysis of this study in its 11th hour, and just happened to pick new variables that just happened to turn out to be the only ones achieving statistical significance. I don’t personally believe that for a second, but what I believe is hardly the point. Belief has no place in such a high stakes enterprise. There are only two things that guarantee the integrity of a Clinical Trial’s analysis – the a priori protocol and the blinding itself. Since there’s no real way to be sure about the blind, the a priori protocol [from before starting the study] is and should be sacrosanct.