study 329 viii – variable variables decoded…

Posted on Thursday 17 September 2015

WARNING AGAIN: Sometimes the devil really is in the details. Here too, we need to get pretty far into the details…

The 528 page Full study report acute is an exhaustive narrative of the entire study including the results in much more detail than the published article – filled with charts and tables. It was released in part by court order in 2004 and in full in 2012. The sections that relate to our questions about the changing outcome variables in the published paper are scattered around. The first reference is on page 44:
Prior to opening the blind, the sponsor and investigators developed a plan to analyze the efficacy data. The plan described a definition of responders and called for additional measures of effectiveness. These included the depression items from the HAM-D and K-SADS-L instruments, and the plan provided for a status of remission…
The reason for the discrepancy mentioned in the last post is simple. They just changed the outcome variables, redefining the meaning of a priori in the process. It’s intended to mean before starting the study. They’ve changed it to mean before the blind is broken [which comes at the end of the study]. They continue on page 49:
Primary Efficacy Variables
The protocol defined the primary efficacy parameters for comparing the efficacy of each active treatment with that of placebo to be:
  • The change from baseline in total HAM-D score at endpoint of the acute phase.
  • The percentage of responders at the endpoint of the acute phase.
Initially the protocol defined a "responder" as a patient whose HAM-D at endpoint was at least 50% lower than the baseline score. This definition was to be used as an "operational" criteria for entry into the continuation phase.

Prior to opening the blind, the sponsor and the investigators developed an analytical plan. Among other issues, this agreed plan included a definition of a "responder" and a "remission" status. The intent was to provide a robust definition of "response" and to describe a status of "remission" in order to provide a rigorous anchor point in analyzing relapses in the continuation phase.

The agreed analytical plan described a "responder" as a patient whose HAM-D score was 8 or less or was reduced from baseline by at least 50%. The remission status was defined as a HAM-D score of 8 or less. The agreed analytical plan also called for the following measure of effectiveness to be included in the analysis: the 9-item depression subscale of the K-SADS-L, the depression item from both the HAM-D and the K-SADS-L, and two methodologies for analyzing the clinical global improvement score: 1) the mean scores and 2) the proportion of patients with rating of "1" or "2" ("very much" or "much improved" respectively). The initial protocol described the K-SADS-L and CGI instruments as secondary measures.

The protocol defined as secondary measures the behavior and functional instruments. These included the Autonomic Function Checklist (AFC), the Self- Perception Profile (SPP), and the Sickness Impact Scale (SIP). The agreed analytical plan included a time to sustained response and various subsidiary covariate analyses of response as secondary analyses.

If you look back at the last post, the name changing, depression-related variables and declared a priori  make more sense. They simply cover-up the fact that they’ve changed the outcome variables by changing the meaning of a priori [without really letting you in on the change]. The terms depression-related variables and plan of analysis appear to have come from an email exchange among the ghost writer and the two SKB authors during editing. And speaking of changes, remember that this is part of Appendix B of that self-same Protocol:

II.  PROTOCOL AMENDMENTS
  No changes to the study protocol will be allowed unless discussed in detail with the SmithKline Beecham (SB) Medical  Monitor and filed as an amendment/modification to this protocol. Any amendment/modification to the protocol will be adhered to by the participating centre (or all participating centres) and will apply to all subjects following approval as appropriate by the Ethical Review Committee or Institutional Review Board.

You might need to take a look at the table again. I’m going to resist the temptation to go on and on about how deceitful that was, since Study 329 will likely become the enduring symbol for why we strictly adhere to the before starting the study meaning of a priori in Clinical Trials going forward. But since we were looking to reanalyze this study, we felt obligated to look for documentation that this plan of analysis actually occurred as described in the CSR. The volumes of archived documents available from the various lawsuits yielded only the Draft Minutes of an April 22, 1997 Teleconference call a few months before the blind was broken. After repeated requests to GSK, we received another – a copy of an email documenting that an SAS programmer was coding for some of the suggested new analysis, again from the period right before the blind was broken. From the perspective of our reanalysis of the data from Study 329, our path was clear. The point was to aim towards adding a correct analysis of the study to the medical literature, so we planned to use the outcome variables outlined by the original a priori protocol [from before starting the study] and to only mention the late addition non-protocol variables in passing without analysis.

Parenthetically, in the three years since I first happened onto the data from Study 329, there has been an intense debate about Data Transparency – the AllTrials Initiative, the EMA, the FDA/NIH, Journals [BMJ], even some PHARMAs, etc. But there’s anything but a consensus about what Data Transparency actually means. The CSR [Clinical Study Report] is often mentioned as the thing to release into the Public Domain for Data Transparency. Every time I read that, I cringe [see post-it notes…]. These CSRs seem to vary a lot from trial to trial, but many, like this one, don’t have the raw data included. In this case, that data was listed as Appendices, but they weren’t released along with the narrative in 2004. And nobody much noticed for 8 years!  The CSR, like the published paper, is only an authored proxy for the data itself. In fact, in the case of Study 329, the CSR is as flawed as the paper itself, if not even more so. Data Opacity reigned.

In this case, we are asked to believe that the authors, uninfluenced by any foreknowledge of the results, decided to completely revamp the outcome analysis of this study in its 11th hour, and just happened to pick new variables that just happened to turn out to be the only ones achieving statistical significance. I don’t personally believe that for a second, but what I believe is hardly the point. Belief has no place in such a high stakes enterprise. There are only two things that guarantee the integrity of a Clinical Trial’s analysis – the a priori protocol and the blinding itself. Since there’s no real way to be sure about the blind, the a priori protocol [from before starting the study] is and should be sacrosanct.

We don’t even have to ask why one might want to change variables along the way – the new ones were significant and the old ones weren’t. But as we’ll soon see, that wasn’t the only change that occurred along the way
  1.  
    Bernard Carroll
    September 17, 2015 | 12:58 PM
     

    In the New York Times yesterday, Study 329 was described as an experiment. Actually, the study has all the hallmarks of an experimercial, a cost-is-no-object exercise driven by a corporate sponsor to create positive publicity for its product in a market niche. This is not clinical science… this is product placement. And if they had to put lipstick on the pig to get what they wanted, well… Meanwhile, the continued stonewalling by AACAP is misguided.

  2.  
    Katie Tierney Higgins RN
    September 17, 2015 | 2:31 PM
     

    Oh dear, could leading academic psychiatrists forget that they are supposed to be administering their medical expertise to assess, and treat live human children and adolescents ?

    I call these responses to Study 329, rhetoric from the Ivory Tower — the translation to everyday language is forthcoming and long overdue– the reality of doctors mistaking patients for *a market* —??

    Where’s the social psychiatrist who has been diagnosing society – on- line–with character/personality [disorders]?? pathology?

Sorry, the comment form is closed at this time.