{"id":59381,"date":"2015-09-17T16:00:19","date_gmt":"2015-09-17T20:00:19","guid":{"rendered":"http:\/\/1boringoldman.com\/?p=59381"},"modified":"2015-09-17T08:41:24","modified_gmt":"2015-09-17T12:41:24","slug":"study-329-ix-mystic-statistics","status":"publish","type":"post","link":"https:\/\/1boringoldman.com\/index.php\/2015\/09\/17\/study-329-ix-mystic-statistics\/","title":{"rendered":"study 329 ix – mystic statistics…"},"content":{"rendered":"\n

Most of us have an incomplete knowledge of Statistical Analysis unless we’ve had formal training and hands-on experience, yet we tend to accept the output from the computer’s statistical packages as if it’s dogma. In academic and commercial laboratories, we count on Statisticians [or trained SAS Programmers] to generate those abstract lettered indices that we discuss as if they’re absolutes – p<\/em>, d<\/em>, k<\/em>, SEM, SD, NNT, OR, etc. And even the experts can’t check things with a pad and pencil. So we’re vulnerable to subtle [and even not so subtle] deceptions. In our RIAT Team’s reanalysis of Study 329, we had decided to follow the a priori protocol, which meant sticking to the protocol defined outcome variables and ignoring those later exploratory variables [in blue in Keller et al<\/em>‘s<\/font> Table 2<\/a>] as discussed earlier.<\/p>\n

The Study 329 protocol is clear and precise about statistical testing: parametric Analysis of Variance for the continuous variables and Logistical Regression for the categorical [yes\/no] variables. They specified a model containing treatment and investigator with contingencies for interactions between them [since I’ve already put the non-stat-savvy set to sleep, I’m going to dumb this down a bit going forward<\/font><\/strong>]. We noticed that our p<\/em> values differed from those in both the Keller et al<\/a> paper and the CSR [Full study report acute<\/a>], even though our open source statistical package [R] is equivalent to their commercial package [SAS] – both available in the Secure Data Portal provided by GSK. While the results for the protocol defined variables were not significant, the numbers still should’ve been close to the same. And there was something else. They were reporting statistics for Paroxetine vs Placebo<\/font><\/em>, Imipramine vs Placebo<\/font><\/em>, and saying that the study was not powered<\/font><\/em> to test Paroxetine vs Imipramine <\/font><\/em>– all pairwise comparisons<\/font>.<\/font><\/em> Why this was important takes a little explaining.<\/p>\n

When a dataset has only two groups [as a study of Paroxetine vs Placebo<\/font><\/em>], pairwise statistical comparisons with something like the familiar t-test are perfectly appropriate. But when you run statistical comparisons on datasets with more than two groups, there’s a two step process. First you test the whole dataset using an OMNIBUS statistical test like Analysis of Variance<\/font> [ANOVA<\/font>]. If the whole dataset is significant, then<\/u> you can run pairwise tests between the various groups to find where the significance lies. But if the OMNIBUS test is not  significant, it means that there are no differences among the groups – and that’s the end of that. The pairwise tests are immaterial no matter how they come out. Keller et al had skipped the OMNIBUS tests altogether<\/font><\/strong> [never mentioned in the protocol, the paper, or the CSR]. Our results were the OMNIBUS statistics and that’s why they were different. With the protocol-defined variables under consideration, it didn’t matter since nothing was significant no matter what your method. So the question became, "Why bother to skip the OMNIBUS statistical tests?"<\/font><\/em><\/p>\n

Since we decided to drop those non-protocol-variables because they were declared post hoc [see the last two posts<\/a>], we had never run the full statistical model analysis on them. But I remembered a spreadsheet we did on a rough pass through this data when we were first getting started. The results are shown here [the OMNIBUS tests are in the far right column and all significant values are shown in red]: <\/div>\n

<\/p>\n

The protocol-specified-variables [white background] are not significant as reported by Keller et al<\/font>. But look at the non-protocol variables [gray background]. Only two were OMNIBUS-significant. And look at the columns measuring strength of effect [EFFECT SIZE, NNT, ODDS RATIO]. Except for the HAM-D DEPRESSED ITEM, those exploratory variables are pretty lame [weak]. While this was a crude first-take without considering the investigator covariate, it suggests that the OMNIBUS statistics didn’t help their cause so they were conveniently ignored. That could offer a plausible explanation for why they skipped the OMNIBUS statistical test altogether [in fact, it’s the only explanation I can think of]. Recalling that spreadsheet, I went back and ran the "full monty" model on these variables and three of them did<\/u> make it under the p<\/em><<\/u>0.05<\/font> wire after all: as expected, the HAM-D DEPRESSED ITEM yielded p<\/em>=0.0032;<\/font> but the others only barely made the cut, HAM-D REMISSION was p<\/em>=0.0504<\/font>, and CGI IMPROVEMENT came in at p<\/em>=0.0493<\/font>. Those last two were barely statistically significant, hardly seeming clinically relevant. And there was something else [see below]. The LOCF dataset for K-SADS-L was very difficult to judge since it was an every other week metric and a number of subjects got off schedule, but for what it’s worth, I could never find the reported significance with various shots at defining the LOCF dataset. Running the full model, I got p<\/em>=0.0833<\/font> OMNIBUS and p<\/em>=0.0662<\/font> for Paroxetine vs Placebo. <\/div>\n

Just one more piece of techno-babble. There’s something more to say about those two minimally significant exploratory variables:<\/p>\n

<\/p>\n

Both of the non-protocol categorical variables were only significant in week 8<\/font><\/em>, suggesting to me that they were probably outliers<\/font><\/em> [flukes]. And, as mentioned earlier, even if you include the rogue non-protocol exploratory variables, applying any correction for multiple variables would wipe out statistical significance for three of the four. That leaves the HAM-D DEPRESSED ITEM as the only statistically significant finding in this entire study – one question on a multi-item rating scale! So in order for Keller et al<\/font><\/em> to reach the conclusion "Paroxetine is generally well tolerated and effective for major depression in adolescents<\/font><\/em>," all three things had a part to play: no correction for multiple variables; redefining a priori<\/em> to mean before the blind is broken<\/em><\/strong> rather than before the study begins<\/em><\/strong>; and ignoring the OMNIBUS statistical test.<\/p>\n

I know these posts are TMI [too much information], so this is the end of all my number chatter. To my way of thinking, Study 329 has become a paradigm, emblematic of the widespread subtle distortion of the tools of scientific analysis in the service of commercial gains in the analysis of Clinical Trials. We wrote this RIAT paper to correct the existing scientific literature, but also to give the clear message that if you publish Clinical Trials that disseminate misinformation to physicians and patients, they might just be coming right back at you. And, in the future, with greater Data Transparency and awareness, it won’t take any fourteen years to make the circuit… <\/div>\n","protected":false},"excerpt":{"rendered":"

Most of us have an incomplete knowledge of Statistical Analysis unless we’ve had formal training and hands-on experience, yet we tend to accept the output from the computer’s statistical packages as if it’s dogma. In academic and commercial laboratories, we count on Statisticians [or trained SAS Programmers] to generate those abstract lettered indices that we […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[5],"tags":[],"class_list":["post-59381","post","type-post","status-publish","format-standard","hentry","category-opinion"],"_links":{"self":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/59381","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/comments?post=59381"}],"version-history":[{"count":38,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/59381\/revisions"}],"predecessor-version":[{"id":60015,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/59381\/revisions\/60015"}],"wp:attachment":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/media?parent=59381"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/categories?post=59381"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/tags?post=59381"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}