{"id":26421,"date":"2012-08-26T16:25:40","date_gmt":"2012-08-26T20:25:40","guid":{"rendered":"http:\/\/1boringoldman.com\/?p=26421"},"modified":"2012-08-26T21:41:34","modified_gmt":"2012-08-27T01:41:34","slug":"the-lesson-of-study-329-conventions-and-protocols","status":"publish","type":"post","link":"https:\/\/1boringoldman.com\/index.php\/2012\/08\/26\/the-lesson-of-study-329-conventions-and-protocols\/","title":{"rendered":"the lesson of Study 329: conventions and protocols\u2026"},"content":{"rendered":"\n

Once the data’s in, one is invariably tempted to play with it – find new things, maybe make it look better. There are some conventions to keep data play in bounds. In a hypothesis driven study like Study 329, one is asked to state the hypothesis and to declare the outcome variables that will prove or disprove the it a priori<\/em> [up-front]. Another temptation is to measure everything under the sun looking for confirmation and report what suits you. As much as we’d like to see the scientist as a detached braniac driven only for the truth, human-ness is just as common in scientists as anyone else. So, in science, there are protocols and conventions in place to hold the science in the road.<\/p>\n

The Study 329 authors knew the rules. The a priori<\/em> protocol was in place, and accurately stated in the text of the article [the red subscripts are mine, for later]:<\/p>\n

 <\/p>\n

There were two Primary Outcome Parameters.  I covered one of them (2) in the first post [the lesson of Study 329: the basics…<\/font><\/a><\/strong><\/u>]. Now, here’s Table 2 from the paper again [the red numbers are mine]: <\/p>\n

<\/p>\n

There are several things of note. First, the Primary Outcome variables are not identified in the Table, instead they are shuffled together with the Secondary Outcomes. In fact, the major one is listed at the end, separation of Paroxetine from Placebo [8<\/font><\/strong>]. Note that both of the Primary Outcomes were not significant<\/font><\/strong> [2<\/font><\/strong> and 8<\/font><\/strong>]. The next thing of note is that there were "five other depression related variables<\/em>" mentioned as secondary outcome measures in the protocol, but there are six<\/em> others in the table. The extra is number 1<\/font><\/strong>. And, by the way, number 1<\/font><\/strong> [HAM-D <<\/u> 8] is just a partial number 2<\/font><\/strong> [HAM-D <<\/u> 8 or 50% reduction in baseline HAM-D]! So they’ve essentially ablated the protocol’s distinctions [and thrown in an extra jury-rigged significant<\/em> parameter]. These are terminal breaks with convention – protocol violations.<\/p>\n

It’s not that these things weren’t pointed out along the way as they created this article. This is from Rosemary Oakes, a SKB statistician [03\/05\/1999] echoed by SKB Exec Jim McCafferty [to Sally Laden]:<\/p>\n

<\/a>
<\/a><\/div>\n

and by the JAACAP reviewers:<\/p>\n

<\/a>
"As this was a pharmaceutical-industry sponsered study, it is likely that there was a primary
outcome measure that was identified a priori<\/em>. If this is the case, the authors should clearly state
what this primary outcome measure was in the METHODS<\/strong>, RESULTS<\/strong>, and COMMENT<\/strong> sections."<\/sup><\/p>\n

And then there was another striking omission, failure to correct for measuring so many parameters [the Bonferroni Correction<\/font><\/strong><\/u><\/a>]. In spite of its funny name, it’s intuitively sensible. If you’re looking at a bunch of parameters the likelihood of finding something by chance increases. So the correction is to up the p value requirement, multiplying by the number of parameters. Here it is mentioned in the JAACAP reviews:<\/p>\n

<\/a>
"Since there was a large number of depression outcome measures used and because a
Bonferroni<\/strong> correction was not employed, this is a particularly important consideration."<\/sup><\/p>\n

And from another feedback letter:<\/p>\n

<\/a><\/p>\n

<\/a>The response to this last comment [right] was written in as marginalia. It says "<?> suggestion dismissed by Keller<\/strike> Ryan<\/strike> <?><\/font><\/strong>" and "since comparisons were a priori<\/em> – I do not think correction is necessary<\/font><\/strong>." The email was sent to Dr. Neal Ryan and it was copied to Dr. Keller so I assume these are their comments At a later time, in a formal response to the JAACAP reviewers addressing this same point, they said:<\/p>\n

<\/a><\/p>\n

Both of these responses are laughable – too far off the point to even matter. The much more likely explanation for their reticence is that the Bonferroni Correction would make their table look very differently – one out of eight staying significant<\/em>:<\/p>\n


[adapted and modified from the original]<\/sup><\/p>\n

So they embedded the crucial Primary Outcomes in a long list to hide their importance [and non-significance], and then ignored the correction that would be expected for a list this long, again Medical Statistics 101. That’s neither oversight nor sloppiness. That’s deliberate deceit. The only<\/u> reason for presenting the data like this is to obscure the failure of the study to show a clinically meaningful response to Paroxetine. The reviewers saw it. Even the people at SKB\/GSK seemed to see the problems. It was the principal authors that appeared to be fighting to to hold on to the obvious sleight of hand. I was surprised at that. This from SKB\/GSK as the paper went to press:<\/p>\n

<\/a><\/p>\n

The available documents are mainly from late in the process, reactions to the drafts. We don’t know much about the communications during the early drafting – between the ghost-writer [Sally Laden], the principals [Keller and Ryan] and SKB\/GSK [McCafferty and Oakes]. But from what we do have, the information flows among them freely. Even being as used to the sheenanigans of the era as I have become, it remains jarring how blurred the boundaries were then…<\/div>\n","protected":false},"excerpt":{"rendered":"

Once the data’s in, one is invariably tempted to play with it – find new things, maybe make it look better. There are some conventions to keep data play in bounds. In a hypothesis driven study like Study 329, one is asked to state the hypothesis and to declare the outcome variables that will prove […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[2],"tags":[],"class_list":["post-26421","post","type-post","status-publish","format-standard","hentry","category-politics"],"_links":{"self":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/26421","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/comments?post=26421"}],"version-history":[{"count":42,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/26421\/revisions"}],"predecessor-version":[{"id":26463,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/26421\/revisions\/26463"}],"wp:attachment":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/media?parent=26421"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/categories?post=26421"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/tags?post=26421"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}