{"id":39878,"date":"2013-09-10T20:27:18","date_gmt":"2013-09-11T00:27:18","guid":{"rendered":"http:\/\/1boringoldman.com\/?p=39878"},"modified":"2013-09-11T18:45:31","modified_gmt":"2013-09-11T22:45:31","slug":"studying-the-studies","status":"publish","type":"post","link":"https:\/\/1boringoldman.com\/index.php\/2013\/09\/10\/studying-the-studies\/","title":{"rendered":"studying the studies…"},"content":{"rendered":"\n

Who would’ve thought that I would end up reading stuff about the Clinical Trial industry? Not me. I guess I could take solace in some old saying like "variety is the spice of life," but whatever path got me here, here I go again. In a Randomized, Double Blind Clinical Trial, the way things are supposed to work is that up front, you have a Protocol, and in that Protocol you outline how the study will be conducted and how you’ll analyze the results at the end of the study. So you make educated guesses about how many subjects you’ll need to prove or disprove the research hypothesis – called power<\/em>. And up front, you declare efficacy variables – the Primary and Secondary Outcome Variables<\/em>, and declare how you’ll analyze them. The reasons are obvious. Clinical Trials are hard to do and involve actual human beings. You compute the power<\/em> to avoid ending up with too little data to say anything at all useful. And anyone who has done any research at all knows why you declare the outcome variables and analytic methods a priori<\/em>. It’s because after you have the data, you can torture<\/em> it and make it say almost anything you want it to say by [excuse my language], masterbating with the numbers. <\/p>\n

And after all that trouble and expense, the temptation to make the data speak is overwhelming. It’s called HARK<\/font><\/strong> [Hypothesis After Data Known] and it’s a no-no<\/em>. What if you find something amazing that you didn’t expect? Great news! Mention it in your article – but then do another study that has this finding as the primary hypothesis using the Randomized Double Blind Clinical Trial techniques to confirm your serendipitous finding. That’s just what scientists do. It’s their nature. This article comes from International Congress on Peer Review and Biomedical Publication in Chicago, and I’m only going to talk about one piece of the article:<\/div>\n
\n
Of Conflicts and Clinical Trials: Researchers Report New Results<\/a><\/div>\n
Science<\/em>Insider<\/font><\/strong><\/div>\n
by Jennifer Couzin-Frankel<\/div>\n
2013-09-10<\/div>\n

…Published Trial Results Often Differ From Those Initially Posted <\/font><\/strong><\/p>\n

Deborah Zarin, director of the database ClinicalTrials.gov at the National Library of Medicine, likes to say that her website is “a window into the sausage factory” — a view that we usually don’t get of how clinical trials work and how they don’t.  Six years ago, ClinicalTrials.gov was tasked by Congress to embark on a new experiment: In addition to trial registrations, many trial sponsors were required to deposit their results in the public database for anyone to access. At the congress, a group from Yale University School of Medicine explored how well the results posted on ClinicalTrials.gov match up with what’s published. What they found was not particularly encouraging.<\/p>\n

Jessica Becker, a medical student, described how she and her Yale colleagues — Harlan Krumholz, Gal Ben-Josef, and Joseph Ross—identified 96 trials published between July 2010 and June 2011, all of them with a ClinicalTrials.gov identification number. They focused on studies that appeared in high-profile journals. Almost three-quarters of the trials analyzed were funded by industry.<\/font><\/strong> All but one trial had at least one discrepancy in how trial details, results, or side effects were reported.<\/p>\n

One big question was whether the same primary endpoints and secondary endpoints appeared in both the final publication and the ClinicalTrials.gov results database. A primary endpoint represents the main goal of a study and the question or questions it was designed to answer. Secondary endpoints are often added to squeeze as much information as possible out of what’s collected, but statistically they can be weaker because the trial wasn’t created with them in mind. Primary endpoints in 14 trials appeared only on ClinicalTrials.gov, while primary endpoints from 10 others were only in the publication. The results described were also different in some cases: For 21% of the primary endpoints, what appeared in the journal wasn’t exactly the outcome described on ClinicalTrials.gov, and in 6%, the Yale group suggested that this difference influenced how the results would be interpreted.<\/p>\n

For secondary endpoints, the difference was even more dramatic: Of more than 2000 secondary endpoints listed across the trials, just 16% appeared the same way in both the public database and the published article along with the same results. Results for dozens of secondary endpoints were inconsistent. “Our findings raise concerns about the accuracy of information in both places, leading us to wonder which to believe,” Becker said. The group hasn’t probed why this is happening: There could be innocent errors on ClinicalTrials.gov or typos in publications. Or authors may promote “more favorable stuff” in what’s printed, she speculated.<\/p>\n

“There are many, many microdecisions” that come with writing up a publication, Zarin says. The uncomfortable results presented by Becker are “part of what motivates the desire” for anonymized information on individual patients, Zarin suggests — exposing that might be the only way to reconcile the discrepancies. Zarin also speculates that researchers might add positive secondary endpoints after the study is completed — a big no-no in the trials world<\/font><\/strong> — to give it a rosier hue, and thus they don’t appear on ClinicalTrials.gov when the study is first registered. Zarin is conducting her own analysis of the ClinicalTrials.gov results database, which now includes results from almost 10,000 trials [150,000 trials are registered on the site]. She says she’s reaching similar outcomes as the Yale group. One question that the Yale team didn’t explore was whether researchers had inputted their results on the site before submitting their paper — something that would allow journal editors or reviewers to play detective and see whether the document they have matches up with what’s in the database…<\/div>\n
hat-tip to pharmagossip<\/a> master spotter<\/sub><\/strong>  <\/div>\n<\/blockquote>\n
In this last few years, I guess studying the studies<\/em> has become my hobby. It wasn’t planned that way. I started off trying to figure out why psychiatric patients were on so many odd medications, something I found out after retiring when I emerged from the cave of my practice to the public world, volunteering in a clinic. But that quickly lead to the whole world of Clinical Drug Trials, something I knew little about. Considering the size and importance of that domain, it’s remarkable that I’d missed it [see 3512!<\/font>…<\/a> or this report from ClinicalTrials.gov – "150,000 trials are registered on the site!<\/em>"].<\/div>\n

This is a well done report [as the research of medical students often turns out to be]. Well powered. Well analyzed. And it answers an important question, How often do people change their analytic techniques, outcome parameters, and [there’s that word again] masterbate with the numbers<\/em>? substituting fantasy for facts<\/em>. The answer is All the time!<\/em> There’s even a confirmatory study underway by Deborah Zarin, the boss at ClinicalTrials.gov. It’s a good show all around and there couldn’t be a more important topic. These Clinical Trials literally determine what gets defined as a Medicine in our modern world. <\/p>\n

Obviously, this is just one more piece of the growing body of evidence that has become a somewhat monotonous topic on this blog – data transparency<\/font><\/strong><\/em> – what Deborah Zarin calls "the desire for anonymized information on individual patients.<\/em>" It’s a conclusion that needs no p-value anymore. It’s just right…<\/div>\n","protected":false},"excerpt":{"rendered":"

Who would’ve thought that I would end up reading stuff about the Clinical Trial industry? Not me. I guess I could take solace in some old saying like "variety is the spice of life," but whatever path got me here, here I go again. In a Randomized, Double Blind Clinical Trial, the way things are […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[2],"tags":[],"class_list":["post-39878","post","type-post","status-publish","format-standard","hentry","category-politics"],"_links":{"self":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/39878","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/comments?post=39878"}],"version-history":[{"count":19,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/39878\/revisions"}],"predecessor-version":[{"id":39926,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/39878\/revisions\/39926"}],"wp:attachment":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/media?parent=39878"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/categories?post=39878"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/tags?post=39878"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}