When Dr. Bernard Carroll comments here, he often uses the term "hand waving" when describing some of the tricky maneuvers used in the clinical trial reports to smooth over shaky logic or rationalize absurdities. It’s a great term, I think originating in the world of stage magicians who use exaggerated gesticulations to distract your attention. My wife’s a figure skating fan, and there it’s called "hand dancing" – dramatic arm and hand gestures to cover up sloppy skating. After reading so many jury-rigged clinical trial reports, I’ve almost come to see the whole narrative as organized around a verbal version of these attempts at artifice, and find myself jotting down the essential pieces in a hastily sketched table on the back of a nearby envelope or piece of scrap-paper. So the basic efficacy table isn’t just a concept or a proposal. It’s an outgrowth of my experience. I guess the formula goes:
article narrative – bullshit = basic efficacy table
One thing the ghost-writers seem to count on is that most doctors look at the abstract, scan the graphs and tables, and move on. I used to see that as virtue – getting through so much material on a regular basis. In my doctor youth, I could do that. But no longer. If a Clinical Trial report or a review article is there to be read, it’s there to be read closely, pencil and envelope back at the ready. At least that’s true of the industry funded clinical trials of psychiatric drugs that I find myself reading these days.
When I drew this diagram, it’s not how things are. It’s how I wish they would be. Step one is the approval of the study Protocol by the Institutional Review Board. At that point, by my reckoning, the trial should be registered [on ClinicalTrials.gov]. There’s no reason at all that the Protocol couldn’t be published at that point. It has been written down for the IRB. Why not make it a part of the registration process. That would mean that a bona fide copy of the a priori declarations would be available from the outset.
adapted from Table 1A inby Deborah A. Zarin, Tony Tse, Rebecca J. Williams, and Sarah CarrNew England Journal of Medicine. 2016 375:1998-2004.
|When does information need to be submitted to or posted on ClinicalTrials.gov?|
|Submission: Within 21 days after enrollment of the first trial participant|
|Posting: Generally, within 30 days after submission. For ACTs of unapproved or uncleared devices, no earlier than FDA approval or clearance and not later than 30 days after FDA approval or clearance (i.e., “delayed posting”), unless a responsible party authorizes posting of submitted information prior to FDA approval or clearance|
|Descriptive information about the trial: e.g., brief title, study design, primary outcome measure information, studies an FDA-regulated device product, device product not approved or cleared by the FDA, post prior to FDA approval or clearance, and study completion date|
|Recruitment information: e.g., eligibility criteria, overall recruitment status,|
|Location and contact information: e.g., name of sponsor, facility information|
|Administrative data: e.g., secondary ID, human-subjects protection review board status|
|Results information reporting|
|When does information need to be submitted to or posted on ClinicalTrials.gov?|
|Standard deadline: Within 12 months after the date of final data collection for the prespecified primary outcome measures (primary completion date)|
|Delayed submission with certification: May be delayed for up to 2 additional years (i.e., up to 3 years total after the primary completion date) for trials certified to be undergoing commercial product development for initial FDA marketing approval or clearance or approval or clearance for a new use|
|Submitting partial results: Deadlines are established for submitting results information for a secondary outcome measure or additional adverse information that has not been collected by the primary completion date|
|Extension request: After receiving and reviewing requests, NIH may extend deadlines for “good cause”|
|Posting: Within 30 days after submission|
|Participant flow: Information about the progress of participants through the trial by treatment group, including the number who started and completed the trial|
|Demographic and baseline characteristics: Demographic and baseline characteristics collected by treatment group or comparison group and for the entire population of participants in the trial, including age, sex and gender, race or ethnicity, and other measures that were assessed at baseline and are used in the analysis of the primary outcome measures|
|Outcomes and statistical analyses: Outcomes and statistical analyses for each primary and secondary outcome measure by treatment group or comparison group, including results of scientifically appropriate statistical analyses performed on these outcomes, if any.|
|Adverse event information: Tables of all anticipated and unanticipated serious adverse events and other adverse events that exceed a 5% frequency threshold within any group, including time frame (or specific period over which adverse event information was collected), adverse-event reporting description (if the adverse-event information collected in the clinical trial is collected on the basis of a different definition of adverse event or serious adverse event from that used in the final rule), collection approach (used for adverse events during the study: systematic or nonsystematic), table with the number and frequency of deaths due to any cause by treatment group or comparison group|
|Protocol and statistical analysis: Protocol and statistical analysis plan to be submitted at time of results information reporting (may optionally be submitted earlier)|
|Administrative data: Administrative information, including a point of contact to obtain more information about the posted summary results information|
First off, anything they do is a step forward. They’ve had the machinery available for two decades, and have done little with it. So they’re finally requiring registration for all the studies, and they pledge to keep up with it. An excellent start.
Initial Registration: They say that they want the submission of the trial registration within three weeks of the initial subject’s enrollment and posting on-line within the 30 days after submission. Of course I’d prefer our "before the study starts" timing, but within the first two months will do. The point is to get it registered before they can look at the results and modify the Protocol – and two months is early enough for me. As for what’s to be posted, they don’t require posting the whole Protocol. That’s a disappointment. I’d prefer anchoring the outcome parameter at the beginning. But at least they do require declaration of the Primary Outcome Variables with registration.
Posting the Results: This has traditionally been the most ignored requirement. They say: "Outcomes and statistical analyses for each primary and secondary outcome measure by treatment group or comparison group, including results of scientifically appropriate statistical analyses performed on these outcomes, if any" and add in "Protocol and statistical analysis plan to be submitted at time of results information reporting (may optionally be submitted earlier)." And I say A+! With that information, I could fill out my entire basic efficacy table, The only thing they left out was the Effect Size and there would be ample information to do that calculation.
And for timing on the Results? I’d have to say "barely passing," if that. "Within 12 months after the date of final data collection for the prespecified primary outcome measures (primary completion date)" and "May be delayed for up to 2 additional years (i.e., up to 3 years total after the primary completion date) for trials certified to be undergoing commercial product development for initial FDA marketing approval or clearance or approval or clearance for a new use." That’s a disappointment, and I can’t see any reason for it. The results are just what they are – they’re the results of the prespecified variables analyzed in the prespecified way. Who needs time for that? But I’ll have to admit that if they were to actually to follow these standards, the improvement would still be dramatic, probably satisfy most of us. With new drugs or new indications, they’d still be early in the drug’s patent life.
the a priori declared primary and secondary outcome variables
the prospectively defined statistical analysis plan
the values, variance, and effect size for those specific parameters