{"id":65329,"date":"2016-05-10T19:07:59","date_gmt":"2016-05-10T23:07:59","guid":{"rendered":"http:\/\/1boringoldman.com\/?p=65329"},"modified":"2016-05-11T06:34:27","modified_gmt":"2016-05-11T10:34:27","slug":"65329","status":"publish","type":"post","link":"https:\/\/1boringoldman.com\/index.php\/2016\/05\/10\/65329\/","title":{"rendered":"the hope diamond&#8230;"},"content":{"rendered":"\n<p align=\"center\"><a href=\"http:\/\/de.slideshare.net\/deevybishop\/what-is-the-reproducibility-crisis-in-science-and-what-can-we-do-about-it\" target=\"_blank\"><\/a><a href=\"http:\/\/de.slideshare.net\/deevybishop\/what-is-the-reproducibility-crisis-in-science-and-what-can-we-do-about-it\" target=\"_blank\"><img loading=\"lazy\" decoding=\"async\" width=\"350\" height=\"200\" border=\"0\" src=\"http:\/\/1boringoldman.com\/images\/bishop.jpg\" \/><\/a><br \/>           <sup>[click image to link to her slides]<\/sup><\/p>\n<p align=\"justify\" class=\"small\"><a target=\"_blank\" href=\"https:\/\/en.wikipedia.org\/wiki\/Dorothy_Bishop_%28psychologist%29\">Dorothy Bishop<\/a> is a Developmental Psychologist who focuses on Dyslexia and other Language Disorders. This is not an article, just the slides from a presentation she gave to the Rhodes Biomedical Association last week on the reproducibility crisis. Her slides tell a story well known to us. And the problem isn&#8217;t the science, it&#8217;s the scientists. She starts with some familiar methods used to distort findings. I&#8217;ve synopsized those opening slides:<\/p>\n<div align=\"center\"><img decoding=\"async\" width=\"300\" border=\"0\" src=\"http:\/\/1boringoldman.com\/images\/bishop.gif\" \/><\/div>\n<ul><span class=\"small\">              <\/p>\n<li><strong><font color=\"#200020\">Publication bias<\/font><\/strong>: burying negative studies<\/li>\n<li><strong><font color=\"#200020\">HARKing<\/font><\/strong>: Hypothesis After Results Known<\/li>\n<li><strong><font color=\"#200020\">p-hacking<\/font><\/strong>: trying different statistical tests on various datasets until you get the result you want<\/li>\n<p>              <\/span><\/ul>\n<div align=\"justify\" class=\"small\">but then she talks about the various people along the way who had written about this. And her history started with <a target=\"_blank\" href=\"https:\/\/en.wikipedia.org\/wiki\/Adriaan_de_Groot\">Adriaan de Groot<\/a> [1956]. <\/div>\n<div align=\"center\"><img decoding=\"async\" width=\"400\" vspace=\"7\" border=\"0\" src=\"http:\/\/1boringoldman.com\/images\/degroot-1.gif\" \/><\/div>\n<div align=\"justify\" class=\"small\">So, on a lark, I Googled Adriaan de Groot, and there was his article full text [put there by Dorothy Bishop]&#8230;<\/div>\n<blockquote>\n<div align=\"center\" class=\"big\"><a target=\"_blank\" href=\"http:\/\/www.ejwagenmakers.com\/inpress\/DeGroot1956_TA.pdf\">The Meaning of &ldquo;Significance&rdquo; for Different Types of Research<\/a><\/div>\n<div align=\"center\" class=\"middle\"><strong><font color=\"#200020\">[Translated and Annotated by Eric&ndash;Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]<\/font><\/strong><\/div>\n<div align=\"center\" class=\"middle>&#8220;>by Dr. A. D. de Groot<\/div>\n<div align=\"center\" class=\"middle\"><em>from the<\/em> Psychological Laboratory of the University of Amsterdam<\/div>\n<div align=\"center\" class=\"middle\">[<a target=\"_blank\" href=\"http:\/\/www.ejwagenmakers.com\/inpress\/DeGroot1956_TA.pdf\">full text on-line<\/a>]          <\/div>\n<p>          <\/p>\n<div align=\"justify\"><em><strong><font color=\"#200020\">Abstract:<\/font><\/strong><\/em><br \/>         Adrianus Dingeman de Groot [1914&ndash;2006] was one of the most influential Dutch psychologists. He became famous for his work &ldquo;Thought and Choice in Chess&rdquo;, but his main contribution was methodological &mdash; De Groot co-founded the Department of Psychological Methods at the University of Amsterdam [together with R. F. van Naerssen], founded one of the leading testing and assessment companies [CITO], and wrote the monograph &ldquo;Methodology&rdquo; that centers on the empirical-scientific cycle: observation&ndash;induction&ndash; deduction&ndash;testing&ndash;evaluation. Here we translate one of De Groot&rsquo;s early articles, published in 1956 in the Dutch journal Nederlands Tijdschrift voor de Psychologie en Haar Grensgebieden . This article is more topical now than it was almost 60 years ago. De Groot stresses the difference between exploratory and confirmatory [&ldquo;hypothesis testing&rdquo;] research and argues that statistical inference is only sensible for the latter: &ldquo;One &lsquo;is allowed&rsquo; to apply statistical tests in exploratory research, just as long as one realizes that they do not have evidential impact&rdquo;. De Groot may have also been one of the first psychologists to argue explicitly for preregistration of experiments and the associated plan of statistical analysis. The appendix provides annotations that connect De Groot&rsquo;s arguments to the current-day debate on transparency and reproducibility in psychological science.<\/div>\n<\/blockquote>\n<div align=\"justify\" class=\"small\">Last week, I called the publication by Jureidini, Amsterdam, and McHenry&nbsp; <a target=\"_blank\" href=\"http:\/\/1boringoldman.com\/index.php\/2016\/05\/05\/the-jewel-in-the-crown\/\">the jewel in the crown&hellip;<\/a> <img decoding=\"async\" hspace=\"4\" height=\"120\" border=\"0\" align=\"right\" src=\"http:\/\/1boringoldman.com\/images\/hope.gif\" \/>to metaphorically emphasize the importance of their article, which introduced subpoenaed internal corporate documents to illustrate the fraudulent underbelly of the 2004 Celexa RCT in adolescents. Well, I need an even greater superlative for the <font color=\"#200020\">De Groot<\/font> article <font color=\"#200020\">Dorothy Bishop<\/font> brings to us from a more naive time &#8211; how about <strong><font color=\"#200020\">the Hope Diamond<\/font><\/strong>? Since you&#8217;re unlikely to read the whole paper without a nudge, here&#8217;s its essence from the translator&#8217;s note in the Appendix:<\/div>\n<blockquote>\n<div align=\"justify\">Specifically, De Groot makes three important interconnected points. The first point is that exploratory analyses invalidate the standard interpretation of outcomes from hypothesis testing procedures. &ldquo;Exploratory investigations differ from hypothesis testing in that the canon of the inductive method of testing is not observed, at least not in its rigid form. The researcher does take as his starting-point certain expectations, a more or less vague theoretical framework; he is indeed out to find certain kinds of relationships in his data, but these have not been antecedently formulated in the form of precisely stated &laquo;<em>testable<\/em>&raquo; hypotheses. Accordingly they cannot, in the strict sense, be put to the test.&rdquo; &laquo;<em>De Groot, 1969, p.  306<\/em>&raquo;.  Indeed, in exploratory work: &ldquo;The characteristic element of &lsquo;trying out whether &#8230;&rsquo; is present, but in such a way that the researcher&rsquo;s attitude in fact boils down to &lsquo;let us see what we can find.&rsquo; Now what is &lsquo;found&rsquo; &mdash; that is, selected &mdash; cannot also be tested on the same materials&rdquo; &laquo;<em>De Groot, 1969, p. 307<\/em>&raquo;&#8230;<\/div>\n<p align=\"justify\">The second, related, point that De Groot makes is the pressing need to distinguish between exploratory and confirmatory &laquo;<em>&ldquo;hypothesis testing&rdquo;<\/em>&raquo; analyses. De Groot reiter- ated this point in his book &ldquo;Methodology&rdquo;: <em><strong><font color=\"#990000\">&ldquo;It is of the utmost importance at all times to maintain a clear distinction between exploration and hypothesis testing . The scientific significance of results will to a large extent depend on the question whether the hypotheses involved had indeed been antecedently formulated, and could therefore be tested against genuinely new materials.<\/font><\/strong><\/em>  Alternatively, they would, entirely or in part, have to be designated as ad hoc hypotheses, which could, emphatically, not yet be tested against &lsquo;new&rsquo; materials.&rdquo; &laquo;<em>De Groot, 1969, p. 52<\/em>&raquo; Indeed, De Groot believed that it was unethical to blur the distinction between exploratory and confirmatory work: <em><strong><font color=\"#990000\">&ldquo;It is a serious offense against the social ethics of science to pass off an exploration as a genuine testing procedure. Unfortunately, this can be done quite easily by making it appear as if the hypotheses had already been formulated before the investigation started. Such misleading practices strike at the roots of &lsquo;open&rsquo; communication among scientists.&rdquo;<\/font><\/strong><\/em> &laquo;<em>De Groot, 1969, p. 52<\/em>&raquo;. This point was later revisited by Kerr &laquo;<em>1998<\/em>&raquo; when he introduced the concept of HARKing &laquo;<em>&ldquo;Hypothesizing After the Results are Known&rdquo;<\/em>&raquo;, as well as by Simmons et al. &laquo;<em>2011<\/em>&raquo;, John et al. &laquo;<em>2012<\/em>&raquo;, and Wagenmakers, Wetzels, Borsboom, and van der Maas &laquo;<em>2011<\/em>&raquo;&#8230;<\/p>\n<div align=\"justify\">The third point that De Groot makes concerns preregistration. De Groot strongly felt that in order for research to qualify as confirmatory &laquo;<em>and, consequently, for statistical inference to be meaningful<\/em>&raquo;, an elaborate preregistration effort is called for: &ldquo;<em><strong><font color=\"#993300\">If an investigation into certain consequences of a theory or hypothesis is to be designed as a genuine testing procedure &laquo;and not for exploration&raquo;, a precise antecedent formulation must be available, which permits testable consequences to be deduced<\/font><\/strong><\/em>.&rdquo; &laquo;<em>De Groot, 1969, p. 69<\/em>&raquo;&#8230;<\/div>\n<\/blockquote>\n<div align=\"justify\" class=\"small\">My version:<\/div>\n<ol><span class=\"small\">  <\/p>\n<li>\n<div align=\"justify\"><em><strong><font color=\"#990000\">Randomized Clinical Trials are not research<\/font><\/strong><\/em> [<em>exporatory<\/em>]<strong><font color=\"#990000\">, they&#8217;re product testing<\/font><\/strong> [<em>confirmatory<\/em>].<\/div>\n<\/li>\n<li>\n<div align=\"justify\"><em><strong><font color=\"#990000\">The a priori Protocol defines the analysis<\/font><\/strong><\/em>.<\/div>\n<\/li>\n<p><\/span><\/ol>\n<div align=\"justify\" class=\"small\"><font color=\"#200020\">Dr. Bernard Carroll<\/font>&#8216;s version was in this <a href=\"http:\/\/1boringoldman.com\/index.php\/2016\/05\/06\/65199\/#comment-264052\" target=\"_blank\">recent comment<\/a>:<\/div>\n<blockquote>\n<div align=\"justify\">There is an obvious way to prevent that kind of data manipulation,  cherry picking, moving of goalposts, HARKing, and glossing over of  adverse events. All it would take is for the FDA to require that <strong>they<\/strong>  analyze the data strictly according to the a priori protocol. That  requirement would apply to any investigational new drug or to any  approved drug being tested for a new indication. Corporations and  investigators would be prohibited from reporting any analyses other than  the FDA analyses. With an a priori protocol and plan of analysis there  should be no room for self-serving &ldquo;creativity&rdquo; by the corporations.<\/div>\n<p align=\"justify\">As things stand, we have a Kabuki theater spectacle. The corporations  don&rsquo;t come clean about what they did and the FDA doesn&rsquo;t call them out  when agency analyses disagree with the corporate line. They may deny or  delay approval, but the FDA doesn&rsquo;t go to the literature like Jureidini,  Amsterdam, and McHenry did here challenge the distorted corporate  analyses reported in the literature.<\/p>\n<div align=\"justify\">This requirement would put an end to creative manipulation of the  clinical trials  literature. As Dr. Mickey often says, this is not high  science but rather product testing. Thus, corporations cannot claim to  be privileged for conducting the statistical analyses. We need a  clinical trials equivalent of the Underwriters Laboratory. Before  licensing, nobody takes the manufacturing corporation&rsquo;s word for it  concerning the safety and performance of X-ray machines or CT scanners  or cardiac defibrillators. Why should we treat drugs any differently?&#8230;<\/div>\n<\/blockquote>\n<div align=\"justify\" class=\"small\">And <font color=\"#200020\">Dr. Adriaan de Groot<\/font> said it back when the world was young [1956]. It&#8217;s the Hope Diamond because it&#8217;s our only hope to stop the craziness of the last thirty five years. <em><strong><font color=\"#990000\">It&#8217;s probably more important than Data Transparency, rigid enforcement of analysis that follows the registered a priori protocol&#8230;<\/font><\/strong><\/em><\/div>\n<div align=\"right\"><sup><strong><font color=\"#200020\">hat tip to Dorothy Bishop&#8230;<\/font><\/strong> &nbsp;<img decoding=\"async\" height=\"35\" border=\"0\" align=\"absmiddle\" src=\"http:\/\/1boringoldman.com\/images\/hat-tip.gif\" \/><\/sup><\/div>\n","protected":false},"excerpt":{"rendered":"<p>[click image to link to her slides] Dorothy Bishop is a Developmental Psychologist who focuses on Dyslexia and other Language Disorders. This is not an article, just the slides from a presentation she gave to the Rhodes Biomedical Association last week on the reproducibility crisis. Her slides tell a story well known to us. And [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[5],"tags":[],"class_list":["post-65329","post","type-post","status-publish","format-standard","hentry","category-opinion"],"_links":{"self":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/65329","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/comments?post=65329"}],"version-history":[{"count":35,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/65329\/revisions"}],"predecessor-version":[{"id":65364,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/posts\/65329\/revisions\/65364"}],"wp:attachment":[{"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/media?parent=65329"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/categories?post=65329"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/1boringoldman.com\/index.php\/wp-json\/wp\/v2\/tags?post=65329"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}